artemist/mes
1
0
Fork 0
Code Issues Pull requests Packages Projects Releases Wiki Activity

doc: Release udpate.

Browse source 
* BOOTSTRAP: Update.
* HACKING: Update.
* NEWS: Update.
* README: Update.
* doc/ANNOUNCE-0.13: New file.
This commit is contained in:
Jan Nieuwenhuizen 2018-04-22 00:45:00 +02:00
parent c6fecdc353
commit 6e32754a98
No known key found for this signature in database
GPG key ID: F3C1A0D9C1D65273
5 changed files with 200 additions and 80 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

147
BOOTSTRAP
View file

@ -2,17 +2,12 @@ bootstrappable.org project -*- org -*-
* What?
** Full source bootstrapping for GuixSD
* Why?
** Reproducibility is essential to Software Freedom
Reproducible builds are a set of software development practices that
create a verifiable path from human readable source code to the binary
code used by computers.
*** What about the compiler?
We have the sources: they always lead to bitwise-same binary, but what
about the compiler?
*** The current way out: Ignore the problem
``recipe for yoghurt: add yoghurt to milk''
A package in GuixSD is can be uniquely identified by the hash of its
source code, its dependencies and its build recipe.
Every package can be built from source, except for the bootstrap
binaries.
*** From the GuixSD manual
@ -23,74 +18,96 @@ The root of this dependency graph is a small set of “bootstrap
binaries”, provided by the ‘(gnu packages bootstrap)’ module. For more
information on bootstrapping, *note Bootstrapping::.
*** New solution: Full source bootstrapping path, Stage0 and Mes
**** GuixSD bootstrap tarballs
$ du -schx $(readlink $(guix build bootstrap-tarballs)/*)
2.1M /gnu/store/9623n4bq6iq5c8cwwdq99qb7d0xj93ym-binutils-static-stripped-tarball-2.28.1/binutils-static-stripped-2.28.1-x86_64-linux.tar.xz
18M /gnu/store/437xwygmmwwpkddcyy1qvjcv4hak89pb-gcc-stripped-tarball-5.5.0/gcc-stripped-5.5.0-x86_64-linux.tar.xz
1.8M /gnu/store/55ccx18a0d1x5y6a575jf1yr0ywizvdg-glibc-stripped-tarball-2.26.105-g0890d5379c/glibc-stripped-2.26.105-g0890d5379c-x86_64-linux.tar.xz
5.7M /gnu/store/bqf0ajclbvnbm0a46819f30804y3ilx0-guile-static-stripped-tarball-2.2.3/guile-static-stripped-2.2.3-x86_64-linux.tar.xz
5.8M /gnu/store/j8yzjmh9sy4gbdfwjrhw46zca43aah6x-static-binaries-tarball-0/static-binaries-0-x86_64-linux.tar.xz
33M total
$ for i in $(readlink $(guix build bootstrap-tarballs)/*);\
do sudo tar xf $i; done
$ du -schx *
130M bin
13M include
54M lib
51M libexec
5.2M share
252M total
* Why?
** Reproducibility is essential to Software Freedom
Reproducible builds are a set of software development practices that
create a verifiable path from human readable source code to the binary
code used by computers.
*** What about the bootstrap binaries and the compilers?
We have the sources: they always lead to bitwise-same binary, but what
about the bootstrap binaries and compilers?
*** The current way out: Ignore the problem
``recipe for yoghurt: add yoghurt to milk''
*** New solution: Full source bootstrapping path
* How?
** Software: Stage0 and Mes
** Stage0
** Software: MesCC-tools, M2-Planet, Mes and MesCC
** MesCC-tools
*** hex.0: amazing ~300 byte self-hosting hex assembler that we consider to be source
*** a M0 macro assembler written in .0
*** a M1 macro assembler written in M0
*** a hex2 linker written in M0
https://github.com/oriansj/mescc-tools
Look:
https://git.savannah.nongnu.org/cgit/stage0.git/tree/Linux%20Bootstrap/hex0.hex
*** hex.0: ~500 byte well-documented, self-hosting hex assembler
Do:
git clone https://git.savannah.nongnu.org/git/stage0.git
cd stage0
make
bin/hex < Linux\ Bootstrap/hex0.hex > hex-1
chmod +x hex-1 # later: bin/exec_enable hex-1
./hex-1 < Linux\ Bootstrap/hex0.hex > hex-2
This 500 byte program is written in ASCII hex. When converted
byte-for-byte from ASCCI to binary we have the only binary seed that
our full source bootstrap path needs.
** Mes
We bless this simple and easily verifyable binary and consider it to
be source.
*** hex1: next level hex assembler written in hex.0
*** M0: a macro assembler written in hex.1
*** M1: a macro assembler written in M0
*** hex2: a hex2 linker written in M0
*** M2-Planet: a self-hosting M2 (C-with-structs) transpiler written in M1
*** Mes: A Scheme interpreter written in C, with cpp transformed into M2
*** MesCC: A C compiler written in Scheme
*** tcc-boot: a patched version of TinyCC
** Mes and MesCC
https://gitlab.com/janneke/mes
*** mes.c: a scheme interpreter prototyped in C ~1400 Lines
*** mescc.scm: a C compiler written in Scheme (uses Nyacc C99 parser in Scheme)
*** mes.M1: this scheme interpreter in annotated M1 assembly
*** mes.c: a Scheme interpreter prototyped in C ~3000 Lines
*** mescc: a C compiler written in Scheme (uses Nyacc C99 parser in Scheme)
*** mes.M2: this Scheme interpreter in preprocessed M2
** Naive recipe
hex + hex.0 => hex-1
hex-1 + M0.0 -> M0
M0 + M1.M0 -> M1.0
M0 + hex2_linker.M0 -> hex2_linker
M1 + mes.M1 -> mes.hex2
hex2_linker + stage0/elf32-header.hex2 + mes.hex2 + elf32-footer.hex2 -> mes
mes + mescc.scm + tcc.c -> tcc.M1 -> tcc.hex2 -> mes-tcc
mes-tcc + gcc.c -> gcc
*done*
** Less naive recipe
*** Use hex2_linker, M1 prototyped in C from
https://github.com/oriansj/mescc-tools
*** Remember that mes.M1 is compiled by mescc from prototyped src/mes.c in C.
*** stage0/mescc-tools: TODO
*** Mes
M1 -f stage0/x86.M1 -f lib/crt1.M1 --LittleEndian --Architecture=1 > lib/crt1.hex2
M1 -f stage0/x86.M1 -f lib/mini-libc-mes.M1 --LittleEndian --Architecture=1 > lib/libc-mes.hex2
M1 -f stage0/x86.M1 -f src/mes.M1 --LittleEndian --Architecture=1 > src/mes.hex2
hex2 --LittleEndian --Architecture=1 --BaseAddress=0x1000000 -f stage0/elf32-header.hex2 -f lib/crt1.hex2 -f lib/libc-mes.hex2 -f src/mes.hex2 -f stage0/elf32-footer-single-main.hex2 > src/mes-mes
exec_enable src/mes-mes
*** Tinycc
./build.sh
./link.sh
* DONE
** stage0: hex.0, M0 done; M1, hex2_linker prototyped in C
** tcc compiled with mescc correctly compiles: int main () {return 42;}
** mes+mescc.scm are mutual self hosting
** during development we run mescc.scm on Guile (mes is slooowww)
** tcc compiled with gcc is known to compile gcc
** TinyCC
https://gitlab.com/janneke/tinycc
* TODO
** fix mescc.scm so that tcc can correctly compile gcc
** fix bootstrap-loops: (Nyacc?, mes.M1?, psyntax.pp?)
** fix MesCC so that tcc can correctly compile gcc
** remove or upstream patches from tcc
** fix bootstrap-loops: (Nyacc?, mes.M12, psyntax.pp?)
** make GNU gcc (8.0?) bootstrappable again, remove [need for] tcc stage
** stage1/2 LISP, FORTH?
** rain1's LISP compiler?
** integrate with GuixSD
** x86_64, arm?
** x86_64, arm, the Hurd
* DONE
** hex.0, hex.1, M0
** M1, hex2_linker prototyped in C
** M2-Planet is now self-hosting, written in M2
** mes+mescc are mutual self-hosting
** patched tcc compiled with mes+mescc
** during development we run mescc on Guile (mes is ~30 times slower)
** tcc compiled with gcc is known to compile gcc
* Contact
** #bootstrappable, #guix on freenode
** bootstrappable.org

10
HACKING
View file

@ -59,8 +59,8 @@ mes.c.
* ROADMAP
** TODO
*** release 0.x, unsorted
- mes+mescc: compile a mes-tcc that can compile gcc-4.7.
- mes: set base-0.scm as default MES_BOOT, drop cat base-0.mes silliness.
- mes+mescc: compile a mes-tcc that can build itself.
- mes-tcc: full source bootstrapped build of gcc-4.7.
- mes: real module support, bonus for supporting Guile's define-module/define-public syntax.
- mes: prepare src/mes.c for M2-Planet transpiler.
- mes: produce functional mes from mes.M1 transpiled by M2-Planet.
@ -100,6 +100,7 @@ mes.c.
- add x86_64, replace GuixSD's bootstrap binaries for x86_64.
- more architectures?
** DONE
*** 0.13: Mes+MesCC now compiles a modified, functional tcc.c (~25,000LOC) in 1h30'.
*** 0.12: Mes+MesCC now compiles mes.c (~3000LOC) in ~4min.
*** 0.11: MesCC now compiles a mes-tcc that passes 26/69 of mescc's C tests.
*** 0.10: Mescc now compiles a mes-tcc that compiles a trivial C to a running a.out.
@ -137,9 +138,10 @@ eenough to work on compiling tinycc's tcc.c albeit a somewhat modified version.
* Bugs
** mes: remove pmatch-car/pmatch-cdr hack.
** mescc: softcode stack frame size, now hardcoded and very large
** mes+mescc: parse tcc.c->tcc.E works, compile tcc.E -> tcc.M1 segfaults.
time GUILE_LOAD_PATH=/home/janneke/src/nyacc/module:$GUILE_LOAD_PATH ../mes/scripts/mescc.mes -E -o tcc.E -I . -I ../mes/lib -I ../mes/include -D 'CONFIG_TCCDIR="usr/lib/tcc"' -D 'CONFIG_TCC_CRTPREFIX="usr/lib:{B}/lib:."' -D 'CONFIG_TCC_ELFINTERP="/gnu/store/70jxsnpffkl7fdb7qv398n8yi1a3w5nx-glibc-2.26.105-g0890d5379c/lib/ld-linux.so.2"' -D 'CONFIG_TCC_LIBPATHS="/home/janneke/src/tinycc/usr/lib:{B}/lib:."' -D 'CONFIG_TCC_SYSINCLUDEPATHS="../mes/include:usr/include:{B}/include"' -D CONFIG_USE_LIBGCC=1 -D 'TCC_LIBGCC="/home/janneke/src/tinycc/usr/lib/libc+tcc-gcc.mlibc-o"' -D CONFIG_TCC_STATIC=1 -D ONE_SOURCE=yes -D TCC_TARGET_I386=1 -D BOOTSTRAP=1 tcc.c
time GUILE_LOAD_PATH=/home/janneke/src/nyacc/module:$GUILE_LOAD_PATH MES_ARENA=200000000 ../mes/scripts/mescc.mes -c -o tcc.M1 tcc.E
time GUILE_LOAD_PATH=/home/janneke/src/nyacc/module:$GUILE_LOAD_PATH ../mes/scripts/mescc -E -o tcc.E -I . -I ../mes/lib -I ../mes/include -D 'CONFIG_TCCDIR="usr/lib/tcc"' -D 'CONFIG_TCC_CRTPREFIX="usr/lib:{B}/lib:."' -D 'CONFIG_TCC_ELFINTERP="/gnu/store/70jxsnpffkl7fdb7qv398n8yi1a3w5nx-glibc-2.26.105-g0890d5379c/lib/ld-linux.so.2"' -D 'CONFIG_TCC_LIBPATHS="/home/janneke/src/tinycc/usr/lib:{B}/lib:."' -D 'CONFIG_TCC_SYSINCLUDEPATHS="../mes/include:usr/include:{B}/include"' -D CONFIG_USE_LIBGCC=1 -D 'TCC_LIBGCC="/home/janneke/src/tinycc/usr/lib/libc+tcc-gcc.mlibc-o"' -D CONFIG_TCC_STATIC=1 -D ONE_SOURCE=yes -D TCC_TARGET_I386=1 -D BOOTSTRAP=1 tcc.c
time GUILE_LOAD_PATH=/home/janneke/src/nyacc/module:$GUILE_LOAD_PATH MES_ARENA=200000000 ../mes/scripts/mescc -c -o tcc.M1 tcc.E
** mes: with-fluids: tests/fluids.test test 7 fails with Mes.
** mescc: 7n-struct-struct-array.c: struct file f = {"first.h"};
** test/match.test ("nyacc-simple"): hygiene problem in match

20
NEWS
View file

@ -10,6 +10,26 @@ Copyright © 2016,2017,2018 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
Please send Mes bug reports to janneke@gnu.org.
* Changes in 0.13 since 0.12
** Core
*** Bootstrapped Mes+MesCC can now compile a patched tcc in ~1h45' (~25,000 LOC).
*** MesCC scripts for Mes and Guile are now merged; executable is: `mescc'.
*** Mes now uses only one arena for stop-and-copy; doubles available size.
*** Mes now has a Guile-like command-line interface (mes.repl has been removed).
*** Mes now boots into a full Scheme by default.
*** Mes can now be compiled (MES_MINI=1) to boot into a minimal Scheme (~2000 cells).
*** Mes now creates less garbage in the reader and in
append2, append_reverse, reverse, reverse!, vector-for-each, vector-to-list, vector-map.
*** 5 new functions
append-reverse, chmod, ioctl, isatty, isspace, last_pair, reverse!.
** Language
*** 3 new functions
char-whitespace?, chmod, isatty?
** Noteworthy bug fixes
*** Two bugs in the jam scraper/garbage collector have been fixed.
*** equal2_p now uses tail call elimination.
*** Escaped characters in strings are now read and write'd correctly.
*** The repl now expands macros again.
* Changes in 0.12 since 0.11
** Core
*** Mes can now run Nyacc (0.80.42) to compile itself in ~4min (~3000 LOC).

22
README
View file

@ -1,21 +1,20 @@
-*- org -*-
Mes -- Maxwell Equations of Software
[[https://gitlab.com/janneke/mes][Mes]] aims to create full source bootstrapping for [[https://www.gnu.org/software/guix][GuixSD]] as part of the
[[http://bootstrappable.org][bootstrappable builds]] effort.
[[https://gitlab.com/janneke/mes][Mes]] aims to help create full source bootstrapping for [[https://www.gnu.org/software/guix][GuixSD]] as part
of the [[http://bootstrappable.org][bootstrappable builds]] effort.
It currently consists of a mutual self-hosting [close to Guile-]
Scheme interpreter prototype in C and a Nyacc-based C compiler in
[Guile] Scheme. This C prototype will be simplified to be transpiled
by [[https://github.com/oriansj/m2-planet][M2-Planet]].
It currently consists of a mutual self-hosting Scheme interpreter
prototype written in C and a Nyacc-based C compiler written in Scheme.
This C prototype is being simplified to be transpiled by [[https://github.com/oriansj/m2-planet][M2-Planet]].
The Scheme interpreter prototype (mes.c) has a Garbage Collector, a
library of loadable Scheme modules-- notably Dominique Boucher's [[https://github.com/schemeway/lalr-scm][LALR]],
Pre-R6RS [[https://www.cs.indiana.edu/chezscheme/syntax-case/old-psyntax.html][portable syntax-case]] with R7RS ellipsis, Matt Wette's [[https://www.nongnu.org/nyacc][Nyacc]],
Guile's [[https://www.gnu.org/software/guile/docs/master/guile.html/PEG-Parsing.html][PEG]] --and test suite just barely enough to support a simple
REPL (repl.mes) and simple C-compiler MesCC (mescc.mes).
REPL and simple C-compiler: MesCC.
MesCC can compile a [[https://gitlab.com/janneke/tinycc][modified TinyCC]] that is close to being
Mes+MesCC can compile a [[https://gitlab.com/janneke/tinycc][modified TinyCC]] that is close to being
self-hosting. A GNU Ccc-compiled tcc is [[https://lists.gnu.org/archive/html/tinycc-devel/2017-05/msg00103.html][known]] to compile GCC.
Mes is inspired by The Maxwell Equations of Software: [[http://www.softwarepreservation.org/projects/LISP/book/LISP%2525201.5%252520Programmers%252520Manual.pdf][LISP-1.5]] -- John
@ -50,9 +49,10 @@ General Public Licence version 3 or later. See the file COPYING.
* REPL it
scripts/repl.mes
src/mes.gcc # default (64 bit) gcc-compiled
src/mes.mlibc-gcc # 32 bit, gcc -nostdlib
src/mes # 32 bit, bootstrapped, mes.M1+MesCC-compiled
* MesCC compiler
scripts/mescc.mes scaffold/main.c > a.out
scripts/mescc -c scaffold/main.c

81
doc/ANNOUNCE-0.13 Normal file
View file

@ -0,0 +1,81 @@
Subject: Mes 0.13 released
I am pleased to announce the release of Mes 0.13, representing 45
commits over 3 weeks. MesCC can now compile a functional tcc when
running on Mes (in ~1h45') or on Guile (in ~3min).
This means that we are getting very close to a full source bootstrap
of tcc: it is now built without gcc, glibc or guile; using only
MesCC-tools and Mes sources and corresponding ascii/binary seeds.
* About
Mes[0] aims to help create full source bootstrapping for GuixSD[1]
as part of the bootstrappable builds[2] project.
It currently consists of a mutual self-hosting Scheme interpreter
prototype in C and a Nyacc-based C compiler in Scheme. This C
prototype is being simplified to be transpiled by M2-Planet[3].
The Scheme interpreter prototype (mes.c) has a Garbage Collector,
a library of loadable Scheme modules-- notably Dominique Boucher's
LALR[4], pre-R6RS portable syntax-case[5] with R7RS ellipsis, Matt
Wette's Nyacc[6], Guile's PEG[7] --and test suite just barely
enough to support a simple REPL (repl.mes) and simple C-compiler
MesCC.
Mes+MesCC can compile a modified TinyCC[8] that is close to being
self-hosting. A GNU Ccc-compiled tcc is known[9] to compile GCC.
Mes is inspired by The Maxwell Equations of Software: LISP-1.5[10]
-- John McCarthy page 13, GNU Guix's source/binary packaging
transparency and Jeremiah Orians's stage0[11] ~300 byte self-hosting
hex assembler.
* Download
git clone https://gitlab.com/janneke/mes
wget https://gitlab.com/janneke/mes/-/archive/v0.13/mes-0.13.tar.gz
Mes runs from the source tree and can also be built, packaged and
installed in Guix[SD] by the usual
guix package -f guix.scm
* Changes in 0.13 since 0.12
** Core
*** Bootstrapped Mes+MesCC can now compile a patched tcc in ~2h30' (~25,000 LOC).
*** MesCC scripts for Mes and Guile are now merged; executable is: `mescc'.
*** Mes now uses only one arena for stop-and-copy; doubles available size.
*** Mes now has a Guile-like command-line interface (mes.repl has been removed).
*** Mes now boots into a full Scheme by default.
*** Mes can now be compiled (MES_MINI=1) to boot into a minimal Scheme (~2000 cells).
*** Mes now creates less garbage in the reader and in
append2, append_reverse, reverse, reverse!, vector-for-each, vector-to-list, vector-map.
*** 5 new functions
append-reverse, chmod, ioctl, isatty, isspace, last_pair, reverse!.
** Language
*** 3 new functions
char-whitespace?, chmod, isatty?
** Noteworthy bug fixes
*** Two bugs in the jam scraper/garbage collector have been fixed.
*** equal2_p now uses tail call elimination.
*** Escaped characters in strings are now read and write'd correctly.
*** The repl now expands macros again.
Greetings,
janneke
[0] https://gitlab.com/janneke/mes
[1] https://www.gnu.org/software/guix
[2] http://bootstrappable.org
[3] https://github.com/oriansj/m2-planet
[4] https://github.com/schemeway/lalr-scm
[5] https://www.cs.indiana.edu/chezscheme/syntax-case/old-psyntax.html
[6] https://www.nongnu.org/nyacc/
[7] https://www.gnu.org/software/guile/docs/master/guile.html/PEG-Parsing.html
[8] https://gitlab.com/janneke/tinycc
[9] https://lists.gnu.org/archive/html/tinycc-devel/2017-05/msg00103.html
[10] http://www.softwarepreservation.org/projects/LISP/book/LISP%25201.5%2520Programmers%2520Manual.pdf
[11] https://github.com/oriansj/stage0