From 6e32754a980acd861e7e225287b8b6eaab71e3a0 Mon Sep 17 00:00:00 2001 From: Jan Nieuwenhuizen Date: Sun, 22 Apr 2018 00:45:00 +0200 Subject: [PATCH] doc: Release udpate. * BOOTSTRAP: Update. * HACKING: Update. * NEWS: Update. * README: Update. * doc/ANNOUNCE-0.13: New file. --- BOOTSTRAP | 147 ++++++++++++++++++++++++++-------------------- HACKING | 10 ++-- NEWS | 20 +++++++ README | 22 +++---- doc/ANNOUNCE-0.13 | 81 +++++++++++++++++++++++++ 5 files changed, 200 insertions(+), 80 deletions(-) create mode 100644 doc/ANNOUNCE-0.13 diff --git a/BOOTSTRAP b/BOOTSTRAP index 618aad4b..427c5d31 100644 --- a/BOOTSTRAP +++ b/BOOTSTRAP @@ -2,17 +2,12 @@ bootstrappable.org project -*- org -*- * What? ** Full source bootstrapping for GuixSD -* Why? -** Reproducibility is essential to Software Freedom -Reproducible builds are a set of software development practices that -create a verifiable path from human readable source code to the binary -code used by computers. -*** What about the compiler? -We have the sources: they always lead to bitwise-same binary, but what -about the compiler? -*** The current way out: Ignore the problem - ``recipe for yoghurt: add yoghurt to milk'' +A package in GuixSD is can be uniquely identified by the hash of its +source code, its dependencies and its build recipe. + +Every package can be built from source, except for the bootstrap +binaries. *** From the GuixSD manual @@ -23,74 +18,96 @@ The root of this dependency graph is a small set of “bootstrap binaries”, provided by the ‘(gnu packages bootstrap)’ module. For more information on bootstrapping, *note Bootstrapping::. -*** New solution: Full source bootstrapping path, Stage0 and Mes +**** GuixSD bootstrap tarballs + +$ du -schx $(readlink $(guix build bootstrap-tarballs)/*) +2.1M /gnu/store/9623n4bq6iq5c8cwwdq99qb7d0xj93ym-binutils-static-stripped-tarball-2.28.1/binutils-static-stripped-2.28.1-x86_64-linux.tar.xz +18M /gnu/store/437xwygmmwwpkddcyy1qvjcv4hak89pb-gcc-stripped-tarball-5.5.0/gcc-stripped-5.5.0-x86_64-linux.tar.xz +1.8M /gnu/store/55ccx18a0d1x5y6a575jf1yr0ywizvdg-glibc-stripped-tarball-2.26.105-g0890d5379c/glibc-stripped-2.26.105-g0890d5379c-x86_64-linux.tar.xz +5.7M /gnu/store/bqf0ajclbvnbm0a46819f30804y3ilx0-guile-static-stripped-tarball-2.2.3/guile-static-stripped-2.2.3-x86_64-linux.tar.xz +5.8M /gnu/store/j8yzjmh9sy4gbdfwjrhw46zca43aah6x-static-binaries-tarball-0/static-binaries-0-x86_64-linux.tar.xz +33M total +$ for i in $(readlink $(guix build bootstrap-tarballs)/*);\ + do sudo tar xf $i; done +$ du -schx * +130M bin +13M include +54M lib +51M libexec +5.2M share +252M total + +* Why? +** Reproducibility is essential to Software Freedom + +Reproducible builds are a set of software development practices that +create a verifiable path from human readable source code to the binary +code used by computers. + +*** What about the bootstrap binaries and the compilers? + +We have the sources: they always lead to bitwise-same binary, but what +about the bootstrap binaries and compilers? + +*** The current way out: Ignore the problem + + ``recipe for yoghurt: add yoghurt to milk'' + +*** New solution: Full source bootstrapping path + * How? -** Software: Stage0 and Mes -** Stage0 +** Software: MesCC-tools, M2-Planet, Mes and MesCC +** MesCC-tools -*** hex.0: amazing ~300 byte self-hosting hex assembler that we consider to be source -*** a M0 macro assembler written in .0 -*** a M1 macro assembler written in M0 -*** a hex2 linker written in M0 + https://github.com/oriansj/mescc-tools -Look: - https://git.savannah.nongnu.org/cgit/stage0.git/tree/Linux%20Bootstrap/hex0.hex +*** hex.0: ~500 byte well-documented, self-hosting hex assembler -Do: - git clone https://git.savannah.nongnu.org/git/stage0.git - cd stage0 - make - bin/hex < Linux\ Bootstrap/hex0.hex > hex-1 - chmod +x hex-1 # later: bin/exec_enable hex-1 - ./hex-1 < Linux\ Bootstrap/hex0.hex > hex-2 +This 500 byte program is written in ASCII hex. When converted +byte-for-byte from ASCCI to binary we have the only binary seed that +our full source bootstrap path needs. -** Mes +We bless this simple and easily verifyable binary and consider it to +be source. + +*** hex1: next level hex assembler written in hex.0 +*** M0: a macro assembler written in hex.1 +*** M1: a macro assembler written in M0 +*** hex2: a hex2 linker written in M0 +*** M2-Planet: a self-hosting M2 (C-with-structs) transpiler written in M1 +*** Mes: A Scheme interpreter written in C, with cpp transformed into M2 +*** MesCC: A C compiler written in Scheme +*** tcc-boot: a patched version of TinyCC + +** Mes and MesCC https://gitlab.com/janneke/mes -*** mes.c: a scheme interpreter prototyped in C ~1400 Lines -*** mescc.scm: a C compiler written in Scheme (uses Nyacc C99 parser in Scheme) -*** mes.M1: this scheme interpreter in annotated M1 assembly +*** mes.c: a Scheme interpreter prototyped in C ~3000 Lines +*** mescc: a C compiler written in Scheme (uses Nyacc C99 parser in Scheme) +*** mes.M2: this Scheme interpreter in preprocessed M2 -** Naive recipe - hex + hex.0 => hex-1 - hex-1 + M0.0 -> M0 - M0 + M1.M0 -> M1.0 - M0 + hex2_linker.M0 -> hex2_linker - M1 + mes.M1 -> mes.hex2 - hex2_linker + stage0/elf32-header.hex2 + mes.hex2 + elf32-footer.hex2 -> mes - mes + mescc.scm + tcc.c -> tcc.M1 -> tcc.hex2 -> mes-tcc - mes-tcc + gcc.c -> gcc - *done* -** Less naive recipe -*** Use hex2_linker, M1 prototyped in C from - https://github.com/oriansj/mescc-tools -*** Remember that mes.M1 is compiled by mescc from prototyped src/mes.c in C. -*** stage0/mescc-tools: TODO -*** Mes - M1 -f stage0/x86.M1 -f lib/crt1.M1 --LittleEndian --Architecture=1 > lib/crt1.hex2 - M1 -f stage0/x86.M1 -f lib/mini-libc-mes.M1 --LittleEndian --Architecture=1 > lib/libc-mes.hex2 - M1 -f stage0/x86.M1 -f src/mes.M1 --LittleEndian --Architecture=1 > src/mes.hex2 - hex2 --LittleEndian --Architecture=1 --BaseAddress=0x1000000 -f stage0/elf32-header.hex2 -f lib/crt1.hex2 -f lib/libc-mes.hex2 -f src/mes.hex2 -f stage0/elf32-footer-single-main.hex2 > src/mes-mes - exec_enable src/mes-mes -*** Tinycc - ./build.sh - ./link.sh -* DONE -** stage0: hex.0, M0 done; M1, hex2_linker prototyped in C -** tcc compiled with mescc correctly compiles: int main () {return 42;} -** mes+mescc.scm are mutual self hosting -** during development we run mescc.scm on Guile (mes is slooowww) -** tcc compiled with gcc is known to compile gcc +** TinyCC + + https://gitlab.com/janneke/tinycc * TODO -** fix mescc.scm so that tcc can correctly compile gcc -** fix bootstrap-loops: (Nyacc?, mes.M1?, psyntax.pp?) +** fix MesCC so that tcc can correctly compile gcc +** remove or upstream patches from tcc +** fix bootstrap-loops: (Nyacc?, mes.M12, psyntax.pp?) ** make GNU gcc (8.0?) bootstrappable again, remove [need for] tcc stage -** stage1/2 LISP, FORTH? -** rain1's LISP compiler? ** integrate with GuixSD -** x86_64, arm? +** x86_64, arm, the Hurd + +* DONE +** hex.0, hex.1, M0 +** M1, hex2_linker prototyped in C +** M2-Planet is now self-hosting, written in M2 +** mes+mescc are mutual self-hosting +** patched tcc compiled with mes+mescc +** during development we run mescc on Guile (mes is ~30 times slower) +** tcc compiled with gcc is known to compile gcc + * Contact ** #bootstrappable, #guix on freenode ** bootstrappable.org diff --git a/HACKING b/HACKING index 0b4620ed..c3dbe7d2 100644 --- a/HACKING +++ b/HACKING @@ -59,8 +59,8 @@ mes.c. * ROADMAP ** TODO *** release 0.x, unsorted - - mes+mescc: compile a mes-tcc that can compile gcc-4.7. - - mes: set base-0.scm as default MES_BOOT, drop cat base-0.mes silliness. + - mes+mescc: compile a mes-tcc that can build itself. + - mes-tcc: full source bootstrapped build of gcc-4.7. - mes: real module support, bonus for supporting Guile's define-module/define-public syntax. - mes: prepare src/mes.c for M2-Planet transpiler. - mes: produce functional mes from mes.M1 transpiled by M2-Planet. @@ -100,6 +100,7 @@ mes.c. - add x86_64, replace GuixSD's bootstrap binaries for x86_64. - more architectures? ** DONE +*** 0.13: Mes+MesCC now compiles a modified, functional tcc.c (~25,000LOC) in 1h30'. *** 0.12: Mes+MesCC now compiles mes.c (~3000LOC) in ~4min. *** 0.11: MesCC now compiles a mes-tcc that passes 26/69 of mescc's C tests. *** 0.10: Mescc now compiles a mes-tcc that compiles a trivial C to a running a.out. @@ -137,9 +138,10 @@ eenough to work on compiling tinycc's tcc.c albeit a somewhat modified version. * Bugs ** mes: remove pmatch-car/pmatch-cdr hack. +** mescc: softcode stack frame size, now hardcoded and very large ** mes+mescc: parse tcc.c->tcc.E works, compile tcc.E -> tcc.M1 segfaults. - time GUILE_LOAD_PATH=/home/janneke/src/nyacc/module:$GUILE_LOAD_PATH ../mes/scripts/mescc.mes -E -o tcc.E -I . -I ../mes/lib -I ../mes/include -D 'CONFIG_TCCDIR="usr/lib/tcc"' -D 'CONFIG_TCC_CRTPREFIX="usr/lib:{B}/lib:."' -D 'CONFIG_TCC_ELFINTERP="/gnu/store/70jxsnpffkl7fdb7qv398n8yi1a3w5nx-glibc-2.26.105-g0890d5379c/lib/ld-linux.so.2"' -D 'CONFIG_TCC_LIBPATHS="/home/janneke/src/tinycc/usr/lib:{B}/lib:."' -D 'CONFIG_TCC_SYSINCLUDEPATHS="../mes/include:usr/include:{B}/include"' -D CONFIG_USE_LIBGCC=1 -D 'TCC_LIBGCC="/home/janneke/src/tinycc/usr/lib/libc+tcc-gcc.mlibc-o"' -D CONFIG_TCC_STATIC=1 -D ONE_SOURCE=yes -D TCC_TARGET_I386=1 -D BOOTSTRAP=1 tcc.c - time GUILE_LOAD_PATH=/home/janneke/src/nyacc/module:$GUILE_LOAD_PATH MES_ARENA=200000000 ../mes/scripts/mescc.mes -c -o tcc.M1 tcc.E + time GUILE_LOAD_PATH=/home/janneke/src/nyacc/module:$GUILE_LOAD_PATH ../mes/scripts/mescc -E -o tcc.E -I . -I ../mes/lib -I ../mes/include -D 'CONFIG_TCCDIR="usr/lib/tcc"' -D 'CONFIG_TCC_CRTPREFIX="usr/lib:{B}/lib:."' -D 'CONFIG_TCC_ELFINTERP="/gnu/store/70jxsnpffkl7fdb7qv398n8yi1a3w5nx-glibc-2.26.105-g0890d5379c/lib/ld-linux.so.2"' -D 'CONFIG_TCC_LIBPATHS="/home/janneke/src/tinycc/usr/lib:{B}/lib:."' -D 'CONFIG_TCC_SYSINCLUDEPATHS="../mes/include:usr/include:{B}/include"' -D CONFIG_USE_LIBGCC=1 -D 'TCC_LIBGCC="/home/janneke/src/tinycc/usr/lib/libc+tcc-gcc.mlibc-o"' -D CONFIG_TCC_STATIC=1 -D ONE_SOURCE=yes -D TCC_TARGET_I386=1 -D BOOTSTRAP=1 tcc.c + time GUILE_LOAD_PATH=/home/janneke/src/nyacc/module:$GUILE_LOAD_PATH MES_ARENA=200000000 ../mes/scripts/mescc -c -o tcc.M1 tcc.E ** mes: with-fluids: tests/fluids.test test 7 fails with Mes. ** mescc: 7n-struct-struct-array.c: struct file f = {"first.h"}; ** test/match.test ("nyacc-simple"): hygiene problem in match diff --git a/NEWS b/NEWS index 844c0015..201d93b7 100644 --- a/NEWS +++ b/NEWS @@ -10,6 +10,26 @@ Copyright © 2016,2017,2018 Jan (janneke) Nieuwenhuizen Please send Mes bug reports to janneke@gnu.org. +* Changes in 0.13 since 0.12 +** Core +*** Bootstrapped Mes+MesCC can now compile a patched tcc in ~1h45' (~25,000 LOC). +*** MesCC scripts for Mes and Guile are now merged; executable is: `mescc'. +*** Mes now uses only one arena for stop-and-copy; doubles available size. +*** Mes now has a Guile-like command-line interface (mes.repl has been removed). +*** Mes now boots into a full Scheme by default. +*** Mes can now be compiled (MES_MINI=1) to boot into a minimal Scheme (~2000 cells). +*** Mes now creates less garbage in the reader and in +append2, append_reverse, reverse, reverse!, vector-for-each, vector-to-list, vector-map. +*** 5 new functions +append-reverse, chmod, ioctl, isatty, isspace, last_pair, reverse!. +** Language +*** 3 new functions +char-whitespace?, chmod, isatty? +** Noteworthy bug fixes +*** Two bugs in the jam scraper/garbage collector have been fixed. +*** equal2_p now uses tail call elimination. +*** Escaped characters in strings are now read and write'd correctly. +*** The repl now expands macros again. * Changes in 0.12 since 0.11 ** Core *** Mes can now run Nyacc (0.80.42) to compile itself in ~4min (~3000 LOC). diff --git a/README b/README index 3ad71d04..599a029c 100644 --- a/README +++ b/README @@ -1,21 +1,20 @@ -*- org -*- Mes -- Maxwell Equations of Software -[[https://gitlab.com/janneke/mes][Mes]] aims to create full source bootstrapping for [[https://www.gnu.org/software/guix][GuixSD]] as part of the -[[http://bootstrappable.org][bootstrappable builds]] effort. +[[https://gitlab.com/janneke/mes][Mes]] aims to help create full source bootstrapping for [[https://www.gnu.org/software/guix][GuixSD]] as part +of the [[http://bootstrappable.org][bootstrappable builds]] effort. -It currently consists of a mutual self-hosting [close to Guile-] -Scheme interpreter prototype in C and a Nyacc-based C compiler in -[Guile] Scheme. This C prototype will be simplified to be transpiled -by [[https://github.com/oriansj/m2-planet][M2-Planet]]. +It currently consists of a mutual self-hosting Scheme interpreter +prototype written in C and a Nyacc-based C compiler written in Scheme. +This C prototype is being simplified to be transpiled by [[https://github.com/oriansj/m2-planet][M2-Planet]]. The Scheme interpreter prototype (mes.c) has a Garbage Collector, a library of loadable Scheme modules-- notably Dominique Boucher's [[https://github.com/schemeway/lalr-scm][LALR]], Pre-R6RS [[https://www.cs.indiana.edu/chezscheme/syntax-case/old-psyntax.html][portable syntax-case]] with R7RS ellipsis, Matt Wette's [[https://www.nongnu.org/nyacc][Nyacc]], Guile's [[https://www.gnu.org/software/guile/docs/master/guile.html/PEG-Parsing.html][PEG]] --and test suite just barely enough to support a simple -REPL (repl.mes) and simple C-compiler MesCC (mescc.mes). +REPL and simple C-compiler: MesCC. -MesCC can compile a [[https://gitlab.com/janneke/tinycc][modified TinyCC]] that is close to being +Mes+MesCC can compile a [[https://gitlab.com/janneke/tinycc][modified TinyCC]] that is close to being self-hosting. A GNU Ccc-compiled tcc is [[https://lists.gnu.org/archive/html/tinycc-devel/2017-05/msg00103.html][known]] to compile GCC. Mes is inspired by The Maxwell Equations of Software: [[http://www.softwarepreservation.org/projects/LISP/book/LISP%2525201.5%252520Programmers%252520Manual.pdf][LISP-1.5]] -- John @@ -50,9 +49,10 @@ General Public Licence version 3 or later. See the file COPYING. * REPL it - scripts/repl.mes + src/mes.gcc # default (64 bit) gcc-compiled + src/mes.mlibc-gcc # 32 bit, gcc -nostdlib + src/mes # 32 bit, bootstrapped, mes.M1+MesCC-compiled * MesCC compiler - scripts/mescc.mes scaffold/main.c > a.out - + scripts/mescc -c scaffold/main.c diff --git a/doc/ANNOUNCE-0.13 b/doc/ANNOUNCE-0.13 new file mode 100644 index 00000000..6e3026fd --- /dev/null +++ b/doc/ANNOUNCE-0.13 @@ -0,0 +1,81 @@ +Subject: Mes 0.13 released + +I am pleased to announce the release of Mes 0.13, representing 45 +commits over 3 weeks. MesCC can now compile a functional tcc when +running on Mes (in ~1h45') or on Guile (in ~3min). + +This means that we are getting very close to a full source bootstrap +of tcc: it is now built without gcc, glibc or guile; using only +MesCC-tools and Mes sources and corresponding ascii/binary seeds. + +* About + + Mes[0] aims to help create full source bootstrapping for GuixSD[1] + as part of the bootstrappable builds[2] project. + + It currently consists of a mutual self-hosting Scheme interpreter + prototype in C and a Nyacc-based C compiler in Scheme. This C + prototype is being simplified to be transpiled by M2-Planet[3]. + + The Scheme interpreter prototype (mes.c) has a Garbage Collector, + a library of loadable Scheme modules-- notably Dominique Boucher's + LALR[4], pre-R6RS portable syntax-case[5] with R7RS ellipsis, Matt + Wette's Nyacc[6], Guile's PEG[7] --and test suite just barely + enough to support a simple REPL (repl.mes) and simple C-compiler + MesCC. + + Mes+MesCC can compile a modified TinyCC[8] that is close to being + self-hosting. A GNU Ccc-compiled tcc is known[9] to compile GCC. + + Mes is inspired by The Maxwell Equations of Software: LISP-1.5[10] + -- John McCarthy page 13, GNU Guix's source/binary packaging + transparency and Jeremiah Orians's stage0[11] ~300 byte self-hosting + hex assembler. + +* Download + + git clone https://gitlab.com/janneke/mes + + wget https://gitlab.com/janneke/mes/-/archive/v0.13/mes-0.13.tar.gz + +Mes runs from the source tree and can also be built, packaged and +installed in Guix[SD] by the usual + + guix package -f guix.scm + +* Changes in 0.13 since 0.12 + ** Core + *** Bootstrapped Mes+MesCC can now compile a patched tcc in ~2h30' (~25,000 LOC). + *** MesCC scripts for Mes and Guile are now merged; executable is: `mescc'. + *** Mes now uses only one arena for stop-and-copy; doubles available size. + *** Mes now has a Guile-like command-line interface (mes.repl has been removed). + *** Mes now boots into a full Scheme by default. + *** Mes can now be compiled (MES_MINI=1) to boot into a minimal Scheme (~2000 cells). + *** Mes now creates less garbage in the reader and in + append2, append_reverse, reverse, reverse!, vector-for-each, vector-to-list, vector-map. + *** 5 new functions + append-reverse, chmod, ioctl, isatty, isspace, last_pair, reverse!. + ** Language + *** 3 new functions + char-whitespace?, chmod, isatty? + ** Noteworthy bug fixes + *** Two bugs in the jam scraper/garbage collector have been fixed. + *** equal2_p now uses tail call elimination. + *** Escaped characters in strings are now read and write'd correctly. + *** The repl now expands macros again. + +Greetings, +janneke + +[0] https://gitlab.com/janneke/mes +[1] https://www.gnu.org/software/guix +[2] http://bootstrappable.org +[3] https://github.com/oriansj/m2-planet +[4] https://github.com/schemeway/lalr-scm +[5] https://www.cs.indiana.edu/chezscheme/syntax-case/old-psyntax.html +[6] https://www.nongnu.org/nyacc/ +[7] https://www.gnu.org/software/guile/docs/master/guile.html/PEG-Parsing.html +[8] https://gitlab.com/janneke/tinycc +[9] https://lists.gnu.org/archive/html/tinycc-devel/2017-05/msg00103.html +[10] http://www.softwarepreservation.org/projects/LISP/book/LISP%25201.5%2520Programmers%2520Manual.pdf +[11] https://github.com/oriansj/stage0