doc: Release update.

* BOOTSTRAP: Release update.
* HACKING: Likewise.
* INSTALL: Likewise.
* NEWS: Likewise.
* PORTING: Likewise.
* README: Likewise.
* ROADMAP: Likewise.
* doc/mes.texi: Likewise.
* doc/announce/ANNOUNCE-0.22: New file.
This commit is contained in:
Jan Nieuwenhuizen 2019-12-09 07:26:16 +01:00
parent d8fca8321e
commit 34feefa559
No known key found for this signature in database
GPG key ID: F3C1A0D9C1D65273
10 changed files with 455 additions and 146 deletions

View file

@ -8,24 +8,24 @@ Copyright © 2016,2017,2018,2019 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
notice and this notice are preserved.
* What?
** Full source bootstrapping for GNU Guix
** Full source bootstrapping for the GNU System
A package in Guix is uniquely identified by the hash of its source code,
its dependencies, and its build recipe.
A package in GNU Guix is uniquely identified by the hash of its source
code, its dependencies, and its build recipe.
Every package can be built from source, except for the bootstrap
binaries.
*** From the Guix manual
*** From the GNU Guix manual
The distribution is fully “bootstrapped” and “self-contained”: each
The distribution is fully “bootstrapped” and “self-contained”: each
package is built based solely on other packages in the distribution.
The root of this dependency graph is a small set of “bootstrap
binaries”, provided by the (gnu packages bootstrap) module. For more
information on bootstrapping, *note Bootstrapping::.
**** Guix bootstrap tarballs
**** Guix v1.0 bootstrap binary seed
$ du -schx $(readlink $(guix build bootstrap-tarballs)/*)
2.1M /gnu/store/9623n4bq6iq5c8cwwdq99qb7d0xj93ym-binutils-static-stripped-tarball-2.28.1/binutils-static-stripped-2.28.1-x86_64-linux.tar.xz
@ -44,6 +44,42 @@ $ du -schx *
5.2M share
252M total
**** Guix Reduced Binary Seed bootstrap binary seed
$ du -schx $(readlink $(guix build bootstrap-tarballs)/*)
5.7M /gnu/store/9f8gi8raqfx9j3l9d00qrrc0jg3r1kyj-guile-static-stripped-tarball-2.2.6/guile-static-stripped-2.2.6-x86_64-linux.tar.xz
80K /gnu/store/b6rjl52hibhmvyw4dg8678pwryhla0h2-linux-libre-headers-stripped-tarball-4.19.56/linux-libre-headers-stripped-4.19.56-x86_64-linux.tar.xz
12K /gnu/store/d7zlxsjcnqilmvqwx7scija9x9bjw8cw-mescc-tools-static-stripped-tarball-0.5.2-0.bb062b0/mescc-tools-static-stripped-0.5.2-0.bb062b0-x86_64-linux.tar.xz
428K /gnu/store/n7zc4kpi8ny6jlfaikkzxlwhc5fvr1vr-mes-minimal-stripped-tarball-0.19/mes-minimal-stripped-0.19-x86_64-linux.tar.xz
6.0M /gnu/store/nv4djwlrljfqmynqr2cqvfwz0ydx7kxb-static-binaries-tarball-0/static-binaries-0-x86_64-linux.tar.xz
13M total
$ for i in $(readlink $(guix build bootstrap-tarballs)/*);\
do sudo tar xf $i; done
Password:
$ du -schx *
93M bin
700K include
38M lib
14M share
145M total
**** Guix Scheme-only bootstrap binary seed
$ du -schx $(readlink $(~/src/guix/wip-bootstrap/pre-inst-env guix build bootstrap-tarballs)/*)
5.7M /gnu/store/1mq2pcd2h7g54xpi2jrgj6ibbi4lgi3c-guile-static-stripped-tarball-2.2.6/guile-static-stripped-2.2.6-x86_64-linux.tar.xz
80K /gnu/store/bl1r2bpk6fam8r2gjvr5mvr48i3dm2hn-linux-libre-headers-stripped-tarball-4.19.56/linux-libre-headers-stripped-4.19.56-x86_64-linux.tar.xz
12K /gnu/store/w0dlz486dhb8aiq8pxm5akllz628fqin-mescc-tools-static-stripped-tarball-0.5.2-0.bb062b0/mescc-tools-static-stripped-0.5.2-0.bb062b0-x86_64-linux.tar.xz
428K /gnu/store/15j6l18q44ymlrh1cfp4s4hc9835xic5-mes-minimal-stripped-tarball-0.19/mes-minimal-stripped-0.19-x86_64-linux.tar.xz
6.2M total
$ for i in $(readlink $(~/src/guix/wip-bootstrap/pre-inst-env guix build bootstrap-tarballs)/*);\
do sudo tar xf $i; done
$ du -schx *
4.9M bin
700K include
38M lib
14M share
57M total
* Why?
** Reproducibility is essential to Software Freedom
@ -92,20 +128,22 @@ be source.
*** mes.c: a Scheme interpreter in ~5,000LOC of simple C
*** mescc: a C compiler written in Scheme (uses Nyacc C99 parser in Scheme)
*** mes.M2: this Scheme interpreter in preprocessed M2
*** mes.M2: this Scheme interpreter in bootstrappable M2
** TinyCC
https://gitlab.com/janneke/tinycc
* TODO
** merge scheme-only bootstrap into Guix.
** reduced binary seed bootstrap into NixOS, Debian.
** remove or upstream patches from tcc-boot
** prepare src/mes.c for M2-Planet transpiler.
** ARM, the Hurd
** fix bootstrap-loops: (Nyacc?, mes.M2, psyntax.pp?)
** make GNU gcc (8.0?) bootstrappable again, remove [need for] tcc stage
* DONE
** upstream mes-boot to Guix.
** reduced binary seed bootstrap into Guix.
** replace Guix bootstrap for x86_64.
** replace Guix bootstrap for x86.
** add full source gcc-4.7 package build.
@ -121,5 +159,6 @@ be source.
** tcc compiled with gcc is known to compile gcc
* Contact
** bug-mes@gnu.org
** #bootstrappable, #guix on freenode
** bootstrappable.org

70
HACKING
View file

@ -1,64 +1,31 @@
-*- org -*-
#+TITLE: Hacking GNU Mes
Copyright © 2016,2017,2018 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
Copying and distribution of this file, with or without modification,
are permitted in any medium without royalty provided the copyright
notice and this notice are preserved.
* SETUP
guix environment -l guix.scm #64 bit + 32bit
CC=i686-unkown-linux-gnu-gcc ./configure
or
./configure
guix environment --system=i686-linux -l guix.scm #32 bit only
or
guix package --profile=~/.config/guix/mes --manifest=build-aux/manifest.scm
. ~/.config/guix/mes/etc/profile
The 64bit bootstrap is not supported; MesCC needs work to be able to
compile a functional tinycc.
* BUILD
There are two major modes to build Mes: true bootstrap and
development.
** DEVELOPMENT BUILD
To help development we assume ./configure sets these variables for make
GNU Guile can be used as a replacement for Mes. It's faster and has
much better error handling.
CC -- gcc (or i686-unknown-linux-gnu-gcc sans libc)
GUILE -- guile
HEX2 -- hex2
MES -- unset
M1 -- M1
prefix -- ""
#+BEGIN_SRC bash
make MES=guile
make check MES=guile
#+END_SRC
Mes is supposed to serve as a full equivalent for Guile, however Mes
is still about 2 to 10 times slower than Guile. That's why we usually
don't use Mes during development, configure --with-cheating.
Configure to compile with Tiny CC
Gcc is used to verify the sanity of our C sources.
i686-unknown-linux-gnu-gcc is used to compare hex/assembly, to test
the gcc variant of Mes C Libirary.
Target prefix: x86-mes-gcc.
gcc -nostdinc,-nostdlib is used to compare hex/assembly, to test the
64bit variant of Mes C Library. Target prefix: x86_64-mes-gcc.
Guile is used to develop MesCC, the C compiler in Scheme that during
bootstrapping will be executed by Mes.
** BOOTSTRAP BUILD
./configure.sh [--prefix=PREFIX]
./bootstrap.sh
./install.sh
In bootstrap mode, we don't have gcc (CC), we don't have a 32 bit gcc,
we have no guile (GUILE)...but we should have hex2, M1, and mes.M1.
That's a bootstrap problem which is currently ignored by using the
mes-seed package. mes.M1 will be produced by M2-Planet from mes.c.
#+BEGIN_SRC bash
./configure CC=tcc --host=i686-unknown-linux-gnu --with-courage
#+END_SRC
* DEBUG
MES_DEBUG=<level> mes
@ -83,7 +50,7 @@ mes-seed package. mes.M1 will be produced by M2-Planet from mes.c.
* Bugs
** mes: performance, Mes is now 2-10x slower than Guile.
** mes/mescc lack support for the Hurd.
** mes/mescc lack support for fork/exec on the Hurd.
** mes: gcc-x86_64 compiled mes segfaults with small arena, or gc_up_arena.
** mes: gcc-x86 compiled, tests/srfi-13.test number->string INT-MIN fails:
test: number->string INT-MIN: fail
@ -214,3 +181,10 @@ https://notabug.org/rain1/hex86/src/master/tests/hex0b3.hex86
https://jamey.thesharps.us/2016/07/15/testing-strategies-for-corrode/
("Randomized testing with Csmith and C-Reduce") [10:58]
** linux syscalls: https://fedora.juszkiewicz.com.pl/syscalls.html
* legalese
Copyright © 2016,2017,2018,2019 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
Copying and distribution of this file, with or without modification,
are permitted in any medium without royalty provided the copyright
notice and this notice are preserved.

23
INSTALL
View file

@ -41,6 +41,19 @@ similar way.
make
#+END_SRC
If you get something like
#+BEGIN_SRC bash
mes.c:(.text+0x8da0): undefined reference to `__stack_chk_fail'
#+END_SRC
then your compiler inserts stack protection code. The Mes C Library
does not support that. Turn it off lik this
#+BEGIN_SRC bash
./configure CFLAGS=-fno-stack-protector
#+END_SRC
** Check it
#+BEGIN_SRC bash
@ -86,3 +99,13 @@ https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/commencement.scm
#+BEGIN_SRC bash
sh install.sh
#+END_SRC
* Regular build and bootstrap combine
To build mes.c using MesCC, configure using --with-bootstrap:
#+BEGIN_SRC bash
./configure --with-bootstrap
make
#+END_SRC
This creates bin/mes-gcc and bin/mes-mescc.

36
NEWS
View file

@ -10,6 +10,41 @@ Copyright © 2016,2017,2018,2019 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
Please send Mes bug reports to bug-mes@gnu.org.
* Changes in 0.22 since 0.21
** Core
*** Mes now builds reproducibly with MesCC, cross distribution
Using --with-bootstrap on x86, a bin/mes-mescc is compiled with
mes+mescc, that shall have sha256sum
9e0bcb1633c58e7bc415f6ea27cee7951d6b0658e13cdc147e992b31a14625fb bin/mes-mescc
This has been verified on GNU Guix, Debian GNU/Linux and NixOS.
*** Mes now builds reproducibly with Guile+MesCC vs Mes+MesCC.
*** Mes now builds reproducibly with Tiny CC-built MesCC.
*** configure now respects the user's CFLAGS, CPPFLAGS and LDFLAGS.
*** Mes now supports mescc-tools 0.5.2 next to 0.6.1.
This allows introducing the Scheme-only bootstrap without updating or
adding new bootstrap binary seeds into GNU Guix.
*** Mes now runs when configured --with-courage on the Hurd:
GNU debian 0.9 GNU-Mach 1.8+git20170609-486/Hurd-0.9 i686-AT386 GNU
Note that mescc does not run; fork and exec are not yet implemented.
*** Mes now configures --with-courage on x86-FreeBSD 12.1
The initial ELF scaffold tests: exit-42.S, 0exit-42.hex2,
body-exit-42.hex2, hello-mes.S, 0hello-mes.hex2, body-hello-mes.hex2
pass.
*** configure now has a --with-bootstrap option.
This adds the reproducible bootstrap build of bin/mes-mescc, using
mes+mescc.
*** configure prefers $CC over gcc; tcc over gcc, gcc over cc.
*** Mes now prints an error when attempting to read a file that does not exist.
*** Mes no longer depends on GIT.
** Noteworthy bug fixes
*** Several annoying build problems were fixed, for non-Guix systems.
*** A bug with buffered-read was fixed.
This means that bash-2.05 can now build glibc-2.2.5; notably it now
successfully executes make-syscall.sh.
*** A bug with execlp, execvp for file names containing a slash was fixed.
This should allow make-3.80 running scripts that have "#! ./move-if-change".
* Changes in 0.21 since 0.20
** Core
*** Mes can now be bootstrapped with Gash and Gash Core Utils.
@ -48,6 +83,7 @@ take-while.
*** 1 new function
__mesabi_uldiv.
** Noteworthy bug fixes
*** map and for-each now support lists with unequal length.
*** interger division has been fixed.
*** isatty now looks at terminfo.
*** signal now uses sigaction correctly for non-x86.

22
PORTING
View file

@ -54,6 +54,28 @@ bootstrap for x86_64 uses x86 mes and that is not expected to change.
The Hurd port can be found in wip-hurd on savannah. For development, we
use a Debian GNU/Hurd vm.
* Porting GNU Mes to FreeBSD
The FreeBSD port can be found in wip-freebsd on savannah.
* Porting the Reduced Binary Seed bootstrap to NixOS
The NixOS port of the Reduced Binary Seed bootstrap lives on the
mes-bootstrap branch in https://github.com/xwvvvvwx/nixpkgs
* Porting the Reduced Binary Seed bootstrap to Debian GNU/Linux
To port the Reduced Binary Seed bootstrap to a traditional distribution
such as Debian, two things need to happen: the bootstrap must be ported
and the distribution build process needs to change to start from a only
binary seed.
The porting aspect is probably easiest: we start by packaging GNU Mes as
a regular package. Having Mes built as a regular package, we can then
build a bootstrap Mes. This bootstrap Mes can be used to build tinycc,
and so on.
Once we have shown that the Debian base system can be bootstrapped from
a Reduced Binary seed, thus significantly reducing the Trusted Computing
Base (TCB), strategic decisions about the build process can start.
* Legalese
Copyright © 2019 Jan (janneke) Nieuwenhuizen <[[mailto:janneke@gnu.org][janneke@gnu.org]]>

22
README
View file

@ -4,20 +4,20 @@
#+SUBTITLE: Maxwell Equations of Software
[[https://www.gnu.org/software/mes][GNU Mes]] is a Scheme interpreter and C compiler for bootstrapping the GNU
system. Since version 0.20 it brings a [[https://guix.gnu.org/blog/2019/guix-reduces-bootstrap-seed-by-50/][Reduced Binary Seed bootstrap]] to
[[https://www.gnu.org/software/guix][GNU Guix]]. This bootstrap has halved the size of opaque, uninspectable
binaries that were needed to bootstrap Guix 1.0. The final goal is to
help create a full source bootstrap as part of the [[http://bootstrappable.org][bootstrappable builds]]
System. Since version 0.22 it has again helped to halve the size of
opaque, uninspectable binary seeds that are currently being used in the
[[https://guix.gnu.org/blog/2019/guix-reduces-bootstrap-seed-by-50/][Reduced Binary Seed bootstrap]] of [[https://www.gnu.org/software/guix][GNU Guix]]. The final goal is to help
create a full source bootstrap as part of the [[http://bootstrappable.org][bootstrappable builds]]
effort for UNIX-like operating systems.
Mes consists of a mutual self-hosting Scheme interpreter written in
~5,000 LOC of simple C, and a C compiler written in Scheme. This mes.c
is [[https://github.com/oriansj/mes-m2][being simplified]] to be transpiled by [[https://github.com/oriansj/m2-planet][M2-Planet]].
The Scheme interpreter is written in ~5,000 LOC of simple C, and the C
compiler written in Scheme and these are mutual self-hosting. This
mes.c is [[https://github.com/oriansj/mes-m2][being simplified]] to be transpiled by [[https://github.com/oriansj/m2-planet][M2-Planet]].
The Scheme interpreter has a Garbage Collector, a library of loadable
Scheme modules-- notably Dominique Boucher's [[https://github.com/schemeway/lalr-scm][LALR]], Pre-R6RS [[https://www.cs.indiana.edu/chezscheme/syntax-case/old-psyntax.html][portable
syntax-case]] with R7RS ellipsis, Matt Wette's [[https://www.nongnu.org/nyacc][Nyacc]] --and test suite just
enough to support a REPL and a C99 compiler: MesCC.
Mes has a Garbage Collector, a library of loadable Scheme modules--
notably Dominique Boucher's [[https://github.com/schemeway/lalr-scm][LALR]], Pre-R6RS [[https://www.cs.indiana.edu/chezscheme/syntax-case/old-psyntax.html][portable syntax-case]] with
R7RS ellipsis, Matt Wette's [[https://www.nongnu.org/nyacc][Nyacc]] --and test suite, just enough to
support a REPL and a C99 compiler: mescc.
Mes+MesCC can compile an only [[http://gitlab.com/janneke/tinycc][lightly patched TinyCC]] that is
self-hosting. Using this tcc and the Mes C library we now have a

31
ROADMAP
View file

@ -9,7 +9,7 @@ Copyright © 2016,2017,2018,2019 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
The goals of the GNU Mes project are two-fold:
- to bootstrap the GNU system purely from source, without using any binary
- to bootstrap the GNU System purely from source, without using any binary
seeds.
- to aid the Reproducible-Builds.org effort; demonstrate the impact and
@ -23,23 +23,32 @@ on Freenode!
* TODO
** release 0.x, unsorted
- Merge the Scheme-only bootstrap which uses Gash and Gash Core
Utilities to remove from bootstrap-binary seed : Awk, Bash, Core
Utilities, Grep, Gzip, Sed, Tar.
- Introduce the Reduced Binaries Seed bootstrap to NixOS (Debian,
Gentoo, ...?)
- Full Source Bootstrap: compile Mes.c using M2-Planet. Jeremiah
Arch, Gentoo, ...?)
- Full Source Bootstrap: compile mes.c using M2-Planet. Jeremiah
branched-out from mes; see https://github.com/oriansj/mes-m2, also see the
branches: wip-m2-merge and wip-m2.
- Scheme-only bootstrap bootstrap-binaries with Gash and Gash Core
Utilities: Awk, Bash, Core Utilities, Grep, Gzip, Make, Sed, Tar.
branches: wip-full-source-bootstrap, wip-m2-merge and wip-m2.
- Full Guile-compatible module support: define-module, define-public,
export, use-modules, ...
- ARMv7/AArch64 support: Mes C Library and MesCC.
- Simplify eval-apply and other core functionality (see mes-m2).
** after release 1.0
- Replace the initial gcc-2.95.3 with gcc-4.6.4.
- Support native x86_64 bootstrap.
- Simplify MesCC ``MesCC should optimize ease of convincing us of its
correctness''
- tcc: remove or upstream patches from tcc-boot.
- tcc: build 0.9.27 directly instead of via 0.9.26, see tinycc
** after release 1.0
- Build Guix packages using Mes: run or simplify (guix build utils),
(guix build gnu-build-system).
- Build Guix packages without guix-daemon. See Ludovic's initrd build
work.
- Bootstrap support for the Hurd. Most needed now are fork and exec.
- Replace the initial gcc-2.95.3 with gcc-4.6.4.
- FreeBSD, *BSD?
wip-bootstrappable@0.9.27 branch
- Bootstrap a `bootstrap-Guile' before bootstrapping tcc?
- Skip tcc: Build gcc using MesCC.
- better garbage collector.
- mes/mescc: proper docstrings, api reference documentation.
@ -51,9 +60,9 @@ on Freenode!
+ get full source syntax-case up (Andre van Tonder?)
https://srfi.schemers.org/srfi-72/srfi-72.html, or
+ ... drop it?
- Support the Hurd. There is a wip-hurd branch; most needed now are fork
and exec.
* DONE
- 0.22 GNU Mes now builds reproducibly cross-platform, runs on the
Hurd and has better support for other distributions and kernels.
- 0.21 GNU Mes now supports a Scheme-only bootstrap and is packaged in
Debian GNU/Linux.
- 0.20 GNU Mes brings the Reduced Binary Source bootstrap to Guix.

View file

@ -37,18 +37,18 @@ Packages are available in Guix master.
GNU Mes[0] brings a Reduced Binary Seed bootstrap[1] to GNU Guix[2].
This bootstrap has halved the size of opaque, uninspectable binaries
that were needed to bootstrap Guix. The final goal is to help create
a full source bootstrap as part of the bootstrappable builds[3] effort
for any interested UNIX-like operating system.
that were needed to bootstrap Guix 1.0. The final goal is to help
create a full source bootstrap as part of the bootstrappable builds[3]
effort for any interested UNIX-like operating system.
It consists of a mutual self-hosting Scheme interpreter written in
~5,000 LOC of simple C and a Nyacc-based C compiler written in Scheme.
This mes.c is being simplified[4] to be transpiled by M2-Planet[5].
Mes consists of a mutual self-hosting Scheme interpreter written in
~5,000 LOC of simple C, and a C compiler written in Scheme. This
mes.c is being simplified[4] to be transpiled by M2-Planet[5].
The Scheme interpreter has a Garbage Collector, a library of loadable
Scheme modules-- notably Dominique Boucher's LALR[6], Pre-R6RS
[portable syntax-case[7] with R7RS ellipsis, Matt Wette's Nyacc[8]
--and test suite just enough to support a REPL and a C99 compiler:
--and test suite, just enough to support a REPL and a C99 compiler:
MesCC.
Mes+MesCC can compile an only lightly patched TinyCC[9] that is

162
doc/announce/ANNOUNCE-0.22 Normal file
View file

@ -0,0 +1,162 @@
Subject: GNU Mes 0.22 released
<#secure method=pgpmime mode=sign>
We are pleased to announce the release of GNU Mes 0.22, representing
57 commits over 8 weeks.
Mes can now be built reproducibly by mes+mescc or guile+mescc accross
distributions such as GNU Guix, Debian GNU/Linux and NixOS. Also, many
build issues were fixed. A big thank you to reproducible-builds.org,
Vagrant Cascadian, Jelle van der Waa, Hannes Mehnert and David Terry!
Mes now runs on the Hurd--mescc does not run yet, many system calls are
missing, especially fork/exec--and initial scaffolding support for
creating FreeBSD binaries was added.
This release better supports bringing the Scheme-only bootstrap to Guix:
it no longer requires an update to the Guix bootstrap seed.
The Scheme-only bootstrap means that Mes can now be built with Gash and
the experimental Gash Core Utils instead of using GNU Awk, GNU Bash, the
GNU Core Utilities, GNU Grep, GNU Gzip, GNU Make, GNU Sed, and GNU Tar.
Also, the Mes C Library now supports bootstrapping those.
We are excited that the Nlnet Foundation[12] is now sponsoring this
work!
Next targets:
- Introduce the Reduced Binaries Seed bootstrap to NixOS (Debian,
Arch, Gentoo, ...?)
- Merge the Scheme-only bootstrap, the wip-bootstrap branch (using
Guile, Gash and Gash Core Utils to remove awk, bash, core utilities,
grep, gzip, make, sed, tar, etc. from the Guix bootstrap binaries)
into Guix core-updates.
- ARM support
- Full Source Bootstrap: compile Mes.c using M2-Planet
- Add full Guile module support
- Reduced Binary Seed bootstrap for ARM
- Support mescc on the Hurd (fork/exec, ...)
Packages are available in Guix master; the cross-distro reproducible
is built using
guix build --system=i686-linux mes-rb5
* About
GNU Mes[0] is a Scheme interpreter and C compiler for bootstrapping the GNU
System. Since version 0.22 it has again helped to halve the size of
opaque, uninspectable binary seeds that are currently being used in the
Reduced Binary Seed bootstrap[1] of GNU Guix[2]. The final goal is to help
create a full source bootstrap as part of the bootstrappable builds[3]
effort for UNIX-like operating systems.
The Scheme interpreter is written in ~5,000 LOC of simple C, and the C
compiler written in Scheme and these are mutual self-hosting. This
mes.c is being simplified[4] to be transpiled by M2-Planet[5].
Mes has a Garbage Collector, a library of loadable Scheme modules--
notably Dominique Boucher's LALR[6], Pre-R6RS [portable syntax-case[7]
with R7RS ellipsis, Matt Wette's Nyacc[8] --and test suite just enough
to support a REPL and a C99 compiler: MesCC.
Mes+MesCC can compile an only lightly patched TinyCC[9] that is
self-hosting. Using this tcc and the Mes C library we now have a
Reduced Binary Seed bootstrap for the gnutools triplet: glibc-2.2.5,
binutils-2.20.1, gcc-2.95.3. This is enough to bootstrap Guix for
i686-linux and x86_64-linux.
Mes is inspired by The Maxwell Equations of Software: LISP-1.5[10] -- John
McCarthy page 13, GNU Guix's source/binary packaging transparency and
Jeremiah Orians's stage0[11] ~500 byte self-hosting hex assembler.
* Download
git clone git://git.savannah.gnu.org/mes.git
Here are the compressed sources and a GPG detached signature[*]:
https://ftp.gnu.org/gnu/mes/mes-0.22.tar.gz
https://ftp.gnu.org/gnu/mes/mes-0.22.tar.gz.sig
Use a mirror for higher download bandwidth:
https://ftpmirror.gnu.org/mes/mes-0.22.tar.gz
https://ftpmirror.gnu.org/mes/mes-0.22.tar.gz.sig
Here are the MD5 and SHA1 checksums:
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx mes-0.22.tar.gz
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx mes-0.22.tar.gz
[*] Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact. First, be sure to download both the .sig file
and the corresponding tarball. Then, run a command like this:
gpg --verify mes-0.22.tar.gz.sig
If that command fails because you don't have the required public key,
then run this command to import it:
gpg --keyserver keys.gnupg.net --recv-keys 1A858392E331EAFDB8C27FFBF3C1A0D9C1D65273
and rerun the 'gpg --verify' command.
* Get informed, get involved
See https://bootstrappable.org
Join #bootstrappable on irc.freenode.net.
* Changes in 0.22 since 0.21
** Core
*** Mes now builds reproducibly with MesCC, cross distribution
Using --with-bootstrap on x86, a bin/mes-mescc is compiled with
mes+mescc, that shall have sha256sum
9e0bcb1633c58e7bc415f6ea27cee7951d6b0658e13cdc147e992b31a14625fb bin/mes-mescc
This has been verified on GNU Guix, Debian GNU/Linux and NixOS.
*** Mes now builds reproducibly with Guile+MesCC vs Mes+MesCC.
*** Mes now builds reproducibly with Tiny CC-built MesCC.
*** configure now respects the user's CFLAGS, CPPFLAGS and LDFLAGS.
*** Mes now supports mescc-tools 0.5.2 next to 0.6.1.
This allows introducing the Scheme-only bootstrap without updating or
adding new bootstrap binary seeds into GNU Guix.
*** Mes now runs when configured --with-courage on the Hurd:
GNU debian 0.9 GNU-Mach 1.8+git20170609-486/Hurd-0.9 i686-AT386 GNU
Note that mescc does not run; fork and exec are not yet implemented.
*** Mes now configures --with-courage on x86-FreeBSD 12.1
The initial ELF scaffold tests: exit-42.S, 0exit-42.hex2,
body-exit-42.hex2, hello-mes.S, 0hello-mes.hex2, body-hello-mes.hex2
pass.
*** configure now has a --with-bootstrap option.
This adds the reproducible bootstrap build of bin/mes-mescc, using
mes+mescc.
*** configure prefers $CC over gcc; tcc over gcc, gcc over cc.
*** Mes now prints an error when attempting to read a file that does not exist.
*** Mes no longer depends on GIT.
** Noteworthy bug fixes
*** Several annoying build problems were fixed, for non-Guix systems.
*** A bug with buffered-read was fixed.
This means that bash-2.05 can now build glibc-2.2.5; notably it now
successfully executes make-syscall.sh.
*** A bug with execlp, execvp for file names containing a slash was fixed.
This should allow make-3.80 running scripts that have "#! ./move-if-change".
Greetings,
janneke and Danny.
[0] https://www.gnu.org/software/mes
[1] https://guix.gnu.org/blog/2019/guix-reduces-bootstrap-seed-by-50/
[2] https://www.gnu.org/software/guix
[3] https://bootstrappable.org
[4] https://github.com/oriansj/mes-m2
[5] https://github.com/oriansj/m2-planet
[6] https://github.com/schemeway/lalr-scm
[7] https://www.cs.indiana.edu/chezscheme/syntax-case/old-psyntax.html
[8] https://www.nongnu.org/nyacc
[9] https://gitlab.com/janneke/tinycc
[10] http://www.softwarepreservation.org/projects/LISP/book/LISP%201.5%20Programmers%20Manual.pdf
[11] https://savannah.nongnu.org/projects/stage0
[12] https://nlnet.nl/project/GNUMes

View file

@ -29,7 +29,7 @@ Documentation License''.
* mes: (mes)Invoking mes. Running Mes, a minimalist Guile lookalike.
* mescc: (mes)Invoking MesCC. Running the MesCC bootstrap compiler.
* mescc: (mes)Invoking mescc. Running the MesCC bootstrap compiler.
@end direntry
@titlepage
@ -48,7 +48,7 @@ Edition @value{EDITION} @*
@contents
@c *********************************************************************
@node Top
@node Top, Introduction, (dir), (dir)
@top GNU Mes
This document describes GNU Mes version @value{VERSION}, a bootstrappable
@ -73,6 +73,7 @@ Software Freedom
* Reproducible Builds:: Reproducibility and free software.
* Bootstrappable Builds:: The freedom to build a software without binary seed.
* Full Source Bootstrap:: Software dependencies worthy of GNU.
* LISP as Maxwell's Equations of Software:: Auditable elegance.
Installation
@ -83,15 +84,15 @@ Installation
Bootstrapping
* The Mes Bootstrap Process:: How Mes will make you yogurt from pure milk.
* Invoking Mes:: Running Mes, a minimalist Guile lookalike.
* Invoking MesCC:: Running the MesCC bootstrap compiler.
* Invoking mes:: Running Mes, a minimalist Guile lookalike.
* Invoking mescc:: Running the MesCC bootstrap compiler.
* Invoking mesar::
Invoking Mes
Invoking mes
* Environment Variables:: If the bits won't change, change their habitat.
Invoking MesCC
Invoking mescc
* MesCC Environment Variables:: There's no NIX like POSIX.
@ -108,7 +109,7 @@ Contributing
@end menu
@c *********************************************************************
@node Introduction
@node Introduction, Installation, Top, Top
@chapter Introduction
@quotation
@ -187,9 +188,10 @@ Freedom 1.
* Reproducible Builds:: Reproducibility and free software.
* Bootstrappable Builds:: The freedom to build a software without binary seed.
* Full Source Bootstrap:: Software dependencies worthy of GNU.
* LISP as Maxwell's Equations of Software:: Auditable elegance.
@end menu
@node Reproducible Builds
@node Reproducible Builds, Bootstrappable Builds, Introduction, Introduction
@section Reproducible Builds
The current Reproducible Builds effort incubated in the Debian
@ -238,7 +240,7 @@ packages could be rebuild bit-identical from source.
@author Martin Uecker
@end quotation
@node Bootstrappable Builds
@node Bootstrappable Builds, Full Source Bootstrap, Reproducible Builds, Introduction
@section Bootstrappable Builds
Software distributions that take reproducible builds seriously are
@ -250,14 +252,14 @@ opaque ascii or binary @emph{seeds} that are injected during build
time. Yes, a package may build reproduciblly from all inspectable
sourcess...but what functionality is programmed in the opaque seed?
@subsection Bootstrap Binaries
@subsection Bootstrap Binary Seed
Possibly one of the most harmless, but certainly by far the biggest
binary seed that all software distributions inject are the so called
@emph{bootstrap binaries}. Bootstrap binaries are the initial binary
@emph{bootstrap binary seed}. Bootstrap binaries are the initial binary
seeds that are used to start building the distribution.
The GNU Guix operating system, version 1.0 had a relatively small
closure of bootstrap binaries: GNU binutils, GNU gcc, GNU Libc, GNU
closure of bootstrap binary seed: GNU binutils, GNU gcc, GNU Libc, GNU
Guile, and ``Static binaries'' (think: bash, bzip2, coreutils, gawk,
grep, gzip, patch, sed, tar, xz).
@ -287,17 +289,17 @@ $ du -schx *
@end example
During the Guix 1.1 development series we managed to create the first
reduction by 50% of the Guix @emph{bootstrap binaries}@footnote{See
reduction by 50% of the Guix @emph{bootstrap binary seed}@footnote{See
@url{https://guix.gnu.org/blog/2019/guix-reduces-bootstrap-seed-by-50/}}.
This was a very important step because the ~250MB @emph{seed} of binary
code was practically non-auditable, which makes it hard to establish
what source code produced them.
@node Full Source Bootstrap
@node Full Source Bootstrap, , Bootstrappable Builds, Introduction
@section Full Source Bootstrap
There is an obvious solution: we cannot allow any binary seeds in our
software stack. Not even in the bootstrap binaries. Maybe that is a
software stack. Not even in the bootstrap binary seed. Maybe that is a
bit too strong: we want to have the absolute minimum of binary seeds and
all binary seeds need to be inspectable and must be reviewed. How big
would the absolute minimal set be?
@ -321,8 +323,8 @@ of the @var{hex} function
@end example
All computer programs look like this: an opaque list of computer codes.
The initial programs that we take for granted---the bootstrap
binaries---are about 250MB of such numbers: think 250,000 pages full of
The initial programs that we take for granted---the bootstrap binary
seed---are about 250MB of such numbers: think 250,000 pages full of
numbers.
Most computers work pretty well so apparently there is not a pressing
@ -407,7 +409,8 @@ nothing to all of the programs that our computer runs. Something that
seemed to be an impossible dream is suddenly starting to look like
``just a couple years of work''.
@subsection LISP as Maxwell's Equations of Software
@node LISP as Maxwell's Equations of Software
@section LISP as Maxwell's Equations of Software
As fate would have it, I stumbled upon this
@url{https://queue.acm.org/detail.cfm?id=1039523, interview with Alan
@ -426,7 +429,7 @@ over.
@end quotation
Our starting point is @file{hex0}, a 500 byte hex assembler and we need
to somehow close the gap to building the bootstrap binaries, esp. GNU
to somehow close the gap to building the bootstrap binary seed, esp. GNU
Gcc and the GNU C Library. What better way to do that than by
leveraging the powers of LISP?
@ -435,8 +438,49 @@ will be indirectly bootstrapped from @file{hex0} and that wields the
magical powers of LISP to close the bootstrap gap, asserting we can
enjoy software Freedom 1.
@subsection Auditable Elegance
@code{eval} and @code{apply} are mutual recursing functions that---using
a few helper functions---describe the core of the universe of computing.
@lisp
(define (apply fn x a)
(cond
((atom fn)
(cond
((eq fn CAR) (caar x))
((eq fn CDR) (cdar x))
((eq fn CONS) (cons (car x) (cadr x)))
((eq fn ATOM) (atom (car x)))
((eq fn EQ) (eq (car x) (cadr x)))
(#t (apply (eval fn a) x a))))
((eq (car fn) LAMBDA)
(eval (caddr fn) (pairlis (cadr fn) x a)))
((eq (car fn) LABEL)
(apply (caddr fn) x
(cons (cons (cadr fn) (caddr fn)) a)))))
@end lisp
@lisp
(define (eval e a)
(cond
((atom e) (cdr (assoc e a)))
((atom (car e))
(cond ((eq (car e) QUOTE) (cadr e))
((eq (car e) COND) (evcon (cdr e) a))
(#t (apply (car e) (evlis (cdr e) a) a))))
(#t (apply (car e) (evlis (cdr e) a) a))))
@end lisp
It will be a big day when our computers are fully bootstrapped from
source. It would be nice if that source code were readable, auditable
and elegant. To be honest, the elegance displayed above that we
achieved at the very start of the Mes project is currently hard to find.
It is our sincerest hope to bring back this level of quality and
elegance..
@c *********************************************************************
@node Installation
@node Installation, Bootstrapping, Introduction, Top
@chapter Installation
@cindex installing Mes
@ -452,7 +496,7 @@ to use it.
* Running the Test Suites:: Testing Mes.
@end menu
@node Regular Requirements
@node Regular Requirements, Bootstrap Requirements, Installation, Installation
@section Regular Requirements
This section lists requirements when building Mes from source. The
@ -476,7 +520,7 @@ Mes is compatible with GNU Guile, so it is possible to share the same
Scheme code between both. Currently Mes only supports the minimal
subset of R5RS and Guile extensions to run MesCC.
@node Bootstrap Requirements
@node Bootstrap Requirements, Running the Test Suites, Regular Requirements, Installation
@section Bootstrap Requirements
This section lists requirements when building Mes as a bootstrap
@ -501,7 +545,7 @@ Bootstrapping Mes depends on the following packages:
@item @url{https://savannah.gnu.org/projects/nyacc/, NYACC}, 0.93.0 or later, including 0.99.0.
@end itemize
@node Running the Test Suites
@node Running the Test Suites, , Bootstrap Requirements, Installation
@section Running the Test Suites
@cindex test suites
@ -544,7 +588,7 @@ CC=gcc CC32=i686-unknown-linux-gnu-gcc MES=guile \
build-aux/test.sh scaffold/tests/00-exit-0
@end example
@node Bootstrapping
@node Bootstrapping, Contributing, Installation, Top
@chapter Bootstrapping
@quotation
@ -584,12 +628,12 @@ responsibility.
@menu
* The Mes Bootstrap Process:: How Mes will make you yogurt from pure milk.
* Invoking Mes:: Running Mes, a minimalist Guile lookalike.
* Invoking MesCC:: Running the MesCC bootstrap compiler.
* Invoking mes:: Running Mes, a minimalist Guile lookalike.
* Invoking mescc:: Running the MesCC bootstrap compiler.
* Invoking mesar::
@end menu
@node The Mes Bootstrap Process
@node The Mes Bootstrap Process, Invoking mes, Bootstrapping, Bootstrapping
@section The Mes Bootstrap Process
The Reduced Binary Seed bootstrap currently adopted by Guix@footnote{See
@ -667,8 +711,8 @@ Although we think these are less essential and thus less interesting
than the GNU toolchain triplet that we focussed on initially, our next
priority is to eleminate these one by one.
@node Invoking Mes
@section Invoking Mes
@node Invoking mes, Invoking mescc, The Mes Bootstrap Process, Bootstrapping
@section Invoking mes
@cindex repl
The @command{mes} command is the Scheme interpreter whose prime
@ -687,13 +731,13 @@ The @var{option}s can be among the following:
@item -s @var{script} @var{arg}@dots{}
@cindex script mode
By default, Mes will read a file named on the command line as a script.
By default, mes will read a file named on the command line as a script.
Any command-line arguments @var{arg}@dots{} following @var{script}
become the script's arguments; the @code{command-line} function returns
a list of strings of the form @code{(@var{script} @var{arg}@dots{})}.
Scripts are read and evaluated as Scheme source code just as the
@code{load} function would. After loading @var{script}, Mes exits.
@code{load} function would. After loading @var{script}, mes exits.
@item -c @var{expr} @var{arg}@dots{}
@cindex evaluate expression, command-line argument
@ -701,7 +745,7 @@ Evaluate @var{expr} as Scheme code, and then exit. Any command-line
arguments @var{arg}@dots{}) following @var{expr} become command-line
arguments; the @code{command-line} function returns a list of strings of
the form @code{(@var{guile} @var{arg}@dots{})}, where @var{mes} is the
path of the Mes executable.
path of the mes executable.
@item -- @var{arg}@dots{}
Run interactively, prompting the user for expressions and evaluating
@ -709,7 +753,7 @@ them. Any command-line arguments @var{arg}@dots{} following the
@option{--} become command-line arguments for the interactive session;
the @code{command-line} function returns a list of strings of the form
@code{(@var{guile} @var{arg}@dots{})}, where @var{mes} is the path of the
Mes executable.
mes executable.
@item -L,--load-path=@var{directory}
Add @var{directory} to the front of Mes module load path. The given
@ -732,10 +776,10 @@ and the command-line arguments---the list provided by the
@code{command-line} function.
@item -h@r{, }--help
Display help on invoking Mes, and then exit.
Display help on invoking mes, and then exit.
@item -v@r{, }--version
Display the current version of Mes, and then exit.
Display the current version of mes%, and then exit.
@end table
@ -743,7 +787,7 @@ Display the current version of Mes, and then exit.
* Environment Variables:: If the bits won't change, change their habitat.
@end menu
@node Environment Variables
@node Environment Variables, , Invoking mes, Invoking mes
@subsection Environment Variables
@cindex environment variables
@cindex shell
@ -752,13 +796,13 @@ Display the current version of Mes, and then exit.
@c Hmm, I expected this paragraph in the Guix manual?
Here are the environment variables (see @pxref{Environment Variables,,,
guile, Guile Reference}) that affect the run-time behavior of
Mes:
mes:
@table @env
@item MES_BOOT
@vindex MES_BOOT
Set @env{MES_BOOT} to change the initial Scheme program that Mes runs.
Set @env{MES_BOOT} to change the initial Scheme program that mes runs.
@item MES_ARENA
@vindex MES_ARENA
@ -821,8 +865,8 @@ Mes uses @var{@strong{GUILE}_LOAD_PATH} for compatibility with Guile.
@end table
@node Invoking MesCC
@section Invoking MesCC
@node Invoking mescc, Invoking mesar, Invoking mes, Bootstrapping
@section Invoking mescc
@example
mescc @var{option}@dots{} @file{FILE}@dots{}
@ -902,14 +946,14 @@ specify LANGUAGE of the following input files
* MesCC Environment Variables:: There's no NIX like POSIX.
@end menu
@node MesCC Environment Variables
@node MesCC Environment Variables, , Invoking mescc, Invoking mescc
@subsection MesCC Environment Variables
@table @env
@item MES
@vindex MES
Setting @env{MES} to a mes-compatible Scheme will run MesCC using that
Setting @env{MES} to a mes-compatible Scheme will run mescc using that
@example
MES=guile mescc -c scaffold/main.c
@end example
@ -931,7 +975,7 @@ during the parsing phase.
@end table
@node Invoking mesar
@node Invoking mesar, , Invoking mescc, Bootstrapping
@section Invoking mesar
@example
@ -961,7 +1005,7 @@ display version and exit
@end table
@c *********************************************************************
@node Contributing
@node Contributing, Acknowledgments, Bootstrapping, Top
@chapter Contributing
@menu
@ -973,7 +1017,7 @@ display version and exit
* Submitting Patches:: Share your work.
@end menu
@node Building from Git
@node Building from Git, Running Mes From the Source Tree, Contributing, Contributing
@section Building from Git
If you want to hack GNU Mes itself, it is recommended to use the latest
@ -997,7 +1041,7 @@ Finally, you have to invoke @code{make check} to run tests
installation instructions (@pxref{Installation}) or send a message to
the @email{bug-mes@@gnu.org} mailing list.
@node Running Mes From the Source Tree
@node Running Mes From the Source Tree, Porting GNU Mes, Building from Git, Contributing
@section Running Mes From the Source Tree
First, you need to have an environment with all the dependencies
@ -1005,7 +1049,7 @@ available (@pxref{Building from Git}), and then simply prefix each
command by @command{./pre-inst-env} (the @file{pre-inst-env} script
lives in the top build tree of Mes).
@node Porting GNU Mes
@node Porting GNU Mes, The Perfect Setup, Running Mes From the Source Tree, Contributing
@section Porting GNU Mes
Mes was written for x86-linux. A 64 bit (x86_64) is almost done, only a
@ -1017,13 +1061,13 @@ An ARM (armv4/armv7l) linux port is underway. A port to GNU/Hurd
Initial scaffold, built by @file{build-aux/build-scaffold.sh}:
@example
@file{lib/x86-mes-gcc/exit-42.S}
@file{lib/x86-mes/elf32-0exit-42.hex2}
@file{lib/x86-mes/elf32-body-exit-42.hex2}
@file{lib/linux/x86-mes-gcc/exit-42.S}
@file{lib/linux/x86-mes/elf32-0exit-42.hex2}
@file{lib/linux/x86-mes/elf32-body-exit-42.hex2}
@file{lib/x86-mes-gcc/hello-mes.S}
@file{lib/x86-mes/elf32-0hello-mes.hex2}
@file{lib/x86-mes/elf32-body-hello-mes.hex2}
@file{lib/linux/x86-mes-gcc/hello-mes.S}
@file{lib/linux/x86-mes/elf32-0hello-mes.hex2}
@file{lib/linux/x86-mes/elf32-body-hello-mes.hex2}
@end example
Porting MesCC:
@ -1038,7 +1082,7 @@ Porting MesCC:
@file{mes/module/mescc/i386/info.mes}
@end example
@node The Perfect Setup
@node The Perfect Setup, Coding Style, Porting GNU Mes, Contributing
@section The Perfect Setup
The Perfect Setup to hack on Mes is basically the perfect setup used
@ -1053,7 +1097,7 @@ on-line documentation (docstrings), context-sensitive completion,
@kbd{M-.} to jump to an object definition, a REPL to try out your code,
and more (@pxref{Introduction,,, geiser, Geiser User Manual}).
@node Coding Style
@node Coding Style, Submitting Patches, The Perfect Setup, Contributing
@section Coding Style
In general our code follows the GNU Coding Standards (@pxref{Top,,,
@ -1089,7 +1133,7 @@ Additionally, in Mes we prefer to format @code{if} statements like this
else))
@end example
@node Submitting Patches
@node Submitting Patches, , Coding Style, Contributing
@section Submitting Patches
Development is done using the Git distributed version control system.
@ -1139,7 +1183,7 @@ Join us on @code{#bootstrappable} on the Freenode IRC network or on
Please send bug reports with full details to @email{bug-mes@@gnu.org}.
@c *********************************************************************
@node Acknowledgments
@node Acknowledgments, Resources, Contributing, Top
@chapter Acknowledgments
We would like to thank the following people for their help: Jeremiah
@ -1151,7 +1195,7 @@ LISP-1.5 and Alan Kay for their inspiring comment on
@url{https://queue.acm.org/detail.cfm?id=1039523, Page 13}.
@c *********************************************************************
@node Resources
@node Resources, GNU Free Documentation License, Acknowledgments, Top
@chapter Resources
@itemize
@ -1185,17 +1229,17 @@ at the freenode IRC network.
@end itemize
@c *********************************************************************
@node GNU Free Documentation License
@node GNU Free Documentation License, Concept Index, Resources, Top
@appendix GNU Free Documentation License
@cindex license, GNU Free Documentation License
@include fdl-1.3.texi
@c *********************************************************************
@node Concept Index
@node Concept Index, Programming Index, GNU Free Documentation License, Top
@unnumbered Concept Index
@printindex cp
@node Programming Index
@node Programming Index, , Concept Index, Top
@unnumbered Programming Index
@syncodeindex tp fn
@syncodeindex vr fn