compile GNU make

[ahgamut]

https://github.com/ahgamut/gnu-make-cosmopolitan

Clone the repo and run superconfigure to obtain make.com.dbg and make.com.

Compilation changes were switch(errno) to if-else, and some linker complaints about getopt.

make.com passes a lot of tests, notable failures are related to the below issue.

make.com -j2 does not work on Linux. I added a SIGSEGV call at the point of failure to obtain the below backtrace, otherwise make.com just exits with the last line.

Uncaught SIGSEGV

7ffe4421ae98 0000004489f5 systemfive_linux+0x15
7ffe4421bcc0 000000430dd6 jobserver_parse_auth+0xf9
7ffe4421bce0 00000041b529 main+0xae6
7ffe4421c840 00000040168f cosmo+0x40
7ffe4421c850 00000040116b _start+0x6a

RAX 0000000000000000 RBX 0000000000000002 RDI 00000000000004b5 ST(0) 0
RCX 00000000004489f5 RDX 0000000000000001 RSI 000000000000000b ST(1) 0
RBP 00007ffe4421bcc0 RSP 00007ffe4421bcc0 RIP 00000000004489f5 ST(2) 0
R8  00000000ffffff00 R9  0000000000000000 R10 0000000000000001 ST(3) 0
R11 0000000000000297 R12 0000000000000002 R13 00007ffe4421c868 ST(4) 0
R14 00007ffe4421c880 R15 00007ffe4421c9f8 CF VF PF AF SF IF

XMM0  00000000000000000000000000000000 XMM8  00000000000000000000000000000000
XMM1  00000000000000000000000000000000 XMM9  00000000000000000000000000000000
XMM2  00000000000000000000000000000000 XMM10 00000000000000000000000000000000
XMM3  ffffffffffffff00000000000000ff00 XMM11 00000000000000000000000000000000
XMM4  00000000000000000000000000000000 XMM12 00000000000000000000000000000000
XMM5  00000000000000000000000000000000 XMM13 00000000000000000000000000000000
XMM6  00000000000000000000000000000000 XMM14 00000000000000000000000000000000
XMM7  00000000000000000000000000000000 XMM15 00000000000000000000000000000000

make.com: *** [Makefile:1445: all-recursive] Error 1

The jobserver_parse_auth function in src/posixos.c calls fcntl(fd, F_GETFD) which always returns -1 -- this causes the above error.

[ahgamut]

I also tried using make.com on Windows: I was able to write the Makefile for a hello world example using the cross9 gcc from the tutorial.

Example Makefile for Hello World compilation on Windows:

CC=../cross9/bin/x86_64-pc-linux-gnu-gcc.exe
OBJCOPY=../cross9/bin/x86_64-pc-linux-gnu-objcopy.exe
COSMO_CFLAGS= -static -fno-pie -no-pie -mno-red-zone -fno-omit-frame-pointer -nostdlib -nostdinc -Wl,--gc-sections -Wl,-z,max-page-size=0x1000 -fuse-ld=bfd -Wl,-T,../libcosmo/ape.lds -include ../libcosmo/cosmopolitan.h ../libcosmo/crt.o ../libcosmo/ape.o ../libcosmo/cosmopolitan.a

all: hello.com

hello.com: hello.com.dbg
	$(OBJCOPY) -SO binary $< $@

hello.com.dbg: hello.c
	$(CC) -o $@ $< $(COSMO_CFLAGS)

I ran it using make.com -f Makefile on Win10 Command Prompt. make.com -F clean doesn't work, because make.com can't find rm or use del.

I also tried to compile Cosmopolitan from source using make.com, but it requires a shell and other utilities which are not present by default on Windows.

[jart]

Nice! If you check it into the codebase I can help you fix the -j flag. Stuff like rm we can write simple replacements. For example, there's a cp and echo command in the examples folder, we could put in build/bootstrap/. There's also examples/unbourne.c which is /bin/sh but it'd require non-trivial effort to get it to work on Windows. Most of our Makefile actually doesn't require /bin/sh at all. We could probably write a shell that just does execvp(argv[1].split()). Speaking of which, I have a great idea now for getting rid of scripts like build/getcompile.

I'm more excited about your Python work in #141. Portable make would be nice to have, but it'd be easier to get Python working well on Windows than it would be to get Almquist Shell working, since UNIX is the shell (it was the first thing Ken Thompson wrote when he created UNIX) and Windows is not UNIX.

[ahgamut]

I'll put together a PR for third_party/gnu-make/ soon.

For /bin/sh I actually got the dash shell to compile, but entering into dash.com from Windows CMD is weird: basic arithmetic works, but it can't find $PATH, and I wasn't able to run anything complex.

I thought Python APE webapps would be nice to have, but they are oddly slow, and there are lots of gotchas involved (for example setup.py scripts sometime wants the glibc version even if the package is a pure-Python wheel).

Ignoring web-based capabilities, both Python 2.7/3.6 just have to pass a lot of tests in their respective suites, and we can add either one to the codebase.

[jart]

Small nit but I'd say name the folder third_party/make/. The unbourne shell is dash by the way! I can help with Python3 performance if you check it in. The tools I usually use are --ftrace and perf record which should work with the binary release build process too.

[jart]

Speaking of Python performance by the way...

How much slower could it be?

[ahgamut]

How much slower could it be?

Are the Python stats in the above benchmark for static or dynamic pages?
Static pages python.com -m http.server perform OK from what I remember.

The slow is in dynamic pages. I simulated Python multithreading using _dummy_thread to get Flask to work: this might have inverted the benefits of using threads. I also used only pure-python packages.

This was a simple Flask app from a couple of weeks ago (GIF playing in realtime):

I may have mixed something up with the APE build/test env above, but generally I don't expect pure CPython to win in performance. I picked Flask because familiarity, but maybe there's a Python web framework that is fast, single-threaded, easy to write and with simple C extension dependencies. I wonder how a Python wrapper to redbean would perform.

[jart]

umask doesn't have a wrapper last time I checked. You probably have to depend on LIBC_SYSV_CALLS.

[ahgamut]

@jart Closing this issue: make is part of the repo and make.com builds the repo on Linux.

[jart]

It's lightning fast! I'm already using it as a replacement for what comes with my distro. Now I can't wait to add the ptrace stuff.

[ahgamut]

I'm already using it as a replacement for what comes with my distro.

For a while now, I've copied make.com and python.com to ~/bin after every build, and I just try out small tasks with them.
I was glad to find out make.com can build the repo (#369).

I don't have a windows system nearby, but I expect #117 might also be solvable after the recent improvements.

We have make.com, zip.com, basic shell commands, and the static GCC toolchain.
Can the repo be fully bootstrapped on Linux?

[jart]

Yes I think so. Having GNU Make was the last big blocker to the repository fully being able to build itself on Linux on its own. There's a few other trivial ones of course, like cp and mkdir. But if the desire is there to make it 100% dependency free, then it can be done.

[ghaerr]

Can the repo be fully bootstrapped on Linux?

I'm running macOS and would sure like to play with Cosmopolitan more... Is the problem getting that working building a static GCC toolchain for macOS, or is there a lot more to it than that?

Thank you!

[ahgamut]

There's a few other trivial ones of course, like cp and mkdir

Is there a list somewhere? For starters we can use python.com: shutil.copy for cp and os.makedirs for mkdir.

[jart]

@ahgamut @ghaerr There should be a list. Would you like to create it? I've enabled the wiki feature on GitHub. Your both welcome to edit it at will with any information you feel might be helpful. Anyone with a GitHub account should have access to make changes there.

@ghaerr You might be able to install an x86_64-linux-gnu toolchain from homebrew. See the instructions in the README file. Please note your mileage may vary. What I recommend doing is SSH'ing into a Linux computer to do development, mostly because that's what I do, so it's the only thing I can 100% support. In the future as the project attracts more developers we'll naturally have a build-anywhere run-anywhere solution. But right now all we've managed to achieve is build-once run-anywhere.

[jart]

Regarding the copy command, one thing that's really cool about cosmopolitan is the cp.com program in the examples folder should copy files at 2x speed compared to Linux distros.

cosmopolitan/examples/cp.c

Lines 74 to 84 in 072e1d2

	int cp(const char *src, const char *dst) {
	if (endswith(dst, "/") || isdirectory(dst)) {
	dst = _gc(xasprintf("%s/%s", dst, basename));
	}
	if (!force && access(dst, W_OK) == -1 && errno != ENOENT) goto OnFail;
	if (copyfile(src, dst, flags) == -1) goto OnFail;
	return 0;
	OnFail:
	fprintf(stderr, "%s %s %s: %s\n", "error: cp", src, dst, strerror(errno));
	return 1;
	}

It's because Cosmopolitan has outstanding polyfills for things like copy_file_range. Most core binutils use sendfile because it's historically been the lowest common denominator in terms of higher performance block copying. But we're able to do it better!

[ahgamut]

@jart make.com MODE= -j4 builds the repo when restricting $PATH to the following commands (in addition to sh itself):

cat
chmod
cp
dd
find
gunzip
gzip
head
mkdir
mv
pwd
sed
touch
uname

Of these, dd is necessary for starting make.com itself (copying /tmp/ape-loader), rest appear in shell scripts or during the build.

dot, clang, gfortran, and xargs are mentioned in build/definitions.mk but they are not required during make.com MODE= -j4.