Separate parsing from reading shell input

[jpco]

NOTE: I've been a bit aggressive with unilaterally merging PRs lately, but this one will not be getting merged without explicit feedback.

The short version: This PR changes the $&parse primitive. Previously, $&parse would take an optional set of prompts and use that to read and parse shell input, producing a parsed command. Now, $&parse takes a command and calls that command once or more in order to read shell input, which it parses. %parse has been rewritten to wrap the new $&parse in such a way that its behavior hasn't changed. This PR also exposes the shell's existing readline logic via the $&readline primitive, and the new $&parse calls this primitive.

The impact of this is that it decouples prompting, reading, parsing, and writing to history. You can see that in these snippets from initial.es. In this first one, we define the %read-line function, which takes a prompt, prints it, and then reads a line. If it exists, the $&readline primitive can implement this function, or it can be done with an echo and a $&read.

if {~ <=$&primitives readline} {
	fn-%read-line = $&readline
} {
	fn %read-line prompt {
		echo -n $prompt
		$&read
	}
}

The second snippet uses this %read-line function, along with the new $&parse and the %write-history hook, to do all the work that would previously happen within %parse:

let (in = (); p = $prompt(1))
unwind-protect {
	$&parse {
		let (r = <={%read-line $p}) {
			in = $in $r
			p = $prompt(2)
			result $r
		}
	}
} {
	if {!~ $#fn-%write-history 0 && !~ $#in 0} {
		%write-history <={%flatten \n $in}
	}
}

Doing this allows arbitrary flexibility in writing to history or prompting -- per-line history writing such as before #65 can be done by people who want that, or a different prompt could be used for each and every line. Something like zsh's TRANSIENT_RPROMPT could be added. You could even add a hook to transform read-in text before parsing it, in order to add !!-style history expansion or old-school string-replacement-based aliases.

This PR also moves both $&parse and $&readline towards being "normal" primitives. In fact, with this PR, all the readline code in the shell is collected into a single file, prim-readline.c. Adding support for an alternative library should only require adding similar primitives in a similar file, as well as some es script to tie things together, which is a much simpler integration story than the dozen or so #if HAVE_READLINE blocks the shell has had till now. (This starts to connect to a potential concept of "extensible primitives", or even "modules", which is in my opinion a very interesting way to begin to direct the shell. Es could inherit loadable modules from Inferno's sh, a shell that arguably descended from es!)

Improving the potential to swap out $&parse may sound strange, but could also be useful. In addition to changes to internals (switching to a hand-written parser for better error reporting, or an incremental parser, etc.), swappable parsers could be a coarse way to allow certain changes to syntax.

A note on the particular API of $&parse -- why not go even simpler and pass $&parse an already-read string which it is to parse? This would make $&parse, effectively, a "push-style" parser. The major issue with this for es is that push-style parsers must be able to be called multiple times over the course of a single parse: it would be called with one line, return a value indicating it needs more input, and then would be called again with the next, and this would repeat until enough has been fed to it that it can return a fully-parsed statement.

Within es, the problem with this pattern is managing parser state across calls. How do we differentiate a second $&parse call just trying to parse two lines from a second, unrelated $&parse call, especially given es lacks opaque handles, so we can't just say $&parse $parser ...? It is much simpler to use a "pull-style" parser, where we give $&parse a way to read input for itself, and have each $&parse call correspond with a complete, returned command.

There are still some corner-cases to handle with this PR. The major remaining problem with it (as evidenced by the test failures) is the fact that $&read, which the shell now actually uses to read input (when not using $&readline), throws an exception when reading a NUL byte, whereas the previous logic skipped a NUL byte and printed a warning to stderr. Other corner-cases include: how $&readline should behave when not reading from a terminal, how to handle multiple $&parses on a single Input.

We may want to add some data for $&parse to pass to reader commands when calling them. One example would be something indicating whether a heredoc is being read, in order to do something like print a different prompt. I'm not in a hurry to add something like this; I think more time is needed to think of what might be included. (I also imagine that a richer API to a curretly-running parser might eventually be useful for something like a tab-completion system, but also don't have any real design for that yet.)