Coupling of Tokens and Scanners

[peteraldous]

I was hoping to add a feature to TokenParsers that allows users to provide a string that represents a single token and get a parser that recognizes that token (this sort of interaction comes essentially for free in a scannerless parser); however, I was only able to make it work in my own code because my TokenParsers.lexical member was a Scanners object with Tokens mixed in. It doesn't work in general because of the separation of Tokens from Scanners (which is what StdLexical does).

Since the whole point of TokenParsers is to read tokens, it seems logical to me that Tokens should be combined into Scanners instead of decoupled; after all, Tokens doesn't provide an interface for accessing tokens, so it can't stand alone. The current design seems to assume that the authors of the TokenParsers subclass will know how TokenParsers.lexical is intended to be used, presumably because the same authors will have written it. Incorporating Tokens into Scanners and requiring TokenParsers.lexical to be a Scanners object would provide a defined interface that would allow for features like the one I described above.

I'm guessing, however, that this has been considered and that the authors of this library decoupled the two classes for a reason. If you're willing, I would love to better understand the rationale behind this design choice.

(My agenda here is that I would like for this library to be a viable option for my undergraduate compilers students. A few improvements, mostly to documentation, will suffice. I'm happy to contribute code and docs, but I want to make sure I understand before I write much of either.)