rosie.rpl

---- -*- Mode: rpl; -*-                                                                             
----
---- rosie.rpl    Some patterns specific to Rosie Pattern Language
----
---- © Copyright IBM Corporation 2016.
---- LICENSE: MIT License (https://opensource.org/licenses/mit-license.html)
---- AUTHOR: Jamie A. Jennings

-- rosie.token will match a token (given the way Rosie tokenizes as it goes) if there is a token.
-- rosie.tokens will advance to the next token and match all of them.

rosie.token = { ![:space:] . {!~ .}* }
rosie.tokens = {~ rosie.token}*

-- Rationale:
--
-- The RPL built-in ~ is bound to the pattern that Rosie uses to advance to the start of the next
-- token, discarding whitespace as it goes.  It's idempotent, so this expression will always fail:
--                                        {~ {!~ .}}
-- It fails because ~ will succeed even if no advancing is needed, such as when the pointer is at
-- the start of a new token or at the end of the input.
--
-- Therefore, if the pointer is at a token already, then to match the token, Rosie must consume a
-- character before checking to see if we are at a token boundary:
--                                        { . {!~ .}*}
-- And since any character except whitespace can conceivably start a token, a good definition for a
-- single token is:
--                                  { ![:space:] . {!~ .}* }
-- I.e. "as long as we're not looking at whitespace, consume characters until the next token
-- boundary."  This definition will fail at the end of input as well as when looking at whitespace,
-- which seems like the right thing.
--
-- Then, a pattern to match Rosie tokens can be made by repeatedly (using *) advancing to the next
-- token (using ~), followed by matching a token:
--                                        {~ token}*
--