Can Many be made to accept an optional separator?

[Mozahler]

If this is the wrong place to ask a question about how to work with the parsing package, please let me know where to redirect.

I've set up a parser (everything has been simplified for reproducibility) that depends upon an item:

public enum Item {
   case item1(Substring)
   case item2(Substring)
   case item3(Substring, Substring)
}

I've written a handful of parsers in this format:

let parseAnItemTwo = Parse(Item.item2) {
   "@@ "
   Prefix { $0 != "\n" }
   "\n"

So far so good. I can hand roll the parsing of a given string when I specify them in the correct order.

But I want to use something similar to OneOf (or an equivalent conditional) to parse a stream of tokens from the input (resulting in Array)

let fullSuite =  OneOf {
      parseAnItemOne
      parseAnItemTwo
      parseAnItemThree
   }
}

I would then run this on my input, which works fine as long as I have a newline between each token

let iterateForTokens = Many {
   fullSuite
} separator: {
   "\n"
}

I need the separator to be optional (there might be newlines between records/tokens, but it's not a requirement).

I tried revising your zeroOrMoreSpaces:

let zeroOrMoreNewlines = Prefix { $0 == "\n" }

let iterateForTokens = Many {
   fullSuite
} separator: {
   zeroOrMoreNewlines
}

This compiles, but produces a run-time error:

The operation couldn’t be completed. (Parsing.ParsingError error 0.)

Again: All tokens in the input stream have unique starting sequences, so explicit delimiters aren't required in the data stream.

[stephencelis]

Can you provide a little more context for folks to help troubleshoot? Full, compiling code for the parser would help, as well as some sample input that fails?

You also might get a better error message if you print it as a string:

do {
  try parser.parse(input)
} catch {
  print("\(error)")
}

Ideally this would be printed automatically, though. Can you let us know how you're getting the error printed like that? Are you invoking this as an executable on the command line, or?

[Mozahler]

Thanks. That was helpful. I was using: print(error.localizedDescription)

Removed the .localizedDescription and got:
error: infinite loop
--> input:6:11
6 | plain text
| ^ expected input to be consumed

Not sure how to reproduce the problem using my simplified version, so I'll wrestle with this and come back.
Thanks for the quick response!

[haikusw]

How do you tell you're at the end of a record without a separator? Just that you've successfully parsed everything needed for a record?

Your sample simplified item parser:

let parseAnItemTwo = Parse(Item.item2) {
   "@@ "
   Prefix { $0 != "\n" }
   "\n"

keeps accumulating characters until it sees a newline. I don't see how it's not going to keep going forever (or until it's consumed all input?) unless there is a delimiter or a max-length specifier or something to bound things.

Even if the delimiter is just a character that is the start of a new record and isn't allowed anywhere the last field of a record, then that could effectively be your delimiter. However it's done, there needs to be some way to know when to start the next record.

[Mozahler]

Hi. -- Apologies for not seeing your response sooner.

As you rightly point out, the input in Many is going to be consuming (including what I have previously defined elsewhere as a delimiter) until a separator is found. I may have confused myself in simplifying my code. How do you distinguish between newline and Optional<newline>? I need to revisit my original intentions.

In hindsight, it's obvious that "separator" is a synonym for "delimiter". I was trying to treat it as though it were a subordinate concept. I didn't think things through.

Thank you for taking the time. Sometimes I miss the forest for the trees.
🤷🏽‍♂️

[Mozahler]

It looks like I can't mark a comment reply as an answer, so let me just say that @haikusw has answered my question.
Thank you both for your time 🙏🏻