Parser combinaison

[mackoj]

Hi,

I'm not really sur how to combine multiple parsers.

Let says for exemple we want to do a parser for Localizable.strings files. It's seems pretty easy it's just "key" = "value";\n but if we want for exemple add the comment above it can become quite tricky to properly implement.

If I want to do two parser one comment parser that could work on multiline comment.

let comment = StartsWith("/*".utf8).take(PrefixUpTo("*/".utf8)).skip(Newline())

and one translation parser

let translation = StartsWith("\"".utf8).take(PrefixUpTo("\"".utf8)) // 1 - key
  .skip(PrefixThrough("=".utf8))
  .skip(PrefixThrough("\"".utf8)).take(PrefixUpTo("\";".utf8))// 2 - value
  .skip(PrefixThrough(";".utf8))

How to combine both parser above ? zip, take, orElse don't seem to work as I hope them to.

If I try to run the translation parser only on a file that have comment it don't quite work either.

let trad = Skip(PrefixThrough("\"".utf8)).take(PrefixUpTo("\"".utf8)) // 1 - key
  .skip(PrefixThrough("=".utf8))
  .skip(PrefixThrough("\"".utf8)).take(PrefixUpTo("\";".utf8))// 2 - value
  .skip(PrefixThrough(";".utf8))

This version don't quite work well either because even if it seems ok the work on multiple line, it can start taking the key on line 1 and then take the value on line 2 and this is bad 😔. I don't understand how to force a parser to be valid on one line only.

Is it possible to combine a parser that work on one line and a parser that work on multiple line ?

var inputLong = """
"sign_in_label" = "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.";


/* %@ will be a price in the local currency. Examples: "$9.99", "9.99€" */
"monthly_price" = "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam. %@";
"format" = "Lorem ipsum";

"Tagline" = "Sed do eiusmod tempor incididunt ut labore";

/* ipsum" refers to "dev profile" / "social media". please keep it to one word when it is possible */
"social" = "Social";
"special" = "Especial";
"""

struct Line {
  let key: String
  let value: String
  let commentaire: String?
}

func parse(_ input : Substring.UTF8View) -> [Line]? {
  let line = Optional.parser(of:StartsWith("/*".utf8).take(PrefixUpTo("*/".utf8)))  // 0 - comment
    .skip(PrefixThrough("\"".utf8)).take(PrefixUpTo("\"".utf8))// 1 - key
    .skip(PrefixThrough("=".utf8))
    .skip(PrefixThrough("\"".utf8)).take(PrefixUpTo("\";".utf8))// 2 - value
    .skip(PrefixThrough(";".utf8))
  
  let converted = line.map { (elem : (Optional<Substring.UTF8View>, Substring.UTF8View, Substring.UTF8View)) -> Line in
    let comment = (elem.0 == nil) ? nil : String(elem.0!)
    return Line(
      key: String(elem.1)!,
      value: String(elem.2)!,
      commentaire: comment?.trimmingCharacters(in: .whitespaces)
    )
  }
  
  let file = Many(converted, separator: Newline())
  let parsedContent = file.parse(input)
  return parsedContent.output
}

let content = parse(inputLong[...].utf8)
for line in content ?? [] {
  print("key: \(line.key)")
  print("value: \(line.value)")
  //  print("comment: \(line.value.commentaire)")
  print("---\n")
}

output

key: sign_in_label
value: Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
---

key: $9.99
value: Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam. %@
---

key: format
value: Lorem ipsum
---

key: Tagline
value: Sed do eiusmod tempor incididunt ut labore
---

key:  refers to 
value: Social
---

key: special
value: Especial
---

The second and the fifth are bad.

Conclusion

In the exemple above I don't see how to separate the comment parser from the translation parser.
What don't work here is that there is no way to express the fact that the translation parser should perform on one line only and the comment parser can be performed on multiple line and have the output of both in one type.

I'm all ears if you have some idea.

Thanks,

[mackoj]

The only way I see I can combine them is by doing one parser.
Like this for exemple but it's still don't work well because it can take the key on line 1 and then take the value on line 2.
I don't understand how to force the translation part to work on one line only and the comment part on multiple lines.

    let line = Optional.parser(of:StartsWith("/*".utf8).take(PrefixUpTo("*/".utf8)))  // 0 - comment
      .skip(PrefixThrough("\"".utf8)).take(PrefixUpTo("\"".utf8))// 1 - key
      .skip(PrefixThrough("=".utf8))
      .skip(PrefixThrough("\"".utf8)).take(PrefixUpTo("\";".utf8))// 2 - value
      .skip(PrefixThrough(";".utf8))

[stephencelis]

@mackoj You're close! I think the problem is that your input has multiple new lines before the comments and your separator is Newline(), which will only attempt to parse a single one. Changing the separator to Many(Newline()) might fix things, or even just Whitespace().

One other suggestion would be to avoid PrefixThrough and PrefixUpTo when using skip with parsers, since in this case it may have made things more difficult to debug. Instead you could be very explicit in what you want to match. This is just a sketch so not sure if it's exactly what you want, but something like this:

let comment = StartsWith("/*".utf8).take(PrefixUpTo("*/".utf8)).skip(StartsWith("*/".utf8))
let literal = StartsWith("\"".utf8).take(PrefixUpTo("\"".utf8)).skip(StartsWith("\"".utf8))
let translation = Skip(Whitespace())
  .take(Optional.parser(of: comment))
  .skip(Whitespace())
  .take(literal)
  .skip(Whitespace())
  .skip(StartsWith("=".utf8))
  .skip(Whitespace())
  .take(literal)
  .skip(Whitespace())
  .skip(StartsWith(";".utf8))
  .skip(Whitespace())

[mackoj]

It worked thanks @stephencelis I wanted to play with it and see if I can use it in a more complex project.
Thanks for the exemple 🙏 I better understand how to approach parsing with this lib now.

[mackoj]

There is other issues with it but I did fix them from your parser.

 "key" = "val;u\"\"\"e"     ;

Thanks again.

[stephencelis]

You might be able to take some of the logic from the example JSON parser, which shows how you can handle things like escape quotes:

swift-parsing/Sources/swift-parsing-benchmark/JSON.swift

Lines 74 to 124 in df12ae9

	// MARK: String
	private let unicode = Prefix<Input>(4) {
	(.init(ascii: "0") ... .init(ascii: "9")).contains($0)
	|| (.init(ascii: "A") ... .init(ascii: "F")).contains($0)
	|| (.init(ascii: "a") ... .init(ascii: "f")).contains($0)
	}
	.compactMap {
	UInt32(String(decoding: $0, as: UTF8.self), radix: 16)
	.flatMap(UnicodeScalar.init)
	.map(Character.init)
	}
	private let escape = StartsWith<Input>(#"\"#.utf8)
	.take(
	StartsWith("\"".utf8).map { "\"" }
	.orElse(StartsWith(#"\"#.utf8).map { #"\"# })
	.orElse(StartsWith("/".utf8).map { "/" })
	.orElse(StartsWith("b".utf8).map { "\u{8}" })
	.orElse(StartsWith("f".utf8).map { "\u{c}" })
	.orElse(StartsWith("n".utf8).map { "\n" })
	.orElse(StartsWith("r".utf8).map { "\r" })
	.orElse(StartsWith("t".utf8).map { "\t" })
	.orElse(unicode)
	)
	private let literal = Prefix<Input>(1...) {
	$0 != .init(ascii: "\"") && $0 != .init(ascii: #"\"#)
	}
	.map { String(decoding: $0, as: UTF8.self) }
	private enum StringFragment {
	case escape(Character)
	case literal(String)
	}
	private let fragment = literal.map(StringFragment.literal)
	.orElse(escape.map(StringFragment.escape))
	private let stringLiteral = StartsWith("\"".utf8)
	.take(
	Many(fragment, into: "") {
	switch $1 {
	case let .escape(character):
	$0.append(character)
	case let .literal(string):
	$0.append(contentsOf: string)
	}
	}
	)
	.skip(StartsWith("\"".utf8))