fix: parse heredocs with an empty body (#309) - #312
Open
livingstaccato wants to merge 1 commit into
Open
Conversation
`a = <<EOF\nEOF` failed to parse. The v8 rewrite turned the optional
newline after the opening marker into a mandatory one, so a marker
immediately followed by its closing delimiter no longer matched the
HEREDOC_TEMPLATE terminal.
The failure is not confined to the empty heredoc. Having failed to match
at that position, the lexer scans on and matches a *later* delimiter, so
an empty heredoc followed by more attributes silently swallows them:
locals {
a = <<EOF
EOF
b = <<EOF
real body
EOF
}
serialized to a single attribute `a` whose value ran to the end of `b`,
with `b` absent from the output. Only a file whose empty heredoc had no
later delimiter to latch onto surfaced as a parse error.
Make the body and its trailing newline a single optional group rather
than restoring the bare `\n?` the rewrite dropped. The group has to be
lazy: a greedy one prefers to match a body, which reintroduces the
run-on above. The delimiter still has to begin its own line, so amplify-education#194's
rule holds and a word merely ending in the delimiter does not terminate
the body.
Cover empty, empty-trimmed and blank-line-only heredocs in the
specialized suite, along with an attribute after them that proves the
lexer stops where it should. `test/helpers/terraform-config/` carried an
empty-heredoc fixture for this before the v8 test reorganisation; nothing
in the current suite replaced it. Without the fix, 6 of the new tests
fail.
livingstaccato
added a commit
to provide-io/pyvider-hcl
that referenced
this pull request
Aug 18, 2026
`<<EOF` immediately followed by its closing marker leaked the raw markers as the value: `_unwrap_heredoc` split the remainder on its last newline, found none, and gave up, so the string fell through to escape processing untouched. The case is unreachable with python-hcl2 8.1.2, whose grammar cannot parse a body-less heredoc at all, so nothing exercised it. Running the suite against a build carrying the fix for that (submitted upstream as amplify-education/python-hcl2#312) surfaced it: with the parser repaired, `x = <<EOF\nEOF` reached the normalizer and came back as the two-line string `<<EOF\nEOF` rather than the empty string HCL specifies. Distinguish having no body lines, which holds "", from having one empty body line, which holds a newline. The new tests drive the normalizer directly rather than through the parser, so they run on the released python-hcl2 as well.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #309.
A heredoc with an empty body failed to parse. The v8 rewrite turned the optional newline after the opening marker into a mandatory one, so a marker immediately followed by its closing delimiter no longer matched
HEREDOC_TEMPLATE.The louder half of the bug
The parse error is the rare case. Having failed to match at that position, the lexer scans on and matches a later delimiter, silently absorbing everything in between:
8.1.2 returns a single attribute whose value runs to the end of
b, withbabsent from the output. No exception. Only a file whose empty heredoc has no later delimiter to latch onto surfaces asUnexpectedToken.The fix
Make the body and its trailing newline one lazy optional group:
Two simpler-looking patches are both wrong, which seems worth recording:
\n?that the rewrite dropped fixes the empty case but keeps the run-on, because the body can still stretch to a later delimiter.\nbefore\s*fixes both of those, but then\s*can match nothing and a word merely ending in the delimiter terminates the body:<<EOF\nsayEOF\nEOFtruncates atsayEOF.The laziness is load-bearing for the same reason: a greedy
?prefers to match a body. The delimiter still has to begin its own line, so the rule from #194 is preserved.Tests
test/helpers/terraform-config/carried an empty-heredoc fixture, added with #117 when this was fixed the first time. The v8 test reorganisation replaced that layout, and no zero-line heredoc appears anywhere in the current suite — which is why nothing caught this.The specialized heredoc fixture now covers empty, empty-trimmed and blank-line-only heredocs, plus an attribute after them that proves the lexer stops where it should. Without the grammar change, 6 of the new tests fail.
nose2 --config tox.ini: 1400 tests, OK.ruffclean.This pull request, and the investigation behind it, were produced by an AI assistant (Claude) working on behalf of the author. Every reproduction, test run and benchmark cited was executed rather than inferred, but please review with that provenance in mind.