Terraform interprets HIL variables in such a way that it allows numeric
attribute names which then get interpreted as numeric indices into a
list. This is used to work around the fact that the splat expressions
don't work for the index operator.
zcl has "full splats" that _do_ support the index operator, but to allow
old Terraform configs to be processed by zcl we'll accept this special
case within attribute-only-splats only.
For the moment this is a special exception made by this specific
implementation of zcl rather than part of the spec, since it's
specifically a pragmatic Terraform migration strategy, but it might get
upgraded to full spec status later if we end up needing to support it
in other host languages.
This requires the scanner to be a little more picky about the ending
of numeric literals, so that they won't absorb the trailing period after
the number in foo.*.baz.1.baz . This is okay because the spec doesn't
allow trailing periods anyway, and this is not actually a change in
final behavior because the parser was already catching this situation
and rejecting it at a later point.
While this does create some ambiguity with arithmetic on variables, like
a-b, this is permitted by HCL and so we'll permit it for zcl too, at the
expense of requiring spaces to be used around minus signs for correct
interpretation.
Previously it was !{, but in real examples this looked confusing since
an exclamation point after a word looks (to humans) like literal
punctuation rather than syntax.
% is not ideal either since it's also the marker traditionally used for
printf, but has the advantage that programmers are already primed for it
to be syntax.
The native parser's ranges don't include any surrounding comments, so we
need to do a little more work to pick them out of the surrounding token
sequences.
This just takes care of _lead_ comments, which are those that appear as
whole line comments above the item in question. Line comments, which
appear after the item on the same line, will follow in a later commit.
There are certain tokens that are _never_ valid, so we might as well
catch them early in the Lex... functions rather than having to handle
them in many different contexts within the parser.
Unfortunately for now when such errors occur they tend to be echoed by
more confusing errors coming from the parser, but we'll accept that for
now.
Previously we were failing to return back to template-scanning mode due
to decrementing "braces" too early, causing the remainder of the template
to be scanned as if it were an expression.
The context where a string literal was found affects what sort of escaping
it can have, so we need to distinguish these cases so that we will only
look for and handle backslash escapes in quoted strings.
In zclwrite we throw away the absolute source position information and
instead just retain the number of spaces before each token. This different
model allows us to rewrite parts of the token sequence without needing
to re-adjust all of the positions, and it also allows us to do simple
indentation and spacing adjustments just by walking through the token
list and adjusting these numbers.
This alternative scanning mode makes the scanner start in template
context rather than normal context. This will be later used by the parser
to allow parsing of standalone templates that aren't embedded inside a
zcl configuration file.
This is important because our syntax for objects uses newlines as the
separator between items, so this is the only signal we'll get that a
given item has ended and another is beginning.
A scanner "mode" decides which state it starts in, allowing us to start
in template mode for parsing top-level templates. However, currently the
only mode implemented is "normal" mode, which is the behavior we had
before.
This requires some extra state-keeping because we allow templates to be
nested inside templates. This takes us outside of the world of regular
languages, but we accept that here because it makes things easier to
deal with down the line in the parser.
The methodology is to keep track of how many braces are open at a given
time and then, when a nested template interpolation begins, record the
current brace level. Then, when a closing brace is encountered, if its
nesting level is at the top of the stack then we pop off the stack and
return to "main" parsing mode.
Ragel's existing idea of calling and returning from machines is important
here too. As this currently stands this is not actually needed, but once
heredocs are in play we will have two possible places to return to at
the end of an interpolation sequence, so the state return stack maintained
by Ragel will determine whether to return to string mode or heredoc mode.
On reflection, it seems easier to maintain the necessary state we need
by doing all of the scanning in a single pass, since we can then just
use local variables within the scanner function.
