A sequence like "foo" is represented in the AST as a TemplateExpr with a
single string literal inside rather than as a string literal node
directly, so we need to recognize that situation during parsing and treat
it as a special case so we can get the intended behavior of representing
that index as a traversal step rather than as a dynamic index operation.
Most of the time this distinction doesn't matter, but it's important for
static analysis use-cases. In particular, hcl.AbsTraversalForExpr will now
accept an expression like foo["bar"] where before it would've rejected it.
This also includes a better error message for when an expression cannot be
recognized as a single traversal. There isn't really any context here to
return a direct reference to the construct that was problematic, which is
what we'd ideally do, but at least this new message includes a summary
of what is allowed and some examples of things that are not allowed as an
aid to understanding what "static variable reference" means.
Fix a bug where diagnostics found when evaluating array individual
elements are ignored, resulting in suppressed errors.
Nomad observed this issue in https://github.com/hashicorp/nomad/issues/5694.
Previously we were using the EndOfLine as a mandatory marker to end the
comment, but that meant that if a comment appeared immediately before EOF
without a newline on the end it would fail to match.
Now we use the :>> operator similarly to how we previously fixed
greediness in the multi-line comment case: it tells Ragel to end the
Comment production if the following pattern matches (if EndOfLine is found)
but also allows the point before EndOfLine to be a final state, in case
EOF shows up there.
The contract for our parser is that in the case of errors our result is
stil valid, though possibly incomplete, so that development tools can
still do analysis of the parts of the result we _were_ able to parse.
However, we were previously failing to meet that contract in the presence
of certain syntax errors during block parsing, where we were producing
a nil body instead of a valid empty one.
Now we'll produce an empty placeholder body if for any reason we don't
have a real one before we return from block parsing, which then allows
analysis tools to see that the containing block was present but makes
its content appear totally empty. This is always done in conjunction with
returning an error, so a calling application will not be mislead into
thinking it is a complete result even though parts are missing.
The TemplateStringLiteral production was not quite right, causing a
literal $ or % immediately followed by " to consume the quotes and any
following characters on the line if there were any more characters on the
line.
Now we match things more precisely, but at the expense of generating some
redundant extra tokens when escapes and literal dollar/percent signs are
present. Those extra tokens don't matter in practice because the resulting
strings get concatenated together anyway, which is proven by the fact
that this changeset includes changes only to the scanner and parser tests,
and not to any of the expression result tests.
While here, I also improved the error message for when the user attempts
to split a quoted string over multiple lines. Previously it was just using
the generic "invalid character" message, which isn't particularly
actionable. Now we'll give the user a couple options of what to do
instead.
This situation is likely to arise if the user attempts to compute an index
using division while expecting HCL to do integer division rather than
float division.
This message is intended therefore to help the user see what fix is needed
(round the number) but because of layering it sadly cannot suggest a
specific remedy because HCL itself has no built-in rounding functionality,
and thus how exactly that is done is a matter for the calling application.
We're accepting that compromise for now to see how common this turns out
to be in practice. My hypothesis is that it'll be seen more when an
application moves from HCL 1 to HCL 2 because HCL 1 would do integer
division in some cases, but then it will be less common once new patterns
are established in the language community. For example, any existing
examples of using integer division to index in Terraform will over time
be replaced with examples showing the use of Terraform's "floor" function.
This is a variant of NewRangeScanner that allows the caller to specify the
start position, which is appropriate when this utility is being used with
a sub-slice of a file, rather than the whole file.
Previously, hclsyntax MissingItemRange() function returned a zero-length
range anchored at the end of the block in question. This commit changes
that to the beginning of the block. In practice, the end of a block is
generally just a "}" and not very useful in error messages.
Evaluate json null values as cty.Null, rather than as unknown value.
Using DynamicPseudoType as the null type as a placeholder for the null
type. Callers may convert the type against schema to get the concrete
type.
The fact that object constructors are newline-sensitive while object for
expressions are not requires some special consideration in the parser. We
previously make a small fix here to delay turning on newline-sensitive
scanning before peeking ahead for a "for" keyword, but that was sufficient
only when the for expression was not already in a newline-sensitive
context.
Now we force newline-sensitive parsing off while we scan for the keyword,
and also again once we begin parsing the for expression, ensuring that
the for expression is always scanned properly regardless of what context
it appears in.
The range was incorrectly being reported as "Context", rather than
"Subject". The Context field has meaning only in conjunction with Subject.
While here, this also tweaks the summary to show the block type name in
quotes, since otherwise the sentence can read oddly for certain block type
names.
These make it easier for calling applications to get the same result as
operators within HCL expressions both for individual attribute accesses
and when processing whole cty.Paths.
(We previously had an Index function which did the same thing for
indexing, and ApplyPath is just a wrapper around calling GetAttr and Index
in a loop.)
When dealing with numbers that have no finite representation in base 2, it
is important that all parsers agree on the expected maximum precision.
Previously we had agreement by convention, but for robustness here we'll
centralize the handling of number parsing to cty.ParseNumberVal, which
uses the same settings as we were previously using in the JSON parser and,
for the native syntax parser, is just a shorthand to the same parsing
we were previously doing with the cty/convert package.
This should cause no behavior change since all of these callers were
previously in agreement with the cty "standard", but this factoring helps
establish that there _is_ a standard here.
This allows using a splat expression to conveniently coerce a
possibly-null scalar into a zero- or one-item tuple, which is helpful
because in HCL we prefer "for each item in sequence" operations over pure
conditionals in many situations just because they compose better in our
declarative language.
For example, in a language that uses the "dynblock" extension we can
turn a possibly-null object into zero or one blocks using its for_each
argument with a splat operation:
dynamic "thingy" {
for_each = maybe_null.*
content {
name = thingy.value.name
}
}
This fixes#66.
A BOM is pointless in a UTF-8 file because it has a fixed encoding
agnostic of host byte ordering, but since Windows tends to use UTF-16
internally lots of Windows software will tend to generate redundant BOM
sequences at the start of UTF-8 files too.
By tolerating a leading BOM we can make life easier for those using such
Windows software, without any significant loss for normal use. This
slightly violates some of our normal assumptions about token positioning
since the BOM occupies bytes but not visible columns, but we'll just
accept that this may cause some slightly-odd behavior for use-cases such
as the diagnostic renderer and hclwrite.
Template sequences are forbidden in block labels, but previously we were
handling them in a very severe way, by bailing out of block parsing early
and leaving the body in the AST as nil.
The rest of HCL doesn't expect to find a nil body, and in any case we can
safely keep parsing the rest of the block after recovering because the
closing quote gives us an unambiguous resume point. Therefore we'll now
process the rest of the block as normal, producing an AST that is complete
aside from having an invalid label string inside of it.
Skipping the template sequence in the returned label entirely creates a
risk that analysis code (which may try to inspect a partial AST on error)
will misinterpret the string as valid, so we generate a placeholder
"${ ... }" or "%{ ... }" sequence in the returned string to make it
clearer in any follow-up output that there was something there in the
original source. Normal callers won't be affected by this because they
don't process the AST when errors are present anyway.
The traversal returned from AbsTraversalForExpr may, for some expression
types, be referring to the same backing array as one stored inside the
node itself, and so previously this function may have inadvertently
corrupted the data associated with an AST node.
This relaxes our previous spec to include a special form from HCL 1:
foo { bar = baz }
Although we normally require each argument to be on a line of its own, as
a special case we allow a block to be defined with a single nested
argument all on one line.
Only one nested argument definition is allowed, and a nested block
definition like "foo { bar {} }" is also disallowed in order to force the
more-readable split of bar {} onto a line of its own.
This is a pragmatic addition for broader compatibility with HCL 1-oriented
input. This single-line usage is not considered idiomatic HCL 2 and may
in future be undone by the formatter, though for now it is left as-is
aside from the spacing around the braces.
This also changes the behavior of the source code formatter to include
spaces on both sides of braces. This mimicks the formatting behavior of
HCL 1 for this situation, and (subjectively) reads better even for other
one-line braced expressions like object constructors and object for
expressions.
This was implemented a long time ago in the original template parser, but
it was missed in the rewrite of the template parser to make it use a
two-stage parsing strategy.
It's implemented as a post-processing step on the result of the first
stage of parsing, which produces a flat sequence of literal strings,
interpolation markers, and control markers, and prior to the second stage
which matches opening and closing control markers to produce an expression
AST.
It's important to do this at parse time rather than eval time since it is
the static layout of the source code that decides the indentation level,
and so an interpolation marker at the start of a line that itself produces
spaces does not affect the result.
The evaluation of this was there but the parsing was still a TODO comment
from early development. Whoops!
Fortunately the existing parser functionality makes this straightforward
since we can just have the traversal parser recursively call itself.
This fixes#63.
When a JSON object is representing an expression, template sequences are
permitted in the property names as well as the values. We must detect
the references here so that applications that do dynamic scope
construction or dependency analysis will get the right result.
Although our API had a place to provide a start position for scanning, it
didn't actually work in practice because the scanner wasn't aware of it
and so it would immediately undo the effect of that start offset when
making the first position adjustment.
Now we'll remember the byte offset we started at and offset the indices
the generate scanner produces so that they are are treated as relative
to that start byte instead of byte zero.
Since we rarely start with a non-zero pos this doesn't affect much, but
one specific thing it affects is the positions of native syntax templates
inside JSON syntax strings.
Due to incorrect diagnostics discipline here, the result of the index
operation was overwriting any diagnostics from either the collection or
the key expressions.
We were previously enabling newline-sensitivity too soon, before checking
for the "for" keyword, and thus seeing the newline token instead of the
for token when peeking ahead.
Now we peek for the for token first and turn on newline-sensitivity only
if it isn't present. This fixes another bug in turn where we would
previously have parsed the for expression itself in newline-sensitive
mode, which we no longer do because we delegate to the for expression
parser sooner.
To make things read better in the normal case, we treat naked identifiers
in the place of object keys as literal strings containing the identifier
text rather than as references. However, this had a couple sub-optimal
implications:
- If a user would try to create a key containing a period, the evaluator
would see that it wasn't a valid keyword and try to resolve it as a
normal scope traversal, causing a confusing error that didn't align
with the user's intent.
- In the rarer case where the attempted key contains a period followed by
a digit, the parser would trip over what seems to be an unexpected
identifier following the colon and produce, again, a confusing error
that doesn't align with what the user intended.
To address the first of these problems, it is now invalid to use a naked
traversal with more than one step as an object key, which allows us to
produce a targeted error message that directs the user to either put the
expression in parentheses to force interpretation as a scope traversal
or in quotes to force interpretation as a literal.
The second problem can't be addressed exactly due to it being a parser
problem, but we improve the situation slightly here by adding an extra
hint to the parse error message in this case so that a user making this
mistake might understand better how the error relates to what they were
trying to express.
Since the result type from a splat is derived from the source value type,
we can't predict our result type when the source type isn't known.
Previously we were incorrectly returning cty.Tuple(cty.DynamicPseudoType)
in this case because of the automatic tuple promotion logic, but now we'll
correctly just give up even before we do _that_ when given a DynamicVal.
Previously we were returning tuple-typed values in all cases, which meant
we had to return no type information at all if the source was an unknown
list since the length of that list is not predictable.
Instead we'll now return a list if the source is a list or set and a tuple
if the source is a tuple, allowing us to return exact type information
when the source value is unknown. This is important for catching type
errors early when accessing attributes across many objects using splat
syntax, since before the presence of a splat operator effectively caused
a total loss of downstream type checking for unknown values in expressions.
The pattern here was being too greedy because by default the longest match
is taken, and "*/" followed by more content is always longer than just
the "*/" to terminate the match.
The :>> symbol is a "finish guard" which tells Ragel to prefer to exit
from any* as soon as "*/" matches, making the match ungreedy.
The result of this is that input files containing more than one multi-line
comment will now tokenize each one separately, whereas before we would
create one long comment including everything from the first /* to the
final */ in the file, effectively causing parts of the file to be
ignored entirely.
* Add tests to cover new behavior introduced in cty
Convert should now not convert null to DynamicPseudoType, and Equals
will always compare Nulls as equal.
* update go.mod
To reduce confusion with object attributes, we switched a while back to
using the term "argument" to refer to the key/value pairs in a body, but
these diagnostic messages were not updated.
Invalid blocks now have empty bodies rather than nil bodies, to avoid the
need to callers to specially handle nils here when they are doing analysis
of a partially-invalid file.
Our usual rule for parse errors is to return a valid-but-incomplete object
along with error diagnostics. This was violating that rule by returning
a nil child body, which callers do not expect to deal with.
Instead, we'll return an *empty* body, so that callers who use the partial
result for careful analyses can still process the block header, without
needing to guard against the body being nil.
This was always a bit of an outlier here because the rest of the API is
intentionally designed to encourage treating AST nodes as immutable.
Although the transforming walkers were functionally correct, they were
causing false positives in the race detector if two walks run concurrently.
We may later introduce something similar to this in the hclwrite package,
where the AST nodes are explicitly mutable.
