When dealing with numbers that have no finite representation in base 2, it
is important that all parsers agree on the expected maximum precision.
Previously we had agreement by convention, but for robustness here we'll
centralize the handling of number parsing to cty.ParseNumberVal, which
uses the same settings as we were previously using in the JSON parser and,
for the native syntax parser, is just a shorthand to the same parsing
we were previously doing with the cty/convert package.
This should cause no behavior change since all of these callers were
previously in agreement with the cty "standard", but this factoring helps
establish that there _is_ a standard here.
This includes a new function cty.ParseNumberVal which centralizes the
standard way to produce a cty.Number from a string so we can be sure to
always get comparable numbers.
When marshalling, the current file index was not stored. Because of
this, a ';' was inserted multiple times for each file, even if the file
did not change.
When unmarshalling, the fileIdx determined by number of ';' was ignored.
Thus, if there were more than one file, all the positions would still
point to the first file.
Fixes setting the MissingItemRange on the remaining body when a
*hclpack.Body is partially decoded. Otherwise when the remaining body is
decoded with missing fields, the diagnostic cannot point to where they
should be set.
Fixes an issue where a nested block would be decoded incorrectly, the
body of the last decoded block overwrites the previously decoded ones.
This was caused by the block being assigned on the stack in the for
loop; when the block is converted to a *hcl.Block, the pointer to Body
will always point to the same block. This caused decoding a new block to
overwrite the bodies of any previously decoded blocks.
This allows using a splat expression to conveniently coerce a
possibly-null scalar into a zero- or one-item tuple, which is helpful
because in HCL we prefer "for each item in sequence" operations over pure
conditionals in many situations just because they compose better in our
declarative language.
For example, in a language that uses the "dynblock" extension we can
turn a possibly-null object into zero or one blocks using its for_each
argument with a splat operation:
dynamic "thingy" {
for_each = maybe_null.*
content {
name = thingy.value.name
}
}
This fixes#66.
Previously we were incorrectly passing down the original forEachCtx down
to nested child blocks for recursive expansion. Instead, we must use the
iteration-specific constructed EvalContext, which then allows any nested
dynamic blocks to use the parent's iterator variable in their for_each or
labels expressions, and thus unpack nested data structures into
corresponding nested block structures:
dynamic "parent" {
for_each = [["a", "b"], []]
content {
dynamic "child" {
for_each = parent.value
content {}
}
}
}
A BOM is pointless in a UTF-8 file because it has a fixed encoding
agnostic of host byte ordering, but since Windows tends to use UTF-16
internally lots of Windows software will tend to generate redundant BOM
sequences at the start of UTF-8 files too.
By tolerating a leading BOM we can make life easier for those using such
Windows software, without any significant loss for normal use. This
slightly violates some of our normal assumptions about token positioning
since the BOM occupies bytes but not visible columns, but we'll just
accept that this may cause some slightly-odd behavior for use-cases such
as the diagnostic renderer and hclwrite.
Template sequences are forbidden in block labels, but previously we were
handling them in a very severe way, by bailing out of block parsing early
and leaving the body in the AST as nil.
The rest of HCL doesn't expect to find a nil body, and in any case we can
safely keep parsing the rest of the block after recovering because the
closing quote gives us an unambiguous resume point. Therefore we'll now
process the rest of the block as normal, producing an AST that is complete
aside from having an invalid label string inside of it.
Skipping the template sequence in the returned label entirely creates a
risk that analysis code (which may try to inspect a partial AST on error)
will misinterpret the string as valid, so we generate a placeholder
"${ ... }" or "%{ ... }" sequence in the returned string to make it
clearer in any follow-up output that there was something there in the
original source. Normal callers won't be affected by this because they
don't process the AST when errors are present anyway.
The traversal returned from AbsTraversalForExpr may, for some expression
types, be referring to the same backing array as one stored inside the
node itself, and so previously this function may have inadvertently
corrupted the data associated with an AST node.
The symmetrical spaces around colons in conditionals are important for
familiarity with C-like languages, so we'll instead accept spaces around
colons in our HCL2-unique "for expression" construct.
Leading whitespace is significant in heredocs, so we'll avoid making any
indentation adjustments for lines between OHeredoc and CHeredoc.
This fixes#31.
Our normal ruleset thinks that the "in" keyword here is a variable
reference and so writes it as "in[y]". Since there's never any reason for
a variable to appear immediately after another variable, we can check
for a preceding identifier as a heuristic to recognize whether in is
probably being used as a keyword rather than as a variable.
This is not exact, but the only time this should be a false positive is
if there were a syntax error in the input, and we don't make any
guarantees about the result in that case anyway.
This fixes#52.
This relaxes our previous spec to include a special form from HCL 1:
foo { bar = baz }
Although we normally require each argument to be on a line of its own, as
a special case we allow a block to be defined with a single nested
argument all on one line.
Only one nested argument definition is allowed, and a nested block
definition like "foo { bar {} }" is also disallowed in order to force the
more-readable split of bar {} onto a line of its own.
This is a pragmatic addition for broader compatibility with HCL 1-oriented
input. This single-line usage is not considered idiomatic HCL 2 and may
in future be undone by the formatter, though for now it is left as-is
aside from the spacing around the braces.
This also changes the behavior of the source code formatter to include
spaces on both sides of braces. This mimicks the formatting behavior of
HCL 1 for this situation, and (subjectively) reads better even for other
one-line braced expressions like object constructors and object for
expressions.
We were taking a pointer to a for loop iterator variable and thus
capturing the final iteration value rather than each one separately. By
using the .Ptr() method instead, we force a copy of the range which we
then take a pointer to.
This was implemented a long time ago in the original template parser, but
it was missed in the rewrite of the template parser to make it use a
two-stage parsing strategy.
It's implemented as a post-processing step on the result of the first
stage of parsing, which produces a flat sequence of literal strings,
interpolation markers, and control markers, and prior to the second stage
which matches opening and closing control markers to produce an expression
AST.
It's important to do this at parse time rather than eval time since it is
the static layout of the source code that decides the indentation level,
and so an interpolation marker at the start of a line that itself produces
spaces does not affect the result.
The evaluation of this was there but the parsing was still a TODO comment
from early development. Whoops!
Fortunately the existing parser functionality makes this straightforward
since we can just have the traversal parser recursively call itself.
This fixes#63.
When a JSON object is representing an expression, template sequences are
permitted in the property names as well as the values. We must detect
the references here so that applications that do dynamic scope
construction or dependency analysis will get the right result.
Although our API had a place to provide a start position for scanning, it
didn't actually work in practice because the scanner wasn't aware of it
and so it would immediately undo the effect of that start offset when
making the first position adjustment.
Now we'll remember the byte offset we started at and offset the indices
the generate scanner produces so that they are are treated as relative
to that start byte instead of byte zero.
Since we rarely start with a non-zero pos this doesn't affect much, but
one specific thing it affects is the positions of native syntax templates
inside JSON syntax strings.
Due to incorrect diagnostics discipline here, the result of the index
operation was overwriting any diagnostics from either the collection or
the key expressions.
We were previously enabling newline-sensitivity too soon, before checking
for the "for" keyword, and thus seeing the newline token instead of the
for token when peeking ahead.
Now we peek for the for token first and turn on newline-sensitivity only
if it isn't present. This fixes another bug in turn where we would
previously have parsed the for expression itself in newline-sensitive
mode, which we no longer do because we delegate to the for expression
parser sooner.
To make things read better in the normal case, we treat naked identifiers
in the place of object keys as literal strings containing the identifier
text rather than as references. However, this had a couple sub-optimal
implications:
- If a user would try to create a key containing a period, the evaluator
would see that it wasn't a valid keyword and try to resolve it as a
normal scope traversal, causing a confusing error that didn't align
with the user's intent.
- In the rarer case where the attempted key contains a period followed by
a digit, the parser would trip over what seems to be an unexpected
identifier following the colon and produce, again, a confusing error
that doesn't align with what the user intended.
To address the first of these problems, it is now invalid to use a naked
traversal with more than one step as an object key, which allows us to
produce a targeted error message that directs the user to either put the
expression in parentheses to force interpretation as a scope traversal
or in quotes to force interpretation as a literal.
The second problem can't be addressed exactly due to it being a parser
problem, but we improve the situation slightly here by adding an extra
hint to the parse error message in this case so that a user making this
mistake might understand better how the error relates to what they were
trying to express.
Since the result type from a splat is derived from the source value type,
we can't predict our result type when the source type isn't known.
Previously we were incorrectly returning cty.Tuple(cty.DynamicPseudoType)
in this case because of the automatic tuple promotion logic, but now we'll
correctly just give up even before we do _that_ when given a DynamicVal.
Previously we were returning tuple-typed values in all cases, which meant
we had to return no type information at all if the source was an unknown
list since the length of that list is not predictable.
Instead we'll now return a list if the source is a list or set and a tuple
if the source is a tuple, allowing us to return exact type information
when the source value is unknown. This is important for catching type
errors early when accessing attributes across many objects using splat
syntax, since before the presence of a splat operator effectively caused
a total loss of downstream type checking for unknown values in expressions.
The pattern here was being too greedy because by default the longest match
is taken, and "*/" followed by more content is always longer than just
the "*/" to terminate the match.
The :>> symbol is a "finish guard" which tells Ragel to prefer to exit
from any* as soon as "*/" matches, making the match ungreedy.
The result of this is that input files containing more than one multi-line
comment will now tokenize each one separately, whereas before we would
create one long comment including everything from the first /* to the
final */ in the file, effectively causing parts of the file to be
ignored entirely.
* Add tests to cover new behavior introduced in cty
Convert should now not convert null to DynamicPseudoType, and Equals
will always compare Nulls as equal.
* update go.mod
This allows us to round-trip Body to JSON and back without any loss as
long as the expression source codes are always valid UTF-8, and we require
that during expression parsing anyway so that is a fine restriction.
The JSON encoding is a little noisy to read due to the extra annotations
required to be lossless (including source ranges) but still relatively
compact due to the base64-VLQ encoding of the source location information.
This allows the static analysis functions in the main HCL package to dig
through our wrapper to get the native expression object needed for most
analyses.
For example, this allows an expression with a native expression source
type whose source contains valid tuple constructor syntax to be used with
hcl.ExprList.
hclpack can potentially be used as an intermediary to more easily bring
non-HCL input into the HCL API, and so allowing a literal value in JSON
format as an expression type means that such a transcoder doesn't need to
worry about formatting values it encounters using HCL syntax and can
instead just encode directly to JSON, but at the expense of then not being
able to use the full expression/template syntax.
This is a straightforward way to get a hclpack.Body in the common case
where the input is already native syntax source code. Since the native
syntax is unambiguous about structure, the whole structure can be packed
in a single pass with no further information.
