This allows using a splat expression to conveniently coerce a
possibly-null scalar into a zero- or one-item tuple, which is helpful
because in HCL we prefer "for each item in sequence" operations over pure
conditionals in many situations just because they compose better in our
declarative language.
For example, in a language that uses the "dynblock" extension we can
turn a possibly-null object into zero or one blocks using its for_each
argument with a splat operation:
dynamic "thingy" {
for_each = maybe_null.*
content {
name = thingy.value.name
}
}
This fixes#66.
A BOM is pointless in a UTF-8 file because it has a fixed encoding
agnostic of host byte ordering, but since Windows tends to use UTF-16
internally lots of Windows software will tend to generate redundant BOM
sequences at the start of UTF-8 files too.
By tolerating a leading BOM we can make life easier for those using such
Windows software, without any significant loss for normal use. This
slightly violates some of our normal assumptions about token positioning
since the BOM occupies bytes but not visible columns, but we'll just
accept that this may cause some slightly-odd behavior for use-cases such
as the diagnostic renderer and hclwrite.
Template sequences are forbidden in block labels, but previously we were
handling them in a very severe way, by bailing out of block parsing early
and leaving the body in the AST as nil.
The rest of HCL doesn't expect to find a nil body, and in any case we can
safely keep parsing the rest of the block after recovering because the
closing quote gives us an unambiguous resume point. Therefore we'll now
process the rest of the block as normal, producing an AST that is complete
aside from having an invalid label string inside of it.
Skipping the template sequence in the returned label entirely creates a
risk that analysis code (which may try to inspect a partial AST on error)
will misinterpret the string as valid, so we generate a placeholder
"${ ... }" or "%{ ... }" sequence in the returned string to make it
clearer in any follow-up output that there was something there in the
original source. Normal callers won't be affected by this because they
don't process the AST when errors are present anyway.
The traversal returned from AbsTraversalForExpr may, for some expression
types, be referring to the same backing array as one stored inside the
node itself, and so previously this function may have inadvertently
corrupted the data associated with an AST node.
This relaxes our previous spec to include a special form from HCL 1:
foo { bar = baz }
Although we normally require each argument to be on a line of its own, as
a special case we allow a block to be defined with a single nested
argument all on one line.
Only one nested argument definition is allowed, and a nested block
definition like "foo { bar {} }" is also disallowed in order to force the
more-readable split of bar {} onto a line of its own.
This is a pragmatic addition for broader compatibility with HCL 1-oriented
input. This single-line usage is not considered idiomatic HCL 2 and may
in future be undone by the formatter, though for now it is left as-is
aside from the spacing around the braces.
This also changes the behavior of the source code formatter to include
spaces on both sides of braces. This mimicks the formatting behavior of
HCL 1 for this situation, and (subjectively) reads better even for other
one-line braced expressions like object constructors and object for
expressions.
This was implemented a long time ago in the original template parser, but
it was missed in the rewrite of the template parser to make it use a
two-stage parsing strategy.
It's implemented as a post-processing step on the result of the first
stage of parsing, which produces a flat sequence of literal strings,
interpolation markers, and control markers, and prior to the second stage
which matches opening and closing control markers to produce an expression
AST.
It's important to do this at parse time rather than eval time since it is
the static layout of the source code that decides the indentation level,
and so an interpolation marker at the start of a line that itself produces
spaces does not affect the result.
The evaluation of this was there but the parsing was still a TODO comment
from early development. Whoops!
Fortunately the existing parser functionality makes this straightforward
since we can just have the traversal parser recursively call itself.
This fixes#63.
When a JSON object is representing an expression, template sequences are
permitted in the property names as well as the values. We must detect
the references here so that applications that do dynamic scope
construction or dependency analysis will get the right result.
Although our API had a place to provide a start position for scanning, it
didn't actually work in practice because the scanner wasn't aware of it
and so it would immediately undo the effect of that start offset when
making the first position adjustment.
Now we'll remember the byte offset we started at and offset the indices
the generate scanner produces so that they are are treated as relative
to that start byte instead of byte zero.
Since we rarely start with a non-zero pos this doesn't affect much, but
one specific thing it affects is the positions of native syntax templates
inside JSON syntax strings.
Due to incorrect diagnostics discipline here, the result of the index
operation was overwriting any diagnostics from either the collection or
the key expressions.
We were previously enabling newline-sensitivity too soon, before checking
for the "for" keyword, and thus seeing the newline token instead of the
for token when peeking ahead.
Now we peek for the for token first and turn on newline-sensitivity only
if it isn't present. This fixes another bug in turn where we would
previously have parsed the for expression itself in newline-sensitive
mode, which we no longer do because we delegate to the for expression
parser sooner.
To make things read better in the normal case, we treat naked identifiers
in the place of object keys as literal strings containing the identifier
text rather than as references. However, this had a couple sub-optimal
implications:
- If a user would try to create a key containing a period, the evaluator
would see that it wasn't a valid keyword and try to resolve it as a
normal scope traversal, causing a confusing error that didn't align
with the user's intent.
- In the rarer case where the attempted key contains a period followed by
a digit, the parser would trip over what seems to be an unexpected
identifier following the colon and produce, again, a confusing error
that doesn't align with what the user intended.
To address the first of these problems, it is now invalid to use a naked
traversal with more than one step as an object key, which allows us to
produce a targeted error message that directs the user to either put the
expression in parentheses to force interpretation as a scope traversal
or in quotes to force interpretation as a literal.
The second problem can't be addressed exactly due to it being a parser
problem, but we improve the situation slightly here by adding an extra
hint to the parse error message in this case so that a user making this
mistake might understand better how the error relates to what they were
trying to express.
Since the result type from a splat is derived from the source value type,
we can't predict our result type when the source type isn't known.
Previously we were incorrectly returning cty.Tuple(cty.DynamicPseudoType)
in this case because of the automatic tuple promotion logic, but now we'll
correctly just give up even before we do _that_ when given a DynamicVal.
Previously we were returning tuple-typed values in all cases, which meant
we had to return no type information at all if the source was an unknown
list since the length of that list is not predictable.
Instead we'll now return a list if the source is a list or set and a tuple
if the source is a tuple, allowing us to return exact type information
when the source value is unknown. This is important for catching type
errors early when accessing attributes across many objects using splat
syntax, since before the presence of a splat operator effectively caused
a total loss of downstream type checking for unknown values in expressions.
The pattern here was being too greedy because by default the longest match
is taken, and "*/" followed by more content is always longer than just
the "*/" to terminate the match.
The :>> symbol is a "finish guard" which tells Ragel to prefer to exit
from any* as soon as "*/" matches, making the match ungreedy.
The result of this is that input files containing more than one multi-line
comment will now tokenize each one separately, whereas before we would
create one long comment including everything from the first /* to the
final */ in the file, effectively causing parts of the file to be
ignored entirely.
* Add tests to cover new behavior introduced in cty
Convert should now not convert null to DynamicPseudoType, and Equals
will always compare Nulls as equal.
* update go.mod
To reduce confusion with object attributes, we switched a while back to
using the term "argument" to refer to the key/value pairs in a body, but
these diagnostic messages were not updated.
Invalid blocks now have empty bodies rather than nil bodies, to avoid the
need to callers to specially handle nils here when they are doing analysis
of a partially-invalid file.
Our usual rule for parse errors is to return a valid-but-incomplete object
along with error diagnostics. This was violating that rule by returning
a nil child body, which callers do not expect to deal with.
Instead, we'll return an *empty* body, so that callers who use the partial
result for careful analyses can still process the block header, without
needing to guard against the body being nil.
This was always a bit of an outlier here because the rest of the API is
intentionally designed to encourage treating AST nodes as immutable.
Although the transforming walkers were functionally correct, they were
causing false positives in the race detector if two walks run concurrently.
We may later introduce something similar to this in the hclwrite package,
where the AST nodes are explicitly mutable.
Although the spec testsuite and associated harness is designed to be
usable by other implementations of HCL not written in Go, it's convenient
to run it as part of our own "go test" test suite here so there isn't
an additional thing to run on each change.
To achieve this, the new package hcl/spectests will build both hcldec and
hclspecsuite from latest source and then run the latter to execute the
test suite, capturing the output and converting it (sloppily) into
testing.T method calls to produce something vaguely reasonable.
Other than the small amount of "parsing" to make it look in the output
like a normal Go test, there's nothing special going on here and so it's
still valid to run the spec suite manually with a build of hcldec from
this codebase, which should produce the same result.
We previously weren't returning appropriate Expression and EvalContext
references inside many of the diagnostics for ForExpr.
First, it was using the top-level expression instead of one of the nested
expressions in many cases. Secondly, it was using the given context
rather than the child context when talking about expressions that get
evaluated once per iteration.
As a result of this reporting we must now produce a new EvalContext for
each iteration, rather than sharing and mutating as we did before, but
in retrospect that's less likely to cause other confusing bugs anyway,
since we don't generally expect EvalContexts to be mutated.
In practice this should never arise because the index operator only works
for lists and maps and they use string keys, but we'll guard against this
anyway and return a placeholder for other values so that the output
doesn't grow unreadably long in that case.
If a diagnostic has an associated Expression and EvalContext then we can
look up the values of any variables referenced in the expression and show
them in the diagnostics message as additional context.
This is particularly useful when dealing with situations where a given
expression is evaluated multiple times with different variables, such as
in a 'for' expression, since each evaluation may produce a different set
of diagnostics.
If a diagnostic occurs while we're evaluating an expression, we'll now
include a reference to that expression in the diagnostic object. We
previously added the corresponding EvalContext here too, and so with these
together it is now possible for a diagnostic renderer to see not only
what was in scope when the problem occurred but also what parts of that
scope the expression was relying on (via method Expression.Variables).
When building tools around HCL configuration files it is useful to be
able to ask what is present at a given position in a file. This set of
new helper functions provide a best-effort implementation of this for
the native syntax only.
It cannot be supported for JSON syntax with these signatures because the
JSON syntax is ambiguous and thus can't be interpreted without a schema
for each structural level. In practice this is not a big loss because
JSON files will usually be generated rather than hand-written anyway, and
so doing automatic analysis and transformation of them would not be
useful: the program that generated the file must be updated instead.
When we're evaluating expressions, we may end up evaluating the same
source-level expression a number of times in different contexts, such as
in a 'for' expression, where each one may produce a different set of
diagnostic messages.
Now we'll attach the EvalContext to each expression diagnostic so that
a diagnostic renderer can potentially show additional information to help
distinguish the different iterations in rendered diagnostics.
During implementation of HCL in other applications, it became clear that
the overloading of the word "attribute" to mean both a key/value pair in
a body and an element within an object value creates confusion.
It's too late to change that in the HCL Go API now, but here we at least
update the diagnostic messages. The new convention is that a key/value
pair within a block is now called an "argument", while an element of an
object is still called an "attribute".
It is unfortunate that the Go-facing API still uses the word "attribute"
for both, but the user experience is the most important thing and in
practice many applications will treat block arguments as one way to set
the attributes of some object anyway, and in that case arguments can be
thought of as the subset of attributes of an object whose values come
from that object's associated block.
This also includes a few other minor terminology tweaks in the diagnostic
messages the reflect how our lexicon has evolved during development and
authoring of user-facing documentation.
We automatically convert from set to list in many other situations, so for
consistency we should accept sets here too and just treat them as
unordered sequences.
This closes#30.
Due to the special handling of the anonymous symbol employed to evaluate
a splat expression, we need to employ a lock on that symbol so that it's
safe for concurrent evaluation.
As before, it's not safe to concurrently evaluate the same expression in
the same context, but it is now safe to do so as long as all concurrent
evaluations have a _distinct_ EvalContext.
This fixes#28.
