HCL uses a number of upstream libraries that implement algorithms defined
in Unicode. This commit is updating those libraries all to versions that
have Unicode 13 support.
The main implication of this for HCL directly is that when it returns
column numbers in source locations it will count characters using the
Unicode 13 definition of "character", which includes various new
multi-codeunit characters added in Unicode 13.
These new version dependencies will also make Unicode 13 support available
for other functionality that HCL callers might use, such as the stdlib
functions in upstream cty, even though HCL itself does not directly use
those.
This adjusts prior behavior that would error to now allow keys that
are marked; those marks are removed in evaluation in order to give
a valid string for the map key, but the resulting object or tuple
is marked as a whole with whatever mark the key contained, ensuring
marks are maintained.
Using marked values in a for expression conditional expression would
previously panic due to calling `.False()` on the result of the
expression.
This commit makes two changes:
- Unmark the conditional expression value before testing if it is false;
- Merge any marks from the conditional into the resulting marks for the
for expression.
When evaluating an HCL expression attempting
to use a marked value as an object key,
return an error rather than falling through
to the cty panic. The error style mimics similar
errors in the area.
Functions which accept multiple parameters can be called with the
expansion operator, `...`. When doing so, we must unmark the expanded
argument value before transforming it into a collection of function
arguments. To ensure that any marks applied to the collection are
preserved, we transfer the collection marks to the individual elements
as we build the argument list.
So far the expression parentheses syntax has been handled entirely in the
parser and has been totally invisible in the AST. That's fine for typical
expression evaluation, but over the years it's led to a few quirky
behaviors in less common situations where we've assumed that all
expressions are covered by the AST itself or by the source ranges that the
AST captures.
In particular, hclwrite assumes that all expressions will have source
ranges that cover their tokens, and it generates an incorrect physical
syntax tree when the AST doesn't uphold that.
After resisting through a few other similar bugs, this commit finally
introduces an explicit AST node for parentheses, which makes the
parentheses explicit in the AST and captures the larger source range that
includes the TokenOParen and the TokenCParen.
This means that parentheses will now be visible as a distinct node when
walking the AST, as reflected in the updated tests here. That may cause
downstream applications that traverse the tree to exhibit different
behaviors but we're not considering that as a "breaking change" because
the Walk function doesn't make any guarantees about the specific AST
shape.
The rules for the splat operator call for it to return an empty tuple
when its operand is null, but this rule was previously being
overridden by another rule that a value whose type is unknown
causes the operator to return an unknown value of unknown
type.
This was initially reported and discussed in Terraform, under
hashicorp/terraform#26746.
If a template expression interpolates values which have marks, we should
apply all of those marks to the output value. This allows template
expressions to function like native cty functions with respect to marks.
It seems to be somewhat common for someone to share HCL code via a forum
or a document and have the well-meaning word processor or CMS replace the
straight quotes with curly quotes, which then lead to confusing errors
when someone copies the result and tries to use it as valid HCL
configuration.
Here we add a special hint for that, giving a tailored error message
instead of the generic "This character is not used within the language"
error message.
HCL has always had some of these special hints implemented here, and they
were originally implemented with special token types to allow the parser
handle them. However, we later refactored to do the check all at once
inside the Lex* family of functions, prior to parsing, so it's now
relatively straightforward to handle it as a special case of TokenInvalid
rather than an entirely new token type. Perhaps later we'll rework the
existing ones to also just use TokenInvalid, but that's a decision for
another day.
Previously functions such as concat() would result in a panic if there
was a null element and a sequence, as in the included test. This PR adds
a check if the error index is outside of the range of arguments and
crafts an error that references the entire function instead of the null
argument.
The following expression caused a panic in hclwrite:
a = foo.*
This was due to the unusual dotted form of a full splat (where the splat
operator is at the end of the expression) being generated with an
invalid source range. In the full splat case, the end of the range was
uninitialized, which caused the token slice to be empty, and thus the
panic.
This commit fixes the bug, adds test coverage, and includes some bonus
tests for other splat expression cases.
HCL uses grapheme cluster segmentation to produce accurate "column"
indications in diagnostic messages and other human-oriented source
location information. Each new major version of Unicode introduces new
codepoints, some of which are defined to combine with other codepoints to
produce a single visible character (grapheme cluster).
We were previously using the rules from Unicode 9.0.0. This change
switches to using the segmentation rules from Unicode 12.0.0, which is
the latest version at the time of this commit and is also the version of
Unicode used for other purposes by the Go 1.14 runtime.
HCL does not use text segmentation results for any purpose that would
affect the meaning of decoded data extracted from HCL files, so this
change will only affect the human-oriented source positions generated for
files containing characters that were newly-introduced in Unicode 10, 11,
or 12. (Machine-oriented uses of source location information are based on
byte offsets and not affected by text segmentation.)
To allow easir adaptation of data already serialized as JSON, HCL native
syntax allows both equals signs _and_ colons for object constructors.
This was already implemented, but not reflected in the pseudo-BNF in
the specification.
Most of the time, the standard expression decoding built in to HCL is
sufficient. Sometimes though, it's useful to be able to customize the
decoding of certain arguments where the application intends to use them
in a very specific way, such as in static analysis.
This extension is an approximate analog of gohcl's support for decoding
into an hcl.Expression, allowing hcldec-based applications and
applications with custom functions to similarly capture and manipulate
the physical expressions used in arguments, rather than their values.
This includes one example use-case: the typeexpr extension now includes
a cty.Function called ConvertFunc that takes a type expression as its
second argument. A type expression is not evaluatable in the usual sense,
but thanks to cty capsule types we _can_ produce a cty.Value from one
and then make use of it inside the function implementation, without
exposing this custom type to the broader language:
convert(["foo"], set(string))
This mechanism is intentionally restricted only to "argument-like"
locations where there is a specific type we are attempting to decode into.
For now, that's hcldec AttrSpec/BlockAttrsSpec -- analogous to gohcl
decoding into hcl.Expression -- and in arguments to functions.
Some HCL callers make the (reasonable) assumption that the overall source
range of an expression will be a superset of all of the ranges of its
child expressions, for purposes such as extraction of source code
snippets, parse tree annotation in hclwrite, text editor analysis
functions like "go to reference", etc.
The IndexExpr type was not previously honoring that assumption, since its
source range was placed around only the bracket portion. That is a good
region to use when reporting errors relating to the index operation, but
it is not a faithful representation of the full extent of the expression.
In order to meet both of these requirements at once, IndexExpr now has
both SrcRange covering the entire expression and BracketRange covering
the index part delimited by brackets. We can then use BracketRange in
our error messages but return SrcRange as the result of the general
Range method that is common to all expression types.
In previous versions we had some bugs around template sequence escapes.
These tests show that they no longer seem to be present, and should
hopefully avoid them regressing in future.
Our error message for the ambiguous situation recommends doing this, but
the parser didn't actually previously allow it. Now we'll accept the form
that the error message recommends.
As before, we also accept a template with an interpolation sequence as
a disambiguation, but the error message doesn't mention that because it's
no longer idiomatic to use an inline string template containing just a
single interpolation sequence.
RouxAntoine
Martin Atkins
Alisdair McDiarmid
Pam Selle
Pam Selle
Rémi Lapeyre
Kristin Laemmert
Masayuki Morita