When a JSON object is representing an expression, template sequences are
permitted in the property names as well as the values. We must detect
the references here so that applications that do dynamic scope
construction or dependency analysis will get the right result.
Although our API had a place to provide a start position for scanning, it
didn't actually work in practice because the scanner wasn't aware of it
and so it would immediately undo the effect of that start offset when
making the first position adjustment.
Now we'll remember the byte offset we started at and offset the indices
the generate scanner produces so that they are are treated as relative
to that start byte instead of byte zero.
Since we rarely start with a non-zero pos this doesn't affect much, but
one specific thing it affects is the positions of native syntax templates
inside JSON syntax strings.
Due to incorrect diagnostics discipline here, the result of the index
operation was overwriting any diagnostics from either the collection or
the key expressions.
We were previously enabling newline-sensitivity too soon, before checking
for the "for" keyword, and thus seeing the newline token instead of the
for token when peeking ahead.
Now we peek for the for token first and turn on newline-sensitivity only
if it isn't present. This fixes another bug in turn where we would
previously have parsed the for expression itself in newline-sensitive
mode, which we no longer do because we delegate to the for expression
parser sooner.
To make things read better in the normal case, we treat naked identifiers
in the place of object keys as literal strings containing the identifier
text rather than as references. However, this had a couple sub-optimal
implications:
- If a user would try to create a key containing a period, the evaluator
would see that it wasn't a valid keyword and try to resolve it as a
normal scope traversal, causing a confusing error that didn't align
with the user's intent.
- In the rarer case where the attempted key contains a period followed by
a digit, the parser would trip over what seems to be an unexpected
identifier following the colon and produce, again, a confusing error
that doesn't align with what the user intended.
To address the first of these problems, it is now invalid to use a naked
traversal with more than one step as an object key, which allows us to
produce a targeted error message that directs the user to either put the
expression in parentheses to force interpretation as a scope traversal
or in quotes to force interpretation as a literal.
The second problem can't be addressed exactly due to it being a parser
problem, but we improve the situation slightly here by adding an extra
hint to the parse error message in this case so that a user making this
mistake might understand better how the error relates to what they were
trying to express.
Since the result type from a splat is derived from the source value type,
we can't predict our result type when the source type isn't known.
Previously we were incorrectly returning cty.Tuple(cty.DynamicPseudoType)
in this case because of the automatic tuple promotion logic, but now we'll
correctly just give up even before we do _that_ when given a DynamicVal.
Previously we were returning tuple-typed values in all cases, which meant
we had to return no type information at all if the source was an unknown
list since the length of that list is not predictable.
Instead we'll now return a list if the source is a list or set and a tuple
if the source is a tuple, allowing us to return exact type information
when the source value is unknown. This is important for catching type
errors early when accessing attributes across many objects using splat
syntax, since before the presence of a splat operator effectively caused
a total loss of downstream type checking for unknown values in expressions.
The pattern here was being too greedy because by default the longest match
is taken, and "*/" followed by more content is always longer than just
the "*/" to terminate the match.
The :>> symbol is a "finish guard" which tells Ragel to prefer to exit
from any* as soon as "*/" matches, making the match ungreedy.
The result of this is that input files containing more than one multi-line
comment will now tokenize each one separately, whereas before we would
create one long comment including everything from the first /* to the
final */ in the file, effectively causing parts of the file to be
ignored entirely.
* Add tests to cover new behavior introduced in cty
Convert should now not convert null to DynamicPseudoType, and Equals
will always compare Nulls as equal.
* update go.mod
This allows us to round-trip Body to JSON and back without any loss as
long as the expression source codes are always valid UTF-8, and we require
that during expression parsing anyway so that is a fine restriction.
The JSON encoding is a little noisy to read due to the extra annotations
required to be lossless (including source ranges) but still relatively
compact due to the base64-VLQ encoding of the source location information.
This allows the static analysis functions in the main HCL package to dig
through our wrapper to get the native expression object needed for most
analyses.
For example, this allows an expression with a native expression source
type whose source contains valid tuple constructor syntax to be used with
hcl.ExprList.
hclpack can potentially be used as an intermediary to more easily bring
non-HCL input into the HCL API, and so allowing a literal value in JSON
format as an expression type means that such a transcoder doesn't need to
worry about formatting values it encounters using HCL syntax and can
instead just encode directly to JSON, but at the expense of then not being
able to use the full expression/template syntax.
This is a straightforward way to get a hclpack.Body in the common case
where the input is already native syntax source code. Since the native
syntax is unambiguous about structure, the whole structure can be packed
in a single pass with no further information.
In most applications it's possible to fully evaluate configuration at the
beginning and work only with resolved values after that, but in some
unusual cases it's necessary to split parsing and decoding between two
separate processes connected by a pipe or network connection.
hclpack is intended to provide compact wire formats for sending bodies
over the network such that they can be decoded and evaluated and get the
same results. This is not something that can happen fully automatically
because a hcl.Body is an abstract node rather than a physical construct,
and so access to the original source code is required to construct such
a representation, and to interpret any source ranges that emerged from
the final evaluation.
To reduce confusion with object attributes, we switched a while back to
using the term "argument" to refer to the key/value pairs in a body, but
these diagnostic messages were not updated.
These are wrappers around the lower-level hclwrite package that are able
to reverse a subset of the behavior of the Decode functions to populate
an hclwrite DOM.
They are not fully symmetrical with DecodeBody because that function can
leave certain parts of the configuration in its opaque form for later
decoding, and these encode functions don't have enough information to
repack that abstract/opaque form into new source code.
In practice we expect that callers using complex techniques like partial
decoding will also use more complex techniques with the hclwrite API
directly, since they will need to coordinate partial _encoding_ of data
that has been portioned off into separate structures, which gohcl is not
equipped to do itself.
This completes the minimal functionality for creating a new file from
scratch, rather than modifying an existing one. This is illustrated by
a new test TestRoundupCreate that uses the API to create a new file in a
similar way to how a calling application might.
There isn't any strong reason for this -- they don't implement io.Reader
and so can't be used in places where a Reader+WriterTo is expected, like
io.Copy -- but go lint thinks that anything called WriteTo with an
io.Writer argument is an attempt to implement WriterTo and so this just
shuts up the linter.
Since this function implicitly creates a new body, this name is more
appropriate and leaves the name "AppendBlock" open for a later method to
append an _existing_ block, such as when moving a block from one file
to another.
Invalid blocks now have empty bodies rather than nil bodies, to avoid the
need to callers to specially handle nils here when they are doing analysis
of a partially-invalid file.
This method allows a caller to generate a nested block within a body.
Since a nested block has its own content body, this now allows for deep
structures to be generated.
Our usual rule for parse errors is to return a valid-but-incomplete object
along with error diagnostics. This was violating that rule by returning
a nil child body, which callers do not expect to deal with.
Instead, we'll return an *empty* body, so that callers who use the partial
result for careful analyses can still process the block header, without
needing to guard against the body being nil.
This was always a bit of an outlier here because the rest of the API is
intentionally designed to encourage treating AST nodes as immutable.
Although the transforming walkers were functionally correct, they were
causing false positives in the race detector if two walks run concurrently.
We may later introduce something similar to this in the hclwrite package,
where the AST nodes are explicitly mutable.
There is precedent for allowing strings containing digits where numbers are expected,
and so this extends that to also allow for boolean values to be given as strings.
This applies only to callers going through the decoder API. Direct access via the AST
will reflect exactly what was given in the input configuration.
