Although the spec testsuite and associated harness is designed to be
usable by other implementations of HCL not written in Go, it's convenient
to run it as part of our own "go test" test suite here so there isn't
an additional thing to run on each change.
To achieve this, the new package hcl/spectests will build both hcldec and
hclspecsuite from latest source and then run the latter to execute the
test suite, capturing the output and converting it (sloppily) into
testing.T method calls to produce something vaguely reasonable.
Other than the small amount of "parsing" to make it look in the output
like a normal Go test, there's nothing special going on here and so it's
still valid to run the spec suite manually with a build of hcldec from
this codebase, which should produce the same result.
We previously weren't returning appropriate Expression and EvalContext
references inside many of the diagnostics for ForExpr.
First, it was using the top-level expression instead of one of the nested
expressions in many cases. Secondly, it was using the given context
rather than the child context when talking about expressions that get
evaluated once per iteration.
As a result of this reporting we must now produce a new EvalContext for
each iteration, rather than sharing and mutating as we did before, but
in retrospect that's less likely to cause other confusing bugs anyway,
since we don't generally expect EvalContexts to be mutated.
In practice this should never arise because the index operator only works
for lists and maps and they use string keys, but we'll guard against this
anyway and return a placeholder for other values so that the output
doesn't grow unreadably long in that case.
If a diagnostic has an associated Expression and EvalContext then we can
look up the values of any variables referenced in the expression and show
them in the diagnostics message as additional context.
This is particularly useful when dealing with situations where a given
expression is evaluated multiple times with different variables, such as
in a 'for' expression, since each evaluation may produce a different set
of diagnostics.
If a diagnostic occurs while we're evaluating an expression, we'll now
include a reference to that expression in the diagnostic object. We
previously added the corresponding EvalContext here too, and so with these
together it is now possible for a diagnostic renderer to see not only
what was in scope when the problem occurred but also what parts of that
scope the expression was relying on (via method Expression.Variables).
When building tools around HCL configuration files it is useful to be
able to ask what is present at a given position in a file. This set of
new helper functions provide a best-effort implementation of this for
the native syntax only.
It cannot be supported for JSON syntax with these signatures because the
JSON syntax is ambiguous and thus can't be interpreted without a schema
for each structural level. In practice this is not a big loss because
JSON files will usually be generated rather than hand-written anyway, and
so doing automatic analysis and transformation of them would not be
useful: the program that generated the file must be updated instead.
When we're evaluating expressions, we may end up evaluating the same
source-level expression a number of times in different contexts, such as
in a 'for' expression, where each one may produce a different set of
diagnostic messages.
Now we'll attach the EvalContext to each expression diagnostic so that
a diagnostic renderer can potentially show additional information to help
distinguish the different iterations in rendered diagnostics.
During implementation of HCL in other applications, it became clear that
the overloading of the word "attribute" to mean both a key/value pair in
a body and an element within an object value creates confusion.
It's too late to change that in the HCL Go API now, but here we at least
update the diagnostic messages. The new convention is that a key/value
pair within a block is now called an "argument", while an element of an
object is still called an "attribute".
It is unfortunate that the Go-facing API still uses the word "attribute"
for both, but the user experience is the most important thing and in
practice many applications will treat block arguments as one way to set
the attributes of some object anyway, and in that case arguments can be
thought of as the subset of attributes of an object whose values come
from that object's associated block.
This also includes a few other minor terminology tweaks in the diagnostic
messages the reflect how our lexicon has evolved during development and
authoring of user-facing documentation.
We automatically convert from set to list in many other situations, so for
consistency we should accept sets here too and just treat them as
unordered sequences.
This closes#30.
Due to the special handling of the anonymous symbol employed to evaluate
a splat expression, we need to employ a lock on that symbol so that it's
safe for concurrent evaluation.
As before, it's not safe to concurrently evaluate the same expression in
the same context, but it is now safe to do so as long as all concurrent
evaluations have a _distinct_ EvalContext.
This fixes#28.
Previously we were only counting a \n as starting a new line, so input
using \r\n endings would get treated as one long line for source-range
purposes.
Now we also consider \r\n to be a newline marker, resetting the column
count to zero and incrementing the line just as we would do for a single
\n. This is made easier because the unicode definition of "grapheme
cluster" considers \r\n to be a single character, so we don't need to
do anything special in order to match it.
Previously, due to how heredoc scanning was implemented, the closing
marker for a heredoc would consume the newline that terminated it. This
was problematic in any context that is newline-sensitive, because it
would cause us to skip the TokenNewline that might terminate e.g. an
attribute definition:
foo = <<EOT
hello
EOT
bar = "hello"
Previously the "foo" attribute would fail to parse properly due to trying
to consume the "bar" definition as part of its expression.
Now we synthetically split the marker token into two parts: the marker
itself and the newline that follows it. This means that using a heredoc
in any context where newlines are sensitive will involuntarily introduce
a newline, but that seems consistent with user expectation based on how
heredocs seem to be used "in the wild".
Implementing the config loader for Terraform led to the addition of some
special static analysis operations for expressions, separate from the
usual action of evaluating an expression to produce a value.
These operations are useful for building application-specific language
constructs within HCL syntax, and so they are now included as part of the
specification in order to help developers of other applications understand
their behaviors and the implications of using them.
This accompanies ExprList, ExprMap, and AbsTraversalForExpr to
complete the set of static analysis interfaces for digging down into the
expression syntax structures without evaluation.
The intent of this function is to be a little like AbsTraversalForExpr
but for function calls. However, it's also similar to ExprList in that
it gives access to the raw expression objects for the arguments, allowing
for recursive analysis.
We recognize and allow naked $ and % sequences by reading ahead one more
character to see if it's a "{" that would introduce an interpolation or
control sequence.
Unfortunately this is problematic in the end condition because it can
"eat" the terminating character and cause the scanner to continue parsing
a template when the user intended the template to end.
Handling this is a bit messy. For the quoted and heredoc situations we
can use Ragel's fhold statement to "backtrack" to before the character
we consumed, which does the trick. For bare templates this is insufficient
because there _is_ no following character and so the scanner detects this
as an error.
Rather than adding even more complexity to the state machine, instead we
just handle as a special case invalid bytes at the top-level of a bare
template, returning them as a TokenStringLit instead of a TokenInvalid.
This then gives the parser what it needs.
The fhold approach causes some odd behavior where an escaped template
introducer character causes a token split and two tokens are emitted
instead of one. This is weird but harmless, since we'll ultimately just
concatenate all of these strings together anyway, and so we allow it
again to avoid making the scanner more complex when it's easy enough to
handle this in the parser where we have more context.
This was allowed in legacy HCL, and although it was never documented as
usable in the Terraform documentation it appears that some Terraform
configurations use this form anyway.
While it is non-ideal to have another edge-case to support/maintain, this
capability adds no ambiguity and doesn't add significant complexity, so
we'll allow it to be pragmatic for existing usage.
Terraform allowed indexing like foo.0.bar to work around HIL limitations,
and so we'll permit that as a pragmatic way to accept existing Terraform
configurations.
However, we can't support this fully because our parser thinks that
chained number indexes, like foo.0.0.bar, are single numbers. Since that
usage in Terraform is very rare (there are very few lists of lists) we
will mark that situation as an error with a helpful message suggesting
to use the modern index syntax instead.
This also turned up a similar bug in the existing legacy index handling
we were doing for splat expressions, which is now handled in the same
way.
We are leaning on the unicode identifier definitions here, but the
specified ID_Start does not include the underscore character and users
seem to expect this to be allowed due to experience with other languages.
Since allowing a leading underscore introduces no ambiguity, we'll allow
it. Calling applications may choose to reject it if they'd rather not have
such weird names.
Previously we missed the '%' character in our "SelfToken" production,
which meant that the modulo operator could not parse properly due to it
being represented as a TokenInvalid.
Due to some earlier limitations of the parser we required each attribute
and block to end with a newline, even if it appeared at the end of a
file. In effect, this required all files to end with a newline character.
This is no longer required and so we'll tolerate that missing newline for
pragmatic reasons.
Elsewhere we are using 512-bit precision as the standard for converting
from a string to a number, since the default is shorter. This is just to
unify JSON parsing with the native syntax processing and the automatic
type conversions in the language, so we don't see different precision
behaviors depending on syntax.
The contract for AbsTraversalForExpr calls for us to interpret an
expression as if it were traversal syntax. Traversal syntax does not have
the special keywords "null", "true" and "false", so we must interpret
these as TraverseRoot rather than as literal values.
Previously this wasn't working because the parser converted these to
literals too early. To make this work properly, we implement
AbsTraversalForExpr on literal expressions and effectively "undo" the
parser's re-interpretation of these keywords to back out to the original
keyword strings.
We also rework how object keys are handled so that we wait until eval time
to decide whether to interpret the key expression as an unquoted literal
string. This allows us to properly support AbsTraversalForExpr on keys
in object constructors, bypassing the string-interpretation behavior in
that case.
This is similar to the ExprList function but for map-like constructs
(object constructors in the native syntax). It allows a more-advanced
calling application to analyze the physical structure of the configuration
directly, rather than analyzing the dynamic results of its expressions.
This is useful when creating what appear to be first-class language
constructs out of the language's grammar elements.
In the JSON syntax, a static map construct is expressed as a direct JSON
object. As with ExprList, this bypasses any dynamic expression evaluation
behavior and requires the user to provide a literal JSON object, though
the calling application is then free to evaluate the key/value expressions
inside in whatever way makes sense.
Previously this was handled in the parser, but the parser now permits
multiple properties with the same name and so we must handle this at the
decoder level instead.
Previously we allowed arrays only at the "leaf" of a set of objects
describing a block and its labels. This is not sufficient because it is
therefore impossible to preserve the relative ordering of a sequence
of blocks that have different block types or labels.
The spec now allows arrays of objects to be used in place of single
objects when that value is representing either an HCL body or a set of
labels on a nested block. This relaxing does not apply to JSON objects
interpreted as expressions or bodies interpreted in dynamic attributes
mode, since there is no requirement to preserve attribute ordering or
support duplicate property names in those scenarios.
This new model imposes additional constraints on the underlying JSON
parser used to interpret JSON HCL: it must now be able to retain the
relative ordering of object keys and accept multiple definitions of the
same key. This requirement is not imposed on _producers_, which are free
to use the allowance for arrays of objects to force ordering and duplicate
keys with JSON-producing libraries that are unable to make these
distinctions.
Since we are now requiring a specialized parser anyway, we also require
that it be able to represent numbers at full precision, whereas before
we made some allowances for implementations to not support this.
The peeker has an "include newlines" stack which the parser manipulates
to switch between the newline-sensitive and non-sensitive scanning modes.
If the parser code fails to manage this stack correctly (for example,
due to a missed call to PopIncludeNewlines) then this causes very
confusing downstream errors that are otherwise difficult to debug.
As an extra debug tool for when errors _are_ detected, when this problem
is encountered during tests we are able to produce a visualization of the
pushes and pops to help the test developer see which pushes and pops
seem out of place.
This is a lot of ugly extra code but it's usually disabled and seems worth
it to allow us to catch quickly bugs that would otherwise be quite
difficult to diagnose.
Previously it was mismanaging the stack by first pushing on "false" and
then trying to undo that by pushing on "true". Instead, it should just
pop off the "false" to return to whatever the previous setting was, since
indexing brackets might already be inside a no-newlines context.
We were previously using an ugly combination of "pretty" and "spew" to
do this, which never really quite worked because of limitations in each
of those.
deep.Equal doesn't produce quite as much detailed information as the
others, but it has the advantage of showing exactly where a difference
exists rather than forcing us to hunt through a noisy diff to find it.
Fuzz testing revealed that there were a few different crashers in the
string literal decoder, which was previously a rather-unweildy
hand-written scanner with manually-implemented lookahead.
Rather than continuing to hand-tweak that code, here instead we use
ragel (which we were already using for the main scanner anyway) to
partition our string literals into tokens that are easier for our
decoder to wrangle.
As a bonus, this also makes our source ranges in our diagnostics more
accurate.
A common pattern is emerging in calling applications of using single-item
absolute traversals to give the impression of static language keywords.
This new function makes that explicitly possible and allows a convenient
pattern for doing so that should improve the readability of a calling
application making use of it.
Calling applications often need to validate strings provided by the user
that will eventually be variable or attribute names in the evaluation
scope, to ensure that they will be evaluable.
Rather than having each application specify its own different subset of
the full set we support (which is derived from Unicode specifications),
we provide a simple function to let callers easily check the validity
of a potential identifier using exactly the same scanning rules we use
within the expression scanner.
To achieve this we actually invoke the scanner and then assert on its
result, which is a pretty expensive way to just check one string but it's
easy to do with code we already have in place and we don't expect this
sort of validation to be going on in a tight loop.
These allow the inclusion of arbitrary unicode codepoints (always encoded
as UTF-8) using a hex representation.
\u expects four digits and can thus represent only characters in the basic
multilingual plane.
\U expects eight digits and can thus represent all unicode characters,
at the cost of being extra-verbose.
Since our parser properly accounts for unicode characters (including
combining sequences) it's recommended to include them literally (UTF-8
encoded) in source code, but these sequences are useful for explicitly
representing non-printable characters that could otherwise appear
invisible in source code, such as zero-width modifier characters.
This fixes#6.
We inherited a restriction from an early zcl prototype here, but it's
far too strict to prohibit tabs entirely and so we'll accept them and
just treat them as spaces for column-counting purposes.
Tabs are still not _advised_, since they add extra complexity for problems
like generating annotated source code snippets (can't necessarily know
how large the tab stop is going to be) or doing surgical updates to
existing source files. The canonical formatting applied by hclwrite's
Format function will still eliminate all tabs, imposing the canonical
style of two spaces per indent level.
This fixes#2.
An earlier iteration of this package was able to optionally use HIL as
its expression engine in place of the hclsyntax expression parser, but
this has since been removed and so this flag no longer has any effect.
Consequently, the public functions ParseWithHIL and ParseFileWithHIL were,
in fact, just using the zclsyntax parser and thus behaving identically to
the Parse and ParseFile functions.
A pattern has emerged of wrapping Expression instances with other
Expressions in order to subtly modify their behavior. A key example of
this is in ext/dynblock, where wrap an expression in order to introduce
our additional iteration variable for expressions in dynamic blocks.
Rather than having each wrapper expression implement wrapping
implementations for our various syntax-level-analysis functions (like
ExprList and AbsTraversalForExpr), instead we define a standard mechanism
to unwrap expressions back to the lowest-level object -- usually an AST
node -- and then use this in all of our analyses that look at the
expression's structure rather than its value.
Terraform is the prime use-case for the dynblock extension, so we'll
include this here currently as a proof-of-concept for Terraform's usage,
but eventually (once Terraform is actually using it) this'll give some
insurance that it doesn't get broken.
In early prototyping the template control sequence introducer was
specified as !{, but that changed to %{ along the way because it seemed
more intuitive and less likely to collide with literal strings.
However, the parser's string literal handling still had remnants of the
old syntax, causing strange quirks in parsing strings that contained
exclamation points.
Now we correctly expect %{ as the control sequence introducer, %%{ as its
escape sequence, and additionally fix a bug where previously template
sequence introduction characters at the end of a string literal would
be silently dropped due to them representing an unterminated escape
sequence.
This fixes#3.
Traversals are always passed by value, so returning a pointer here is
inconsistent with how hcl.TraverseIndex is used elsewhere and thus makes
life inconvenient for callers making type assertions.
In complex expressions it can be hard to determine which portion is
relevant when we print a diagnostic message. To address this, when color
is enabled we bold and underline the "subject" portion of the source code,
which then makes it stand out within the full lines of code we print
in the snippet.
Now that we have helper methods for computing relationships between
ranges, we can eliminate all of the tricky line-counting and byte-counting
code here and instead use the higher-level operations.
The result is a single loop using the RangeScanner.
This is a convenience wrapper around SourceRange.Overlap that also
calculates the ranges in the receiver that _aren't_ overlapping with the
given range.
This is useful when, for example, partitioning a portion of source code
to insert markers to highlight the location of an error, as we do when
printing code snippets as part of diagnostic output.
This is a generalization of RangeBetween that finds a single range that
covers the full extent of both given ranges, possibly also including some
additional content between the ranges if they do not overlap.
RangeScanner has an interface similar to bufio.Scanner for partitioning
a buffer into tokens, but it returns the hcl.Range of each token along
with that token so that the caller can see where the token fits in
relation to the entire source file.
The main intended use-case for this is to partition a source file into
lines for the purpose of printing a source code snippet in diagnostic
output. Having the source location information is important in that case
to recognize which lines belong to the subject and context of each
diagnostic.
This is useful, for example, when printing source snippets to the terminal
as part of diagnostics, in order to detect the portion of the source code
that coincides with the subject or context of each diagnostic.
This can be useful, for example, when using Expression.Variables to
pre-validate all of the referenced variables before evaluation, so that
the traversal source ranges can be included in any generated diagnostics.
As an extra level of confidence in addition to the unit tests, this
integration test verifies that a certain set of features that Terraform
uses are able to work properly together.
Terraform is used as an example here just because it's a more advanced
consumer of HCL and thus it exercises some codepaths that most
applications don't need, such as ExprList and AbsTraversalForExpr.
This helper allows a calling application to require a given expression be
some sort of list constructor (tuple constructor in native syntax, or
array in JSON) and peel off that outer level of list to obtain a slice
of the Expression objects inside.
This is useful in rare cases where the calling application needs to
extract the expressions within the list without evaluating the entire list
expression first. For example, the expressions that result from this
function might be passed into AbsTraversalForExpr in situations where the
caller requires a static list of static traversals that will never
actually be evaluated.
These functions permit a calling application to recognize when an
expression represents a static absolute traversal and obtain that
traversal. This allows for the unusual-but-valid case where an application
wishes to access the expression source rather than its resulting value,
when the expression source is something that can be understood as a
traversal.
An example use-case is an attribute that takes a list of other attributes
it depends on, expressed as traversals. In this case the calling
application needs to access the attribute names themselves rather than
their values, e.g. to build some sort of dependency graph to gradually
populate the scope for evaluation.
Sometimes we want an expression that just wraps a static value, e.g. for
testing or to provide a default value for a missing attribute.
StaticExpr gives us a convenient way to do that, returning a value that
implements the Expression interface by returning just the given static
value.
This is a super-invasive update since the "zcl" package in particular
is referenced all over.
There are probably still a few zcl references hanging around in comments,
etc but this takes care of most of it.
