"HCL Configuration Language" will feel redundant to anyone who thinks
they know what "HCL" is supposed to stand for, even though our docs don't
actually expand the abbreviation at all. This new version is also still
redundant with that interpretation, but at least it emphasizes that it's
a toolkit for creating configuration languages rather than a configuration
language in its own right.
Git isn't available in the Read The Docs build, but we don't actually need
it there anyway because we are told the version number that Read The Docs
thinks it is building.
Although our underlying parse tree retains all of the token content, it
doesn't necessarily retain all of the spacing information under editing,
and so formatting on save ensures that we'll produce a canonical result
even if some edits have been applied that have changed the expected
alignment of objects, etc.
When nested attributes are of type cty.DynamicPseudoType, a block spec
that is backed by a cty collection is annoying to use because it requires
all of the blocks to have homogenous types for such attributes.
These new specs are similar to BlockListSpec and BlockMapSpec
respectively, but permit each nested block result to have its own distinct
type.
In return for this flexibility, we lose the ability to predict the exact
type of the result: these specs must just indicate their type as being
cty.DynamicPseudoType themselves, since we need to know how many blocks
there are and what types are inside them before we can know our final
result type.
Our BlockList, BlockSet, and BlockMap specs all produce cty collection
values, which require all elements to have a homogeneous type. If the
nested spec contained an attribute of type cty.DynamicPseudoType, that
would create the risk of each element having a different type, which would
previously have caused decoding to panic.
Now we either handle this during decode (BlockList, BlockSet) or forbid
it outright (BlockMap) to prevent that crash. BlockMap could _potentially_
also handle this during decode, but that would require a more significant
reorganization of its implementation than I want to take on right now,
and decoding dynamically-typed values inside collections is an edge case
anyway.
However, since this repository is only a temporary home for this module,
the go.mod file carries a warning to that effect. It will move to its more
permanent home as it transitions from experimental to released.
Although the spec testsuite and associated harness is designed to be
usable by other implementations of HCL not written in Go, it's convenient
to run it as part of our own "go test" test suite here so there isn't
an additional thing to run on each change.
To achieve this, the new package hcl/spectests will build both hcldec and
hclspecsuite from latest source and then run the latter to execute the
test suite, capturing the output and converting it (sloppily) into
testing.T method calls to produce something vaguely reasonable.
Other than the small amount of "parsing" to make it look in the output
like a normal Go test, there's nothing special going on here and so it's
still valid to run the spec suite manually with a build of hcldec from
this codebase, which should produce the same result.
When a test file declares one or more expected diagnostics, we check those
instead of checking the result value. The severities and source ranges
must match.
We don't test the error messages themselves because they are not part of
the specification and may vary between implementations or, in future, be
translated into other languages.
The harness can now run tests that decode successfully and compare the
result with a given value. Further work is required in later commits to
deal with other cases, such as tests that intentionally produce errors.
By default we generate human-readable diagnostics on the assumption that
the caller is a simple program that is capturing stdin via a pipe and
letting stderr go to the terminal.
More sophisticated callers may wish to analyze the diagnostics themselves
and perhaps present them in a different way, such as via a GUI.
This option skips the usual decoding step and instead prints out a JSON-
formatted list of the variables that are referenced by the configuration.
In simple cases this is not required, but for more complex use-cases it
can be useful to first analyze the input to see which variables need to
be in the scope, then construct a suitable set of variables before finally
decoding the input. For example, some of the variable values may be
expensive to produce.
This is the hcldec interface to Body.JustAttributes, producing a map whose
keys are the child attribute names and whose values are the results of
evaluating those expressions.
We can't just expose a JustAttributes-style spec directly here because
it's not really compatible with how hcldec thinks about things, but we
can expose a spec that decodes a specific child block because that can
then compose properly with other specs at the same level without
interfering with their operation.
The primary use for this is to allow the use of the block syntax to define
a map:
dynamic_stuff {
foo = "bar"
}
JustAttributes is normally used in static analysis situations such as
enumerating the contents of a block to decide what to include in the
final EvalContext. That's not really possible with the hcldec model
because both structural decoding and expression evaluation happen
together. Therefore the use of this is pretty limited: it's useful if you
want to be compatible with an existing format based on legacy HCL where a
map was conventionally defined using block syntax, relying on the fact
that HCL did not make a strong distinction between attribute and block
syntax.
For now, this is the only way to set an attribute, and so attributes can
only be set to literal values.
Later this will be generalized so that this is just a helper wrapper
around a "SetAttribute" method that just uses a given expression, which
then helps by constructing the expression from the value first.
The original prototype of hclwrite tried to track both the tokens and
the AST as two parallel data structures. This quickly exploded in
complexity, leading to lots of messy code to manage keeping those two
structures in sync.
This new approach melds the two structures together, creating first a
physical token tree (made of "node" objects, and hidden from the caller)
and then attaching the AST nodes to that token tree as additional sidecar
data.
The result is much easier to work with, leading to less code in the parser
and considerably less complex data structures in the parser's tests.
This commit is enough to reach feature parity with the previous prototype,
but it remains a prototype. With a more usable foundation, we'll evolve
this into a more complete implementation in subsequent commits.
We previously weren't returning appropriate Expression and EvalContext
references inside many of the diagnostics for ForExpr.
First, it was using the top-level expression instead of one of the nested
expressions in many cases. Secondly, it was using the given context
rather than the child context when talking about expressions that get
evaluated once per iteration.
As a result of this reporting we must now produce a new EvalContext for
each iteration, rather than sharing and mutating as we did before, but
in retrospect that's less likely to cause other confusing bugs anyway,
since we don't generally expect EvalContexts to be mutated.
In practice this should never arise because the index operator only works
for lists and maps and they use string keys, but we'll guard against this
anyway and return a placeholder for other values so that the output
doesn't grow unreadably long in that case.
If a diagnostic has an associated Expression and EvalContext then we can
look up the values of any variables referenced in the expression and show
them in the diagnostics message as additional context.
This is particularly useful when dealing with situations where a given
expression is evaluated multiple times with different variables, such as
in a 'for' expression, since each evaluation may produce a different set
of diagnostics.
If a diagnostic occurs while we're evaluating an expression, we'll now
include a reference to that expression in the diagnostic object. We
previously added the corresponding EvalContext here too, and so with these
together it is now possible for a diagnostic renderer to see not only
what was in scope when the problem occurred but also what parts of that
scope the expression was relying on (via method Expression.Variables).
When building tools around HCL configuration files it is useful to be
able to ask what is present at a given position in a file. This set of
new helper functions provide a best-effort implementation of this for
the native syntax only.
It cannot be supported for JSON syntax with these signatures because the
JSON syntax is ambiguous and thus can't be interpreted without a schema
for each structural level. In practice this is not a big loss because
JSON files will usually be generated rather than hand-written anyway, and
so doing automatic analysis and transformation of them would not be
useful: the program that generated the file must be updated instead.
When we're evaluating expressions, we may end up evaluating the same
source-level expression a number of times in different contexts, such as
in a 'for' expression, where each one may produce a different set of
diagnostic messages.
Now we'll attach the EvalContext to each expression diagnostic so that
a diagnostic renderer can potentially show additional information to help
distinguish the different iterations in rendered diagnostics.
During implementation of HCL in other applications, it became clear that
the overloading of the word "attribute" to mean both a key/value pair in
a body and an element within an object value creates confusion.
It's too late to change that in the HCL Go API now, but here we at least
update the diagnostic messages. The new convention is that a key/value
pair within a block is now called an "argument", while an element of an
object is still called an "attribute".
It is unfortunate that the Go-facing API still uses the word "attribute"
for both, but the user experience is the most important thing and in
practice many applications will treat block arguments as one way to set
the attributes of some object anyway, and in that case arguments can be
thought of as the subset of attributes of an object whose values come
from that object's associated block.
This also includes a few other minor terminology tweaks in the diagnostic
messages the reflect how our lexicon has evolved during development and
authoring of user-facing documentation.
This will allow for use-cases such as renaming a variable (changing the
content of the first token) and replacing variable references with
constant values that they evaluate to for debug purposes.
This is for the core HCL syntax, so it doesn't include any
application-specific keyword highlighting, etc.
The structural, expression, and template languages are separated into
different grammar definitions so that they can be used independently, but
they embed each other as needed to complete the language.
This is just a first pass, really. There are probably some bugs here, and
also some missing features.
