Lately we've made some changes that have affected the behavior of the
specsuite tests, causing them to fail. Much of this was caused by changes
to the harness itself (based on hcldec), although one break in particular
here was an intentional change to the implementation of modulo in
upstream cty to make it produce a more accurate result when used with
a fractional divisor.
When we did the automatic rewriting from the hashicorp/hcl2 import paths
to hashicorp/hcl/v2 import paths, our automatic rewrite script did not
run "go fmt" afterwards and so left some imports misordered as a result
of the slightly-differently-shaped directory structure in the new
repository.
We remove properties whose values are null by default in order to produce
output that is more convenient to consume in the common case.
However, sometimes those nulls are significant, so we'll allow the user
to opt in to retaining them, at the expense of producing a result that is
more noisy if the spec contains lots of optional attributes that are not
set.
When I tried a spec with quoted types, I got this error:
`A type is required, not string.`
Later in the document, the types don’t have quotes, so I figured that’s what’s expected.
We were taking a pointer to a for loop iterator variable and thus
capturing the final iteration value rather than each one separately. By
using the .Ptr() method instead, we force a copy of the range which we
then take a pointer to.
Our BlockList, BlockSet, and BlockMap specs all produce cty collection
values, which require all elements to have a homogeneous type. If the
nested spec contained an attribute of type cty.DynamicPseudoType, that
would create the risk of each element having a different type, which would
previously have caused decoding to panic.
Now we either handle this during decode (BlockList, BlockSet) or forbid
it outright (BlockMap) to prevent that crash. BlockMap could _potentially_
also handle this during decode, but that would require a more significant
reorganization of its implementation than I want to take on right now,
and decoding dynamically-typed values inside collections is an edge case
anyway.
Although the spec testsuite and associated harness is designed to be
usable by other implementations of HCL not written in Go, it's convenient
to run it as part of our own "go test" test suite here so there isn't
an additional thing to run on each change.
To achieve this, the new package hcl/spectests will build both hcldec and
hclspecsuite from latest source and then run the latter to execute the
test suite, capturing the output and converting it (sloppily) into
testing.T method calls to produce something vaguely reasonable.
Other than the small amount of "parsing" to make it look in the output
like a normal Go test, there's nothing special going on here and so it's
still valid to run the spec suite manually with a build of hcldec from
this codebase, which should produce the same result.
When a test file declares one or more expected diagnostics, we check those
instead of checking the result value. The severities and source ranges
must match.
We don't test the error messages themselves because they are not part of
the specification and may vary between implementations or, in future, be
translated into other languages.
The harness can now run tests that decode successfully and compare the
result with a given value. Further work is required in later commits to
deal with other cases, such as tests that intentionally produce errors.
By default we generate human-readable diagnostics on the assumption that
the caller is a simple program that is capturing stdin via a pipe and
letting stderr go to the terminal.
More sophisticated callers may wish to analyze the diagnostics themselves
and perhaps present them in a different way, such as via a GUI.
This option skips the usual decoding step and instead prints out a JSON-
formatted list of the variables that are referenced by the configuration.
In simple cases this is not required, but for more complex use-cases it
can be useful to first analyze the input to see which variables need to
be in the scope, then construct a suitable set of variables before finally
decoding the input. For example, some of the variable values may be
expensive to produce.
This is the hcldec interface to Body.JustAttributes, producing a map whose
keys are the child attribute names and whose values are the results of
evaluating those expressions.
We can't just expose a JustAttributes-style spec directly here because
it's not really compatible with how hcldec thinks about things, but we
can expose a spec that decodes a specific child block because that can
then compose properly with other specs at the same level without
interfering with their operation.
The primary use for this is to allow the use of the block syntax to define
a map:
dynamic_stuff {
foo = "bar"
}
JustAttributes is normally used in static analysis situations such as
enumerating the contents of a block to decide what to include in the
final EvalContext. That's not really possible with the hcldec model
because both structural decoding and expression evaluation happen
together. Therefore the use of this is pretty limited: it's useful if you
want to be compatible with an existing format based on legacy HCL where a
map was conventionally defined using block syntax, relying on the fact
that HCL did not make a strong distinction between attribute and block
syntax.
Now that we have the necessary functions to deal with this in the
low-level HCL API, it's more intuitive to use bare identifiers for these
parameter names. This reinforces the idea that they are symbols being
defined rather than arbitrary string expressions.
In a few specific portions of the spec format it's convenient to have
access to some of the functions defined in the cty stdlib. Here we allow
them to be used when constructing the value for a "literal" spec and in
the result expression for a "transform" spec.
This new spec type allows evaluating an arbitrary expression on the
result of a nested spec, for situations where the a value must be
transformed in some way.
This is essentially a CLI wrapper around the hcldec package, accepting a
decoding specification via a HCL-based language and using it to translate
input HCL files into JSON values while performing basic structural and
type validation of the input files.
This is a super-invasive update since the "zcl" package in particular
is referenced all over.
There are probably still a few zcl references hanging around in comments,
etc but this takes care of most of it.
The main "zcl" package requires a bit more care because of how many
callers it has and because of its two subpackages, so we'll take care
of that one separately.
This applies the simple native syntax reformatting function to one or
more files. It does not support JSON or any other syntax.
Calling applications might provide their own versions of this that e.g.
can format an entire directory by matching on filename patterns, but
this serves as an example and a utility for single files.
Martin Atkins
Masayuki Morita
Buck Doyle