This is essentially a CLI wrapper around the hcldec package, accepting a
decoding specification via a HCL-based language and using it to translate
input HCL files into JSON values while performing basic structural and
type validation of the input files.
A common pattern is emerging in calling applications of using single-item
absolute traversals to give the impression of static language keywords.
This new function makes that explicitly possible and allows a convenient
pattern for doing so that should improve the readability of a calling
application making use of it.
Calling applications often need to validate strings provided by the user
that will eventually be variable or attribute names in the evaluation
scope, to ensure that they will be evaluable.
Rather than having each application specify its own different subset of
the full set we support (which is derived from Unicode specifications),
we provide a simple function to let callers easily check the validity
of a potential identifier using exactly the same scanning rules we use
within the expression scanner.
To achieve this we actually invoke the scanner and then assert on its
result, which is a pretty expensive way to just check one string but it's
easy to do with code we already have in place and we don't expect this
sort of validation to be going on in a tight loop.
The readme was previously unclear about the fact that HCL is not a configuration language in itself but rather a toolkit for defining and parsing configuration languages.
It may still not be totally clear, but it is hopefully clearer than it was.
These allow the inclusion of arbitrary unicode codepoints (always encoded
as UTF-8) using a hex representation.
\u expects four digits and can thus represent only characters in the basic
multilingual plane.
\U expects eight digits and can thus represent all unicode characters,
at the cost of being extra-verbose.
Since our parser properly accounts for unicode characters (including
combining sequences) it's recommended to include them literally (UTF-8
encoded) in source code, but these sequences are useful for explicitly
representing non-printable characters that could otherwise appear
invisible in source code, such as zero-width modifier characters.
This fixes#6.
We inherited a restriction from an early zcl prototype here, but it's
far too strict to prohibit tabs entirely and so we'll accept them and
just treat them as spaces for column-counting purposes.
Tabs are still not _advised_, since they add extra complexity for problems
like generating annotated source code snippets (can't necessarily know
how large the tab stop is going to be) or doing surgical updates to
existing source files. The canonical formatting applied by hclwrite's
Format function will still eliminate all tabs, imposing the canonical
style of two spaces per indent level.
This fixes#2.
An earlier iteration of this package was able to optionally use HIL as
its expression engine in place of the hclsyntax expression parser, but
this has since been removed and so this flag no longer has any effect.
Consequently, the public functions ParseWithHIL and ParseFileWithHIL were,
in fact, just using the zclsyntax parser and thus behaving identically to
the Parse and ParseFile functions.
A pattern has emerged of wrapping Expression instances with other
Expressions in order to subtly modify their behavior. A key example of
this is in ext/dynblock, where wrap an expression in order to introduce
our additional iteration variable for expressions in dynamic blocks.
Rather than having each wrapper expression implement wrapping
implementations for our various syntax-level-analysis functions (like
ExprList and AbsTraversalForExpr), instead we define a standard mechanism
to unwrap expressions back to the lowest-level object -- usually an AST
node -- and then use this in all of our analyses that look at the
expression's structure rather than its value.
Terraform is the prime use-case for the dynblock extension, so we'll
include this here currently as a proof-of-concept for Terraform's usage,
but eventually (once Terraform is actually using it) this'll give some
insurance that it doesn't get broken.
For applications already using hcldec, a decoder specification can be used
to automatically drive the recursive variable detection walk that begins
with WalkForEachVariables, allowing all "for_each" and "labels" variables
in a recursive block structure to be detected in a single call.
This function returns a map describing all of the child block types
declared inside a spec. This can be used for recursive decoding of bodies
using the low-level HCL API, though in most cases callers should just use
Decode which does recursive decoding of an entire nested structure in
a single call.
The previous ForEachVariables method was flawed because it didn't have
enough information to properly analyze child blocks. Since the core HCL
API requires a schema for any body analysis, and since a schema only
describes one level of configuration structure at a time, we must require
callers to drive a recursive walk through their nested block structure so
that the correct schema can be provided at each level.
This API is rather more complex than is ideal, but is the best we can do
with the HCL Body API as currently defined, and it's currently defined
that way in order to properly support ambiguous syntaxes like JSON.
This extension allows an application to support dynamic generation of
child blocks based on expressions in certain contexts. This is done using
a new block type called "dynamic", which contains an iteration value
(which must be a collection) and a specification of how to construct a
child block for each element of that collection.
To assist in testing code that depends on hcl.ExprList and
hcl.AbsTraversalForExpr we now implement the necessary interfaces on our
existing MockExprLiteral and MockExprVariable, as well as adding new
functions MockExprList and MockExprTraversal that more directly serve
those interfaces with full functionality.
In early prototyping the template control sequence introducer was
specified as !{, but that changed to %{ along the way because it seemed
more intuitive and less likely to collide with literal strings.
However, the parser's string literal handling still had remnants of the
old syntax, causing strange quirks in parsing strings that contained
exclamation points.
Now we correctly expect %{ as the control sequence introducer, %%{ as its
escape sequence, and additionally fix a bug where previously template
sequence introduction characters at the end of a string literal would
be silently dropped due to them representing an unterminated escape
sequence.
This fixes#3.
Traversals are always passed by value, so returning a pointer here is
inconsistent with how hcl.TraverseIndex is used elsewhere and thus makes
life inconvenient for callers making type assertions.
In complex expressions it can be hard to determine which portion is
relevant when we print a diagnostic message. To address this, when color
is enabled we bold and underline the "subject" portion of the source code,
which then makes it stand out within the full lines of code we print
in the snippet.
Now that we have helper methods for computing relationships between
ranges, we can eliminate all of the tricky line-counting and byte-counting
code here and instead use the higher-level operations.
The result is a single loop using the RangeScanner.
This is a convenience wrapper around SourceRange.Overlap that also
calculates the ranges in the receiver that _aren't_ overlapping with the
given range.
This is useful when, for example, partitioning a portion of source code
to insert markers to highlight the location of an error, as we do when
printing code snippets as part of diagnostic output.
This is a generalization of RangeBetween that finds a single range that
covers the full extent of both given ranges, possibly also including some
additional content between the ranges if they do not overlap.
RangeScanner has an interface similar to bufio.Scanner for partitioning
a buffer into tokens, but it returns the hcl.Range of each token along
with that token so that the caller can see where the token fits in
relation to the entire source file.
The main intended use-case for this is to partition a source file into
lines for the purpose of printing a source code snippet in diagnostic
output. Having the source location information is important in that case
to recognize which lines belong to the subject and context of each
diagnostic.
This is useful, for example, when printing source snippets to the terminal
as part of diagnostics, in order to detect the portion of the source code
that coincides with the subject or context of each diagnostic.
This can be useful, for example, when using Expression.Variables to
pre-validate all of the referenced variables before evaluation, so that
the traversal source ranges can be included in any generated diagnostics.
As an extra level of confidence in addition to the unit tests, this
integration test verifies that a certain set of features that Terraform
uses are able to work properly together.
Terraform is used as an example here just because it's a more advanced
consumer of HCL and thus it exercises some codepaths that most
applications don't need, such as ExprList and AbsTraversalForExpr.
This helper allows a calling application to require a given expression be
some sort of list constructor (tuple constructor in native syntax, or
array in JSON) and peel off that outer level of list to obtain a slice
of the Expression objects inside.
This is useful in rare cases where the calling application needs to
extract the expressions within the list without evaluating the entire list
expression first. For example, the expressions that result from this
function might be passed into AbsTraversalForExpr in situations where the
caller requires a static list of static traversals that will never
actually be evaluated.
These functions permit a calling application to recognize when an
expression represents a static absolute traversal and obtain that
traversal. This allows for the unusual-but-valid case where an application
wishes to access the expression source rather than its resulting value,
when the expression source is something that can be understood as a
traversal.
An example use-case is an attribute that takes a list of other attributes
it depends on, expressed as traversals. In this case the calling
application needs to access the attribute names themselves rather than
their values, e.g. to build some sort of dependency graph to gradually
populate the scope for evaluation.
This function returns the type of value that should be returned when
decoding the given spec. As well as being generally useful to the caller
for book-keeping purposes, this also allows us to return correct type
information when we are returning null and empty values, where before we
were leaning a little too much on cty.DynamicPseudoType.
A BlockLabelSpec can be placed in the nested spec structure of one of the
block specs to require and obtain labels on that block.
This is a more generic methodology than BlockMapSpec since it allows the
result to be a list or set with the labels inside the values, rather than
forcing all the label tuples to be unique and losing the ordering by
collapsing into a map structure.
