HCL uses a number of upstream libraries that implement algorithms defined
in Unicode. This commit is updating those libraries all to versions that
have Unicode 13 support.
The main implication of this for HCL directly is that when it returns
column numbers in source locations it will count characters using the
Unicode 13 definition of "character", which includes various new
multi-codeunit characters added in Unicode 13.
These new version dependencies will also make Unicode 13 support available
for other functionality that HCL callers might use, such as the stdlib
functions in upstream cty, even though HCL itself does not directly use
those.
HCL uses grapheme cluster segmentation to produce accurate "column"
indications in diagnostic messages and other human-oriented source
location information. Each new major version of Unicode introduces new
codepoints, some of which are defined to combine with other codepoints to
produce a single visible character (grapheme cluster).
We were previously using the rules from Unicode 9.0.0. This change
switches to using the segmentation rules from Unicode 12.0.0, which is
the latest version at the time of this commit and is also the version of
Unicode used for other purposes by the Go 1.14 runtime.
HCL does not use text segmentation results for any purpose that would
affect the meaning of decoded data extracted from HCL files, so this
change will only affect the human-oriented source positions generated for
files containing characters that were newly-introduced in Unicode 10, 11,
or 12. (Machine-oriented uses of source location information are based on
byte offsets and not affected by text segmentation.)
Our normal ruleset thinks that the "in" keyword here is a variable
reference and so writes it as "in[y]". Since there's never any reason for
a variable to appear immediately after another variable, we can check
for a preceding identifier as a heuristic to recognize whether in is
probably being used as a keyword rather than as a variable.
This is not exact, but the only time this should be a false positive is
if there were a syntax error in the input, and we don't make any
guarantees about the result in that case anyway.
This fixes#52.
There isn't any strong reason for this -- they don't implement io.Reader
and so can't be used in places where a Reader+WriterTo is expected, like
io.Copy -- but go lint thinks that anything called WriteTo with an
io.Writer argument is an attempt to implement WriterTo and so this just
shuts up the linter.
For now, this is the only way to set an attribute, and so attributes can
only be set to literal values.
Later this will be generalized so that this is just a helper wrapper
around a "SetAttribute" method that just uses a given expression, which
then helps by constructing the expression from the value first.
The original prototype of hclwrite tried to track both the tokens and
the AST as two parallel data structures. This quickly exploded in
complexity, leading to lots of messy code to manage keeping those two
structures in sync.
This new approach melds the two structures together, creating first a
physical token tree (made of "node" objects, and hidden from the caller)
and then attaching the AST nodes to that token tree as additional sidecar
data.
The result is much easier to work with, leading to less code in the parser
and considerably less complex data structures in the parser's tests.
This commit is enough to reach feature parity with the previous prototype,
but it remains a prototype. With a more usable foundation, we'll evolve
this into a more complete implementation in subsequent commits.
This is a super-invasive update since the "zcl" package in particular
is referenced all over.
There are probably still a few zcl references hanging around in comments,
etc but this takes care of most of it.
The main "zcl" package requires a bit more care because of how many
callers it has and because of its two subpackages, so we'll take care
of that one separately.
Martin Atkins
Fatih Arslan