Commit Graph

310 Commits

Author SHA1 Message Date
Nicholas Jackson
23fc060132 gohcl: allow optional attributes to be specified via struct tag
Previously we required optional attributes to be specified as pointers so that we could represent the empty vs. absent distinction.

For applications that don't need to make that distinction, representing "optional" as a struct tag is more convenient.
2018-02-17 10:36:04 -08:00
Martin Atkins
eea3a14a71 hcl/json: allow more flexible use of arrays when describing bodies
Previously we allowed arrays only at the "leaf" of a set of objects
describing a block and its labels. This is not sufficient because it is
therefore impossible to preserve the relative ordering of a sequence
of blocks that have different block types or labels.

The spec now allows arrays of objects to be used in place of single
objects when that value is representing either an HCL body or a set of
labels on a nested block. This relaxing does not apply to JSON objects
interpreted as expressions or bodies interpreted in dynamic attributes
mode, since there is no requirement to preserve attribute ordering or
support duplicate property names in those scenarios.

This new model imposes additional constraints on the underlying JSON
parser used to interpret JSON HCL: it must now be able to retain the
relative ordering of object keys and accept multiple definitions of the
same key. This requirement is not imposed on _producers_, which are free
to use the allowance for arrays of objects to force ordering and duplicate
keys with JSON-producing libraries that are unable to make these
distinctions.

Since we are now requiring a specialized parser anyway, we also require
that it be able to represent numbers at full precision, whereas before
we made some allowances for implementations to not support this.
2018-02-17 10:26:58 -08:00
Martin Atkins
77dc2cba20 hcl/json: fuzzing utilities 2018-02-16 21:18:25 -08:00
Martin Atkins
f87a794800 hclsyntax: check for and report incorrect peeker stack discipline
The peeker has an "include newlines" stack which the parser manipulates
to switch between the newline-sensitive and non-sensitive scanning modes.
If the parser code fails to manage this stack correctly (for example,
due to a missed call to PopIncludeNewlines) then this causes very
confusing downstream errors that are otherwise difficult to debug.

As an extra debug tool for when errors _are_ detected, when this problem
is encountered during tests we are able to produce a visualization of the
pushes and pops to help the test developer see which pushes and pops
seem out of place.

This is a lot of ugly extra code but it's usually disabled and seems worth
it to allow us to catch quickly bugs that would otherwise be quite
difficult to diagnose.
2018-02-16 17:37:22 -08:00
Martin Atkins
9dfc220a4b hclsyntax: index expression parsing properly manages "include newlines"
Previously it was mismanaging the stack by first pushing on "false" and
then trying to undo that by pushing on "true". Instead, it should just
pop off the "false" to return to whatever the previous setting was, since
indexing brackets might already be inside a no-newlines context.
2018-02-16 16:45:42 -08:00
Martin Atkins
9301cd2ad5 hclsyntax: use go-test/deep for comparing parse test results
We were previously using an ugly combination of "pretty" and "spew" to
do this, which never really quite worked because of limitations in each
of those.

deep.Equal doesn't produce quite as much detailed information as the
others, but it has the advantage of showing exactly where a difference
exists rather than forcing us to hunt through a noisy diff to find it.
2018-02-16 16:44:03 -08:00
Martin Atkins
5ca9713bf0 hclsyntax: prevent ragel line comments becoming package docs 2018-02-04 19:01:48 -08:00
Martin Atkins
cfd802163b hclsyntax: rewrite string literal decoder with ragel
Fuzz testing revealed that there were a few different crashers in the
string literal decoder, which was previously a rather-unweildy
hand-written scanner with manually-implemented lookahead.

Rather than continuing to hand-tweak that code, here instead we use
ragel (which we were already using for the main scanner anyway) to
partition our string literals into tokens that are easier for our
decoder to wrangle.

As a bonus, this also makes our source ranges in our diagnostics more
accurate.
2018-02-04 19:01:48 -08:00
Martin Atkins
93a7008e3d hclsyntax: helpers for fuzz testing with go-fuzz 2018-02-04 18:55:25 -08:00
Martin Atkins
18a92d222b ext/userfunc: use bare identifiers for param names
Now that we have the necessary functions to deal with this in the
low-level HCL API, it's more intuitive to use bare identifiers for these
parameter names. This reinforces the idea that they are symbols being
defined rather than arbitrary string expressions.
2018-02-04 11:20:42 -08:00
Martin Atkins
2ddf8b4b8c cmd/hcldec: allow spec file to define variables and functions
The spec file can now additionally define default variables and functions
for the eval context used to evaluate the input file.
2018-02-04 11:05:23 -08:00
Martin Atkins
6c3ae68a0e cmd/hcldec: make cty stdlib functions available to specs
In a few specific portions of the spec format it's convenient to have
access to some of the functions defined in the cty stdlib. Here we allow
them to be used when constructing the value for a "literal" spec and in
the result expression for a "transform" spec.
2018-02-04 10:33:35 -08:00
Martin Atkins
1ba92ee170 cmd/hcldec: "transform" spec type
This new spec type allows evaluating an arbitrary expression on the
result of a nested spec, for situations where the a value must be
transformed in some way.
2018-02-04 09:59:20 -08:00
Martin Atkins
f65a097d17 cmd/hcldec: decode "array" blocks
These were missed on the previous pass, causing a disagreement with the
documentation.
2018-02-04 09:45:28 -08:00
Martin Atkins
ee147d9ee6 cmd/hcldec: Command-line tool for converting HCL config to JSON
This is essentially a CLI wrapper around the hcldec package, accepting a
decoding specification via a HCL-based language and using it to translate
input HCL files into JSON values while performing basic structural and
type validation of the input files.
2018-02-03 15:37:11 -08:00
Martin Atkins
102e698035 hcl: ExprAsKeyword function
A common pattern is emerging in calling applications of using single-item
absolute traversals to give the impression of static language keywords.
This new function makes that explicitly possible and allows a convenient
pattern for doing so that should improve the readability of a calling
application making use of it.
2018-02-03 08:55:50 -08:00
Martin Atkins
9f91684a1f hclsyntax: ValidIdentifier function
Calling applications often need to validate strings provided by the user
that will eventually be variable or attribute names in the evaluation
scope, to ensure that they will be evaluable.

Rather than having each application specify its own different subset of
the full set we support (which is derived from Unicode specifications),
we provide a simple function to let callers easily check the validity
of a potential identifier using exactly the same scanning rules we use
within the expression scanner.

To achieve this we actually invoke the scanner and then assert on its
result, which is a pretty expensive way to just check one string but it's
easy to do with code we already have in place and we don't expect this
sort of validation to be going on in a tight loop.
2018-02-02 08:09:40 -08:00
Martin Atkins
f70b6b00c8 hclwrite: update remaining stale references to "zcl" 2018-01-27 11:03:05 -08:00
Martin Atkins
fc404031a0 hcltest: update a remaining stale reference to "zcl" 2018-01-27 11:03:05 -08:00
Martin Atkins
076aa5aafc hcl: update stale references to "zcl" 2018-01-27 11:03:05 -08:00
Martin Atkins
305d5f96d1 json: update stale references to "zcl" 2018-01-27 11:03:05 -08:00
Martin Atkins
5a7ff3bca2 hclsyntax: rename our ragel scanner to "hcltok"
It was previously called "zcltok" after the prototype implementation that
HCL was forked from.
2018-01-27 11:03:05 -08:00
Martin Atkins
385f330d4f hclsyntax: update stale references to "zcl" in comments 2018-01-27 11:03:05 -08:00
Martin Atkins
1f3c1e1b14 gohcl: update remaining reference to "zcl" 2018-01-27 11:03:05 -08:00
Martin Atkins
b1a8ce3a09 ext/userfunc: update stale references to "zcl" 2018-01-27 11:03:05 -08:00
Martin Atkins
19d9927d4b ext/include: update stale reference to "zcl" 2018-01-27 11:03:05 -08:00
Martin Atkins
a59f60c1db hclfmt: Have this tool call itself by the correct name
Its usage output was still reporting "zclfmt", which is a remnant of the
experimental project that HCL was forked from.
2018-01-27 11:03:05 -08:00
Martin Atkins
22bc7a98cb
README: Clarify that HCL is just syntax
The readme was previously unclear about the fact that HCL is not a configuration language in itself but rather a toolkit for defining and parsing configuration languages.

It may still not be totally clear, but it is hopefully clearer than it was.
2018-01-27 10:28:15 -08:00
Martin Atkins
a1c55afeca hclsyntax: \uxxxx and \Uxxxxxxxx escape sequences in string literals
These allow the inclusion of arbitrary unicode codepoints (always encoded
as UTF-8) using a hex representation.

\u expects four digits and can thus represent only characters in the basic
multilingual plane.

\U expects eight digits and can thus represent all unicode characters,
at the cost of being extra-verbose.

Since our parser properly accounts for unicode characters (including
combining sequences) it's recommended to include them literally (UTF-8
encoded) in source code, but these sequences are useful for explicitly
representing non-printable characters that could otherwise appear
invisible in source code, such as zero-width modifier characters.

This fixes #6.
2018-01-27 10:20:22 -08:00
Martin Atkins
f0bf2b15ae hclsyntax: permit tabs and treat them like spaces
We inherited a restriction from an early zcl prototype here, but it's
far too strict to prohibit tabs entirely and so we'll accept them and
just treat them as spaces for column-counting purposes.

Tabs are still not _advised_, since they add extra complexity for problems
like generating annotated source code snippets (can't necessarily know
how large the tab stop is going to be) or doing surgical updates to
existing source files. The canonical formatting applied by hclwrite's
Format function will still eliminate all tabs, imposing the canonical
style of two spaces per indent level.

This fixes #2.
2018-01-27 09:30:36 -08:00
Martin Atkins
88bf362f0c hclsyntax: update generators to use HCL package path
Since these have Go source code embedded in strings, they were not found
during the original big zcl to HCL rename.
2018-01-27 09:26:56 -08:00
Martin Atkins
678f7e6781 json: remove non-functional HIL parsing functions
An earlier iteration of this package was able to optionally use HIL as
its expression engine in place of the hclsyntax expression parser, but
this has since been removed and so this flag no longer has any effect.

Consequently, the public functions ParseWithHIL and ParseFileWithHIL were,
in fact, just using the zclsyntax parser and thus behaving identically to
the Parse and ParseFile functions.
2018-01-27 09:15:53 -08:00
Martin Atkins
34e27c038a hcl: UnwrapExpression and UnwrapExpressionUntil
A pattern has emerged of wrapping Expression instances with other
Expressions in order to subtly modify their behavior. A key example of
this is in ext/dynblock, where wrap an expression in order to introduce
our additional iteration variable for expressions in dynamic blocks.

Rather than having each wrapper expression implement wrapping
implementations for our various syntax-level-analysis functions (like
ExprList and AbsTraversalForExpr), instead we define a standard mechanism
to unwrap expressions back to the lowest-level object -- usually an AST
node -- and then use this in all of our analyses that look at the
expression's structure rather than its value.
2018-01-27 09:10:18 -08:00
Martin Atkins
60b539d5d7 integrationtest: include dynblock usage in the "terraformlike" test
Terraform is the prime use-case for the dynblock extension, so we'll
include this here currently as a proof-of-concept for Terraform's usage,
but eventually (once Terraform is actually using it) this'll give some
insurance that it doesn't get broken.
2018-01-27 09:10:18 -08:00
Martin Atkins
d6fc633aa0 ext/dynblock: ForEachVariablesHCLDec helper
For applications already using hcldec, a decoder specification can be used
to automatically drive the recursive variable detection walk that begins
with WalkForEachVariables, allowing all "for_each" and "labels" variables
in a recursive block structure to be detected in a single call.
2018-01-27 09:10:18 -08:00
Martin Atkins
130b3c5105 hcldec: New function ChildBlockTypes
This function returns a map describing all of the child block types
declared inside a spec. This can be used for recursive decoding of bodies
using the low-level HCL API, though in most cases callers should just use
Decode which does recursive decoding of an entire nested structure in
a single call.
2018-01-27 09:10:18 -08:00
Martin Atkins
45c6cc83f0 ext/dynblock: A more arduous way to find variables required to expand
The previous ForEachVariables method was flawed because it didn't have
enough information to properly analyze child blocks. Since the core HCL
API requires a schema for any body analysis, and since a schema only
describes one level of configuration structure at a time, we must require
callers to drive a recursive walk through their nested block structure so
that the correct schema can be provided at each level.

This API is rather more complex than is ideal, but is the best we can do
with the HCL Body API as currently defined, and it's currently defined
that way in order to properly support ambiguous syntaxes like JSON.
2018-01-27 09:10:18 -08:00
Martin Atkins
da95646a33 ext/dynblock: dynamic blocks extension
This extension allows an application to support dynamic generation of
child blocks based on expressions in certain contexts. This is done using
a new block type called "dynamic", which contains an iteration value
(which must be a collection) and a specification of how to construct a
child block for each element of that collection.
2018-01-27 09:10:18 -08:00
Fatih Arslan
f87600a7d9 update zclsyntax to hclsyntax in various comments and strings
These are remnants of the project HCL was forked from.
2018-01-23 21:54:38 -08:00
Martin Atkins
613331e829 hcltest: new implementations for ExprList and AbsTraversalForExpr
To assist in testing code that depends on hcl.ExprList and
hcl.AbsTraversalForExpr we now implement the necessary interfaces on our
existing MockExprLiteral and MockExprVariable, as well as adding new
functions MockExprList and MockExprTraversal that more directly serve
those interfaces with full functionality.
2018-01-20 11:20:50 -08:00
Martin Atkins
83451bb547 hcl/hclsyntax: correctly handle %{ sequence escapes
In early prototyping the template control sequence introducer was
specified as !{, but that changed to %{ along the way because it seemed
more intuitive and less likely to collide with literal strings.

However, the parser's string literal handling still had remnants of the
old syntax, causing strange quirks in parsing strings that contained
exclamation points.

Now we correctly expect %{ as the control sequence introducer, %%{ as its
escape sequence, and additionally fix a bug where previously template
sequence introduction characters at the end of a string literal would
be silently dropped due to them representing an unterminated escape
sequence.

This fixes #3.
2018-01-19 08:11:25 -08:00
Martin Atkins
0daeda39ff hcl/hclsyntax: return hcl.TraverseIndex, not pointer to one
Traversals are always passed by value, so returning a pointer here is
inconsistent with how hcl.TraverseIndex is used elsewhere and thus makes
life inconvenient for callers making type assertions.
2018-01-18 08:12:05 -08:00
Martin Atkins
883a81b490 hcl: highlight the subject when printing a diagnostic source snippet
In complex expressions it can be hard to determine which portion is
relevant when we print a diagnostic message. To address this, when color
is enabled we bold and underline the "subject" portion of the source code,
which then makes it stand out within the full lines of code we print
in the snippet.
2018-01-14 12:25:35 -08:00
Martin Atkins
14cfe59a52 hcl: Simplify the text DiagnosticWriter using new Range functions
Now that we have helper methods for computing relationships between
ranges, we can eliminate all of the tricky line-counting and byte-counting
code here and instead use the higher-level operations.

The result is a single loop using the RangeScanner.
2018-01-14 12:07:33 -08:00
Martin Atkins
368a3f81c0 hcl: SourceRange.PartitionAround
This is a convenience wrapper around SourceRange.Overlap that also
calculates the ranges in the receiver that _aren't_ overlapping with the
given range.

This is useful when, for example, partitioning a portion of source code
to insert markers to highlight the location of an error, as we do when
printing code snippets as part of diagnostic output.
2018-01-14 11:51:05 -08:00
Martin Atkins
1365a2cfe5 hcl: RangeOver function
This is a generalization of RangeBetween that finds a single range that
covers the full extent of both given ranges, possibly also including some
additional content between the ranges if they do not overlap.
2018-01-14 11:33:23 -08:00
Martin Atkins
d34d4686fb hcl: RangeScanner helper
RangeScanner has an interface similar to bufio.Scanner for partitioning
a buffer into tokens, but it returns the hcl.Range of each token along
with that token so that the caller can see where the token fits in
relation to the entire source file.

The main intended use-case for this is to partition a source file into
lines for the purpose of printing a source code snippet in diagnostic
output. Having the source location information is important in that case
to recognize which lines belong to the subject and context of each
diagnostic.
2018-01-14 11:24:19 -08:00
Martin Atkins
11e4972f13 hcl: Helper methods for detecting overlaps in ranges
This is useful, for example, when printing source snippets to the terminal
as part of diagnostics, in order to detect the portion of the source code
that coincides with the subject or context of each diagnostic.
2018-01-14 10:08:39 -08:00
Martin Atkins
600e8726ec hcl: Export the source range for a Traversal
This can be useful, for example, when using Expression.Variables to
pre-validate all of the referenced variables before evaluation, so that
the traversal source ranges can be included in any generated diagnostics.
2018-01-13 23:01:11 -08:00
Martin Atkins
f3af67f344 integrationtest: exercise some features Terraform uses
As an extra level of confidence in addition to the unit tests, this
integration test verifies that a certain set of features that Terraform
uses are able to work properly together.

Terraform is used as an example here just because it's a more advanced
consumer of HCL and thus it exercises some codepaths that most
applications don't need, such as ExprList and AbsTraversalForExpr.
2018-01-13 00:58:50 -08:00