Lately we've made some changes that have affected the behavior of the
specsuite tests, causing them to fail. Much of this was caused by changes
to the harness itself (based on hcldec), although one break in particular
here was an intentional change to the implementation of modulo in
upstream cty to make it produce a more accurate result when used with
a fractional divisor.
When we did the automatic rewriting from the hashicorp/hcl2 import paths
to hashicorp/hcl/v2 import paths, our automatic rewrite script did not
run "go fmt" afterwards and so left some imports misordered as a result
of the slightly-differently-shaped directory structure in the new
repository.
We were taking a pointer to a for loop iterator variable and thus
capturing the final iteration value rather than each one separately. By
using the .Ptr() method instead, we force a copy of the range which we
then take a pointer to.
Although the spec testsuite and associated harness is designed to be
usable by other implementations of HCL not written in Go, it's convenient
to run it as part of our own "go test" test suite here so there isn't
an additional thing to run on each change.
To achieve this, the new package hcl/spectests will build both hcldec and
hclspecsuite from latest source and then run the latter to execute the
test suite, capturing the output and converting it (sloppily) into
testing.T method calls to produce something vaguely reasonable.
Other than the small amount of "parsing" to make it look in the output
like a normal Go test, there's nothing special going on here and so it's
still valid to run the spec suite manually with a build of hcldec from
this codebase, which should produce the same result.
When a test file declares one or more expected diagnostics, we check those
instead of checking the result value. The severities and source ranges
must match.
We don't test the error messages themselves because they are not part of
the specification and may vary between implementations or, in future, be
translated into other languages.
The harness can now run tests that decode successfully and compare the
result with a given value. Further work is required in later commits to
deal with other cases, such as tests that intentionally produce errors.
