go/hcl
2
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

guide: A section on dynamic decoding using hcldec

Browse Source
This commit is contained in:
Martin Atkins 2018-08-26 10:03:28 -07:00
parent 3b190b826e
commit b737ad87a3
3 changed files with 245 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
guide/go.rst
View File

@ -23,5 +23,6 @@ a Go program.
go_parsing
go_diagnostics
go_decoding_gohcl
go_decoding_hcldec
.. _Go: https://golang.org/

4
guide/go_decoding_gohcl.rst
View File

@ -1,5 +1,7 @@
.. go:package:: gohcl
.. _go-decoding-gohcl:
Decoding Into Native Go Values
==============================
@ -53,6 +55,8 @@ The sections below discuss some additional decoding use-cases. For full details
on the `gohcl` package, see
`the godoc reference <https://godoc.org/github.com/hashicorp/hcl2/gohcl>`_.
.. _go-decoding-gohcl-evalcontext:
Variables and Functions
-----------------------

240
guide/go_decoding_hcldec.rst Normal file
View File

@ -0,0 +1,240 @@
.. go:package:: hcldec
.. _go-decoding-hcldec:
Decoding With Dynamic Schema
============================
In section :ref:`go-decoding-gohcl`, we saw the most straightforward way to
access the content from an HCL file, decoding directly into a Go value whose
type is known at application compile time.
For some applications, it is not possible to know the schema of the entire
configuration when the application is built. For example, `HashiCorp Terraform`_
uses HCL as the foundation of its configuration language, but parts of the
configuration are handled by plugins loaded dynamically at runtime, and so
the schemas for these portions cannot be encoded directly in the Terraform
source code.
HCL's ``hcldec`` package offers a different approach to decoding that allows
schemas to be created at runtime, and the result to be decoded into
dynamically-typed data structures.
The sections below are an overview of the main parts of package ``hcldec``.
For full details, see
`the package godoc <https://godoc.org/github.com/hashicorp/hcl2/hcldec>`_.
.. _`HashiCorp Terraform`: https://www.terraform.io/
Decoder Specification
---------------------
Whereas :go:pkg:`gohcl` infers the expected schema by using reflection against
the given value, ``hcldec`` obtains schema through a decoding *specification*,
which is a set of instructions for mapping HCL constructs onto a dynamic
data structure.
The ``hcldec`` package contains a number of different specifications, each
implementing :go:type:`hcldec.Spec` and having a ``Spec`` suffix on its name.
Each spec has two distinct functions:
* Adding zero or more validation constraints on the input configuration file.
* Producing a result value based on some elements from the input file.
The most common pattern is for the top-level spec to be a
:go:type:`hcldec.ObjectSpec` with nested specifications defining either blocks
or attributes, depending on whether the configuration file will be
block-structured or flat.
.. code-block:: go
spec := hcldec.ObjectSpec{
"io_mode": &hcldec.AttrSpec{
Name: "io_mode",
Type: cty.String,
},
"services": &hcldec.BlockMapSpec{
TypeName: "service",
LabelNames: []string{"type", "name"},
Nested: hcldec.ObjectSpec{
"listen_addr": &hcldec.AttrSpec{
Name: "listen_addr",
Type: cty.String,
Required: true,
},
"processes": &hcldec.BlockMapSpec{
TypeName: "service",
LabelNames: []string{"name"},
Nested: hcldec.ObjectSpec{
"command": &hcldec.AttrSpec{
Name: "command",
Type: cty.List(cty.String),
Required: true,
},
},
},
},
},
}
val, moreDiags := hcldec.Decode(f.Body, spec, nil)
diags = append(diags, moreDiags...)
The above specification expects a configuration shaped like our example in
:ref:`intro`, and calls for it to be decoded into a dynamic data structure
that would have the following shape if serialized to JSON:
.. code-block:: JSON
{
"io_mode": "async",
"services": {
"http": {
"web_proxy": {
"listen_addr": "127.0.0.1:8080",
"processes": {
"main": {
"command": ["/usr/local/bin/awesome-app", "server"]
},
"mgmt": {
"command": ["/usr/local/bin/awesome-app", "mgmt"]
}
}
}
}
}
}
Types and Values With ``cty``
-----------------------------
HCL's expression interpreter is implemented in terms of another library called
:go:pkg:`cty`, which provides a type system which HCL builds on and a robust
representation of dynamic values in that type system. You could think of
:go:pkg:`cty` as being a bit like Go's own :go:pkg:`reflect`, but for the
results of HCL expressions rather than Go programs.
The full details of this system can be found in
`its own repository <https://github.com/zclconf/go-cty>`_, but this section
will cover the most important highlights, because ``hcldec`` specifications
include :go:pkg:`cty` types (as seen in the above example) and its results are
:go:pkg:`cty` values.
``hcldec`` works directly with :go:pkg:`cty` — as opposed to converting values
directly into Go native types — because the functionality of the :go:pkg:`cty`
packages then allows further processing of those values without any loss of
fidelity or range. For example, :go:pkg:`cty` defines a JSON encoding of its
values that can be decoded losslessly as long as both sides agree on the value
type that is expected, which is a useful capability in systems where some sort
of RPC barrier separates the main program from its plugins.
Types are instances of :go:type:`cty.Type`, and are constructed from functions
and variables in :go:pkg:`cty` as shown in the above example, where the string
attributes are typed as ``cty.String``, which is a primitive type, and the list
attribute is typed as ``cty.List(cty.String)``, which constructs a new list
type with string elements.
Values are instances of :go:type:`cty.Value`, and can also be constructed from
functions in :go:pkg:`cty`, using the functions that include ``Val`` in their
names or using the operation methods available on :go:type:`cty.Value`.
In most cases you will eventually want to use the resulting data as native Go
types, to pass it to non-:go:pkg:`cty`-aware code. To do this, see the guides
on
`Converting between types <https://github.com/zclconf/go-cty/blob/master/docs/convert.md>`_
(staying within :go:pkg:`cty`) and
`Converting to and from native Go values <https://github.com/zclconf/go-cty/blob/master/docs/gocty.md>`_.
Partial Decoding
----------------
Because the ``hcldec`` result is always a value, the input is always entirely
processed in a single call, unlike with :go:pkg:`gohcl`.
However, both :go:pkg:`gohcl` and :go:pkg:`hcldec` take :go:type:`hcl.Body` as
the representation of input, and so it is possible and common to mix them both
in the same program.
A common situation is that :go:pkg:`gohcl` is used in the main program to
decode the top level of configuration, which then allows the main program to
determine which plugins need to be loaded to process the leaf portions of
configuration. In this case, the portions that will be interpreted by plugins
are retained as opaque :go:type:`hcl.Body` until the plugins have been loaded,
and then each plugin provides its :go:type:`hcldec.Spec` to allow decoding the
plugin-specific configuration into a :go:type:`cty.Value` which be
transmitted to the plugin for further processing.
In our example from :ref:`intro`, perhaps each of the different service types
is managed by a plugin, and so the main program would decode the block headers
to learn which plugins are needed, but process the block bodies dynamically:
.. code-block:: go
type ServiceConfig struct {
Type string `hcl:"type,label"`
Name string `hcl:"name,label"`
PluginConfig hcl.Body `hcl:",remain"`
}
type Config struct {
IOMode string `hcl:"io_mode"`
Services []ServiceConfig `hcl:"service,block"`
}
var c Config
moreDiags := gohcl.DecodeBody(f.Body, nil, &c)
diags = append(diags, moreDiags...)
if moreDiags.HasErrors() {
// (show diags in the UI)
return
}
for _, sc := range c.Services {
pluginName := block.Type
// Totally-hypothetical plugin manager (not part of HCL)
plugin, err := pluginMgr.GetPlugin(pluginName)
if err != nil {
diags = diags.Append(&hcl.Diagnostic{ /* ... */ })
continue
}
spec := plugin.ConfigSpec() // returns hcldec.Spec
// Decode the block body using the plugin's given specification
configVal, moreDiags := hcldec.Decode(sc.PluginConfig, spec, nil)
diags = append(diags, moreDiags...)
if moreDiags.HasErrors() {
continue
}
// Again, hypothetical API within your application itself, and not
// part of HCL. Perhaps plugin system serializes configVal as JSON
// and sends it over to the plugin.
svc := plugin.NewService(configVal)
serviceMgr.AddService(sc.Name, svc)
}
Variables and Functions
-----------------------
The final argument to ``hcldec.Decode`` is an expression evaluation context,
just as with ``gohcl.DecodeBlock``.
This object can be constructed using
:ref:`the gohcl helper function <go-decoding-gohcl-evalcontext>` as before if desired, but
you can also choose to work directly with :go:type:`hcl.EvalContext` as
discussed in :ref:`expression-eval`:
.. code-block:: go
ctx := &hcl.EvalContext{
Variables: map[string]cty.Value{
"pid": cty.NumberIntVal(int64(os.Getpid())),
},
}
val, moreDiags := hcldec.Decode(f.Body, spec, ctx)
diags = append(diags, moreDiags...)
As you can see, this lower-level API also uses :go:pkg:`cty`, so it can be
particularly convenient in situations where the result of dynamically decoding
one block must be available to expressions in another block.