hcl/spec.md

HCL Syntax-Agnostic Information Model

This is the specification for the general information model (abstract types and semantics) for hcl. HCL is a system for defining configuration languages for applications. The HCL information model is designed to support multiple concrete syntaxes for configuration, each with a mapping to the model defined in this specification.

The two primary syntaxes intended for use in conjunction with this model are the HCL native syntax and the JSON syntax. In principle other syntaxes are possible as long as either their language model is sufficiently rich to express the concepts described in this specification or the language targets a well-defined subset of the specification.

Structural Elements

The primary structural element is the body, which is a container representing a set of zero or more attributes and a set of zero or more blocks.

A configuration file is the top-level object, and will usually be produced by reading a file from disk and parsing it as a particular syntax. A configuration file has its own body, representing the top-level attributes and blocks.

An attribute is a name and value pair associated with a body. Attribute names are unique within a given body. Attribute values are provided as expressions, which are discussed in detail in a later section.

A block is a nested structure that has a type name, zero or more string labels (e.g. identifiers), and a nested body.

Together the structural elements create a hierarchical data structure, with attributes intended to represent the direct properties of a particular object in the calling application, and blocks intended to represent child objects of a particular object.

Body Content

To support the expression of the HCL concepts in languages whose information model is a subset of HCL's, such as JSON, a body is an opaque container whose content can only be accessed by providing information on the expected structure of the content.

The specification for each syntax must describe how its physical constructs are mapped on to body content given a schema. For syntaxes that have first-class syntax distinguishing attributes and bodies this can be relatively straightforward, while more detailed mapping rules may be required in syntaxes where the representation of attributes vs. blocks is ambiguous.

Schema-driven Processing

Schema-driven processing is the primary way to access body content. A body schema is a description of what is expected within a particular body, which can then be used to extract the body content, which then provides access to the specific attributes and blocks requested.

A body schema consists of a list of attribute schemata and block header schemata:

• An attribute schema provides the name of an attribute and whether its presence is required.

• A block header schema provides a block type name and the semantic names assigned to each of the labels of that block type, if any.

Within a schema, it is an error to request the same attribute name twice or to request a block type whose name is also an attribute name. While this can in principle be supported in some syntaxes, in other syntaxes the attribute and block namespaces are combined and so an attribute cannot coexist with a block whose type name is identical to the attribute name.

The result of applying a body schema to a body is body content, which consists of an attribute map and a block sequence:

• The attribute map is a map data structure whose keys are attribute names and whose values are expressions that represent the corresponding attribute values.

• The block sequence is an ordered sequence of blocks, with each specifying a block type name, the sequence of labels specified for the block, and the body object (not body content) representing the block's own body.

After obtaining body content, the calling application may continue processing by evaluating attribute expressions and/or recursively applying further schema-driven processing to the child block bodies.

Note: The body schema is intentionally minimal, to reduce the set of mapping rules that must be defined for each syntax. Higher-level utility libraries may be provided to assist in the construction of a schema and perform additional processing, such as automatically evaluating attribute expressions and assigning their result values into a data structure, or recursively applying a schema to child blocks. Such utilities are not part of this core specification and will vary depending on the capabilities and idiom of the implementation language.

Dynamic Attributes Processing

The schema-driven processing model is useful when the expected structure of a body is known a priori by the calling application. Some blocks are instead more free-form, such as a user-provided set of arbitrary key/value pairs.

The alternative dynamic attributes processing mode allows for this more ad-hoc approach. Processing in this mode behaves as if a schema had been constructed without any block header schemata and with an attribute schema for each distinct key provided within the physical representation of the body.

The means by which distinct keys are identified is dependent on the physical syntax; this processing mode assumes that the syntax has a way to enumerate keys provided by the author and identify expressions that correspond with those keys, but does not define the means by which this is done.

The result of dynamic attributes processing is an attribute map as defined in the previous section. No block sequence is produced in this processing mode.

Partial Processing of Body Content

Under schema-driven processing, by default the given schema is assumed to be exhaustive, such that any attribute or block not matched by schema elements is considered an error. This allows feedback about unsupported attributes and blocks (such as typos) to be provided.

An alternative is partial processing, where any additional elements within the body are not considered an error.

Under partial processing, the result is both body content as described above and a new body that represents any body elements that remain after the schema has been processed.

Specifically:

• Any attribute whose name is specified in the schema is returned in body content and elided from the new body.

• Any block whose type is specified in the schema is returned in body content and elided from the new body.

• Any attribute or block not meeting the above conditions is placed into the new body, unmodified.

The new body can then be recursively processed using any of the body processing models. This facility allows different subsets of body content to be processed by different parts of the calling application.

Processing a body in two steps — first partial processing of a source body, then exhaustive processing of the returned body — is equivalent to single-step processing with a schema that is the union of the schemata used across the two steps.

Expressions

Attribute values are represented by expressions. Depending on the concrete syntax in use, an expression may just be a literal value or it may describe a computation in terms of literal values, variables, and functions.

Each syntax defines its own representation of expressions. For syntaxes based in languages that do not have any non-literal expression syntax, it is recommended to embed the template language from the native syntax e.g. as a post-processing step on string literals.

Expression Evaluation

In order to obtain a concrete value, each expression must be evaluated. Evaluation is performed in terms of an evaluation context, which consists of the following:

• An evaluation mode, which is defined below.
• A variable scope, which provides a set of named variables for use in expressions.
• A function table, which provides a set of named functions for use in expressions.

The evaluation mode allows for two different interpretations of an expression:

• In literal-only mode, variables and functions are not available and it is assumed that the calling application's intent is to treat the attribute value as a literal.

• In full expression mode, variables and functions are defined and it is assumed that the calling application wishes to provide a full expression language for definition of the attribute value.

The actual behavior of these two modes depends on the syntax in use. For languages with first-class expression syntax, these two modes may be considered equivalent, with literal-only mode simply not defining any variables or functions. For languages that embed arbitrary expressions via string templates, literal-only mode may disable such processing, allowing literal strings to pass through without interpretation as templates.

Since literal-only mode does not support variables and functions, it is an error for the calling application to enable this mode and yet provide a variable scope and/or function table.

Values and Value Types

The result of expression evaluation is a value. Each value has a type, which is dynamically determined during evaluation. The variable scope in the evaluation context is a map from variable name to value, using the same definition of value.

The type system for HCL values is intended to be of a level abstraction suitable for configuration of various applications. A well-defined, implementation-language-agnostic type system is defined to allow for consistent processing of configuration across many implementation languages. Concrete implementations may provide additional functionality to lower HCL values and types to corresponding native language types, which may then impose additional constraints on the values outside of the scope of this specification.

Two values are equal if and only if they have identical types and their values are equal according to the rules of their shared type.

Primitive Types

The primitive types are string, bool, and number.

A string is a sequence of unicode characters. Two strings are equal if NFC normalization (UAX#15 of each string produces two identical sequences of characters. NFC normalization ensures that, for example, a precomposed combination of a latin letter and a diacritic compares equal with the letter followed by a combining diacritic.

The bool type has only two non-null values: true and false. Two bool values are equal if and only if they are either both true or both false.

A number is an arbitrary-precision floating point value. An implementation must make the full-precision values available to the calling application for interpretation into any suitable number representation. An implementation may in practice implement numbers with limited precision so long as the following constraints are met:

• Integers are represented with at least 256 bits.
• Non-integer numbers are represented as floating point values with a mantissa of at least 256 bits and a signed binary exponent of at least 16 bits.
• An error is produced if an integer value given in source cannot be represented precisely.
• An error is produced if a non-integer value cannot be represented due to overflow.
• A non-integer number is rounded to the nearest possible value when a value is of too high a precision to be represented.

The number type also requires representation of both positive and negative infinity. A "not a number" (NaN) value is not provided nor used.

Two number values are equal if they are numerically equal to the precision associated with the number. Positive infinity and negative infinity are equal to themselves but not to each other. Positive infinity is greater than any other number value, and negative infinity is less than any other number value.

Some syntaxes may be unable to represent numeric literals of arbitrary precision. This must be defined in the syntax specification as part of its description of mapping numeric literals to HCL values.

Structural Types

Structural types are types that are constructed by combining other types. Each distinct combination of other types is itself a distinct type. There are two structural type kinds:

• Object types are constructed of a set of named attributes, each of which has a type. Attribute names are always strings. (Object attributes are a distinct idea from body attributes, though calling applications may choose to blur the distinction by use of common naming schemes.)
• Tuple types are constructed of a sequence of elements, each of which has a type.

Values of structural types are compared for equality in terms of their attributes or elements. A structural type value is equal to another if and only if all of the corresponding attributes or elements are equal.

Two structural types are identical if they are of the same kind and have attributes or elements with identical types.

Collection Types

Collection types are types that combine together an arbitrary number of values of some other single type. There are three collection type kinds:

• List types represent ordered sequences of values of their element type.
• Map types represent values of their element type accessed via string keys.
• Set types represent unordered sets of distinct values of their element type.

For each of these kinds and each distinct element type there is a distinct collection type. For example, "list of string" is a distinct type from "set of string", and "list of number" is a distinct type from "list of string".

Values of collection types are compared for equality in terms of their elements. A collection type value is equal to another if and only if both have the same number of elements and their corresponding elements are equal.

Two collection types are identical if they are of the same kind and have the same element type.

Null values

Each type has a null value. The null value of a type represents the absence of a value, but with type information retained to allow for type checking.

Null values are used primarily to represent the conditional absence of a body attribute. In a syntax with a conditional operator, one of the result values of that conditional may be null to indicate that the attribute should be considered not present in that case.

Calling applications should consider an attribute with a null value as equivalent to the value not being present at all.

A null value of a particular type is equal to itself.

Unknown Values and the Dynamic Pseudo-type

An unknown value is a placeholder for a value that is not yet known. Operations on unknown values themselves return unknown values that have a type appropriate to the operation. For example, adding together two unknown numbers yields an unknown number, while comparing two unknown values of any type for equality yields an unknown bool.

Each type has a distinct unknown value. For example, an unknown number is a distinct value from an unknown string.

The dynamic pseudo-type is a placeholder for a type that is not yet known. The only values of this type are its null value and its unknown value. It is referred to as a pseudo-type because it should not be considered a type in its own right, but rather as a placeholder for a type yet to be established. The unknown value of the dynamic pseudo-type is referred to as the dynamic value.

Operations on values of the dynamic pseudo-type behave as if it is a value of the expected type, optimistically assuming that once the value and type are known they will be valid for the operation. For example, adding together a number and the dynamic value produces an unknown number.

Unknown values and the dynamic pseudo-type can be used as a mechanism for partial type checking and semantic checking: by evaluating an expression with all variables set to an unknown value, the expression can be evaluated to produce an unknown value of a given type, or produce an error if any operation is provably invalid with only type information.

Unknown values and the dynamic pseudo-type must never be returned from operations unless at least one operand is unknown or dynamic. Calling applications are guaranteed that unless the global scope includes unknown values, or the function table includes functions that return unknown values, no expression will evaluate to an unknown value. The calling application is thus in total control over the use and meaning of unknown values.

The dynamic pseudo-type is identical only to itself.

Capsule Types

A capsule type is a custom type defined by the calling application. A value of a capsule type is considered opaque to HCL, but may be accepted by functions provided by the calling application.

A particular capsule type is identical only to itself. The equality of two values of the same capsule type is defined by the calling application. No other operations are supported for values of capsule types.

Support for capsule types in a HCL implementation is optional. Capsule types are intended to allow calling applications to pass through values that are not part of the standard type system. For example, an application that deals with raw binary data may define a capsule type representing a byte array, and provide functions that produce or operate on byte arrays.

Type Specifications

In certain situations it is necessary to define expectations about the expected type of a value. Whereas two types have a commutative identity relationship, a type has a non-commutative matches relationship with a type specification. A type specification is, in practice, just a different interpretation of a type such that:

• Any type matches any type that it is identical to.

• Any type matches the dynamic pseudo-type.

For example, given a type specification "list of dynamic pseudo-type", the concrete types "list of string" and "list of map" match, but the type "set of string" does not.

Functions and Function Calls

The evaluation context used to evaluate an expression includes a function table, which represents an application-defined set of named functions available for use in expressions.

Each syntax defines whether function calls are supported and how they are physically represented in source code, but the semantics of function calls are defined here to ensure consistent results across syntaxes and to allow applications to provide functions that are interoperable with all syntaxes.

A function is defined from the following elements:

• Zero or more positional parameters, each with a name used for documentation, a type specification for expected argument values, and a flag for whether each of null values, unknown values, and values of the dynamic pseudo-type are accepted.

• Zero or one variadic parameters, with the same structure as the positional parameters, which if present collects any additional arguments provided at the function call site.

• A result type definition, which specifies the value type returned for each valid sequence of argument values.

• A result value definition, which specifies the value returned for each valid sequence of argument values.

A function call, regardless of source syntax, consists of a sequence of argument values. The argument values are each mapped to a corresponding parameter as follows:

• For each of the function's positional parameters in sequence, take the next argument. If there are no more arguments, the call is erroneous.

• If the function has a variadic parameter, take all remaining arguments that where not yet assigned to a positional parameter and collect them into a sequence of variadic arguments that each correspond to the variadic parameter.

• If the function has no variadic parameter, it is an error if any arguments remain after taking one argument for each positional parameter.

After mapping each argument to a parameter, semantic checking proceeds for each argument:

• If the argument value corresponding to a parameter does not match the parameter's type specification, the call is erroneous.

• If the argument value corresponding to a parameter is null and the parameter is not specified as accepting nulls, the call is erroneous.

• If the argument value corresponding to a parameter is the dynamic value and the parameter is not specified as accepting values of the dynamic pseudo-type, the call is valid but its result type is forced to be the dynamic pseudo type.

• If neither of the above conditions holds for any argument, the call is valid and the function's value type definition is used to determine the call's result type. A function may vary its result type depending on the argument values as well as the argument types; for example, a function that decodes a JSON value will return a different result type depending on the data structure described by the given JSON source code.

If semantic checking succeeds without error, the call is executed:

• For each argument, if its value is unknown and its corresponding parameter is not specified as accepting unknowns, the result value is forced to be an unknown value of the result type.

• If the previous condition does not apply, the function's result value definition is used to determine the call's result value.

The result of a function call expression is either an error, if one of the erroneous conditions above applies, or the result value.

Type Conversions and Unification

Values given in configuration may not always match the expectations of the operations applied to them or to the calling application. In such situations, automatic type conversion is attempted as a convenience to the user.

Along with conversions to a specified type, it is sometimes necessary to ensure that a selection of values are all of the same type, without any constraint on which type that is. This is the process of type unification, which attempts to find the most general type that all of the given types can be converted to.

Both type conversions and unification are defined in the syntax-agnostic model to ensure consistency of behavior between syntaxes.

Type conversions are broadly characterized into two categories: safe and unsafe. A conversion is "safe" if any distinct value of the source type has a corresponding distinct value in the target type. A conversion is "unsafe" if either the target type values are not distinct (information may be lost in conversion) or if some values of the source type do not have any corresponding value in the target type. An unsafe conversion may result in an error.

A given type can always be converted to itself, which is a no-op.

Conversion of Null Values

All null values are safely convertable to a null value of any other type, regardless of other type-specific rules specified in the sections below.

Conversion to and from the Dynamic Pseudo-type

Conversion from the dynamic pseudo-type to any other type always succeeds, producing an unknown value of the target type.

Conversion of any value to the dynamic pseudo-type is a no-op. The result is the input value, verbatim. This is the only situation where the conversion result value is not of the given target type.

Primitive Type Conversions

Bidirectional conversions are available between the string and number types, and between the string and boolean types.

The bool value true corresponds to the string containing the characters "true", while the bool value false corresponds to the string containing the characters "false". Conversion from bool to string is safe, while the converse is unsafe. The strings "1" and "0" are alternative string representations of true and false respectively. It is an error to convert a string other than the four in this paragraph to type bool.

A number value is converted to string by translating its integer portion into a sequence of decimal digits (0 through 9), and then if it has a non-zero fractional part, a period . followed by a sequence of decimal digits representing its fractional part. No exponent portion is included. The number is converted at its full precision. Conversion from number to string is safe.

A string is converted to a number value by reversing the above mapping. No exponent portion is allowed. Conversion from string to number is unsafe. It is an error to convert a string that does not comply with the expected syntax to type number.

No direct conversion is available between the bool and number types.

Collection and Structural Type Conversions

Conversion from set types to list types is safe, as long as their element types are safely convertable. If the element types are unsafely convertable, then the collection conversion is also unsafe. Each set element becomes a corresponding list element, in an undefined order. Although no particular ordering is required, implementations should produce list elements in a consistent order for a given input set, as a convenience to calling applications.

Conversion from list types to set types is unsafe, as long as their element types are convertable. Each distinct list item becomes a distinct set item. If two list items are equal, one of the two is lost in the conversion.

Conversion from tuple types to list types permitted if all of the tuple element types are convertable to the target list element type. The safety of the conversion depends on the safety of each of the element conversions. Each element in turn is converted to the list element type, producing a list of identical length.

Conversion from tuple types to set types is permitted, behaving as if the tuple type was first converted to a list of the same element type and then that list converted to the target set type.

Conversion from object types to map types is permitted if all of the object attribute types are convertable to the target map element type. The safety of the conversion depends on the safety of each of the attribute conversions. Each attribute in turn is converted to the map element type, and map element keys are set to the name of each corresponding object attribute.

Conversion from list and set types to tuple types is permitted, following the opposite steps as the converse conversions. Such conversions are unsafe. It is an error to convert a list or set to a tuple type whose number of elements does not match the list or set length.

Conversion from map types to object types is permitted if each map key corresponds to an attribute in the target object type. It is an error to convert from a map value whose set of keys does not exactly match the target type's attributes. The conversion takes the opposite steps of the converse conversion.

Conversion from one object type to another is permitted as long as the common attribute names have convertable types. Any attribute present in the target type but not in the source type is populated with a null value of the appropriate type.

Conversion from one tuple type to another is permitted as long as the tuples have the same length and the elements have convertable types.

Type Unification

Type unification is an operation that takes a list of types and attempts to find a single type to which they can all be converted. Since some type pairs have bidirectional conversions, preference is given to safe conversions. In technical terms, all possible types are arranged into a lattice, from which a most general supertype is selected where possible.

The type resulting from type unification may be one of the input types, or it may be an entirely new type produced by combination of two or more input types.

The following rules do not guarantee a valid result. In addition to these rules, unification fails if any of the given types are not convertable (per the above rules) to the selected result type.

The following unification rules apply transitively. That is, if a rule is defined from A to B, and one from B to C, then A can unify to C.

Number and bool types both unify with string by preferring string.

Two collection types of the same kind unify according to the unification of their element types.

List and set types unify by preferring the list type.

Map and object types unify by preferring the object type.

List, set and tuple types unify by preferring the tuple type.

The dynamic pseudo-type unifies with any other type by selecting that other type. The dynamic pseudo-type is the result type only if all input types are the dynamic pseudo-type.

Two object types unify by constructing a new type whose attributes are the union of those of the two input types. Any common attributes themselves have their types unified.

Two tuple types of the same length unify constructing a new type of the same length whose elements are the unification of the corresponding elements in the two input types.

Static Analysis

In most applications, full expression evaluation is sufficient for understanding the provided configuration. However, some specialized applications require more direct access to the physical structures in the expressions, which can for example allow the construction of new language constructs in terms of the existing syntax elements.

Since static analysis analyses the physical structure of configuration, the details will vary depending on syntax. Each syntax must decide which of its physical structures corresponds to the following analyses, producing error diagnostics if they are applied to inappropriate expressions.

The following are the required static analysis functions:

• Static List: Require list/tuple construction syntax to be used and return a list of expressions for each of the elements given.

• Static Map: Require map/object construction syntax to be used and return a list of key/value pairs -- both expressions -- for each of the elements given. The usual constraint that a map key must be a string must not apply to this analysis, thus allowing applications to interpret arbitrary keys as they see fit.

• Static Call: Require function call syntax to be used and return an object describing the called function name and a list of expressions representing each of the call arguments.

• Static Traversal: Require a reference to a symbol in the variable scope and return a description of the path from the root scope to the accessed attribute or index.

The intent of a calling application using these features is to require a more rigid interpretation of the configuration than in expression evaluation. Syntax implementations should make use of the extra contextual information provided in order to make an intuitive mapping onto the constructs of the underlying syntax, possibly interpreting the expression slightly differently than it would be interpreted in normal evaluation.

Each syntax must define which of its expression elements each of the analyses above applies to, and how those analyses behave given those expression elements.

Implementation Considerations

Implementations of this specification are free to adopt any strategy that produces behavior consistent with the specification. This non-normative section describes some possible implementation strategies that are consistent with the goals of this specification.

Language-agnosticism

The language-agnosticism of this specification assumes that certain behaviors are implemented separately for each syntax:

• Matching of a body schema with the physical elements of a body in the source language, to determine correspondence between physical constructs and schema elements.

• Implementing the dynamic attributes body processing mode by either interpreting all physical constructs as attributes or producing an error if non-attribute constructs are present.

• Providing an evaluation function for all possible expressions that produces a value given an evaluation context.

• Providing the static analysis functionality described above in a manner that makes sense within the convention of the syntax.

The suggested implementation strategy is to use an implementation language's closest concept to an abstract type, virtual type or interface type to represent both Body and Expression. Each language-specific implementation can then provide an implementation of each of these types wrapping AST nodes or other physical constructs from the language parser.