This applies both two the whole of bare expressions and to any nested
expressions within parentheses. The latter means that an attribute value
can span over multiple lines if it's wrapped in parens:
foo = (
1 + 2 + 3 + 4 + 5
)
The ~ character can be used at the start and end of interpolation
sequences to trim off whitespace in neighboring literals, with the goal
of allowing extra whitespace to be included for readability without
including it in the resulting string.
Previously we were failing to return back to template-scanning mode due
to decrementing "braces" too early, causing the remainder of the template
to be scanned as if it were an expression.
Conditional expression parsing is ostensibly implemented, but since it
depends on the rest of the expression parsers -- not yet implemented --
it cannot be tested in isolation.
Also includes an initial implementation of the conditional expression
node, but this is also not yet tested and so may need further revisions
once we're in a better position to test it.
This rewrite of decodeQuotedLit, now called decodeStringLit, is able to
handle both cases with a single function, and also now correctly handles
situations where double-$ and double-! are not followed immediately by
a { symbol, and must thus be treated literally.
The context where a string literal was found affects what sort of escaping
it can have, so we need to distinguish these cases so that we will only
look for and handle backslash escapes in quoted strings.
This recovery method attempts to place the peeker directly after the
newline indicating the end of the current body item. It does this by
counting open and close bracketing constructs and then returning when
a newline is encountered with no bracketing constructs open.
It's designed for use in the "header" part of a body item, with no
bracketing constructs open yet. It _might_ work in other situations, but
is likely to end up choosing the wrong end point if used in the middle
of a bracketed expression that itself contains newlines.
These are the top-level interface to parsing configuration files in the
native zcl syntax, similarly to the existing methods for parsing other
syntaxes.
The parser has some recovery heuristics but they will not always achieve
the best result. To prevent unsuccessful recovery from causing a cascade
of confusing follow-on errors, we'll track in the parser whether recovery
has been attempted and then the specific sub-parsers can use this to
skip certain "unexpected token type" errors in recovery situations,
assuming instead that an earlier error will cover the problem and that
we just want to bail out quickly.
The scanner produces complicated sequences for quoted strings due to the
template language, but sometimes we just want a simple string with no
interpolations.
This will make it easier for the parser to walk through the token sequence
with one token of lookahead, maintaining state of where we've got to and
also allowing us to switch into and out of the newline-sensitive mode
as we transition to and from newline-sensitive constructs.
Mirroring the Lex... triplet of functions, callers here can choose to
parse in one of three modes:
- ParseConfig is the main entry point and parses what we might consider
a whole config file.
- ParseExpression parses a sequence as an isolated expression, which
may be useful in implementing a REPL to inspect configuration or
application state.
- ParseTemplate parses a sequence as an isolated template, which may be
useful in parsing external files as templates, for situations where the
input is too large to be conveniently embedded within configuration.
In zclwrite we throw away the absolute source position information and
instead just retain the number of spaces before each token. This different
model allows us to rewrite parts of the token sequence without needing
to re-adjust all of the positions, and it also allows us to do simple
indentation and spacing adjustments just by walking through the token
list and adjusting these numbers.
This LexConfig, LexExpression and LexTemplate set of functions allow
outside callers to use the scanner in isolation, skipping the parser.
This may be useful for use-cases such as syntax highlighting, separate
parsers (such as the one in zclwrite), and so forth. Most callers should
use the parser (once implemented) though, to get a semantic AST.
This uses a separate, lower-level AST than the parser so that it can
retain the raw tokens and make surgical changes. As a consequence it
has much less semantic detail than the parser AST, and in particular
doesn't represent the structure of expressions except to retain variable
references to enable global rename operations.
This alternative scanning mode makes the scanner start in template
context rather than normal context. This will be later used by the parser
to allow parsing of standalone templates that aren't embedded inside a
zcl configuration file.
This is important because our syntax for objects uses newlines as the
separator between items, so this is the only signal we'll get that a
given item has ended and another is beginning.
A scanner "mode" decides which state it starts in, allowing us to start
in template mode for parsing top-level templates. However, currently the
only mode implemented is "normal" mode, which is the behavior we had
before.
This requires some extra state-keeping because we allow templates to be
nested inside templates. This takes us outside of the world of regular
languages, but we accept that here because it makes things easier to
deal with down the line in the parser.
The methodology is to keep track of how many braces are open at a given
time and then, when a nested template interpolation begins, record the
current brace level. Then, when a closing brace is encountered, if its
nesting level is at the top of the stack then we pop off the stack and
return to "main" parsing mode.
Ragel's existing idea of calling and returning from machines is important
here too. As this currently stands this is not actually needed, but once
heredocs are in play we will have two possible places to return to at
the end of an interpolation sequence, so the state return stack maintained
by Ragel will determine whether to return to string mode or heredoc mode.
