Commit Graph

9 Commits

Author SHA1 Message Date
Martin Atkins
f1c91b8eea zclsyntax: detect newlines during scanning
This is important because our syntax for objects uses newlines as the
separator between items, so this is the only signal we'll get that a
given item has ended and another is beginning.
2017-05-29 09:25:21 -07:00
Martin Atkins
861d0a1b4e zclsyntax: scanning of comments 2017-05-29 09:13:35 -07:00
Martin Atkins
2551856d22 zclsyntax: heredoc support in the scanner 2017-05-29 08:55:53 -07:00
Martin Atkins
4a939a2b46 zclsyntax: allow scanner to support multiple modes
A scanner "mode" decides which state it starts in, allowing us to start
in template mode for parsing top-level templates. However, currently the
only mode implemented is "normal" mode, which is the behavior we had
before.
2017-05-28 15:44:22 -07:00
Martin Atkins
18e45ec05c zclsyntax: scanner support of string templates
This requires some extra state-keeping because we allow templates to be
nested inside templates. This takes us outside of the world of regular
languages, but we accept that here because it makes things easier to
deal with down the line in the parser.

The methodology is to keep track of how many braces are open at a given
time and then, when a nested template interpolation begins, record the
current brace level. Then, when a closing brace is encountered, if its
nesting level is at the top of the stack then we pop off the stack and
return to "main" parsing mode.

Ragel's existing idea of calling and returning from machines is important
here too. As this currently stands this is not actually needed, but once
heredocs are in play we will have two possible places to return to at
the end of an interpolation sequence, so the state return stack maintained
by Ragel will determine whether to return to string mode or heredoc mode.
2017-05-28 15:33:01 -07:00
Martin Atkins
5898b36695 zclsyntax: scan single-character tokens that represent themselves
For convenience we use the rune values of these tokens as their token
enum values, so we can handle them all via a single rule.
2017-05-28 09:34:20 -07:00
Martin Atkins
53944335f1 zclsyntax: identifiers in the scanner 2017-05-28 09:16:53 -07:00
Martin Atkins
6362354c87 zclsyntax: scanning of numeric literals 2017-05-28 08:56:43 -07:00
Martin Atkins
187d7b8045 zclsyntax: re-organize and simplify the scanner 2017-05-28 08:38:13 -07:00