Commit Graph

11 Commits

Author SHA1 Message Date
Martin Atkins
861d0a1b4e zclsyntax: scanning of comments 2017-05-29 09:13:35 -07:00
Martin Atkins
2551856d22 zclsyntax: heredoc support in the scanner 2017-05-29 08:55:53 -07:00
Martin Atkins
4a939a2b46 zclsyntax: allow scanner to support multiple modes
A scanner "mode" decides which state it starts in, allowing us to start
in template mode for parsing top-level templates. However, currently the
only mode implemented is "normal" mode, which is the behavior we had
before.
2017-05-28 15:44:22 -07:00
Martin Atkins
da874ba0ff zclsyntax: disallow newlines in string templates
Heredocs will deal with the multiline string case once implemented.
2017-05-28 15:35:55 -07:00
Martin Atkins
18e45ec05c zclsyntax: scanner support of string templates
This requires some extra state-keeping because we allow templates to be
nested inside templates. This takes us outside of the world of regular
languages, but we accept that here because it makes things easier to
deal with down the line in the parser.

The methodology is to keep track of how many braces are open at a given
time and then, when a nested template interpolation begins, record the
current brace level. Then, when a closing brace is encountered, if its
nesting level is at the top of the stack then we pop off the stack and
return to "main" parsing mode.

Ragel's existing idea of calling and returning from machines is important
here too. As this currently stands this is not actually needed, but once
heredocs are in play we will have two possible places to return to at
the end of an interpolation sequence, so the state return stack maintained
by Ragel will determine whether to return to string mode or heredoc mode.
2017-05-28 15:33:01 -07:00
Martin Atkins
c50317a2e0 zclsyntax: scan multi-character punctuation tokens 2017-05-28 09:40:58 -07:00
Martin Atkins
5898b36695 zclsyntax: scan single-character tokens that represent themselves
For convenience we use the rune values of these tokens as their token
enum values, so we can handle them all via a single rule.
2017-05-28 09:34:20 -07:00
Martin Atkins
53944335f1 zclsyntax: identifiers in the scanner 2017-05-28 09:16:53 -07:00
Martin Atkins
6362354c87 zclsyntax: scanning of numeric literals 2017-05-28 08:56:43 -07:00
Martin Atkins
187d7b8045 zclsyntax: re-organize and simplify the scanner 2017-05-28 08:38:13 -07:00
Martin Atkins
76c0ca70f0 zclsyntax: scanner to return whole token slice at once
On reflection, it seems easier to maintain the necessary state we need
by doing all of the scanning in a single pass, since we can then just
use local variables within the scanner function.
2017-05-28 07:11:24 -07:00