Commit Graph

7 Commits

Author SHA1 Message Date
Martin Atkins
4a939a2b46 zclsyntax: allow scanner to support multiple modes
A scanner "mode" decides which state it starts in, allowing us to start
in template mode for parsing top-level templates. However, currently the
only mode implemented is "normal" mode, which is the behavior we had
before.
2017-05-28 15:44:22 -07:00
Martin Atkins
bb87c7a0f9 zclsyntax: heredoc to be separate start/end tokens
Just as we have OQuote and CQuote, we need the same for heredocs so that
we can parse their contents as templates that may span multiple tokens.
2017-05-28 09:36:32 -07:00
Martin Atkins
187d7b8045 zclsyntax: re-organize and simplify the scanner 2017-05-28 08:38:13 -07:00
Martin Atkins
b8db08bf04 zclsyntax: use stringer for TokenType stringification 2017-05-28 07:38:17 -07:00
Martin Atkins
63fcfd6b7d zclsyntax: include a token for the end of a template sequence
Although this end symbol appears as just a close-brace in source, it's
worth differentiating it because the scanner must differentiate it anyway
(to recognize moving back into template-scanning mode) and it avoids the
parser from having to similarly re-recognize the difference.
2017-05-28 07:20:39 -07:00
Martin Atkins
76c0ca70f0 zclsyntax: scanner to return whole token slice at once
On reflection, it seems easier to maintain the necessary state we need
by doing all of the scanning in a single pass, since we can then just
use local variables within the scanner function.
2017-05-28 07:11:24 -07:00
Martin Atkins
e65eafbe83 zclsyntax: define the initial set of language tokens for the scanner 2017-05-27 19:00:00 -07:00