Okay so I have an idea. Basically undo the context tokenizer and make the
bottom tokenizer just normal tokens again. Get rid of areas and then just
have something which reads the input tokens and builds a state from it. I
can have a special modified token state kind of thing where I can turn
<<< into something such as
<. I do not really need two
tokenizer depths and it would just be filling in basic class structure
anyway. What I can do is have the lower level tokenizer and the basic
structure on top. This will make things easier to use.
I think a location aware interface would be best to reduce duplicate constructors and such.
An EOF token would be useful so that null pointers are not used for the token, makes it easier to parse.
Actually, always generating EOF would probably be the best so that peeking could work on an infinite input stream. But this could be an artifact of the layering system.
Actually, I think decomposition could be automatic mostly if it follows a
period because that would result in something like
However thinking about it, the left side of the generic will always have
be along since it will be split by another token so that is only needed for
the right hand operations.