Okay so, I was thinking... ExpandedToken is the same as Token except it has comments attached. So I am just going to merge the two.


Also, I was thinking of moving the expanding code.


Tokenizer would best have the expanding code with a note in the class that it performs such things. Then I do not need to worry about such things for the most part. Then on top of the TokenSource I can have a BufferedTokenSource which does the marking, setting, and resetting as needed.


I can do the same with comments in Tokenizer too, just having those attached to input tokens accordingly on read.