Okay so slight issue with the expanding tokenizer. I need to handle diverging token sources. Because say if I perform a successful read of a split area I need to set the source to use that divergent area. It also needs to support peeking and such. So I think I can just go with another class and use a single object for that. But the question is, what is the best way to do that? I basically need a means to set success or failure. Will need to figure that out. Also for a bunch of elements the major class structure I had does not really make that much sense when it comes to inheriting, however it is much better compared to before. I think what would be better is having elements of the source file in an element package then having lexical just be the parsers. That way I can structure things a bit better and just have parsers be in other classes without much consideration. Actually maybe I do not have to do this at all, just support marking and setting like input streams and readers. I think that would be far better in the long run and likely easier to debug. Also additionally this could just be done in another class that provides tokens rather than modifying ExpandingSource even more. Basically it will allow there to be sub-marks which can be closed as such. But there is no way to tell if a close was a success or failure. That would still be rather manual.


So just a change from ExpandingSplit to a source which can be marked and such. They will all operate from the same object but it will just check the close order and marking order as needed. But marks can just actually operate on a single stream and there can just be multiple marks and sets. But each sub-mark can just have a commit() method which just sets the parent stream position to the given position. This would be taken on successful runs. But that might not always be the case however. Just that this is a bit more complicated than it seems.