DISCLAIMER: These notes are from the defunct k8 project which precedes SquirrelJME. The notes for SquirrelJME start on 2016/02/26! The k8 project was effectively a Java SE 8 operating system and as such all of the notes are in the context of that scope. That project is no longer my goal as SquirrelJME is the spiritual successor to it.


My wake time slowly moves forward in time, however I thought of this idea of how I can make my tokenizer simpler by using inner classes in a set of sort. An enum cannot be used because it would be static, so I would have to add the enumerations to a set along with the inner classes. The classes should only decode a basic set of values, such as comments, literals, words, etc.


Actually a map.


I will just take the input code and produce a raw basic set of tokens which will then be passed through another decoder which will handle the logic better.


Brackets used for binary shifts and such will have to be split into multiple tokens then possibly merged back later. Note that this is only for extgreater{} and extless. However this only needs to be done for merged symbols that are extless, extgreater, extless extless, extgreater extgreater, extgreater extgreater extgreater.