DISCLAIMER: These notes are from the defunct k8 project which precedes SquirrelJME. The notes for SquirrelJME start on 2016/02/26! The k8 project was effectively a Java SE 8 operating system and as such all of the notes are in the context of that scope. That project is no longer my goal as SquirrelJME is the spiritual successor to it.
I moved my laptop to a makeshift desk, it feels much better now. For the tokenization code all I need to do is start parsing of tokens now using either standard stuff or regular expression brute force. The only thing I need to determine is when a token ends, that is there must be a valid separation sequence like a separator or operator. Or perhaps just a general separation token which acts like a separated operator of sorts.