History

SquirrelJME is an idea and execution that extends from previous ideas and projects. The base idea extends from my desire to write an operating system to run on systems, since such things have fascinated me.

Old Projects (2011/01/15 - 2016/02/25)

These are old defunct projects which I have previously worked on.

I will say that it can take years before you can find a project you enjoy working on and one that inspires you. Keep searching. Just be careful where you step and do not take in more than you can handle.

HandOS (2011/01/15 - 2011/01/22)

This was an attempt at creating an open source implementation of Palm OS. Palm OS was not actively used much often and additionally existing emulators for Palm OS are sub-par in that they were either old or were horribly out of date. The goal of the project was to run m68k and ARM Palm OS applications on the host system.

The project was fully structured and had all the default applications in place as placeholders for when they were ready.

However, the project could not actually be compiled, it was mostly pretty much just placeholders, some reference documentation, and notes. It did include some tests which showed that threading was indeed possible on Palm OS like modern systems.

MPA Orange-Sapphire (2011/12/12 - 2011/12/13)

This was a very short project which never really took off. It was intended to be an operating system written in a BASIC-like language. Obviously BASIC is rather horrible compared to the languages of today.

MPAYAOS (2012/02/20 - 2012/02/29)

Standing for Yet Another Operating System.

This was a short attempt at another operating system, it never really got past the bootloader stage. It was written in C.

HandNIX (2012/05/28 - 2012/05/29)

Standing for Hand UNIX Environment.

This was an attempt to make a UNIX-like environment for Palm OS so that I could do development for my projects on Palm OS. It existed as a single C source code file.

Only a few things were implemented. The major thing which was implemented was a basic cooperative tasked system which was based on Palm OS's versions of setjmp and longjmp. There also was a basic terminal emulator which did not support any escape sequences.

Every POSIX command that existed on the system was referenced via a function pointer and the intention was to have virtual files (via #!) which would execute those internal commands when requested. The environment would have started on the shell (sh) which would then be available to execute other programs.

HandOS 6.3 (2012/05/31 - 2012/08/17)

HandOS is a clone of Palm OS, however the main purpose of this project was to implement the only released in a simulated environment: Palm OS Cobalt.

It was written in C and it required the Palm OS SDK for compilation.

I did not have headers for Palm OS 6, so I was writing the headers from scratch.

MPAPCOS (2012/09/30 - 2013/01/16)

Standing for Posix Compatible Operating System.

This was an attempt to write an operating system which was compatible with POSIX:2008 and could run such programs. A C preprocessor was written and some basic utilities were supported, but generally that was as far as it got.

HandOS 5.6 (2013/01/23 - 2013/02/05)

This was another attempt at writing a Palm OS clone, it did not get very far. There was basic project structure but not much else.

MPAMSEK (2013/03/31 - 2013/04/14)

Standing for Multi-System Emulation Kernel.

This project was I believe to write an operating system which could run many other operating systems in a very bare bones emulated environment to run as if it were on native hardware. Basically, you would boot into a system and instead of running an OS with an emulator running on it, the emulator would be directly running on the hardware.

I only really got a basic system launch on PowerPC's Open Firmware. I was able to print to the system's framebuffer console but not much else.

MPAOSKP (2013/04/14 - 2013/05/15)

This project is based off MPAMSEK.

I am not too sure what the acronym stood for, very likely it stood for Operating System Kernel Project.

I wrote a memory allocator but it never really got past the boot loader and early kernel initialization stages.

PowerPC and x86 worked on Open Firmware.

I failed to get SPARC working, I could just not get the bootloader to run properly and it usually ended up just crashing or not working at all. It may have been related to register windows.

I thought about writing a C compiler but I gave up quickly and abandoned that idea because that would be too much work with too little of a benefit.

POSIX Environment for Palm OS (2013/07/25 - 2013/07/27)

This project was essentially similar to an earlier but was a POSIX environment for Palm OS. By reverse engineering bits of Palm OS previously I determined that some things were very possible. For Palm OS 5 I had pretty much a concept background threading and server system. Basically a daemon would be launched in the background which would keep the environment running. There would be a connection made to the daemon with locks and semaphores. There would then be a terminal interface which would show what was being output to the console.

There would be threads in the background running tasks. However to access the server process a pointer would have to be remembered. This is where the feature manager was used to keep a persistent handle in memory so the daemon could be connected to. I determined that it was very important to be able to switch programs outside of the environment.

For SquirrelJME on Palm OS, I have an idea to do something very similar so that it runs in the background while other applications may be used.

POSIX Environment for Palm OS (2014/10/12 - 2014/10/12)

This project never went past a few files and headers. The purpose of this project was the same as the previous.

k5 (2013/12/25 - 2014/04/25)

This project had 503 commits.

This is basically the start of my delving into Java virtual machines. It was written in Java and was pretty bloated when it came to the compiler, using very abstract concepts for compilation. It was to target Java SE but I am not too sure which version it was for. Very likely it was Java 7 as Java 8 was released shortly before this project ended.

I never really got past the JIT stage in the project. It was also quite massive of a project.

Anything that was made for documentation was written in LaTeX, which is far less friendly compared to Markdown.

k8/Squirrnix (2014/04/26 - 2016/02/25)

This is the direct predecessor to SquirrelJME. Utilities and classes in SquirrelJME have been derived from this project. Generally if I find a class which is missing in SquirrelJME I check this project and copy all of the code from there since the bulk of it is just placeholder.

This project had a massive number of commits: 9041.

It intended to implement the entirety of Java SE 8 with all of its profiles. This was for the most part a very unrealistic goal. As with k5, it never really got past the JIT stage. The massive requirements in place were unrealistic. This project was doomed for failure even when it began but I was a bit foolish to take up this task.

SquirrelJME

Starting Background

SquirrelJME stems from the following ideas:

I had a growing fascination with J2ME. I had an old cell phone which could run J2ME applications. I would make a few concept projects but they did not last long because programming for J2ME is quite horrible as it is very lacking when it comes to Java. I did have the idea of writing a virtual machine that could run J2ME applications so that I could use some again.

Another thing is that the write once and run anywhere aspect of Java pretty much fascinated me also. The idea of writing a program once and having it work pretty much everywhere was indeed interesting. I wanted to do this with k8 but it was not really a feasible thing to do because it was so gigantic.

However, there was much frustration with my then current project k8. It was gigantic and the number of classes to implement was essentially an impossible mountain to cross. There was a growing sense of doubt that the project would never get anywhere because it had unrealistic goals that would be impossible for a single woman to obtain all by herself. I calculated that it could take perhaps up to a decade to implement the core classes of Java SE 8. By a decade's time, Java SE 8 would be obsolete and nobody would be using it anyway. So this compounded a wasted effort. Additionally being a third-party VM, there would quite literally be very little interest in it, so it could not be for fame. Basically it was too much and too ambitious of a project and sadly it took two years to realize that.

Near the end of the project's life span I was a very depressed person, I felt like the project was a complete waste of time and that life was rather meaningless. I felt trapped working on the project with no escape. My life was melting away into nothingness and worthlessness. I did not want to admit that the project was a failure and would be going nowhere, I was naive.

In February of 2016 I watched Hack Summit 2016, it was pretty much the first online conference which I have visited. It was really exciting and it did move me a bit. But I think the one talk which really moved me was the one by Sarah Allen (you can see her on YouTube here: https://www.youtube.com/watch?v=S1iYeThQO-g). It really resonated with me and it has really given me the light shining through my own fog to realize that my own project was ruining my life. After that I decided that my own project was not something I really wanted to do, that my ambitious goals were not even caring for myself. I decided that I should care more about myself and that I had to cut my project loose.

When that happened, I was still so very much interested in Java and having write once and run anywhere, portability and virtual machines. I found Java to be very awesome. I knew that k8 would never fit on the systems I would have liked to target. I got the motivation to push Java as small as it can go. I knew the gigantic size of k8 was too much, so I decided to switch to the small scale. I remember my fascination with J2ME and I wanted to clone that environment because I found it interesting. Java was seen as this gigantic bloated monster that ate so much CPU and so much memory. But working with the JVM so much made me realize how wrong it was. Java was in fact very small and very light. J2ME was very light and very simplistic.

The first thing to do is determine has J2ME been updated ever? To my surprise it actually was updated for Java 8. So this was quite something. Although it was not really a microscopic Java 8 VM it was Java 7, which was good enough. It added extra classes and was in general quite cool. JSR360, which is the specification for Java ME 8 was proposed on October 2, 2012. The final release was made on April 30, 2014. Java SE 8 being released on March 18, 2014. So Java ME 8 came out very shortly after Java SE 8. However with Android and iOS being everywhere it never really stood a chance and Java ME 8 never really got the chance to actually be anything.

I decided to study Java ME 8, to see if it was indeed viable. I took a quick glance at it and then deeper dives into it. Java ME 8 imported quite a number of things from Java SE 7. It basically was bringing it up to date with the modern Java language rather than languishing in the past. It did remove some things but, in the essence of compatibility those could still be brought in. These removed things were stuff such as LCDUI, which were pretty much nowhere. It took a bit to verify compatibility but the changes made do not have an impact on backwards compatibility. As such this was a bonus.

One added bonus that I liked is that Java ME was so small compared to Java SE. It is very tiny and extremely compact. Basically what is in the language is the bare minimum sane choice. I will say that there are some errors and perhaps some omissions but in general it has been rather good. The thought of Java SE being implemented and the gigantic task I knew was impossible and the small size of Java ME is actually is within reach. The small number of classes, up around 512 classes for the standard makes it very reasonable. This estimated 512 classes is about half the size of Java SE 8's compact1 and does not include things such as SQL. I really liked the fact that it gave you the minimum set of classes and you could just bring your own classes in if you care about it. Personally, I do not really care for SQL and stuff like CORBA because I do not often use them at all.

Naming and Mascot

The name SquirrelJME comes from the fact that I personally find squirrels to be very interesting and curious animals, not to mention that they are cute. Where I used to live I would see squirrels all the time so they were quite common, although where I live currently as of this writing I do not often see squirrels.

Originally my plan for a mascot at least for k5 or k8 would be to have a pink bunny rabbit in reference to The Legend of Zelda: A Link To The Past. In this game there is a region that you can enter called the Dark World where every human who enters is transformed into an animal that reflects their inner self. I then switched my idea to having the mascot be albino because these animals are a bit more rare (since they are more easily predated in nature because their bright white fur stands out). Then when SquirrelJME has come around I decided to keep the same mascot but make him a squirrel instead. So the green tunic that Link wears is a reference to the Zelda series.

It was difficult finding a name for him, as such I initially decided on just placeholder name for awhile. I thought about Lexer but that was quite long and slightly complicated, so I shorted it to Lex. It was nice and for the most part the name stuck since I was happy with it. Therefor it was never changed.

He was uploaded on 2016/05/07 to the repository.

He has subsequently been redrawn on 2018/10/19 by Kat Adam-MacEwen, where he ended up very handsome. She is quite a wonderful artist.

The C based virtual machine RatufaCoat was named by meepingsnesroms of the Mu project. Ratufa being the sub-species of squirrels in southeast Asia.

Gaining a Sister Project

On 2018/11/03 the developer of Mu, meepingsnesroms, wanted to become a sister project of SquirrelJME, so this was very exciting! The link to her repository is in the readme! We will be cooperating mostly by word of mouth by linking to each other's projects to hopefully gain interest in both of our projects.

The First Release

The first version of SquirrelJME was released along with Mu on December 25, 2018. The repository had 10,571 commits. This version spent two years in development.

This release was mostly just a demo to show that SquirrelJME exists and that it is being worked on. It is very primitive and not much can be done with it. From the release though there has been slowly increasing interest in the project since there are not many emulators that support J2ME that exist and are actively worked on.

The Second Release

This is the currently being developed release, it is expected to be at most in development for two years and six months in development.

2019

The history from January to June was taken from a note, although it was never completed.

January

This is the month that was right after 0.2.0 was released. I started the initial work on SummerCoat which has taken pretty much the entire year and more to develop, it is a really complex system and it really shows. From this run I had the initial SCRF compiler which would eventually be thrown out and which would result in the current compiler that I wrote. I even had a base project setup for RatufaCoat at the middle of January. I also performed some refactoring of the compiler stuff as well making it cleaner because it did get a bit messy. I did get method processing put in but that was later split apart because wow it was complex. Also added a bunch of new tests and implemented parts of the class library since they were needed for the VM to function. I figured at this time that I should write a compiler which is not all that optimized to keep it simpler, although today I do have basic optimation in place. For CircleCI I put in JUnit reporting for the tests that ran which has been around for awhile now. Also for the tests that run within the VM, the profiler snapshots are stored as well so that way I can determine how fast stuff has been running accordingly. I defined an API level for 0.4.0 but eventually later in the year I determined that the API stuff is too complex, so I opted for the much simpler system call stuff. I added system properties support for SpringCoat since I needed to enable or disable debugging.

February

More work was done with SCRF. Added the ability to launch tasks with system properties and such. SummerCoat VM went through a slight refactor as well. Some of the stuff that I worked on this month have somewhat survived to the end of the year such as the stack map stuff which worked well. I mostly just implemented a ton of methods. Not much here this month.

March

This month was very rough for me because I had gotten laid off at the start of the month and it was devastating to me. I pretty much still worked on this project because it helped me get through the rough times and it also acted as a resume for me as well. I was in and out of depression so I did not exactly work on this every single day as I was job hunting as well.

One big thing I did was remove tool-scrf and put it in tool-classfile and built it up from that. I know in 2020 I will be pretty much splitting that back out and writing a far more streamlined compiler which hopefully is faster and more efficient as well. This is where the MinimizedClassFile started from so it has quite been around. It is still around and it has always been improving from this point. SummerCoat had this class library kind of thing that basically cached single classes and such I believe in memory. I know with the compiler refactoring I brought stuff back in from SCRF since it still worked just fine. For the class file stuff I implemented more things that were missing such as the exception handler. I had all of the narrow and long stuff which was a bit confusing, I know later this eventually was removed because it was just impossible to handle while keeping things simpler. I know today that everything is 32-bit for the most part. I also worked on the graphics stuff too, which is interesting. This was even the advanced graphics stuff. This graphics stuff needs to be refactored but that will get done eventually. I also worked on Squirrel Quarrel when it was a Starcraft clone still, but now today it is more of its own original game that I have planned. You could kind of play it and command units I think and such. For graphics I added a serializer so that it could be done by the host VM, being faster. I may do that again for SummerCoat although it would be a bit more difficult. This was also when I was more going twords https://squirreljme.cc/ instead of my standard domain that I use. I also am actually glad the complex type handling was removed from SummerCoat. Even reference queues and such were their own instructions, but now that is different.

April

At the start of this month I did some fixing of the class parser. I did lots more compiler work. More library methods were implemented.

I renamed my Twitter and used my MultiPhasicApps handle for SquirrelJME stuff, which actually helped me much.

I worked on the graphics stuff. Implemented some base record store stuff which was nice. Refactored XPM and PNG image loading. Added a Null Audio system.

Lots of more compiler work.

Later in the month I completely refactored SummerCoat, it pretty much should be the same as it is today just more worked on.

Also corrected testing stuff.

At the end of the month I put a bunch of work into the fonts too, I did so much with them! I actually still need to finish the symbol fonts a bunch too.

Also kinda glad I do not have class IDs and such today as well.

May

More font stuff, yay!

This month I added the concept of BootRAM and this has actually simplified things making things easier to load and such.

Added some more implementations to methods as well and HTTP support!

Also big refactor of the testing framework, which ended up going rather nicely. I also added my first sorting algorithm which was shell sort. But I guess now there is just block sort which does nothing? Oh wait no there is ShellSort.

June

The notes stopped before June, so this and anything following was written much later on at the end of 2020.

This month there was much work done on RatufaCoat and SummerCoat since they are shared. The compiler has seen a number of changes as well. There was much going back and forth, fixing things, and debugging things. A compiler and virtual machine combination is very complex. Additionally at the end of the month SummerCoat got profiling output.

July

This was the month where I was hired for my second programming job, as such from work there has been a great reduction in available time to work on SquirrelJME. However money is important and I was a single month away before I would have ran out.

At the start of the month I worked on Squirrel Quarrel a bit, I am assuming at this point I was to make it a turn based strategy although it may have been a real time strategy.

I did port RatufaCoat partially to Palm OS however it was not really functional and there were a number of limitations. Other than that, there was more minor work on SummerCoat/RatufaCoat. I decided it would be best to split the constant pool into a static pool and a run-time pool, that is still here to this day but it is something that has simplified and reduced the need to load everything at once.

But due to my new job, the month mostly just had progress at the start.

August

Due to my job, there was little progress this month. I worked a bit on the compiler, refactoring it and otherwise.

September

This month more work was done on the compiler, mostly with fixing bugs, making it better, and other such things. I did do much more than what I did the previous month however.

Unrelated to other things, I did put in support for a demo that requires a proprietary JVM to run a game.

October

Worked more on SummerCoat and attempting to get it booting up and otherwise. This would prove to be very difficult for me to do.

Otherwise, not much was done this month.

November

More work on SummerCoat and such, there were optimizations and fixes.

Most of this was done during the holidays where there were few days off.

December

Very much this was more work on SummerCoat. It was much work and I was getting more and more burnt out from working on this.

2020

Virtually everyone who has spent some time in 2020 will know how rough the year has been, there were no exceptions for myself.

January

I worked a bit more on SummerCoat but slowed down greatly, I decided to implement some other classes and such.

February

This was a difficult month... my close friend Emily passed away at the start of the month. I was devastated and was broken from it, I remember being at work and hearing the news and I sunk in my chair. I remember going outside and sitting on the small bench employees went to for their break. I broke down and cried. I remember calling my fiance and asking her to pick me up early from work. I knew she could tell that something was very wrong. I stuttered to tell her. My fiance comforted me so much. I still think of Emily often and I feel an emptiness inside of me when I do so. Typing this is difficult, but... I will remember her always. She was a close friend to me and we would talk so often. I do know that she was very thankful to have me as a friend. I hope that she is resting in peace.

Being devastated, I did my best to distract myself.

I remember just sitting down and was thinking how many months have gone by working on SummerCoat and such, being in an endless loop. I decided to change how I was doing SquirrelJME to use more modern development tools. I had always previously wrote it using a basic terminal and a basic text editor. I decided to make the switch to IntelliJ, which I can definitely say made development much easier. The first most important thing to do was to setup the Gradle build system, which I was doing. I never went with Gradle in 2016 because it was too limited, however in 2020 it has gone far enough in development to where it could work well with SquirrelJME. I did major cleaning up and otherwise. Organizing things and setting up testing like I had before. I setup a basic JNI wrapper for native calls since I need to do SquirrelJME things in Java SE to run them properly. I worked a bunch on the dependency system and otherwise, building a SquirrelJME plugin that could do everything I needed. I additionally started development in branches so I could do multiple things back and forth, this would help at times with ADHD.

March

SquirrelJME for the time was keeping me distracted much, my fiance was there for me as well. At the start of this month I married her, so from now on she is my wife. I know she has helped me get through this year so many times and I love her very much. We married the day before lockdown, we were very lucky.

For SquirrelJME, I continued on the Gradle work. By the second I had already merged in all of Gradle changes. It did feel good the direction this was going and it was the start of taking a new path. I know there were many thoughts to when I started SquirrelJME, how it was basically started from the realization that I was going nowhere with my SummerCoat work. So that was shelved for awhile.

I started work on bringing up SpringCoat. This would involve a large number of refactors and otherwise, to improve the old virtual machine I was intending to originally remove. I decided it would be best to keep SpringCoat because it is an additional vector into making sure things are operating. At least with the optimizations I have done it much quicker. I effectively continued it throughout the month. I made much progress. By the end of the month much was done but there was still a way to go.

April

This month started with continuing the refactoring and upbringing of SpringCoat. For the most part it was continuing but at a slower pace.

May

For the most part, I continued work on bringing up SpringCoat. However after about two months of working on it I realized that I was falling into the same pit as before with SummerCoat. So I decided to backpedal and start a new branch which was effectively cherry picking what I did. This was the second half of the month.

I had previously rewritten TAC, which that was cherry picked. I picked up some other things, but I suppose a bunch of it was dropped. This is okay though. This cherry picking was merged into trunk.

I additionally at the end of the month did a bunch of syntax and miscellaneous cleanup, this was a nice way to organize.

At the very end of the month, I decided that I needed to do this another way. So SpringCoat was very high-level and SummerCoat was very low-level, but it was difficult to bridge these together. So I decided on an idea of having a Mid-Level Emulation (MLE) which would bridge the gap. This is still around to the time of writing, so it has definitely been a good decision.

June

The first half of the month, little progress was made as I was mostly thinking about the best way to do things. It was thinking from nothing again, but I eventually figured it out. I did so some refactorings and such naturally.

However the second half of the month, I did much more work. I improved testing. I wrote new APIs for SpringCoat and put these on the emulation layer. I decided that it would be best if SpringCoat not do any Assembly stuff since that was complicated and I did not want to handle that for now. I did end up doing much work on SpringCoat as well. The speed increased more and it was much better.

At the very end of the month, I worked on utilities to replace the old shell scripts so that I could develop purely on Windows rather than a Linux VM.

I continued to do much more work on Mid-Level SpringCoat and eventually merged everything in accordingly! It was a very nice feeling.

July

This month was my birthday month but also I started work on Mid-Level LCDUI Refactoring and work on access control. Also the ability to launch tasks was added as well in the access control branch. Otherwise the Mid-Level LCDUI work is basically a complete refactoring of the LCDUI code. Before I had something that was too high and native while another that just used the framebuffer. So I decided to take a path that would allow both. For now it is based on Swing but it is very possible for it to be backed on a Framebuffer. Naturally for what is running on RetroArch will be using the framebuffer along with any system that lacks native widgets. However, for the systems that support it, it will effectively reduce the work that is needed to be done to make everything consistent to make it feel like it would be for that operating system.

The access control stuff carried on a bit and I decided I could really use what I wrote via the tasks, so I forked off that branch and merged that into trunk. I then merged trunk into the other branch. So this way whatever work I done could be used. Since access control is not too important. But otherwise, I made improvements there.

Near the end of the month I started on a slight refactor to RatufaCoat and switching it to CMake. Using CLion which is a very nice IDE, it should make it easier to work on it. It took about two days to do all of this so it was a really short-lived branch. But other than that, it did pave the way for refactoring of RatufaCoat. Regardless though I took a short break on the LCDUI stuff since I have been working on it for awhile.

On the 28th, I spent a day working on a user guide and a FAQ to answer some questions. This gets synced to GitHub accordingly. Something I feel is important since people tend to ask a bunch of questions about this.

August

This month I continued work on the LCDUI refactoring. I also started a new branch for refactoring the handling of the many different virtual machines and the means for testing it. This work has been very helpful as I still use it and it is helping me progress and making sure everything is tested.

I also additionally started work on refactoring RatufaCoat which is ongoing but that is expected as I wanted to start some of it through interest despite SummerCoat itself not being ready. Of course once SummerCoat is ready I will likely work on this.

But majorly the month was spent on the build refactoring.

September

This continues the build refactoring and by the second week of the month I merged it in. It has been very beneficial. But from doing this and taking a break from the LCDUI work, I continued work on that. Pretty much the entire rest of the month was working on this.

October

This month continued the LCDUI work. At the start of the month I removed a bunch of the old code as it was no longer needed. Other than the LCDUI work I did some additional changes to the CI/CD pipelines. Progress on the LCDUI was scattered though and was generally slow to progress.

November

This month at the start was more of the same, working on the LCDUI. However after about half the month the branch was merged in. It was not 100% however it was at a level I was good with and I did not want to drag it on for more. So I took what I had and merged it in.

By the end of the month I started work on building the SummerCoat ROM, this was a combined image file and otherwise that would be needed by SummerCoat. This was mostly just to get the tasks running but not a ROM that could actually be ran. However I did merge this in.

After that, at the very end of the month I started working on Mid-Level SummerCoat. Like MLE SpringCoat, this was a refactor and simplification of SummerCoat so I could get things running again.

December

This month I continued work on Mid-Level SummerCoat, although work has been scattered. I have been making very good progress on this however, since whatever progress I make is good.

Other than that, at the end of the month I started work on doing the completion counts which would be used to determine how far I am with standard API implementations and otherwise.

Near the very end of the month, publishing was added to Maven Local, BinTray, and OSS SonaType. This would allow documentation to potentially be downloaded or browser, but will mean that SquirrelJME can be consumed by others.

2021

To be written...