Ok, so there will be a garbage collector interface and some classes, and then another project will have the actual implementation.
One thing that will be complex though, is when threads are active and running that local variables can point to objects. Due to this, going through the call stack will be extremely complex and potentially a very expensive sequence because objects would have to be checked for each potential position in the stack to make sure that the given integers are objects.
As for native object layout and references, there will be a singular chain of positions which contain object information. There would then be a pointer to the actual object data (and array elements). Although doubly referenced this would allow all the objects to easily be traversed (because they would be in a chain) and would also allow the objects to be compacted if they are not "locked". Compaction however, could be a complex process. Before an object is accessed (such as to determine if it is an instance of a class) a flag can be set to indicate that it should not be moved elsewhere (the data pointer of the reference). When the operation completes, the flag is cleared so the object may be moved around. Actual interaction in this case is very short so the flag should only be set for short durations. Due to multi-threading however and context switching, there would need to be an atomic update to the usage count of an object before its pointer can safely be used. There would also have to be a usage count. If there is an error and the usage count is never decremented then the object may never get garbage collected, so there will have to be some kind of guard regardless of kernel/program exceptions being thrown so that the count lowers.
Back to what I was previously writing, I could have two stacks: one for objects and the other for everything else. However that would complicate memory allocation and require another set of registers. So it would be best if it were kept to a single stack. So, instead there could be a reference zone which is used to store references that are in a given position. The stack would be structured in a way where it can easily be scanned. The start of the stack would indicate its length and the number of references in the reference zone. This way, objects which are not in the reference zone do not have to be checked at all to determine if they point to objects. So due to this, any reference which is set to an object will have to be placed in the reference zone.
The garbage collector will need to have a generic object window kind of thing.
Basically it is an interface which acts as a rover over the objects which
are currently allocated. One window would go over the objects which are
associated with variables running in methods. Another window would go over the
static fields inside of a class. There would just be these two windows as that
is all that is needed. If the window reaches an Object, it would potentially
have fields. If a class has object fields, then they are scanned with the
mark and sweep algorithm. Generally any field which is not strongly reached
by a variable or static field can be garbage collected. I would have to handle
cases where a field refers to
this. In these cases the object must not be
marked, otherwise any objects could just not be garbage collected by just
referring to themselves. However cyclic references with one or more objects
will not be garbage collected unless no class static or local points to those
given objects. Another rover that would be needed would be one that can go
over arrays of objects also.
I could also probably use a kind of allocation scheme of sorts. This way there can be a completely shared interface which is used to access any and all objects that exist.
For the queue, I should make it concrete and then have associated object providers and such which are implemented by providers of objects.
When it comes to object allocation, I should have differing allocators potentially or just an interface and a standard one. I would propose that the standard and rerecording interpreters use the allocators themselves. In fact the allocators could act as access to object memory spaces. So rather than having a bunch of discrete actual objects in the interpreter, there is really just a pool that is used and such. The size of the memory pool could be fixed or it could grow along with the application. However to be friendly to other systems, the allocator would have to be able to detect system memory usage and the amount of memory that the VM is using so that GCs happen when system memory gets a bit low. This way another program for example can claim that memory. Also, if a system supports it there could even be garbage collection callbacks performed.
One thing to consider however is that when it comes to the rerecording allocator, that it must be deterministic. I would suppose then that the garbage collector that would run on the allocator would run when requested by a running program. When the allocator determines that memory is a bit shrinking and not as available, it can alert the launcher that it may be time for a garbage collection. This way in the rerecording interpreter the GC events and such can be recorded by the launcher and executed from there.
What I then need object representation access, some pointers, other ranges, and other kind of memory regions that could be used. The memory interface would need to handle cases where the layout of memory is dynamic and potentially where code and data are stored in differing regions. There would also have to be a compact and generic means to modify memory bytes, values, and such.