I've been lurking here for a while and I have to say this is awesome progress.
Perhaps you know this already based on implementation of the XMAP, but most SGI systems had WID planes that allow per-window bit depth and color map selection. Think of it as a per-pixel MUX for interpreting the contents of the 24-bit color buffer. This is something unique to SGI that dates back to the earliest machines where you could trade color depth for interaction (24-bit single-buffered vs 8 or 12-bit double-buffered). Or it could be used to provide a 4k color index mode.
I suspect the above issues may be with interpretation of the bits in the buffer. For example, borders were often rendered in the native depth for the root window, but the content could be in another mode based on the WID bits. In this case, the inner window is probably being interpreted with the wrong bit depth or the wrong color map entries.
There was a tool called 'mag' let you explore the frame buffer to see what's what. It would be interesting to see what it thinks those windows are.
One other thing... some systems like the Indigo implemented WID planes in an RLE list to save memory. It was only 64kx4, so it was possible to overwhelm it by doing things like allocating round windows.
