So, me and a user by the name of Happy are nosing around at the N64 again. He pointed out an RSP DMA bug which I've since fixed this afternoon, but interestingly enough this bug was masking another bug which causes Ocarina of Time 1.2 - and a number of ROMs using the CIC-6105- not to boot.
The reason is a clever little bit of copy protection, which I'm in the process of accommodating in the N64 driver. I'll let Happy explain it:
The code works like this.
1. CPU sets up transfer of RSP code to IMEM.
2. CPU starts RSP.
3. RSP grabs the SP SEMAPHORE. Meanwhile CPU starts PI transfer.
4. CPU continually polls the PI status until it sees that the PI is no longer busy, i.e. transfer complete. When CPU sees this it releases the SP SEMAPHORE. Meanwhile, the RSP is sitting in a polling loop waiting for the signal from SP SEMAPHORE. Each iteration of the loop decrements a counter.
5. When the RSP starts the validation task (after OS and game have started) the counter value is still there. It starts at a value of 0x200000 and is expected to end up with a value between 0x4C000 and 0x4FFFF.
It you want a look at the code for the RSP counter loop, set up the debugger to pause on the RSP on boot, ('focus 1' and then 'step') The loop is at IMEM 11E8. You can change the value after the loop finishes and see that the game boots properly.
As an interesting aside - and an unfortunate one, for performance reasons - with the current CPU interleave of 600Hz, the R4300's time slice lasts long enough that the DMA completes and it releases the semaphore before the RSP even gets to it, causing an R5 value of 0x200000, which is way out of range and causes it to stop.
I'm currently tweaking the interleave values, and with any luck something good will soon happen!