Last night, a guy by the name of Peter Lemon (aka [krom]) hopped onto #messdev, pointing me at a whole suite of pretty comprehensive hardware tests that he wrote and ran on a real NTSC N64 in order for MarathonMan to improve CEN64, which to my knowledge is the only working low-level N64 emulator other than MESS.
These tests can be found here: https://github.com/PeterLemon/N64/
So far I've gotten through all of the CPU and CP1 tests, the Framebuffer and Input tests, and most of the 16BPP RDP tests. Here are the passes and failures as I remember them:
CPUTest/CPU: All pass
CPUTest/CP1/CEIL: 64-bit CEIL operations fail to properly sign-extend results out to 64 bits.
CPUTest/CP1/FLOOR: 64-bit FLOOR operations fail to properly sign-extend results out to 64 bits.
CPUTest/CP1/TRUNC: 64-bit TRUNC operations fail to properly sign-extend results out to 64 bits.
CPUTest/CP1/ROUND: 64-bit ROUND operations fail to properly sign-extend results out to 64 bits.
CPUTest/CP1/NEG: Result for 0.0 fails to return -0.0 (0x80000000), returns 0.0 (0x00000000) instead.
Framebuffer: All pass
Input: All pass
RDP/16BPP/Rectangle/FillRectangle: All pass
RDP/16BPP/Rectangle/TextureRectangle/TLUT: CopyRGBA4 and CopyRGBA8 pass, all Cycle1 variants fail to display anything.
RDP/16BPP/Rectangle/TextureRectangle: Other than the above, Copy variant passes, all Cycle1 variants fail to display anything.
RDP/16BPP/Triangle/Cube: Fail due to triangles often having unusual horizontal spikes at their minima and maxima as the cubes rotate.
RDP/16BPP/Triangle/FillTriangle: All pass
RDP/16BPP/Triangle/FillZBufferTriangle320x240: Fail due to several of the red triangles on the top row having their tips incorrectly truncated against the green triangles.
RDP/16BPP/Triangle/Plot: All pass
RDP/16BPP/Triangle/Rotate: All pass
RDP/16BPP/Triangle/TextureTriangle: All fail. Cycle1 tests fail to display anything. The tests that do display something indicate that the two triangles do not align properly across the shared edge, in the form of a visible texture seam/offset.
I'll resume testing tonight after work, but I have to say how great it is that someone finally made a comprehensive suite of one-shot tests running on actual hardware, much more comprehensive and much more useful than my initial crack at it 6-7 years ago.
I also have to say that I'm glad I was right: I already knew that SGI's internal documentation on the RSP refers to opcodes that were intended to accelerate DCT, which makes sense given that they had the chipset more or less done and then shopped it around to console manufacturers to see who would be interested, so if it ended up being put in a CD-ROM-based console, VCD decoding would be a good selling point. I've long theorized that the opcodes made it into production silicon but were simply never documented for any developers, but everyone I've talked to said that that was unlikely. In fact, [krom]'s tests have indicated that these opcodes are indeed still present in the production silicon and work as described.