Well, that will change when I release Shinobi X (if/when I decide to do so). Still doesn't make much of a difference since the FM handling didn't really get worse than what was there before, it just does it wrong in a different way.
I've examined FM a bit more while working with Shinobi X, and while I wasn't able to make any significant progress on getting it to sound correct, I was able to get MDXSL and MDYSL working properly. I've verified that these are correct by comparing their values with the fm_layer and fm_gen values given by ssfinfo and they are consistent with the tables in the SCSP manual. To achieve this, I changed the FM ring buffer pointer to decrement rather than increment as well as making it update regardless of whether a slot is active or not.
While investigating FM issues, I uncovered a few more (relatively minor) problems. I fixed a small LPSLNK problem (wasn't handling this quite correctly before). Also, I made a change to the DSP input mix level, which improves the effect balance. Previously, DSP output was a bit quiet compared to the direct output. I resolved this by shifting left what was the previous input by 2. I believe this is correct since the LPANTABLE/RPANTABLE calculations contain a multiplier of 4.0, and an additional left shift of 2 is in line with the fact that the MIXS input is 20 bits.
Below is a upgraded patch to AOSDK 1.1.6. I've commented out the FM-synth stuff because it isn't correct yet and right now Shinobi X sounds better with it disabled.
Changes: -Corrected handling of the FM ring-buffer pointer - MDXSL and MDYSL now handled correctly -Incorporated MDL into modulation code (not yet handled correctly, but it's something to start with) -Changed frequency-step calculation to use UINT64 to fix rollover problems caused by large OCT values -Improved DSP mix (I believe it's correct now) -Fixed a minor LPSLNK problem