I'm giving up for the moment. I was never that good with signal processing anyway smile Some loose thoughts:
- as I understand it the accumulator multiplication is X (24b sample value) times Y (.12b signed fixed point)
- for the SOA tune it helps to scale Y down a bit (taken from COEF table)
- that still doesn't help all that much, and some other files still play wrong (like first Sakura Wars 4 one)
- if you scale down MIXS input, the sound will still be clipped and hiss, but won't die completly (wave looking wrong though, so it's not really about input level)

I've tried several modifications, like saturating v, v+B (note, those never get out of 26 bit range anyway). It doesn't seem to be memory read/write problem either - tested on separate table to store full ints rather then packed ones.

That particular SOA program is rather simple, all it does is take MIXS, read some memory, write some, use COEF only, writes output. No fancy addressing or latching registers. It does use MEMS and TEMP though. If you zero whole TEMP range on DSP step entry it behaves a bit like if you lowered MIXS level - sounds better but wave still going up and down and has clipped areas.
Based on this I'd say it's either ACC that is computed wrong or TEMP addressing is incorrect. I've tried many things on both, no results (like getting TEMP address just a bit wrong and you loose one or both sound channels). Perhaps there is a way to control MDEC_CT value?

I'll test few more silly ideas of mine but I don't expect any positive changes. If anyone wants to give it a try, be my guest. At this point I'm not touching DSP anymore unless something comes to me...