First of all, the equation is wrong. If it was like (1.0 - (f + q)) it'd make more sense, but that's still no good. Most of the result comes from f * in part - if the f is too big it won't do much filtering. If it's too small it will all but mute that channel. Since it's only 13 bits I think that ((filtervalue>>8)^0x1F) is a bit of overkill (shifts up to 31 bits).
(1.0 - f + q) looks right to me. It produces a stable lowpass filter for all values of f and q between 0.0 and 1.0. (1.0 - (f + q)) has more of a bandpass characteristic.
As for the ((filtervalue>>8)^0x1F) part, it looks a bit weird to me, too - causes more than half the filter values to produce filters w/ zero (-infinity dB) gain.
Originally Posted by Deunan Knute
In short: I'm lost. I've said it before that signal processing is not my forte - I don't really get the math behind IIR filters. I've tried some random changes, alas only ended up with unstable vibrato or noise generators Any ideas?
Not sure exactly what's going wrong. Are you scaling the f the q values properly (to account for their fixed-point representation)?