Anyone who felt that the scanlines and shadow mask seeemed a bit "off" was right - the scanlines did not line up to the individual pixels. Additionally, I was bilinear filtering the texture during multiple stages rather than point sampling until the last possible moment. Sync up to the MESS SVN tree and copy the shaders into your MAME hlsl directory for goodness.
The upshot of this is that you can really crank the shadow mask and scanline settings without inducing anywhere near as much rainbow moiré as before.
Don't believe me? Here's a before (left) and after (right) with the same settings, no oversampling, no prescale, full alpha on scanlines and shadow mask: http://icips.us/emulation/crt/scanmask_before_after.png