I can't get access to any HLSL filters. Stuff like screen curvature and vector. It said that I need PS3.0...whatever that means. I've tried installing directx9...multiple times...but nothing works. Oh, well.
Something something shaders for pixels version 3. Clearly that means your PC's video card does not contain the first model PlayStation 3's hardware (PS3.0 with PS2 Emotion Engine and Graphics Synthesiser chips) in order for HLSL to work.
Okay, but to actually be somewhat helpful, OP, your hardware's too old. By far.
Depends. Maybe running with -verbose and pasting the log would give enough info to actually help. Or OP describing the hardware that they have.
Yeah, we don't know what MAME version, what hardware, what OS, or anything.