WAV is the only format which MAME currently supports for saving cassette images. This is partly because MAME emulates cassette tape interfaces on a lower level than many other emulators. Decoding data from the serial bitstreams MAME is designed to output would be difficult at best, though it would likely be more tractable for CPU-to-tape-drive interfaces based on USARTs and such than purely bitbanged interfaces such as the ZX Spectrum ULA which provide a much wider latitude for highly nonstandard formats.