After a day of working with this I'm impressed.
The emulation gets almost all of the issues with using a 9600 baud connection on a real Rainbow correct (it's easy to overrun it at that rate). I'd hoped I could set the baud rate to 19200 and get 1920cps transfers, but no matter the baud rate I'm maxed out at ~(speedup * 170)cps. Real rainbow hardware has similar performance, at least when transferring to Venix (I max out at around 180-185cps).
You emulate the bad baud rate stuff as well. When Venix is running its default getty at '7' is is using 2400 baud. The default serial rate is 9600 for MAME, so you get garbage characters, exactly like you would when you do this IRL.
I get several interrupts off messages. More than I get on real hardware, but in similar circumstances. I think there's a race or two left in the Venix winnie or ca drivers that we're faithfully emulating.
Venix still has all the annoying issues of being a V7 port coupled with the unforgiving (by modern standards) error paths in the installation.
So color me impressed. I wish there was a way to "improve" on some of these issues, but I can't think of any that don't reduce the fidelity of the emulation. I may implement a deep invisible rx and tx FIFO for the upd7201 though and see if never dropping a character improves the speed, or reveals more bugs.