You should be able to boot vbswx1.flp. It's a copy of the 'recovery' disk that's been enhanced to recognize any winchester drive size. The vxfer*flp disks will work too, but are the simple version and only recognize an RD51 10MB drive (or at least, it assumes it's a RD51, regardless of what it really is).
http://www.vintagecomputer.net/digital/VENIX/VENIX_Install.pdf has basics. The BSW stuff is a little different, but not that different. Mostly, it requests more disks at the end of the process to load the enhancements from BSW. I think that it's decent. I have no power right now, but if it's not so good, I'll walk through things on MESS and post screen shots.
http://www.vintagecomputer.net/browse_thread_record.cfm?id=675&tid=5 has some more details as well, though the xhomer project linked from there seems to be off the air.
I'd love to have a faster MAME/MESS rainbow running as well.
Hmmm, I'll have to run through what I've done in the past... there's a step to create the raw image, wutil it with a venix partition, marking it bootable, then booting venix restore...
../git/mame/mame64 rainbow -flop1 ~/git/venix/dist/rb/vbswx1.flp
gives me a screen that asks if I want to format the Winchester Drive then gives me an error because I don't have a hard dive connected.... I'll try to write more later.