I still haven't nailed down the crash. I *think* it's a bitmap boundary issue but then, if I fix it then the right screen doesn't like it.

This:
http://goliathindustries.com/vb/VBDiagrams.html
and iFixit has a nice teardown here:
http://www.ifixit.com/Teardown/Nintendo-Virtual-Boy-Teardown/3540/1

gives these components:
Nintendo VUE-VPU, 160pin
Nintendo NVC-VUE, NEC, 100pin
Nintendo VRM-VUE, 70pin
Nintendo VSU-VUE ATT, 28pin
Toshiba TC511664BJ-80 128 KB of DRAM, 40pin 1mByte D-Ram (EDO?) 1mb x 8
Toshiba TC511632FL-70 / )(Link LH5P1632N-70Y, 40pin 512kByte PSram 512k x 8

So:
Nintendo VUE-VPU, 160pin - Video Processor?
Nintendo NVC-VUE, NEC, 100pin - CPU?
Nintendo VRM-VUE, 70pin - VRAM?
Nintendo VSU-VUE ATT, 28pin - Audio Processor?

Yes?

As said elsewhere:

Given typical Nintendo patterns of the time, it's possible audio is inside the CPU and at least 2 of the other chips are video (SNES had separate sprite and tilemap chips, similar to the Saturn, but glued together better).

VRM-VUE - vram ?

Now I got this: http://code.google.com/p/devkitvb/
stating: NVC (a customized version of NEC v810)
... so the 100-pin chip is the CPU? Weird.

Since there are a bunch of undocumented opcodes in this one, it's possible that the V810 is actually a custom one, just like the Sufami main CPU is actually.

given that the part number above has 'NEC' in the name of the 100 pin chip, and we know it's based off the NEC v810, I don't see why that's so weird ;-)

given that the part number above has 'NEC' in the name of the 100 pin chip, and we know it's based off the NEC v810, I don't see why that's so weird ;-)

Given that the NES, GB(C) and SNES CPUs are all custom parts (especially the GB, which aside from executing mostly the same instruction set has almost nothing in common with a stock Z80) it's not surprising that the VB CPU is as well.

Preliminary Affine Mode implemented: