Update: issue seems to be related to the Track 0 signal, used by the upd765 code to check if a recalibrate function is successful. The failure occurs on the second recalibrate function, so it works the first time, but not the second, even though both the FDC and drive say the current track is 0.
Changing line 309 of machine/upd765.c to:
if (fdc->pcn[fdc->drive] != 0)
fixes the x68k, however, I think the real issue is more with flopdrv.c somewhere.