I'll agree, it looks suspect. It parses sprites once backwards rather than checking twice.
anomie's doc indicates that sprite stuff occurs in two phases. First is a "left to right" scan starting at 0 or "First Sprite", depending on Priority Rotation. This makes the short-list of 32 sprites. This is done during the previous scanline's display period (scanline 0 is not rendered, but rather caches the OBJ data)
Then, during h-blanks, the 32 sprite array is cached backwards (31 ->0), and 34 tiles are grabbed, left-to-right within a sprite (after flipping).
Totally unlike what I see in MESS. Good call. MESS only does RTO calculation when sprites are enabled. Check the rendering state at the time of the RTO test.