do you refer to setting the flag or to clearing it or both? iirc, the clearing part is already outside video update (it's inside a timer that get set either by the scanline or the hblank timer); the setting part is done in snes_update_objects_rto which gets called before scanline drawing and that can be moved outside video_update if necessary. if the latter, can you suggest a smarter place to move snes_update_objects_rto at?
The only valid place is inside the read/write handler itself.
which handlers? the current emulation works as follows:
at the end of each scanline we update the OAM parameters that should be used for the next line; then we pass through OAM RAM, checking which sprites will be drawn in the scanline (and setting the flags if there are too many sprites); finally we draw the scanline (both BG layers and sprites). notice that RTO flags are not reset until next frame (and indeed the flag reset happens in a timer which is called at the beginning of the frame).
how do you suggest to re-implement it without breaking sprite limits?