The 'out of memory' comes from machine.c (failure to allocate memory).

The problem can be replicated simply by doing a hard reset, it is not necessary to load another cart.

The failing command is on line 3757 of src/mame/video/n64.c, in the video_start function.

state->m_rdp->AuxBuf[1] = auto_alloc_array_clear(machine, UINT8, EXTENT_AUX_COUNT);