is SM64 still crashing on OSX when you try to enter the castle?
My experience has been that N64 games will randomly crash regardless of OS and regardless of what you do. I suspect either the RSP or main CPU at some point runs a little longer than the other expects, causing the other to panic.