If you are emulating a computer system in MESS, you will need to toggle the full/partial emulation mode to be able to exit a game (it's mapped to Scroll Lock by default). If it's not working on something like the NES driver however, I have no idea; maybe doing a simple control remap in the MESS options could work for you.
Another thing which might present a slight 'problem' is using the new UI, which adds a Snes9x style menu bar at the top. IIRC, the new UI was slightly different than the MAME style interface; you might have better results using the classic MAME-like UI if you're using a MAME cabinet. This is a simple change if you want the MAME style interface, just open up .\ini\mess.ini and change newui 0 to newui 1. It's right down the bottom of my INI file. This will give you the in-game tab menu and everything else you're used to in MAME.