You can do it in MacMAMEinfoX - go to Preferences, then click "Options" button, select the "Video 2" tab, and then choose between "Full Screen" and "Windowed" with the radio buttons.