An alias to the "OpenGL Overlays" folder is not recognised in '~/Documents/MacMAME'. However, a symlink to it or an original folder does work. (i.e. the "OpenGL Overlays" popup is not populated in 0.87 unless that folder is a symlink or a real folder, not an alias.)
(Hey Brad,
I remember reading your code a while ago which was intended to handle aliases of this folder too. Maybe you accidentally used only my patch instead or your code too? If you recall, the patch I sent you only used the unix API for resolving the "OpenGl Overlays" folder, whereas your code (at least the portion you sent me) used carbon to handle an alias.)

ck