These are known issues. I meant to put out an advisory about it but I've been running the betas so long I missed the release day.
-video bgfx with backend metal will give a gray screen. MAME is running (you can hear the sounds), but no video draws. It's unclear if this is BGFX's fault or if SDL isn't properly setting up the view.
I'm not sure if this is due to Apple breakage or a bug in bgfx. Updating to latest bx/bgfx/bimg causes the problem to change to a deadlock instead. This deadlock can be mitigated by defining BGFX_CONFIG_MULTITHREADED=0. See https://github.com/bkaradzic/bgfx/issues/2036
for more details and a different workaround.
-video bgfx with backend OpenGL crashes due to a Clang compiler bug. I know this because the MAME subset that comes with the AMPLE Apple II-only frontend for macOS is built with an older compiler and BGFX with OpenGL works on Big Sur in that version.
This is not a Clang compiler bug. The likely reason that the version built with an older compiler works, is because it is also built with an older SDK, and Apple is likely using some workaround meant for older programs that cannot be patched and rebuilt.
As the crash log indicates, the crash happens because bgfx is trying to set up OpenGL on a thread that is not the main thread. macOS no longer allows this. This appears to be low priority for bgfx: see the comments at https://github.com/bkaradzic/bgfx/pull/2019
— this specific PR was immediately reverted. Setting BGFX_CONFIG_MULTITHREADED=0 mitigates this problem as well.
EDIT: I can confirm that setting BGFX_CONFIG_MULTITHREADED=0 (as a define in 3rdparty.lua) fixes the problem in the version of bgfx shipped with MAME.