This is very helpful, thanks! Looks like a driver issue indeed. For comparison this is what I am getting on my another machine (Ryzen 5 5600x, NVIDIA RTX 2070). X.org only as there is no Wayland for nvidia yet:
Code
$ mame -video bgfx -bgfx_backend vulkan -bgfx_screen_chains default -window -str 90 -nothrottle -sound none -noautosave umk3
Average speed: 928.13% (89 seconds)
$ mame -video bgfx -bgfx_backend vulkan -bgfx_screen_chains default -nowindow -str 90 -nothrottle -sound none -noautosave umk3
Average speed: 900.49% (89 seconds)
$ mame -video bgfx -bgfx_backend vulkan -bgfx_screen_chains hlsl -window -str 90 -nothrottle -sound none -noautosave umk3
Average speed: 917.46% (89 seconds)
$ mame -video bgfx -bgfx_backend vulkan -bgfx_screen_chains hlsl -nowindow -str 90 -nothrottle -sound none -noautosave umk3
Average speed: 843.10% (89 seconds)
$ mame -video bgfx -bgfx_backend opengl -bgfx_screen_chains default -window -str 90 -nothrottle -sound none -noautosave umk3
Average speed: 933.27% (89 seconds)
$ mame -video bgfx -bgfx_backend opengl -bgfx_screen_chains default -nowindow -str 90 -nothrottle -sound none -noautosave umk3
Average speed: 898.42% (89 seconds)
$ mame -video bgfx -bgfx_backend opengl -bgfx_screen_chains hlsl -window -str 90 -nothrottle -sound none -noautosave umk3
Average speed: 920.90% (89 seconds)
$ mame -video bgfx -bgfx_backend opengl -bgfx_screen_chains hlsl -nowindow -str 90 -nothrottle -sound none -noautosave umk3
Average speed: 890.63% (89 seconds)
Vulkan and OpenGL are within a margin of error from each other, and there is no significant cost to either hlsl chain or to going fullscreen. I guess the AMD mesa drivers have still some way to go.