Make a mame.ini by using this command. mame64 -cc
Then paste these in there over the old settings.
# OSD VIDEO OPTIONS
# DIRECT3D POST-PROCESSING OPTIONS
If you want rounded corners. I using these below.
Just 0.0 those out. If you don't any of this showing. Should look like this.
You will get a hang of the settings. Take a while.