Make a mame.ini by using this command. mame64 -cc
Then paste these in there over the old settings.
#
# OSD VIDEO OPTIONS
#
video d3d
#
# DIRECT3D POST-PROCESSING OPTIONS
#
hlsl_enable 1
If you want rounded corners. I using these below.
distort_corner 0.02
round_corner 0.10
smooth_border 0.01
Just 0.0 those out. If you don't any of this showing. Should look like this.
distort_corner 0.0
round_corner 0.0
smooth_border 0.0
You will get a hang of the settings. Take a while.