xinyingho, if you obtain a satisfactory result, could you share a screenshot of a Game Gear driver using HLSL and running a game that uses SMS compatibility mode? There are few GG games using that mode, or you can test with SMS games on a GG driver using the Master Gear (mgear) adapter, like done by the following example:
mame gamegear -cart mgear -cart2 roms/sms/altbeast.zip
Note: currently, the mgear subslot (cart2) requires a direct path to the ROM file. I sent a minor patch to etabeta sometime ago that adds software list support to the device, but while he doesn't find free time for MAME we still need to provide the path.
In SMS mode, the GG LCD display scales the image through subpixel rendering. The driver has a scaling algorithm implemented, but the result is somewhat off the mark. I think this is caused by the pixel-level rendering of the driver when HLSL is not used. I'm curious to see if HLSL will look more faithful to the real image produced by the GG LCD display. For reference, the following link points to a comparison image of how a SMS game looks on SMS and GG:http://retrorgb.com/images/SMSonGG02.jpg
(from webpage http://retrorgb.com/smsongg.html