First, this new answers are all very interesting and may are the most accurate way i can think of, to reach proper simulation of vector monitors. Sadly this is beyond of any possibilty to help you or others from my side

.
Long before i did start this thread, i had a nice discussion on a arcade-convention with guys that owned the originals at their homes and man-caves. We discussed what MAME could do, to improve the appearance of vector-games. Obviously the first thing, was the mimic of the beam and the phosphors, but it also was the first thing what we know, it would be very difficult to do, till the point "maybe impossible with satisfying results". So we focused on other things, that maybe could be done, with less effort.
These things came out:
1. seeing the points from drawn lines. (you could do this even with old HLSL, but the picture would be very dark, with unnatural thin lines) it would look like this:
With the latest changes from Jezze, it is possible to show the points more clearly and with enough brightness. I now have more the problem, to show them colored for colored games (i.e. a bright red point for a red line). Everything is fine with b/w games.
2. A glow that doesnt look to "artificial". You can see a good example of what i mean, if you look at the deathstar explosion of Starwars. The glow should have a circle/donut form, but looks more hexagonal instead.
3. Some "wobble/shake" filter to mimic the "stable"

screen of vector games. This gets really extreme on the cheap hardware of a Vectrex. Just watch this video at 15min. :
https://www.youtube.com/watch?v=12juB-ySTWoSee how the whole screen/playfield is shaking

? It gets even worse with Clean Sweep. Same stuff happens with the arcade stuff, but way less and not to that degree.
I am not sure, but such a filter could maybe also help the phosphor filter. It seems that the phosphor filter applies only to objects in motion (ok, obvious), but it has downsides for some cases. If you look at the scrolling text in StarWars and use some higher phosphor settings (0.60-0.80), you will see that the phosphor effect turns off, at the moment where the text comes to halt. This looks very strange and i guess it wouldnt look so strange, if we would have a wobble/shake filter, as the text would be still in "little" motion and the phosphor filter wouldnt stop working abruptly.
StarWars Logo with no motion:
StarWars Logo, with motion:
All the images here, are already scaled down and not full quality (i.e. you barely see the shadowmask etc.), but i hope they are good enough, to explain the examples properly

.