The board is actually 25MHz, and after correcting that and trying to find the factor of error to get the right table entry, I find that a factor of 5 identifies a 20MHz board and a factor of 4 a 36MHz board, so it isn't just a simple divider or two that needs to be corrected.
I believe that the factor is 4, 2 from the counter and 2 from the prescaler, where I count both raising and falling edges currently, does that makes sense? However, if that is correct I am still off by some meassure. Here is the table:
As you can see the difference between the 25MHz I want to get and the 20MHz I get is only 0x200 shifted 4 so times 16 = 0x2000 which is far from the 20% I would expect. How does that work? Any hint in the right direction appreciated, even smaller insults.