The AVI format itself has a cutoff in the 1-2 GB region as well. Either use a compressing codec or a screen capture program.

Alternatively, the program's dead (we've been unable to contact the author for 3 years now) so nobody will feel bad if you use some other emulator.