Since this is somewhat of a popular question, I thought I'd make a single post with instructions for both.
How to record a movie:Open Nestopia and start the game you wish to record. Once the game is running, go to File -> Movie player -> File. Now you can name the file if you wish, but if you simply click the Browse button, you will see the file will already be named (same as rom name) and will be saved to the state folder in the Nestopia directory by default. Start the game and go back to File -> Movie player and select record. When finished, just go back and click stop.
How to export to .avi:Open Nestopia and start the game you made a recording of then go to File -> Movie player -> File and select the .nsv that corresponds to the game. Go back to the movie player and notice the export to .avi is now selectable. The conversion process will take some time so be patient. When the conversion is finished, the .avi will be found in the State folder of the Nestopia directory.
Hope this helps