I have never used it on a machine with so much memory! (Only time I have used more than 16GB involves a server that typically has its own built in memory checkers)
You use the memtest86+ tag, but state memtest86 in the question - I would just say make sure you are using Memtest86+ and not Memtest86 (as memtest86+ is updated, memtest86 isn't).
Hopefully you are giving it a good few minutes to start - I know when I have run it on machines with 8GB, it can sometimes look like it has frozen for up to 5 minutes before it starts - I am not sure if this effect is exponential with the more memory you put on.
After this, the reason for it not working is either down to the motherboard not being compatible or too much memory, If the motherboard is not compatible, you are out of luck - short of waiting for a new release, but if it is just too much memory, you may want to try running it with 4GB's (or the maximum it allows) at a time.
Alternatively, look at an alternate memory checker such as Microsoft diagnostics. From first looks, it seems less impressive but it works very well on all machines compatible with Windows. If you have an installed copy of Windows Vista/Windows 7 or an install disk, you have this utility.
The easiest way to get in to it is to press F8 at startup and start it from there, or look at this page which has the many other ways.
And, most importantly, enjoy your new system, it sounds impressive!
Did you get memtest from here?...http://www.memtest.org/
– Moab – 2011-01-26T23:04:14.910@Moab: You nailed it--I didn't realize I had an old version. It's running fine now. Unfortunately I can't accept a comment as the right answer. – Loren Pechtel – 2011-01-27T02:49:47.193
@ Loren, I made the comment an answer, glad it worked for you. – Moab – 2011-01-27T15:23:14.073