Fixing a broken NTVDM

Question

_Problem:

About a week ago, the NTVDM on an XP installation broke.

_Symptoms:

The first symptom was that a 16-bit executable that is run during an auto-run batch file would hang and peg the CPU at 100% until ntvdm.exe is killed, at which point the batch file continues running (albeit without the work that the executable was supposed to have done). The next symptom was that a different 16-bit executable that is run as part of a shutdown batch script would hang, causing the script to fail.

Interestingly, they do sometimes work (~30%). In other words, if a program is run (from the Run dialog) 10 times, it might run correctly about 2-4 times and hang the rest of the time. This is mostly just with simpler programs that print something out or such as opposed to more complicated ones with UIs like edit.com or command.com which seem to always hang. A few select programs that are exceedingly simple (e.g., crlf.com: 578 bytes, ret.com: 1 byte, etc.) always seem to run and return without problem.

_Tests:

Some tests show that the problem is with the NTVDM itself and not just the specific executables (which worked just fine until recently anyway). For example, running edit.com opens a console and displays the editor, but is unresponsive to the keyboard and must be killed. In fact, even running command.com won’t work; it opens a console and shows the prompt then just sits there, unresponsive.

I checked all the related NTVDM files (and numerous other OS files) and they are all correct. I also checked several related registry entires and they all check out as well. The programs run fine on other XP installations (which is not a surprise since command.com and edit.com work as well); so it is that specific machine. The environment has not changed since the NTVDM stopped working, but I checked it anyway and there is nothing overtly problematic; temp and tmp point to existing directories with no LFN names, no broken variables with nulls or such, and while the path is long, it has been such for some time.

Files verified: ntd*, ntv*, dos*, redir.exe, Mscdexnt.exe, *.sys

Registry entires verified: *\Software\Microsoft\Command processor, *ControlSet*\Control\WoW, *\Software\Microsoft\Windows NT\CurrentVersion\WOW*

The NTVDM and 16-bit apps work just fine in safe-mode on the machine in question; it is only in normal mode that they hang.

_Question:

Can anybody think of anything that could break the NTVDM in a manner that produces these symptoms?

(To be clear; it is NOT the common error “The NTVDM CPU has encountered an illegal instruction.” Also, please don’t suggest using Dosbox or other VM software because that is not applicable, and moreover, the programs in question worked before, not to mention that Windows’ own 16-bit programs won’t work.)