Why Win16 application layout is behaving strangely under Win10 32bit?

Question

I have an old C Win16 application that is behaving badly when run in Windows 10 32bit. Unfortunately, I do not have the source code for the application so I cannot debug issues from there. The application was working normally in Windows 7 but in Windows 10, it starts by using NTVDM but for some reason the GUI has the following issues:

Issue 1. The application opens a popup dialog when it starts. Mouse interaction works so that I can drag the popup dialog around the application window but the popup itself has no focus and I cannot click any of the buttons or input text to the dialog. I can use still Alt+F to access File-menu in the top application bar while the popup is open.

Issue 2. The application can also be configured to open without the popup window. In this case the top menu-bar and status bars are displayed out of place and do not respond to any input from the keyboard.

In both cases, clicking on the main application window buttons does nothing and I need to use task manager to kill ntvdm.exe to close the application.

So far, I have tried different compatibility mode settings as well as trying to run the application with an alternative VDM (WineVDM). Compatibility mode does nothing and the application launches using WineVDM but has same layout issues that using NTVDM has. This seems to indicate that the problem is not in VDM but has more to do with legacy graphics rendering differences between Win7 and Win10.

What should I try next? Is there a way to identify which GUI framework was used?

Edit: I'm currently running this in a virtual machine "Win10 32bit" inside a 64bit Windows 10 machine and the display adapter is currently "Microsoft Hyper-V Video". Tried changing this to "Microsoft Basic Display Adapter" but that didn't help.

Edit2: Read somewhere that Win10 creators update broke GDI applications so tested with an earlier version of windows (1607). So the question actually changes to "how can I get GDI applications to run correctly after creators update".

Answer 1

I didn't find a way to fix the layout issues but furtunately was able to go around it.

At some point during the study, I found out that people had had issues with GDI rendered applications after the 2017 Windows 10 Creators Update (version 1703). Font rendering performance was in some ways affected and people had other issues as well. Further study revealed that in the version 1703, Microsoft made some large changes to the Desktop Window Manager (DWM) that controls how the applications draw to the screen. With DWM, the applications draw their contents into a separate memory-buffer for each application and that is used e.g. to display the real-time toolbar thumbnails in modern Windows applications. The old way is for the application to draw directly into the screen buffer. Update 1703 made it impossible to disable DWM for any application and deprecated parts of the API that is used for controlling it separately for each application.

I tested the problematic application with Windows 10 Anniversary Update (version 1607) and it worked! The hair-pulling inducing layout issues were gone and the application ran like a charm. The problem of course is that version 1607 of Windows 10 has even shorter support window than the latest version of Windows 7. Fortunately, Microsoft gives an option to use long-term support branches for its Windows Enterprise editions and those reflect the status of the “normal” release from the same year. I tested with Windows 10 Enterprise 2016 LTSB edition that has extended support until 2026 – and the application worked!

Case closed for now.