A few years later, the "world has changed", so
The free version of RealVNC (v. 6.17.1113, Nov 2017, x64) works smoothly for me. In particular from Windows 7 Professional to OS X Yosemite (10.11) and to OS X High Sierra (10.13 w/ latest updates), the "OS X Authentication" that Mac OS builtin VNC requires, with username and password and no tweaks to the VNC server side, posed no problem. These are security types 30 and 35: see https://vncdotool.readthedocs.io/en/0.8.0/rfbproto.html#security
Although performance was clearly lacking. It's using ZRLE compression; perhaps JPEG or something else is superior, and perhaps professional RealVNC is optimized.
TightVNC and TigerVNC both fail complaining those are unknown codes.
[And do everything over a VPN for heaven's sake if going through the Internet.]