2

There is a small group of us working on Amazon Web Services (AWS), we use mac books as our desktop machines and VNC into servers running on AWS. The problem started when we upgraded our VPN from PPTP to L2TP (Centos 6 with Openswan). My colleague who set it up had no problems with the new VPN. However, within a few minutes of connecting my VNC viewer sessions went dark and reported:

RFB Protocol Error: Bad hextile data or RFB Protocol Error: Bad xrle data

and other of the encoding protocols used by RFB (Remote Frame Buffer). This continued every minute or so depending on activity, with a delay while it corrected itself of up to 10 seconds. Clearly unusable. I went back to the PPTP VPN and continued without problem. Yesterday however, we connected another colleague to the L2TP VPN and he saw the same issues as me, so now we have to fix it.

The only discernible difference was that the two of us having problems with the VPN have mid-2014 15-inch Mac Book Pro with retina display (bought early 2015 with Yosemite) whereas my colleague that does not have this problem has a Mac Book Air (13-inch, Mid 2013) upgraded to Yosemite.

Searches for RFB protocol error have turned up nothing and other than that I don't even know what to search for or even ask about.

Any help appreciated.

Ross Addinall
  • 41
  • 1
  • 1
  • 4

3 Answers3

1

The most likely cause of this setting if you are using a VPN would be an MTU size set too large, causing packet fragmentation.
When this occurs some protocols will encounter problems, and is known to cause some issues with the RFB protocol.
If you have the PV AMI, then it may be worth testing if adjusting the MTU on both sides of the connection to a lower value would prevent problems.

user2392903
  • 11
  • 1
0

Not the answer you want to hear, but here are my experiences.
It's now 2021, in the middle of the covid pandemic and this was never solved on my side but our setup is the opposite: I have Macbook 2019 which was set with Screen Sharing while everyone else in the group, 10 people, all use Windows machines either at home or at the office and they all use VNC 5.0.1 to connect to that Macbook. Yeah, company scrimping on a meager Macbook and paying me much more to "keep it alive"....
I do have a Macbook at home, first, an early 2010 and now (2021) I have upgraded to a 2015 and I can tell you it's not a Mac version that does that. We had the problem before on all versions of OSX running on the "problematic" machine ( oh really ? Did I say problematic ??? No, sorry, just kidding).
I haven't had the chance to test with other VNC flavors, such as Tight VNC or Ultra VNC.
I have in my office a Windows Desktop VNC Viewer 5.0.1 and 4.6.3 - I prefer 4.6.3 for its interface but both show the same error once in a while. From home, I don't need to connect to the Mac but when I do, to revive it, I use Mac native "Connect To Server (CMD-K)" and happily it works. Usually just the fact of logging in and logging off, is enough to make it available again to other people.
My conclusion is "I have to live with that" since neither Apple or VNC will look into it, or rather, Apple makes changes (or may not change at all) and move on... while VNC will not look either .... and I'm just happy to keep that paycheck coming at the end of the month.

AlexD
  • 101
  • 1
0

After a lot of searching I found this:

Corrupted MAC on input error while logged on an Open Swan VPN #90

The symptoms of the problem are completely different, but the hardware setup (MBP with retina) was very similar. At the very bottom, the solution appears to be to move the VPN server from a PV (paravirtual) AMI (Amazon Machine Image) to an HVM (Hardware Virtual Machine ?).

I looked here:
Linux AMI Virtualization Types
and here:
www opswat com/blog/aws-2015-why-you-need-switch-pv-hvm
(moderator please re-instate link if deemed acceptable)

So, I went to find out if our AMI's were PV or HVM. You can tell by looking in the machine details on the AWS management page, there is a field 'Virtualization' which is either 'hvm' or 'paravirtual'. So I went ahead and found a Centos 6 HVM image and rebuilt the L2TP server (I used Libreswan).

Now I am glad to report that the RFB Protocol errors have gone for me and my colleague and everything is working well (for a day at least).

I don't understand the interaction between the latest MacBooks, PV/AMI instances and VNC RFB, but for now it's working.

Ross Addinall
  • 41
  • 1
  • 1
  • 4