What techniques can I use for audio and/or video chat that don't use SIP and complex layered architectures?

Question

Point
A few years ago a friend and I were messing around and we created a super-simple video chat link by running VLC twice on both our computers: one "watched" the local webcam and cloned the video data into an RTMP stream, while the other copy tuned into the remote RTMP feed. We both made the remote window bigger than the local one and voilà, insta-video chat.

Background
It was awesome. Because the whole thing used TCP instead of UDP, we didn't see any frameskipping - we just had to deal with transmit delays, which were incredibly tiny because we could precisely configure the source video quality and transmission bitrate, and since the data was point-to-point and there were no relays involved there was virtually zero latency.

Once we'd proved that we could actually make this work we were both promptly distracted with other interesting experiments so didn't go any further, but it would have been trivial to add PuTTY in to the mix with a couple of port-redirects in there - and suddenly the whole thing would have been in an SSL tunnel.

This system was quite fiddly to use, though: if the remote link wavered once, the local copy of VLC would go "nope" and go kaput. Both of our connections were awesome so this didn't happen (IIRC), but it was always a potential occurance. And configuring everything took 30 clicks. I imagine I could have set up a decent autoconfiguration via commandline flags, but that wouldn't have fixed the connection dropping.

Question
I'm wondering what out there doesn't use SIP, doesn't require weird proxies, doesn't rely on a pile of "horribly complicated architectures and protocols you have to setup and configure because... tradition!", but does provide no-fuss video and audio transport through a thin, resource-light transport/encryption layer that's incredibly quick to setup.

If it worked on Android too that'd be great too, but I primarily want this to work on Linux.

NOTE that I said "what techniques can I use" in the title - I'm not looking for product or service recommendations but generic concepts I can apply using a choice of different tools.