How to set up an automatic failover reverse proxy for two identical Icecast master servers?

Question

I'm trying to do simple everything double with a reverse proxy in the front high availability for an Icecast master streaming server (i.e. I'm not talking about Icecast relays here). So, three VMs:

• 2 identical self-contained Icecast VMs (each with a local MPD music source and a local nginx frontend for correct headers)
• a single load balancer / reverse proxy nginx VM.

My question is – how do I configure the reverse proxy to do automatic failover in case one of the Icecast VMs goes down with as little interruption for the stream client?

Illustration:

                             /--- [ local nginx A <-> icecast master A <- mpd A]
-> [nginx reverse proxy] ---<
                             \--- [ local nginx B <-> icecast master B <- mpd B]

I first tried this simple tutorial to set up a reverse proxy nginx after which I could listen to the stream by opening the nginx VM.

upstream backend  {
  ip_hash; # try to send the same clients to the same servers
  server 1.2.3.4;
  server 1.2.3.5 max_fails=1  fail_timeout=15s;
}
server {
  location / {
    proxy_pass  http://backend;
  }
}

When I stop the Icecast service on the Icecast VM that's the final endpoint, though, the client doesn't failover to the good Icecast. Not even after a refresh for some reason. I tried experimenting with various ip_hash, max_fails, fail_timeout options, different response header properties, buffer sizes etc. that other sites mentioned, but nothing worked. I feel a bit like I'm fishing in the dark here and that there should be some obvious solution for streaming failover, given the number of popular radio stations out there. Any advice about how best to set this up or some good resources? Do I want 302 redirects or a real proxy pass?

I'm open to suggestions based on HAProxy if that's a better way to go too.

Answer 1

As you discovered, reverse proxying Icecast is not a good idea. There is no real benefit, while there are several downsides and you still have a single point of failure: your frontend.

Icecast is a very stable and reliable server and typical strategies employed for serving web pages over HTTP don't necessarily apply to it.

Your effort would be probably better spent at getting to know Icecast, its configuration and limitations. That said, if properly configured Icecast will easily saturate a 1 GBit/s connection and serve well over 20.000 concurrent listeners. Tests actually indicate that it scales well beyond that, but there might be corner cases.

After that ask yourself: What problem in terms of availability am I really trying to solve and the answer to the problem will most likely not be 'reverse proxy'.