Podman rootful containers not accessible by haproxy loadbalancer

Question

I have created two networks in podman, "backend" and "frontend".

NAME      VERSION  PLUGINS
podman    0.4.0    bridge,portmap,firewall,tuning
backend   0.4.0    bridge,portmap,firewall,dnsname
frontend  0.4.0    bridge,portmap,firewall,dnsname

I have a MS Sql Server container running in "backend" network using the following command:

podman run -d -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=TestS01Pass' --name mssqlserver -v sqlvolume:/var/opt/mssql --network backend mcr.microsoft.com/mssql/server:2019-latest

I also have three .netcore web apps (productapp1, productapp2, productapp3) which are assigned to both "backend" and "frontend" networks. Please see below the content of the dockerfile for them:

FROM mcr.microsoft.com/dotnet/core/aspnet:3.1
COPY dist /app
WORKDIR /app
EXPOSE 80/tcp
ENTRYPOINT [ "dotnet", "DockerSample.dll" ]

And these are the commands I've used to create them:

podman create --name productapp1 --network backend,frontend docker-sample
podman create --name productapp2 --network backend,frontend docker-sample
podman create --name productapp3 --network backend,frontend docker-sample

I also have an haproxy container that's assigned to the "frontend" network using the following command:

podman run -d --name loadbalancer --network frontend --volume $(pwd)/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg -p 3000:80 --privileged haproxy:latest

The configuration for haproxy is as follows:

defaults
        timeout connect 5000
        timeout client 50000
        timeout server 50000
frontend localnodes
        bind *:80
        mode http
        default_backend mvc
        stats enable
        stats uri /stats
        stats refresh 1s

backend mvc
        mode http
        balance roundrobin
        server mvc1 productapp1:80
        server mvc2 productapp2:80
        server mvc3 productapp3:80

By looking at logs for web apps, I can confirm that they're working as expected without any problem. Please see below the logs for one of the web app containers:

warn: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[60]
      Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed.
Applying Migrations...
Seed Data Not Required...
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: http://localhost:5000
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /app

The problem is when I navigate to http://localhost:3000, I receive the 503 Service Unavailable message. (No server is available to handle this request.)

I ran the following command on one of webapps and verified the port for mssqlserver is accessible:

podman exec -it productapp1 /bin/nc -zvw3 mssqlserver 1433

The result is:

DNS fwd/rev mismatch: mssqlserver != mssqlserver.dns.podman
mssqlserver [10.89.1.55] 1433 (?) open

But if I run the same command for one of web apps:

podman exec -it productapp1 /bin/nc -zvw3 productapp2 80
podman exec -it productapp1 /bin/nc -zvw3 productapp2 5000

Both returns the connection refused message:

DNS fwd/rev mismatch: productapp2 != productapp2.dns.podman
productapp2 [10.89.1.57] 80 (?) : Connection refused

DNS fwd/rev mismatch: productapp2 != productapp2.dns.podman
productapp2 [10.89.1.57] 5000 (?) : Connection refused

I wonder if anyone could shed a light on this as I've been searching and reading a lot and yet can't figure out why something this simple shouldn't work.

Really appreciated.

Thanks.

Update 1: I forgot to mention that I have tried haproxy with the following configuration too:

defaults
   timeout connect 5000
   timeout client 50000
   timeout server 50000
frontend localnodes
   bind *:80
   mode http
   default_backend mvc
   stats enable
   stats uri /stats
   stats refresh 1s
    
backend mvc
   mode http
   balance roundrobin
   server mvc1 productapp1:5000
   server mvc2 productapp2:5000
   server mvc3 productapp3:5000

Update 2: The following is the content of my launchSettings.json

{
  "iisSettings": {
    "windowsAuthentication": false,
    "anonymousAuthentication": true,
    "iisExpress": {
      "applicationUrl": "http://localhost:30113",
      "sslPort": 44371
    }
  },
  "profiles": {
    "IIS Express": {
      "commandName": "IISExpress",
      "launchBrowser": true,
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development"
      }
    },
    "DockerSample": {
      "commandName": "Project",
      "launchBrowser": true,
      "applicationUrl": "https://localhost:5001;http://localhost:5000",
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development"
      }
    }
  }
}

I also tried creating the containers with -e ASPNETCORE_URLS=http://+:5000 but still I'm getting the same error.

Update 3: Updated launchSettings.json to:

{
  "iisSettings": {
    "windowsAuthentication": false,
    "anonymousAuthentication": true,
    "iisExpress": {
      "applicationUrl": "http://localhost:30113",
      "sslPort": 44371
    }
  },
  "profiles": {
    "IIS Express": {
      "commandName": "IISExpress",
      "launchBrowser": true,
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development"
      }
    },
    "DockerSample": {
      "commandName": "Project",
      "launchBrowser": true,
      "applicationUrl": "https://+:5001;http://+:5000",
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development"
      }
    }
  }
}

Update 4: After helps from Michael Hampton, I managed to get the port 5000 open for my web app containers. The logs for my web app containers look like this now:

warn: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[60]
      Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed.
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: http://[::]:5000
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /app

I can also netcat this port from other containers:

DNS fwd/rev mismatch: productapp2 != productapp2.dns.podman
productapp2 [10.89.1.82] 5000 (?) open

And I can now navigate to my web apps as expected.

You need to check your application code. In particular you need to check that it is binding to the port you expect, and did not receive an error when attempting to do so. Use `podman logs ` to check logs, and open the code in your preferred IDE or editor to check the code itself. — Michael Hampton, Jul 09 '20 at 11:05
@MichaelHampton Thanks for your reply. As I've mentioned in my question, I have checked the logs for the web app containers and can confirm they've started properly. I have tested the web app (which is a super simple test app) and mssqlserver without assigning a network to their containers and they just work fine. I cannot access the web apps when I assign the mentioned networks to their containers. — mdx0111, Jul 09 '20 at 13:28

score 0 · Accepted Answer · answered Jul 09 '20 at 15:32

0

Your logs say the app is listening to port 5000, but you have configured haproxy to try to connect to it on port 80! This isn't going to work. Reconfigure haproxy to connect to the correct port.

        server mvc1 productapp1:5000
        server mvc2 productapp2:5000
        server mvc3 productapp3:5000

Your logs also say the web app is only listening to localhost, thus it will only accept connections from its own container, not other containers in the pod. How you fix this depends on the specifics of the app. I guess you should be looking at Properties/launchSettings.json if you are using the ASP.NET Core sample app.

answered Jul 09 '20 at 15:32

Michael Hampton

237,123
42
477
940

Sorry, I forgot to mention that I have also tried the haproxy config with port 5000. Thanks for the suggestion on launchSettings.json. I'll have a play to see if I can do anything with that. The problem is I can netcat mssqlserver container on port 1433 but can not netcat web app containers on port 5000 nor 80. – mdx0111 Jul 09 '20 at 15:51
@mdx0111 Right, because it is only listening on localhost, so you can't access it from outside its own container. This is fine if you are running the app directly on your local machine, but it's useless when you containerize it. You should have something like `"applicationUrl": "http://+:5000",` for your desired profile. – Michael Hampton Jul 09 '20 at 16:01
I checked my launchSettings.json and can confirm it has the following line: "applicationUrl": "https://localhost:5001;http://localhost:5000" I also passed -e ASPNETCORE_URLS=http://+:5000 when creating the web app containers but still I'm getting the same error. I'm sure there's something silly I'm missing cause mssqlserver works like a treat but the containers I've created don't! Your comments and help is really appreciated. – mdx0111 Jul 09 '20 at 16:14
@mdx0111 Yes, for the third time, the problem is `localhost`! Change it to a `+`. I can't tell if your environment variable overrides launchSettings.json or not. Maybe it doesn't. Check the logs to be sure, or just fix the launchSettings.json which you'll probably end up having to do anyway. – Michael Hampton Jul 09 '20 at 16:15
Ah! Stupid me! I'll change it right away! Thanks buddy! – mdx0111 Jul 09 '20 at 16:18
I'm afraid it's still the same! Eventhough I have changed the launchSetting.json, it's still saying localhost in the logs. – mdx0111 Jul 09 '20 at 16:23
@mdx0111 Hm, is it trying to use some other profile defined in there? Or maybe it's getting it somewhere else entirely. I am not that familiar with ASP.NET Core, and this isn't a programming site anyway, so you may have to seek help from somewhere more appropriate to get this changed. – Michael Hampton Jul 09 '20 at 16:42
Thanks to your helps I finally got port 5000 open on my containers! Phew! I'm still unable to hit the web apps though. :D – mdx0111 Jul 09 '20 at 17:01
Turned out it just needed a restart of my load balancer. It works now, thanks a lot @MichaelHampton – mdx0111 Jul 10 '20 at 04:41

Podman rootful containers not accessible by haproxy loadbalancer

1 Answers1