0

About my setup:

  1. I've got an HAProxy configuration for my WordPress environments.
  2. I've got multiple server pools for the "front-end" of each WordPress website.
  3. I force everyone to connect to one specific server for anything in the "back-end" of each WordPress website. For now... This just makes things simple for file syncing and database replication.

My problem:

My clients are uploading massive amounts of photos to their websites in one big bulk upload. Most of the time, the files are uploading as expected. However, randomly and sporadically some of the photos are failing to upload with a fairly generic error code in WordPress of "HTTP Error".

The photos which fail to upload are never the same photo file. It's completely random in each case.

I've been trouble shooting this problem for about a week or so now, and can't seem to hunt down the problem.

  1. I've verified the DNS is correct and functioning as intended. The user is resolving to the HAProxy server, and the connection is forwarded to the appropriate server hosting the "back-end" of the WordPress website.
  2. I've thoroughly checked Apache, PHP, and MySQL configurations on the server hosting the "back-end" of each WordPress website.
  3. I've checked out everything related to WordPress from head to toe.

The only thing I can think of at this point is that there's a problem with my HAProxy configuration. I feel like there's too many requests being made too quickly, and the HAProxy Server isn't keeping up with the communication to and from the server hosting the "back-end" of the WordPress website (where the bulk uploads are taking place).

I'd like to solve this problem related to HAProxy. If you feel like my configuration could be simplified or improved upon elsewhere (unrelated to this particular problem), I'm open for improvements overall with my current HAProxy configuration, but the main focus is on this "HTTP Error" problem related to bulk uploading of files.

Here's my HAProxy 1.6.9 configuration:

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http
    retries 3
    option redispatch
    maxconn 2000
    timeout connect 5000
    timeout check 5000
    timeout client 300000
    timeout server 300000

frontend http-in
    bind *:80
    option  httplog
    option http-server-close

    acl has_domain hdr(host) -m found
    acl has_www hdr_beg(host) -i www.

    acl has_admin path_beg /wp-admin
    acl has_login path_beg /wp-login.php
    acl has_custom_login path_beg /manage

    acl has_server1 hdr_beg(host) -i server1.

    use_backend admin_servers if has_domain has_www has_admin or has_domain has_www has_login or has_domain has_www has_custom_login

    use_backend live_servers if has_domain !has_admin !has_login !has_custom_login or has_www !has_admin !has_login !has_custom_login

    use_backend default_servers if has_server1

    default_backend default_servers


backend default_servers
    mode http
    stats enable
    stats uri /haproxy?stats
    balance roundrobin
    option httpclose
    option forwardfor
    cookie SERVERID insert indirect nocache
    server server1 1.1.1.1:80 check cookie server1

backend admin_servers
    mode http
    stats enable
    stats uri /haproxy?stats
    balance roundrobin
    option httpclose
    option forwardfor
    cookie SERVERID insert indirect nocache
    server server1 1.1.1.1:80 check cookie server1

backend live_servers
    mode http
    stats enable
    stats uri /haproxy?stats
    balance roundrobin
    option httpclose
    option forwardfor
    cookie SERVERID insert indirect nocache
    server server1 1.1.1.1:80 check cookie server1 weight 200
    server server2 2.2.2.2:80 check cookie server2 weight 25 maxconn 256
Michael Ecklund
  • 251
  • 2
  • 5
  • 13
  • probably you need to have sticky sessions in your haproxy - you are balancing round robin and this could cause clients to move server during http put, if you clients are coming from different sites then you could balance based on there source address 'balance source' - there are lots of options – Sum1sAdmin Oct 19 '16 at 17:00
  • @Sum1sAdmin Thanks for the tip. It does make sense to make that adjustment. However, it still didn't improve or solve the problem. – Michael Ecklund Oct 20 '16 at 14:49

1 Answers1

0

After a TON of digging... I've solved the problem. Turns out the problem here wasn't related to HAProxy.

As mentioned in the question. There were HTTP Errors (Internal Server Error - 500) while uploading files to a WordPress website. The error was always random and never consistent.

These WordPress websites were in a WordPress Multisite configuration.

The problem was an related to HTTP. So there's something which WordPress and my web server (Apache) both relied on, that wasn't working properly. The only common link that came to mind was the .htaccess file.

WordPress Multisite has two network modes:

  1. Subdirectory Installtion
  2. Subdomain Installation

Depending on which network mode you configure WordPress Multisite to use, it requires a different set of rewrite rules for your .htaccess file.

My WordPress Multisite was configured as a "subdirectory" installation.

I replaced the rewrite rules for a "subdirectory" installation with the appropriate rewrite rules for a "subdomain" installation. I then also updated wp-config.php to change define('SUBDOMAIN_INSTALL', false); to this define('SUBDOMAIN_INSTALL', true);

The HTTP Error on file uploads had then completely vanished and bulk uploads began to upload at a 100% success rate once again.

Michael Ecklund
  • 251
  • 2
  • 5
  • 13