How to figure out what is bad about a 400 bad request, on an Apache-server

Question

The overarching question

How do I see what is 'bad' about a 400 - bad request?

Info about the error

When I click around the WordPress-backend, then between 3 and 7 requests (out of 95-100) give me a: 400 - Bad request-error, upon every page load. But it varies which request that fails, from refresh to refresh?!
I've been battling this issue for 8 hours (ish). I feel like I've tried everything.
It happens both in Chrome and Firefox
It doesn't happen on another computer with an identical setup.
It's mainly static files, where the requests fail (but not exclusively).
There are 10-15 other developers also working on it, they haven't seen it either (same branch, almost identical database, same Vagrant setup).

System info

WordPress-installation
Apache-server
Running locally on a Vagrant box (Ubuntu VirtualBox)

Solution attempt 1: See tendencies between which files it fails to load

I can't see any tendencies there. Here are 4 random refreshes.

Refresh1

api-client-1740.js
backbone.min.js
styleGuide-1740.js
escape-html.js
admin-script.min.js
quicktags.js
admin-bar.js
hooks.js
debug-bar-js.dev.js
font-awesome.css
nav-menus.css
list-tables.css

Refresh2

nav-menus.css
about.css
admin-menu.css

Refresh3

tags-suggest.js
i18n.js
wp-polyfill.js
themes.css
list-tables.css

Refresh4

"Image loadingAnimation.gif"
flame.png
api-client-1740.js
element.min.js
admin-script.min.js
moment.min.js
mouse.min.js
lodash.min.js
underscore.min.js
hoverintent-js.min.js
adminbar-1740.css
admin-global-1740.css

And sometimes none of the requests fails.

Solution attempt 2: Inspect the request in the network-tab

I can find a reason for the 400 Bad request in there. Here is an example of a request that returned 400 Bad request (where I've redacted sensitive info).

Request

GET /wp/wp-admin/load-styles.php?c=0&dir=ltr&load%5Bchunk_0%5D=dashicons,admin-bar,site-health,common,forms,admin-menu,dashboard,list-tables,edit,revisions,media,themes,about,nav-menus,wp-poi&load%5Bchunk_1%5D=nter,widgets,site-icon,l10n,buttons,wp-auth-check&ver=5.8.3 HTTP/1.1
Host: mydomain.test
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0
Accept: text/css,*/*;q=0.1
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://mydomain.test/wp/wp-admin/
DNT: 1
Connection: keep-alive
Cookie: wordpress_a8d2d12312312379974dc3f916b356d3=MYWORDPRESSUSERNAME%7C1612378456%7C8xaOuvvv7N7Aq4PzHreiIfjKNoRt6NInOxbwvWW9nYL%7Cc70240d7f46d0dc5b6112554da95f69cee61180ebb2b3a93326cb42d02937f3f; wordpress_test_cookie=WP%20Cookie%20check; wordpress_logged_in_a8d2dfda29fb0e79974dc3f916b356d3=MYWORDPRESSUSERNAME%7C1612378456%7C8xaOuvvv7N7Aq4PzHreiIfjKNoRt6NInOxbwvWW9nYL%7Cd16311d6028a3550a8777754a99a1e2d09bffb05f45f36854038387bc0fdce43; wp-settings-2=posts_list_mode%3Dlist%26ampampampamplibraryContent%3Dbrowse%26ampampampampeditor%3Dtinymce%26ampampampampmfold%3Do%26libraryContent%3Dbrowse; wp-settings-time-2=1652706157; CookieConsent={stamp:%27bs7eydGl1MNS0O2BJZiPnLkz4NSuHl+9jWJceT1L8gnBXxa3B+MlsA==%27%2Cnecessary:true%2Cpreferences:true%2Cstatistics:true%2Cmarketing:true%2Cver:1%2Cutc:1652701927276%2Ciab2:%27CPZD8gAPZD8gACGABBENCPCsAP_AAH_AAAAAIwtd_X__bX9j-_5_bft0eY1P9_r3_-QzjhfNs-8F3L_W_L0Xw2E7NF36pq4KuR4Eu3LBIQNlHMHUTUmwaokVrzHsak2cpyNKJ7LEknMZO2dYGH9Pn9lDuYKY7_5___bx3D-v_t_-39T378Xf3_d5_2_--vCfV599jbn9fV_7_9nP___9v-_8_________gjAASYal5AF2JY4MmkaRQogRhWEhVAoAKKAYWiKwAcHBTsrAJdQQsAEAqAjAiBBiCjBgEAAgEASERASAFggEQBEAgABAAiAQgAImAQWAFgYBAAKAaFiAFAAIEhBkQERymBARIlFBLZWIJQV7GmEAdZYAUCiMioAESAAAkBASFg5jgCQEuFkgSYoXyAEYIUAAAAA.YAAAAAAAAAAA%27%2Cgacm:%271~AAAAAAAAAAACAABAAAgAIAAABAAhAAAACAAAAAAAQAAQAAAAAAABBBAAIAAAAAAAAAAAAQAAAIBAAAAAIgMAAAAAAAgAAAACAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAgAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAQAAAAAAFAAAABAAAAAAAAAAAARBAAAAAAAAAACAAABAAAAAAAAAAEAAAAAAABAAAAAEAAAAAAAAAAAAAAAAAAACBAAAAAAAAAAAAQAAAAAAAAAAgAAAAAAAQAAAAAAAAAAAAAAAACAgBAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAAAAAAAAAAAAAAAAAAAgAAAAAAAEAAAAAAAQAAgAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAgAAAAAAkQAAAAAAAAAAAAAAAAQ=%27%2Cregion:%27dk%27}; modal_page_counter_cookie=1; utag_main=v_id:0180ccb62e2b002302377014532c05054001000f00fbe$_fd:2$_se:1$_ss:1$_st:1652709110390$dc_visit:2$ses_id:1652707310390%3Bexp-session$_pn:1%3Bexp-session$dc_event:1%3Bexp-session; user_obj=eyJ0b2tlbiI6ImQ1YWFhYWI5NzQyZTc2M2FlZjEyYzhjNDBhNGE3NWRhODA3MmZhOTc4MDNkZDdhMWJkNmRlNzYzODRhMjM4MDgiLCJzdWJzY3JpcHRpb25fYXJlYXMiOnsiMCI6IjlkYjk2OTcxLTFiNmItNDM0NC05OWFiLTg5YTVjM2NkYmU5MSIsIjNhMGYxODU3LWMzOTMtNGI4Ny1hNjIwLTE4YzEyMTA2OWMdfiI6IjNhMGYxODU3LWMzOTMtNGI4Ny1hNjIwLTE4YzEyMTA2OWM3ZiIsIjNhMDg2OGEzLThiZWEtNGE1Mi1hMGUwLTI4OWE0ZTNkMzIxMSI6IjNhMDg2OGEzLThiZWEtNGE1Mi1hMGUwLTI4OWE0ZTNkMzIxMSIsIjlkYjk2OTcxLTFiNmItNDM0NC05OWFiLTg5YTVjM2NkYmU5MSI6IjlkYjk2OTcxLTFiNmItNDM0NC05OWFiLTg5YTVjM2NkYmU5MSIsIjE0ZmU1ZDI0LWQxYTYtNGNmNy04NmNhLTgzZTlhNDgzNDkIJGkdf4jE0ZmU1ZDI0LWQxYTYtNGNmNy04NmNhLTgzZTlhNDgzNDk5OCIsIjcwZjI1NjRjLTI2ZTItNDU5Yy05MmQ2LTMwOTUwMWQ4MTc1OSI6IjcwZjI1NjRjLTI2ZTItNDU5Yy05MmQ2LTMwOTUwMWQ4MTc1OSIsIjc3ZDgzMGZlLTYwNzYtNDRjMy05MTZmLTMwNzUwODNkZDg1MiI6Ijc3ZDgzMGZlLTYwNzYtNDRjMy05MTZmLTMwNzUwODNkZDg1MiIsIjMxNjNiY2RhLTIxYWEtNDhlZS1hYmFlLTI2NjAzZTA3YjU4NSI6IjMxNjNiY2RhLT5YWEtNDhlZS1hYmFlLTI2NjAzZTA3YjU4NSIsIjQwMzcyMzEzLTZkYTItNDJhZi04NzBlLWRmMGE5MDU2MmZmNiI6IjQwMzcyMzEzLTZkYTItNDJhZi04NzBlLWRmMGE5MDU2MmZmNiIsImE3ZGQzMjc2LTJiNDMtNDg2NS1iNGViLTFiZTEwMDk3NmQ5NiI6ImE3ZGQzMjc2LTJiNDMtNDg2NS1iNGViLTFiZTEwMDk3NmQ5NiIsImUxY2ExNzc2LTU0NTUtNDc4MC04OTMzLTgxM2M0NjlmZmZhMSI6ImUxY2ExNzc2LTU0NTUtNDc4MC071TMzLTgxM2M0NjlmZmZhMSIsImY2ZWQ5NmVhLWU4MjAtNDg3My1iMmM2LTcwYzRjODc1ZjUyMCI6ImY2ZWQ5NmVhLWU4MjAtNDg3My1iMmM2LTcwYzRjODc1ZjUyMCIsImY3YjViYjRjLTZmYWUtNDdhYi05YTgzLWUyNGVjMzllZWY1ZSI6ImY3YjViYjRjLTZmYWUtNDdhYi05YTgzLWUyNGVjMzllZWY1ZSJ9LCJlbWFpbCI6InpvZCtzdWJzY3JpcHRpb250eXBlMkBwZXl0ei5kayIsInN1YnNjcmlwdGlvbl90eXBlIjoiMiJ9; user_token=d5aaaab9742e763aef12c8c40a4a75da8072fa97803dd7a1bd6de76384a23808; wordpress_test_cookie=WP%20Cookie%20check; wordpress_logged_in_a8d2d12312312379974dc3f916b356d3=MYWORDPRESSUSERNAME%7C1612378456%7C8xaOuvvv7N7Aq4PzHreiIfjKNoRt6NInOxbwvWW9nYL%7Cc70240d7f46d0dc5b6112554da95f69cee61180ebb2b3a93326cb42d02937f3f
Pragma: no-cache
Cache-Control: no-cache

Response headers

HTTP/1.1 400 Bad Request
Date: Mon, 16 May 2022 13:27:53 GMT
Server: Apache/2.4.41 (Ubuntu)
Content-Length: 299
Connection: close
Content-Type: text/html; charset=iso-8859-1

Response

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
<hr>
<address>Apache/2.4.41 (Ubuntu) Server at vagrant Port 80</address>
</body></html>

I can't see out of that, why that request was 'bad'. Nor why the server couldn't understand it.

Solution attempt 3: Check the logs

I checked these:

/var/log/apache2/access.log
/var/log/apache2/error.log
/var/log/apache2/other_vhosts_access.log

I even tried only outputting lines containing the word 400, with this command:

tail -f access.log error.log other_vhosts_access.log | awk '/400/'

And I could get them (from the other_vhost_access.log-file). But I can't see any reason for it to be 'bad' either:

In Firefox

vagrant:80 123.123.123.1 - - [16/May/2022:15:27:53 +0200] "GET /wp/wp-admin/js/common.min.js HTTP/1.1" 400 481 "http://mydomain.test/wp/wp-admin/, http://mydomain.test/wp/wp-admin/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0, Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0"

In Chrome

vagrant:80 123.123.123.1 - - [16/May/2022:15:45:24 +0200] "POST /es/query/ HTTP/1.1" 400 481 "http://mydomain.test/, http://mydomain.test/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36, Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36"

Solution attempt 4: Change LogLevel on the Apache-server

As advised here I tried Changing the LogLevel on the Apache-server.

But I didn't see any better logs in access.log, error.log nor other_vhost_access.log.

Solution attempt 5: Debug using `tcpdump`

I also tried running tcpdump from inside the box and output it all using the command:

13:01:42.102438 IP (tos 0x0, ttl 64, id 54239, offset 0, flags [DF], proto TCP (6), length 52)
    123.123.123.12.80 > 123.123.123.4.57340: Flags [.], cksum 0xc3db (incorrect -> 0xe5ed), seq 114948251, ack 2535382125, win 505, options [nop,nop,TS val 243067183 ecr 1641029207], length 0
13:01:42.102475 IP (tos 0x0, ttl 64, id 8630, offset 0, flags [DF], proto TCP (6), length 52)
    123.123.123.12.80 > 123.123.123.4.57334: Flags [.], cksum 0xc3db (incorrect -> 0xbf8a), seq 2046057008, ack 256413570, win 505, options [nop,nop,TS val 243067183 ecr 3257787508], length 0
13:01:42.106114 IP (tos 0x2,ECT(0), ttl 64, id 54240, offset 0, flags [DF], proto TCP (6), length 533)
    123.123.123.12.80 > 123.123.123.4.57340: Flags [P.], cksum 0xc5bc (incorrect -> 0xe068), seq 114948251:114948732, ack 2535382125, win 505, options [nop,nop,TS val 243067186 ecr 1641029207], length 481: HTTP, length: 481
    HTTP/1.1 400 Bad Request
    Date: Mon, 16 May 2022 11:01:42 GMT
    Server: Apache/2.4.41 (Ubuntu)
    Content-Length: 299
    Connection: close
    Content-Type: text/html; charset=iso-8859-1

    <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
    <html><head>
    <title>400 Bad Request</title>
    </head><body>
    <h1>Bad Request</h1>
    <p>Your browser sent a request that this server could not understand.<br />
    </p>
    <hr>
    <address>Apache/2.4.41 (Ubuntu) Server at vagrant Port 80</address>
    </body></html>
13:01:42.106394 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    123.123.123.4.57340 > 123.123.123.12.80: Flags [.], cksum 0xddfb (correct), seq 2535382125, ack 114948732, win 2051, options [nop,nop,TS val 1641029211 ecr 243067186], length 0
13:01:42.109623 IP (tos 0x0, ttl 64, id 54241, offset 0, flags [DF], proto TCP (6), length 52)
    123.123.123.12.80 > 123.123.123.4.57340: Flags [F.], cksum 0xc3db (incorrect -> 0xe400), seq 114948732, ack 2535382125, win 505, options [nop,nop,TS val 243067190 ecr 1641029211], length 0

And again, I can see the 400 Bad request-error. But I can't see what's bad about it.

Solution attempt 6: Look into headers / cookies

Since the errors only occurred when I was logged in, then I tried looking at the header and cookies. But I couldn't see anything spectacular there.

I even compared them to the headers/cookies on another machine, where this is running flawlessly, and nothing sprung in my eye.

... And still... Even if it was this, then why don't I get a better error message?

Solution attempt 7: Server performance - activity monitor

I checked the disk-space, CPU-usage and memory usage on the server (via bpytop) when I refreshed a page. But it looks normal.

Untried attempts

Downgrade and reinstall and/or upgrade WordPress core.
Stop and start apache
Restart the Ubuntu-server.

I've not done this, since I would really like to find out, how to see what is wrong here (instead of just trying to fix it).

Possibly the Apache virtualHost entry specifies (error) log files in a different location, or you configured something like: https://serverfault.com/q/997624/960939 because AFAIK otherwise 400 errors should be logged. When you're using wordpress it's very likely that you have Rewrite rules active, maybe those don't function correctly - https://httpd.apache.org/docs/2.4/mod/mod_rewrite.html#logging — Rob, May 16 '22 at 15:58
Look at the full unfiltered error.log at the time the error happens. Don't filter for "400" - there's no reason the message telling you what's going wrong has to contain that string. — Tilman Schmidt, May 16 '22 at 16:41
Don't split your efforts into "attempts". You should consider all factors at the same time: access log, error log, network (the actual "bytes" of the request and response), client presentation. You have to isolate the entries of all of these that correspond to the same single wrong request. Also, you may configure Apache to log more verbose. Also, you've seen the "bad request" itself — replicate that request with complete headers using curl or even "telnet webserver 80" or openssl s_client (in case of HTTPS), see for yourself if it's really bad request or transient problem on web server side. — Nikita Kipriyanov, May 16 '22 at 19:04