How to defragment or minimize a "directory file" on ext4?

Question

I have quite an old home directory on HDD. There were lots of files created and removed and now the size of /home/username is 995328 bytes. Please note, I mean the size of the "directory file" (every directory is a special file that contains links to other files and directories), not the total size of files inside. In other words, that's the size of "." (dot) directory you see in ls -la output.

My home directory only has 244 files+dirs inside. On another PC I have 203 files+dirs in the home dir, and the "." dir is only 4096 bytes. This may not be of concern but I noticed a huge performance degradation when played Natural Selection 2. In particular, it keeps compiled shaders and mods in "~/.config/Natural Selection 2" and so each time it accesses a file there it has to traverse this huge ~1 Mb home directory.

To speed things up I moved its directory to /opt/confs (my root partition is on SSD, the game itself also resides there, in /opt) and it helped a lot. However, other players still loaded before me, they're on Windows and SSD as well. I also tried to mount --bind that directory from "/opt/confs/Natural Selection 2" to "~/.config/Natural Selection 2" but I didn't see any improvement.

What DID help is creating a new user in /opt/home (i.e. on SSD) and using it for launching the game so /home/username isn't used anymore. I had a huge improvement, like twofold, 37 seconds instead of 1 minute 15 seconds. I have 32 Gb of RAM (only 10-12 is occupied at any time) so all of this should be heavily cached and HDD speed shouldn't interfere with the path traversing.

I think the home directory file is now full of holes left after removing old files and .* directories and thus it's slow to seek. I couldn't find a way do make it smaller and faster other than to create a new directory and move the files and directories into it, remove the now empty old home dir and rename this new one. But that's a hacky way and if this is really a problem, there should be a way to deal with sparse directories. Especially when everyone tells that ext filesystems don't need defragmentation at all. Well, for files it's true but seems that no one cares about directories.

edit: forgot to test another cases to rule out other possibilities. I moved the home directory of that user to /home (which is on HDD) and the config directory of the game to /opt/confs (SSD). So now the path is like / (SSD) /home/ (HDD) /newuser/.config/ (still HDD) Natural Selection 2 (symlink to /opt/confs/Natural Selection 2, SSD). Still loads pretty fast, no difference with pure SSD path (43 secs). Then I did a control measurement: I moved the new user's home dir into my main user's homedir so its path is /home/rkfg/newuser where /home/rkfg is the problematic fragmented directory. The game's config directory is still symlinked to SSD, everything else is the same. And the long loading times are back: 1 minute 39 seconds! Now it has to traverse through the large directory file and since the game does it dozens of times per second it all adds up. Yes, it's not very optimized in this part, hopefully it will get better. But the point is: huge and fragmented directory files DO produce bigger seek times. And the cache doesn't help much despite all optimizations.