I've heard the argument against DNS-over-HTTPS that it is supposed to be a security nightmare for network defenders because it enables encrypted DNS over port 443, compared to DNS-over-TLS which goes through port 853.
These network defenders are possibly corporate environments that rely on plaintext DNS inspection to enforce policies. Assuming that devices fallback to plaintext DNS if DoH/DoT are unavailable, the network administrators could block port 853 with little risk because it is only used by DoT. On the other hand, if they simply block port 443, then all HTTPS websites will become unavailable.
Similarly, if they see an influx of DoT traffic, it could indicate an anomaly. If some similar traffic spikes occur with DoH, then it might not be possible to directly distinguish HTTPS from DoH traffic.
As for the question from the title:
Why is DNS-over-HTTPS such a big security nightmare compared to DNS-over-TLS?
This should probably be worded as "Why is DNS-over-HTTPS seen as a security nightmare compared to DNS-over-TLS?". DoH and DoT are pretty similar on a protocol level, in both cases DNS messages are encrypted. See also my Cloudflare blog post explaining DNS encryption where I describe the technical protocol details, deployment choices, and various expectations from individuals and organizations.
Historically, the operating system has been accepting whatever DNS resolver was advertised by the local network. This is typically configured by the corporate network administrator, or the ISP. They expect to have the ability to provide services such as malware blocking, parental filtering, blocking of illegal content, and in some cases query logging.
DoH and DoT are great in protecting the privacy and integrity of DNS queries in untrusted environments such as airport Wi-Fi or even snooping/interference from the local government. However since it was emerging technology, not all existing DNS resolvers have support for it.
That put early adopters such as Mozilla in a difficult position, should they abandon the idea of improving privacy, or should they select a DNS resolver who supports DoH with a strong privacy policy? They ended up with the latter, but that meant that the default DNS resolver provided by the operating system was initially ignored. This is probably the reason for the negative pushback against DoH from ISPs and governments. If DoT was deployed in a similar way, I would have expected a similar criticism.
To conclude, I don't think that DoH is such a "security nightmare" as claimed. It is just that some organizations are concerned about losing control over DNS. Previously it was centrally controlled by the operating system, but as DoH/DoT is still pretty new, there is no real standard on configuring it so many applications have their own mechanisms to do so. This is probably the "nightmare" that some admins have, the extra complexity that they have to go through to ensure that their filtering policies are applied.