This will certainly come over as controversial, but nevertheless it's how I see it...
There is a bit of a misunderstanding in the necessity of HTTPS which was most probably spread deliberately. Like with every half-truth, there is some truth in the arguments, but also a whole lot of lies.
HTTPS (or TLS) does have some very useful and desirable properties (authentication and confidentiality) which are absolutely mandatory for some (think banking), and arguably necessary for quite a few, maybe the majority of, services. Anything that contains personal identification data, basically.
That being the case, there are many things for which HTTPS is utterly unnecessary, and on the other hand, HTTPS used improperly, i.e. with mixed content, can be pretty insecure (almost like no HTTPS), which was the justification for HTTPS everywhere in the first place. And yes, in the light of some sites back in the days that indeed offered a mandatory HTTPS kind of service with mixed content, it certainly did have some merit.
That, and then there's of course a good amount of paranoia that some people have and that is being actively promoted, about the whole world actually being interested in every little, unimportant thing in the unimportant little lives of everybody. Sure enough, after posting everything you've done today (with photos and geotag!) on Instagram where literally the whole world can read it, good job at having done so securely, via an encrypted channel. Also, it's important that nobody finds out about what you do on the internet in general. That, and there's this conspiracy where they alter news articles and feed you with false information to, uh... I don't know what for (actually, there is some truth in that, too, because that is just what e.g. Google does -- only at a different level, what's being changed is not the actual contents, but which content you're being shown at all, but that's irrespective of HTTPS being used). The silver bullet HTTPS prevents all these bad things! So clearly, everything needs to be HTTPS/TLS.
Regardless, even when used properly, HTTPS still fails to provide the service that you wish. For one reason, the entire chain of certificates works on the assumption that you can "trust" someone (say, Comodo) who makes money from selling certificates, without actually having a reason to trust them. And then, not just governments but also large enterprises (and schools, and antivirus, and who knows who else...) actively subvert the certificate chain by installing root certificates for the only purpose of, well, effectively breaking the system.
So, no, communications are not guaranteed to be confidential, and no, they are not authenticated in a reliable way. Not as much as you would think, anyway. Using your employer's laptop? Using your kid's computer? Lost cause. Antivirus installed on your computer? All bets are open.
But at least you know that a site is safe, the green thingie on your browser tells you so, and it warns about risky sites. Right, everybody can get a non-green certificate for free (avoiding the scary warning), and a green-badge certificate for very little money. It has absolutely no meaning.
I seriously hope you have TLS enabled for accessing your gmail account, too. Because, you know, that makes it secure, you don't want someone on the wire to read your mails, do you. Sure enough, Google will not read your entirely unencrypted mails while they are stored on their server. Sure enough, being a US company, they will not provide the contents to a particular governmental organization.
Now the real reason why you must have HTTPS everywhere is that companies like Google, Microsoft, or Amazon, and with them all providers who sell bandwidth, want that.
They do not want everybody and their grandmother to set up a credit-card computer as a transparent web proxy which not only reduces your bandwidth consumption by caching resources, but also filters out their advertizing and tracking stuff. Sure, you can always add a browser plugin which does the same thing. Except, you must maintain it on every computer that you have in your house, and on some (Fire TV) it is outright impossible without rendering the device unusable, or you must root it (think Android phone) which also isn't necessarily destruction-free (thank you so much for Samsung Knox, so awesome).
Luckily, you can just cut the crap globally, for all devices within your network by having a transparent proxy right behind your cable/dsl modem, which costs you 20€ and 3 mins setup. Oh heck, what a catastrophe! You are to download exactly the version they want (including "personalization"), and when they want it, including all beacons and whatnot. So that is the true reason why you need HTTPS everywhere.
Ironically, the companies that promoted HTTPS and emphasized how e.g. TLS not only hides the actual content, but also the exact URL that you clicked on (like http://somesite.com/dirty_porn_pic.jpg
) and such... in reality they are exactly the ones who go to any length to fingerprint your system, identify you, keep an infinite history, track every single click you do, and collect every possible piece of information including where you go and when, and your heartbeat. Or, the contents of any file on your computer.
Ever wondered how Amazon does it so they incidentially recommend XYZ on your PC after you searched Google for XYZ on your phone five mins earlier? Different company, different device, one supposedly cannot possibly know both devices are owned by the same person. I did in fact wonder how it's done, since in my understanding, whatever they need to do to achieve that certainly is not compliant with the law (in the EU at least). But apparently, that's not a hindrance.
HTTPS actually helps in doing all these borderline-legitimate things, both by providing a false sense of security, by obscuring what is being sent, and by no longer making people ask: "Hey, what's that encrypted traffic coming from my device anyway!?". Because, you know, all traffic should be encrypted, that's a good thing. Encrypted stuff is not suspicious, it's probably harmless. Nobody is hiding something.
1I'm not sure how this plays into this, if at all, but in some cases, HTTPS is not supported by a browser or OS, in which case HTTP is required. I've gotten complaints when my website automatically rewrote HTTPS to HTTP, because this made my site entirely inaccessible to some people. Now my site defaults to HTTP, although it does support HTTP. Ideally, in the future I'll have it default to HTTPS, but not rewrite HTTP to HTTPS, which should never be done as a general best practice. – InterLinked – 2019-10-29T12:41:14.020
1
@InterLinked there are ways for your server to automatically upgrade to HTTPS only if it is correctly supported (and continue using HTTP for archaic noncompliant clients) - see https://scotthelme.co.uk/we-dont-do-https-for-backwards-compatibility/ for example
– Matija Nalis – 2019-11-13T17:22:55.603@MatijaNalis Thanks for the link, I'll look into it. I'm curious if it works for browsers that claim to support SSL but don't do it properly. For example, support for SSL in Chrome 49 on Windows XP is basically nonexistent - you just get SSL certificate errors. Pages work perfectly in Internet Explorer and Firefox. HTTP also works, but many (maybe not all) HTTPS sites do not work. I'd need a solution that would handle this kind of situation, too. – InterLinked – 2019-11-13T18:20:38.163
@InterLinked Some of them should work - for example https image one. Browser will try to get 1x1 pixel image over HTTPS, and only if it succeeds it will receive HSTS header which will force all future connections to that domain to use HTTPS. If TLS failed for any reason, the browser would not receive HSTS headers over it, and would continue using website via HTTP (only without showing 1x1 https image). If you want quick test, I've implemented some (but not this 1x1 image) of the upgrade methods here
– Matija Nalis – 2019-11-13T23:52:30.743@InterLinked When you say rewrite, do you also mean redirect? Because this is what I'm doing:
Redirect permanent / https://${subname}.${name}.${tld}/
, does this not qualify as a general best practice? – Chazy Chaz – 2019-11-14T14:41:49.840Well, I've tried both rewrite using htaccess and other methods of redirecting. It's not a general best practice to redirect unconditionally because it won't work for all clients. – InterLinked – 2019-11-14T21:48:16.763