I can't block port 443 due to https traffic.
MiTM is not possible. We cannot make everyone trust our own certificate.
That's not completely true, even when you can't terminate the HTTPS connections on the firewall. In general, it's possible to filter traffic based on the (plain text) server name indication (SNI) in ClientHello
.
Then, if a VPN is using non-standard traffic on port 443, you could block everything on port 443 that doesn't meet the TLS standards. That won't break HTTPS, but may affect other non-standard services, requiring exceptions for their IP addresses.
The only problem with these measures is that they are not compatible with TLS 1.3, because it has the ability to encrypt the SNI. For TLS 1.3 you can only block connections based on DNS names (requiring additional and inaccurate DNS lookups), and/or block DNS requests to certain domains (not affecting solutions using hard-coded IP addresses). However, this approach might also work for blocking VPN solutions that rely on a working DNS for their domain(s).
I have a firewall (pfsense) acting as a gateway
The documentation on pfSense's transparent proxy features are quite limited, but it does tell pfSense is using Squid to implement these features, and Squid can do a lot of things. The SNI inspection should be possible with the current pfSense's Squid versions (since 0.4.35).
The Squid feature is called SslBump Peek and Splice. With this feature you can perform actions based on the SNI, and there are two actions that can either block or allow the connection without intercepting it:
As the documention comes with clear configuration examples, the rest is left as homework. It's also likely that the pfSense's web interface doesn't have these features, but they should be accessible through configuration files. (Some commercial solutions are more intuitive with this, but using open source is a decision that usually comes with the requirement of learning more.)