From my understanding the vulnerability requires the download of the actual exploit payload. JNDI provides a way to do using LDAP and (with older versions of Java only?) with RMI. The server must return a serialized Java object which on deserialization leads to attacker controlled code execution.
This means that limiting outgoing connections to prevent loading the exploit can mitigate the vulnerability. But it is not possible to just block specific ports - the attacker controlled server which hosts the exploit payload can run on arbitrary ports.
It might be possible with a firewall capable of application layer inspection (i.e. DPI or application proxies) to restrict the outgoing protocols, i.e. LDAP and RMI should not be allowed. Alternatively one might employ a HTTP proxy and restrict outgoing traffic to the proxy. This should help since LDAP and RMI do not work over a HTTP proxy.
The most robust mitigation though is not just deny selected outgoing traffic using a blacklist but instead only allow the actually expected traffic with a restrictive whitelist.