Which of the two following has a stronger cryptographic strength in withstanding brute-force attacks?
- WPA/WPA2 Enterprise using 2048-bit RSA public-key
- WPA/WPA2 Personal using 63-character Passphrase
Which of the two following has a stronger cryptographic strength in withstanding brute-force attacks?
Ignoring the benefits and drawbacks of each protocol and focusing exclusively on the difficulty of a brute force attack, the answer is that they are both identical. The reason for this is that the only difference between the two is how the key is derived. When using WPA2 Enterprise, a random 128-bit key is generated and exchanged using public key cryptography. For WPA2 Personal using a passphrase of any length, a 128-bit key is derived from the passphrase itself using a key-strengthening algorithm called PBKDF2.* Regardless of how it is generated, the session key will have a keyspace of up to 2128. For a 63-character random passphrase, the entire 2128 keyspace of the session key is used.
If on the other hand you are referring to any type of cryptoanalytic attack, the answer is that the WPA2 Enterprise option using RSA 2048 provides less security. RSA 2048 is approximately equivalent to a 112-bit key. The reason for this is that attacking RSA is not done using brute force, but by attempting to factor a very large composite number. RSA uses the so-called integer factorization problem, a problem in mathematics which states that, while it is easy to multiply two huge prime numbers, factoring the resulting composite number back into the original two prime numbers is very, very hard.†
The most effective algorithm for factoring large numbers is called GNFS, which is significantly faster than an exhaustive search, but is still far too computationally-intensive to perform on realistic semiprimes. When you are using a 63-character passphrase on the other hand, it is directly converted into a 128-bit key by PBKDF2 using 4096 iterations of HMAC-SHA1. If the 63-character passphrase is chosen randomly from a US keyboard with 95 printable keys, it gives a keyspace of 9563, which is far greater than 2128 (it has the equivalent security of log2(9563) ≈ 413.9 bits). Because the session key is only 128 bits, the entire keyspace of the 63 keys is not used, only that of the 128-bit session key.
In practice, WPA2 Enterprise using public key cryptography is more secure. This is not because it is more difficult to attack with cryptanalysis, but because it provides other security benefits such as forward secrecy, which ensures that an attacker who compromises one session will not be able to then retroactively decrypt previously recorded sessions, because each key is generated randomly for each session. WPA2 Personal on the other hand uses the passphrase to derive the key, and as long as the passphrase remains the same, any data encrypted under that passphrase can be decrypted as soon as you learn what it is. Because WPA2 Enterprise allows the key to be generated entirely randomly, there is no risk of using a poor passphrase that can be broken.
It is often not easy to maintain a fully random 63-character password among multiple devices. If you are using the complete 95 character set from a US keyboard, you only need 20 random characters to fully utilize the entire keyspace of the cipher, as 9520 is just barely above 2128. Adding more characters would only be useful if you expect them not to be entirely random.
You should be aware that WPA3 has been announced. It is a new protocol which vastly improves wireless security. It provides individualized encryption even on open networks, allowing people to securely use hotel and airport hotspots without worrying about passive eavesdropping, and allowing people to share their key with guests without allowing them to read their traffic. It does this by dissociating the user-supplied key from the encryption. In other words, the key you provide is only used to authenticate yourself to the hotspot. Public key cryptography will automatically generate a per-use random session key. WPA3 comes with other improvements as well, such as a larger 192-bit key. The protocol will be slowly rolled out to new devices during 2018.
Note that I am assuming WPA2 is configured to use CCMP which uses AES128 in CCM mode, rather than the broken TKIP which uses the much weaker RC4 stream cipher.
* It is slightly more complicated than this. The PSK is not directly plugged into the cipher after going through PBKDF2. Instead, a 4-way handshake is done which relies on the secrecy of the PSK. It suffices to say that knowledge of the PSK allows anyone who has captured traffic (including the 4-way handshake) to decrypt the session.
† Integer factorization is an open problem in mathematics. It is possible that a classical algorithm will be discovered that can factor a large semiprime in polynomial time. Additionally, quantum computers running Shor's algorithm will do this in polynomial time. There are no publicly known cryptoanalytic quantum computers in production.