But say I want the server to encrypt data using the private key and decrypt it using the public key, as a way of distributing data that can be verified to have come from the right server.
You can do this - this is, at a very simplistic level, how RSA signing works (note, simplistic - there is a bit more to it).
Rather than modify the code to allow this, can I simply publish the private key and keep the public key secret? Does this affect the security of the system?
You don't need to publish the private key at all - RSA is a trapdoor permutation which means:
- If you encrypt with a public key, you can decrypt with the private key.
- If you encrypt with a private key, you can decrypt with a public key.
Thus, RSA supports doing both signing and encryption relying on the end user having only the public key.
In your case, if the client wishes to verify data came from the server, you apply the second case of RSA and decrypt the signature data using the public key you already have.
Furthermore, because it is a permutation, you shouldn't need to modify your code at all. Both keys should work using the same function. I would expect any decent crypto library would have APIs for verifying signatures according to the varying standards that exist - one of these would probably be a good bet.
RSA Labs provide a nice explanation of this.
If you want to extend this between servers, or verify client communication - generate keys for each entity and swap the public ones. The process can then be used at both ends.
Theoretically speaking, e and d are interchangeable (which is why RSA works)(one must be designated secret and kept secret) but p and q must always be kept secret as these allow you to derive d from e and vice versa. However, you need to be extremely careful in your understanding of the private key - does your software store p/q in the private key? If so, you can't publish it as is. Also, when I say interchangeable - once you publish one of that pair (e or d along with your modulus n) you must guard the other with your life. Practically speaking as Graeme linked to in the comments e is often chosed as a small/fixed value. My comment on e/d being interchangeable clearly does not apply when e is easily determined. Doing this sort of thing therefore has the potential for confusion and mis-implementation. Use a third-party library/don't start publishing private keys.