You do not have to generate the CSR on the server where it is going to be used, but for certificates only used on a single server, it is highly recommended that you generate the keys on the server and never let it leave that host.
You will need to let Apache know where the private key can be found however.
Think of the private and public key as a set of unique English-Korean dictionaries.
The browser speaks English only - so does Apache
When the browser wants to send something to Apache in secrecy, it obtains the English-Korean dictionary (the certificate/public key) from Apache and translates the secret message to Korean.
When Apache receives the secret message, it needs the Korean-English dictionary (the private key) to translate it in order to make sense of it.
Without the private key, everything sent to the web server is pure korean and apache won't know how to respond