Passports does not encrypt data, instead it uses Basic Access Control, as a authentication layer, to prevent access to passport data without authorization. The authentication data required to access the passport is written in the MRZ, so ideally, you would need to optically read the passport to get the "password" to open the RFID chip.
However, passport data is x509 signed, so you can still use it for verification if you want to authenticate a user via its passport, to ensure personal information and passport photo is not fraudulent.
Note that you must encrypt the data in transit. Note also that someone can copy the details from the passport and send to your server to impersonate a user if you only do static authentication.
However, passports also support a form of dynamic authentication to do challenge-response authentication of the passport, to ensure it has not being copied, but that requires a Active cooperation of your server, eg your server must actively talk to the passport through a link , for example a mobile phone with NFC that is held against the passport in several seconds during the "signup" process.
Basically, its the passport that signs your challenge using a certificate that is embedded in the passport, and thus you can verify the response is correctly signed, and the certificate used for signing is signed by the static certificate that you can verify against the ICAO root.
A mobile phone can be a excellent resource to allow anyone to signup for your service with a passport. They just photo the MRZ with the camera, then they hold the phone against the passport while your server talks to the passport to verify its authenticity, and then, the signup is complete. I guess you then only save the photo and name of the user.
Depending on your security requirements, it might be enough with static authentication.
Note that the password cannot be used itself for any signing or encrypting activites, so if you for example would want the user to be able to use his passport to sign or encrypt anything, you will have to create a key for the user and link it to the user.