However, I've done some rudimentary research on my own, and I cannot wrap my head around the question, is having a keystore and a truststore on the (classloader) directory of my application a security vulnerability? Could it ever be one?
I am aware that keystore stores the private and public key of the server, and truststore stores the public key of the server, which it verifies & uses when contacting the server. How can I protect my keystore's and truststore's certificate password, when they must be on the application's directory? Does it need to be protected, even?
Actually, the truststore contains a certificate (or several, one of) which is used to validate the server's certificate or (in general) chain. Normally this is a cert of a CA which issued the server cert directly or (now more common) indirectly e.g. root CA (cert in truststore) issued cert for intermediate CA (in handshake) which issued cert for server (ditto). Using a self-signed cert is a special case where the server cert is also its root, and thus put in the client's truststore.
This information is indeed vital to security. If an attacker can steal your keystore data, they can impersonate you to others, or if you use (or used) ciphersuites in 1.2 that DON'T support forward secrecy (i.e. 'plain RSA' suites/keyexchange) they can decrypt eavesdropped connections to you. If they can alter your truststore data, they can impersonate others to you.
Whether and how this is a risk depends on what possible attacks and attackers you must defend against, or as usually phrased here, "what is your threat model?" If your endpoints are themselves secure, and the application containing the security data can be deployed to them securely, you're fine. If attackers can run things on your system(s) or otherwise access your file(s) it gets more complicated, and there are no generic or very easy answers; in general most of the answers that actually work involve hardware, either custom or platform, and thus aren't cross-platform and often aren't cheap. See for examples
Where to store a server side encryption key?
https://crypto.stackexchange.com/questions/35530/where-and-how-to-store-private-keys-in-web-applications-for-private-messaging-wi
https://stackoverflow.com/questions/3038608/how-do-i-store-a-java-keystore-password
What encryption algorithm should my keystore use? I'm heading for really strong encryption, future proofing as much as possible along with keeping up as much backwards compatibility as I can without reducing the application's security.
Don't use JKS, which was designed deliberately weak because of US export restrictions back in the 1990s and was the default through j8; higher versions default to PKCS12, though keytool still accepts JKS and JCEKS automatically. (It warns about the better standardness of PKCS12, but not the better security.)
Java's version of PKCS12 (like most) uses 3/3DES for privatekeys, which is what you should care about; (again like most) it 'encrypts' the certbag with RC2-40, which is very weak and basically useless because certs aren't secret. While not the theoretically strongest possible, 3/3DES is strong enough against any near-term conventional attack. What experts fear sometime vaguely-soon-ish is quantum attacks, but no one yet knows what the 'correct' solution for post-quantum crypto (PQC) is; the few experimental algorithms that exist now (1) aren't used for keystores (2) may well be changed, perhaps incompatibly (3) may turn out to have problems. If you really want, you could use PKCS12 and superencrypt the file with (some) PQC; BouncyCastle has a few options in a special provider, if you (can) use that. That way even if the PQC turns out to be useless you still have the 3DES, and if you need to change it you can just re-encrypt the file without needing to understand the contents.
Is there an issue with the certificates being self-signed considering I'm solely using them between peers of the same application?
Less so than otherwise. It does mean you can't use PKI to revoke them, but must have some other means, that works reliably -- unless you don't need revocation because all your endpoints are controlled, top to bottom, by people who never make mistakes or have the personal situation change (e.g. marry, divorce, get a job, lose a job, etc).
Considering I'm doing Java, do SSLSockets/SSLServerSockets create a "brand new session" for every new connection, as in, do they reuse the private or public key? Are private keys generated when making a handshake with a client?
SSL and TLS through 1.2 allows session resumption, which does an 'abbreviated' handshake that reuses the session 'master secret' from a previous connection's full handshake, as long as (in both senses) both endpoints have cached it. Most servers implement this and (AFAIK all) browsers use it extensively; Java JSSE implements it by default, although you can limit the number (size) or timeout. There is also a less-used option which stores a 'ticket' on the client only and uses that; see What are the differences between Session Ticket and Session ID based Session Resumption? SSL/TLS Session Resumption with Session Tickets and (my) https://crypto.stackexchange.com/questions/15209/compare-rfc-5246-sessionid-re-use-versus-rfc-5077-session-resumption . JSSE does not implement this. Either of these eliminates all the public-key crypto operations on the new session (when successful).
TLS 1.3 eliminates both of these and instead has a new mechanism which reuses the authentication but optionally does a new keyexchange (XXDHE); see my answer on the crypto.SX link above. This eliminates all or only some of the PKC (again when successful). In versions that implement TLS 1.3 (j11+) JSSE does implement this, apparently (per the debug log) only the with-XXDHE option, but there was at least one bug, see https://stackoverflow.com/questions/53913758/java-11-https-connection-fails-with-ssl-handshakeexception-while-using-jsoup .
For full handshakes in TLS 1.2 (and below) which use 'ephemeral' keyexchange (DHE or ECDHE in the ciphersuite), and for all full (non-PSK) handshakes in 1.3 as well as PSK handshakes that elect XXDHE (which as noted above Java does) both endpoints generate an ephemeral keypair used in the keyexchange. Otherwise not. (The 'export' ciphersuites in SSLv3 and older TLS could sometimes use a uncertificated keypair that might, or might not, be ephemeral, but those ciphersuites were obsoleted long ago, even before the older protocol versions that supported them.) If you are concerned because RSA keypair generation is expensive, don't be; DHE or ECDHE (XXDHE) keypair generation is cheap, at least in CPU; it does require some randomness, but SSL/TLS always requires a random nonce for the protocol, and also random IVs if you use a CBC-mode ciphersuite in 1.2 or 1.1, and I'm pretty sure JSSE reuses the RNG (SecureRandom).
