If I encrypt 160 bits (20 bytes) of data using an RSA 1024 private key, will the result always be 128 bytes? This seems to be the case in my experiements, but I want to be sure.
For example,
/bin/echo -n "foo" | openssl dgst -sha1 -sign privateKey.der -keyform DER > enc.txt
always seems to result in a 128 byte file enc.txt even after I generate new public/private keys (the private key saved into the file privateKey.der).
Basically yes. That's how RSA works, as described in the standard.
On encryption, the input data is first "padded", i.e. expanded with some randomness and structure, and then turned into a big integer value m in the 0..n-1 range (n being the modulus). That value is then raised to the power e (the public exponent) modulo n, yielding another integer in the 0..n-1 range. The result is encoded into bytes with what the RFC calls I2OSP (big-endian unsigned convention). For a 1024-bit RSA modulus, you will always get a 128-byte result.
Similarly, a RSA signature, for a 1024-bit key, always has length exactly 128 bytes.
Mandatory reminder: no, you are not "encrypting with the private key". The whole notion of "encrypting with the private key" is a flawed analogy, which works only for RSA, and actually does not work for RSA, precisely because it completely fails at taking padding into account. You are signing. And, indeed, you use the -sign command-line flag. RSA is two algorithms, which happen to share some common mathematical structure, but not all of it; things will be clearer if you keep thinking about RSA encryption and RSA signatures as distinct process.
