When I encrypt the data with a public key, decryptor asks for private key twice. Where does the second key come from?
Most likely the key used for encryption has a subkey and encryption is done for both key and subkey.
To disable the subkey (i.e. exclude it from encryption process), set it's Enabled property to false.