Skip to content

CWE-780: Use of RSA Algorithm without OAEP - Java

Overview

Using RSA encryption without OAEP (Optimal Asymmetric Encryption Padding) enables padding oracle attacks, chosen ciphertext attacks, and message malleability. In Java, this commonly occurs when using Cipher.getInstance("RSA") without specifying the padding mode, which defaults to the insecure PKCS#1 v1.5 padding - confirmed on JDK 26 with the SunJCE provider, where ciphertext from the bare "RSA" transformation round-trips through "RSA/ECB/PKCS1Padding".

Switching to an OAEP transformation name is most of the fix but not all of it. OAEP has two digests, and the transformation name sets only one: "RSA/ECB/OAEPWithSHA-256AndMGF1Padding" leaves MGF1 on SHA-1 unless an OAEPParameterSpec says otherwise.

Primary Defence: Encrypt and decrypt with an explicit OAEPParameterSpec that names SHA-256 for both the OAEP digest and MGF1, passed to Cipher.init() on each side. Never rely on the default "RSA" transformation, and use hybrid encryption (RSA-OAEP for the key + AES-GCM for the data) for payloads larger than the RSA size limit.

Common Vulnerable Patterns

Default RSA Transformation (Uses PKCS#1 v1.5)

// VULNERABLE - Defaults to "RSA/ECB/PKCS1Padding" (weak!)
import javax.crypto.Cipher;
import java.security.PublicKey;

Cipher cipher = Cipher.getInstance("RSA");
cipher.init(Cipher.ENCRYPT_MODE, publicKey);
byte[] encrypted = cipher.doFinal(plaintext);
// This uses PKCS#1 v1.5, vulnerable to Bleichenbacher's attack!

Why this is vulnerable:

  • Cipher.getInstance("RSA") defaults to RSA/ECB/PKCS1Padding (PKCS#1 v1.5), not OAEP.
  • PKCS#1 v1.5 padding is susceptible to padding oracle and chosen-ciphertext attacks.
  • The Bleichenbacher attack (1998) breaks PKCS#1 v1.5 when decryption success/failure can be observed.
  • An attacker can adaptively probe decryption to recover plaintext without the private key.

OAEP Transformation Name Without an OAEPParameterSpec

// VULNERABLE - the name sets the OAEP digest; MGF1 stays on SHA-1
Cipher cipher = Cipher.getInstance("RSA/ECB/OAEPWithSHA-256AndMGF1Padding");
cipher.init(Cipher.ENCRYPT_MODE, publicKey);   // no OAEPParameterSpec
byte[] encrypted = cipher.doFinal(plaintext);
// cipher.getParameters() reports: MD: SHA-256 / MGF: MGF1SHA-1

Why this is vulnerable: the transformation name carries one digest and OAEP needs two - the message digest and the one MGF1 runs on. SunJCE reads SHA-256 from the name for the first and leaves the second at its default of SHA-1. Measured on JDK 26, cipher.getParameters() after this init() prints MD: SHA-256, MGF: MGF1SHA-1, and ciphertext produced this way fails with BadPaddingException against a decryptor configured with MGF1ParameterSpec.SHA256, succeeding only against MGF1ParameterSpec.SHA1.

The consequence is interoperability before it is security. A Java service encrypting this way disagrees with any peer using SHA-256 for both digests, which is what every other ecosystem produces: Go's rsa.EncryptOAEP takes one hash and uses it for both, .NET's RSAEncryptionPadding.OaepSHA256 sets both, and Python's padding.OAEP has no defaults at all so both are named at the call site. The error on the peer says only that padding failed. Every review artifact says SHA-256: the code, the transformation string, the ticket. Nothing except getParameters() says otherwise. The reverse direction fails too, so a ciphertext written with a full SHA-256 spec cannot be read back by the bare name.

Using Deprecated SHA-1 with OAEP

// VULNERABLE - SHA-1 is deprecated
Cipher cipher = Cipher.getInstance("RSA/ECB/OAEPWithSHA-1AndMGF1Padding");
cipher.init(Cipher.ENCRYPT_MODE, publicKey);
byte[] encrypted = cipher.doFinal(plaintext);
// SHA-1 is cryptographically broken, use SHA-256 or better

Why this is vulnerable:

  • OAEP is correct here; the digest is the finding. The 2017 collisions do not break OAEP-SHA-1, whose proof treats the hash as a random oracle rather than as collision resistant.
  • OAEP's security proof assumes a strong hash; SHA-1's weaknesses undermine that proof.
  • Modern standards prohibit SHA-1 for new systems; use SHA-256, SHA-384, or SHA-512.

Encrypting Large Data Directly

// VULNERABLE - Will throw exception or fail
// Max plaintext for 3072-bit RSA with SHA-256 OAEP: 318 bytes
byte[] largeData = new byte[1000];
Cipher cipher = Cipher.getInstance("RSA/ECB/OAEPPadding");
cipher.init(Cipher.ENCRYPT_MODE, publicKey, new OAEPParameterSpec(
    "SHA-256", "MGF1", MGF1ParameterSpec.SHA256, PSource.PSpecified.DEFAULT));
byte[] encrypted = cipher.doFinal(largeData);  // javax.crypto.IllegalBlockSizeException!

Why this is vulnerable:

  • RSA can only encrypt data smaller than the modulus minus padding overhead (~318 bytes for a 3072-bit key with SHA-256 OAEP).
  • Larger inputs throw IllegalBlockSizeException.
  • The safe pattern is hybrid encryption (AES-GCM for data + RSA-OAEP for the key), not a larger RSA key.

Secure Patterns

// SECURE - RSA-OAEP with both digests named explicitly
import javax.crypto.Cipher;
import javax.crypto.spec.OAEPParameterSpec;   // javax.crypto.spec, not java.security.spec
import javax.crypto.spec.PSource;
import java.security.KeyPair;
import java.security.KeyPairGenerator;
import java.security.PrivateKey;
import java.security.PublicKey;
import java.security.SecureRandom;
import java.security.spec.MGF1ParameterSpec;

public class SecureRSAExample {

    // One definition of the parameters, handed to every init() call, so the
    // encrypt and decrypt sides cannot drift apart.
    public static OAEPParameterSpec oaepSha256() {
        return new OAEPParameterSpec(
            "SHA-256",                  // OAEP message digest
            "MGF1",                     // Mask generation function
            MGF1ParameterSpec.SHA256,   // MGF1 digest - SHA-1 if this is omitted
            PSource.PSpecified.DEFAULT  // Empty label
        );
    }

    public static KeyPair generateKeyPair() throws Exception {
        KeyPairGenerator keyGen = KeyPairGenerator.getInstance("RSA");
        keyGen.initialize(3072, new SecureRandom());  // NIST rates 2048-bit at 112-bit strength and accepts it only through 2030; 3072-bit gives 128-bit and stays valid beyond
        return keyGen.generateKeyPair();
    }

    public static byte[] encryptOAEP(byte[] plaintext, PublicKey publicKey) throws Exception {
        Cipher cipher = Cipher.getInstance("RSA/ECB/OAEPPadding");
        cipher.init(Cipher.ENCRYPT_MODE, publicKey, oaepSha256());
        return cipher.doFinal(plaintext);
    }

    public static byte[] decryptOAEP(byte[] ciphertext, PrivateKey privateKey) throws Exception {
        Cipher cipher = Cipher.getInstance("RSA/ECB/OAEPPadding");
        cipher.init(Cipher.DECRYPT_MODE, privateKey, oaepSha256());
        return cipher.doFinal(ciphertext);
    }
}

Why this works: OAEP makes RSA probabilistic, so identical plaintexts do not produce identical ciphertexts and there is no padding-valid signal for a Bleichenbacher probe to iterate on. The spec is passed rather than the longer transformation name because OAEP has two digest parameters and the name sets only one, so "RSA/ECB/OAEPPadding" plus an OAEPParameterSpec is the combination where nothing is left to a provider default. Both digests are then SHA-256, which is what every non-Java implementation of OAEP-SHA-256 assumes and what makes the ciphertext readable outside the JVM.

Passing the same spec to init() on the decrypt side is not symmetry for its own sake. The parameters are not carried in the ciphertext, so a decryptor that omits them silently uses MGF1-SHA-1 and fails with BadPaddingException on every message.

OAEPParameterSpec lives in javax.crypto.spec, not java.security.spec alongside MGF1ParameterSpec - the two are imported from different packages, which is easy to get wrong because the class names suggest otherwise.

Hybrid Encryption for Large Data

// SECURE - Hybrid encryption combining AES-GCM with RSA-OAEP
import javax.crypto.Cipher;
import javax.crypto.KeyGenerator;
import javax.crypto.SecretKey;
import javax.crypto.spec.GCMParameterSpec;
import javax.crypto.spec.SecretKeySpec;
import java.security.SecureRandom;

// Step 1: Generate random AES key
KeyGenerator keyGen = KeyGenerator.getInstance("AES");
keyGen.init(256, new SecureRandom());
SecretKey aesKey = keyGen.generateKey();

// Step 2: Encrypt large data with AES-GCM
byte[] iv = new byte[12];
new SecureRandom().nextBytes(iv);
Cipher aesCipher = Cipher.getInstance("AES/GCM/NoPadding");
aesCipher.init(Cipher.ENCRYPT_MODE, aesKey, new GCMParameterSpec(128, iv));
byte[] encryptedData = aesCipher.doFinal(largeData);

// Step 3: Encrypt AES key with RSA-OAEP, same explicit spec as above
Cipher rsaCipher = Cipher.getInstance("RSA/ECB/OAEPPadding");
rsaCipher.init(Cipher.ENCRYPT_MODE, publicKey, SecureRSAExample.oaepSha256());
byte[] encryptedKey = rsaCipher.doFinal(aesKey.getEncoded());

// Send: encryptedKey, iv, encryptedData

// Receiving side - unwrap the key, then the data
Cipher rsaOut = Cipher.getInstance("RSA/ECB/OAEPPadding");
rsaOut.init(Cipher.DECRYPT_MODE, privateKey, SecureRSAExample.oaepSha256());
SecretKey recovered = new SecretKeySpec(rsaOut.doFinal(encryptedKey), "AES");

Cipher aesOut = Cipher.getInstance("AES/GCM/NoPadding");
aesOut.init(Cipher.DECRYPT_MODE, recovered, new GCMParameterSpec(128, iv));
byte[] plaintext = aesOut.doFinal(encryptedData);   // equals largeData

Why this works: RSA's size limits make direct encryption impractical for large data. AES-GCM efficiently encrypts bulk data and provides integrity checks. RSA-OAEP wraps only the short ephemeral AES key, so the only value RSA has to carry is small enough to fit. The iv has to travel with the ciphertext - it is not secret, but without it the recovered AES key decrypts nothing - and the RSA step uses the same explicit OAEPParameterSpec as the direct case, because the transformation name alone would put MGF1 back on SHA-1 here too.

Considerations

Check whether the call site signs rather than encrypts. OAEP is an encryption padding, so a finding raised against RSA signing is not asking for it. Signing needs Signature.getInstance("RSASSA-PSS") with a PSSParameterSpec, and applying OAEP there is not a weaker fix but a meaningless one. Confirm which operation the code performs before choosing the padding, because the two look similar in a diff and the scanner rule often does not distinguish them.

Testing

  • Round trip: decryptOAEP(encryptOAEP(plaintext, pub), priv) returns a byte array equal to plaintext. Assert on Arrays.equals, not on the call completing - an encrypt-only test passes with parameters no decryptor can match.
  • Both digests are SHA-256: after init(), cipher.getParameters().toString() contains MGF: MGF1SHA-256. On JDK 26 the bare "RSA/ECB/OAEPWithSHA-256AndMGF1Padding" transformation reports MGF1SHA-1 here, and nothing else in the code distinguishes the two.
  • Cross-implementation decrypt: ciphertext from the Java encryptor decrypts in the peer that will actually consume it - Python padding.OAEP(mgf=MGF1(SHA256()), algorithm=SHA256()), Go rsa.DecryptOAEP(sha256.New(), ...), or .NET RSAEncryptionPadding.OaepSHA256. A BadPaddingException on the peer is the MGF1 mismatch, not a corrupt message.
  • Boundary input: a 318-byte payload encrypts on a 3072-bit key and 319 bytes throws IllegalBlockSizeException: Data must not be longer than 318 bytes (measured on JDK 26). Assert the largest payload the system actually sends is under that, or that it takes the hybrid path.
  • Probabilistic output: two encryptions of the same plaintext produce different ciphertexts. Equal output means the padding is not OAEP.
  • Uniform failures: a tampered ciphertext, a ciphertext for a different key, and a truncated one all reach the caller as the same generic error. Distinct messages or status codes per case are what makes an oracle usable.

Common Pitfalls

  • Fixing the encrypt call site but leaving a decrypt method (or a different service) that still calls Cipher.getInstance("RSA") or "RSA/ECB/PKCS1Padding" for backward compatibility with older messages - the JCE provider will decrypt PKCS#1 v1.5 ciphertexts on that path, so the padding-oracle attack the OAEP change was meant to close is still reachable through it.
  • Catching BadPaddingException from cipher.doFinal() and returning or logging the exception's message or type to the caller - a JCE provider's decrypt failures for invalid OAEP padding versus other errors are supposed to be indistinguishable, but re-throwing distinct messages or handling them in separate catch blocks with different response codes can rebuild a Manger-style oracle.
  • Passing the wrong hash to OAEPParameterSpec's MGF1 parameter while updating the main digest (e.g. OAEPParameterSpec("SHA-256", "MGF1", MGF1ParameterSpec.SHA1, ...)) - this is a legal RFC 8017 parameter set and SunJCE encrypts with it without complaint, so nothing rejects the mismatch; it just produces ciphertext no standard OAEP-SHA-256 decryptor can read. Omitting the spec entirely gives the same result by a different route, since that is what the bare transformation name already does.
  • Encrypting data over the ~318-byte limit by Base64-encoding or compressing it first, hoping it fits under the OAEP ceiling, instead of switching to hybrid encryption - this only delays the IllegalBlockSizeException to a larger input size and does not address RSA's fundamental unsuitability for bulk data.

Additional Resources