picoCTF Solutions
picoCTF 2026CryptographyEasy

StegoRSA picoCTF 2026 Solution

Published: March 20, 2026

Description

A message has been encrypted using RSA. The public key is gone... but someone might have been careless with the private key. Can you recover it and decrypt the message? Download the flag.enc and image.jpg .

Setup

Download flag.enc

Download flag.enc and image.jpg.

Examine the image metadata - something may be hidden there.

bash
exiftool image.jpg
bash
strings image.jpg

Solution

Walk me through it
  1. Step 1Extract the RSA private key from the image
    The key is in the EXIF Comment as a hex string. Default exiftool truncates long fields, so use -b to dump the full raw value to a file.
    bash
    exiftool image.jpg
    bash
    exiftool -b -Comment image.jpg > key.hex
    Learn more

    Why `-b` matters here. By default exiftool wraps long values for terminal display and applies print conversions, so a multi-kilobyte hex string in the Comment field gets visually truncated and you grab the wrong slice. -b (binary mode) bypasses print conversion and dumps the raw field value, which is exactly what you want when piping into a file or hex decoder.

    EXIF metadata was originally designed for camera settings (shutter, aperture, GPS, timestamp), but the spec includes free-form text fields like Comment, UserComment, ImageDescription, and Artist that can hold arbitrary data. Hiding a key here is light steganography: the image renders normally; only metadata inspection reveals the payload.

    Real-world organizations strip metadata before publishing images (mat2, ImageMagick's -strip) precisely because camera model, GPS, and software version leak operational details. See the Steganography Tools guide for a broader EXIF/strings/binwalk workflow.

  2. Step 2Decode the private key
    The hex string is a hex-encoded PEM file. Decode and verify it parses cleanly before relying on it.
    python
    python3 -c "
import binascii
hex_key = open('key.hex').read().strip()
pem = binascii.unhexlify(hex_key).decode()
open('private.pem', 'w').write(pem)
print('Key written to private.pem')
"
    bash
    # Verify the key parses (catches truncation / wrong-field extraction):
    bash
    openssl rsa -text -noout -in private.pem | head
    Learn more

    What hex-encoded PEM looks like. A real PEM file is plain ASCII:

    -----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEA0Z9vM5...
...
-----END RSA PRIVATE KEY-----

    Hex-encoded, every byte becomes two ASCII hex digits, so the dashes, newlines, and base64 body all show up as a long string of [0-9a-f]. The file starts with 2d2d2d2d2d424547494e (which is "-----BEGIN" in hex). That pattern is the giveaway: if you see a hex blob whose first bytes decode to -----BEGIN, run unhexlify and you have a PEM.

    PEM is base64-encoded DER wrapped in -----BEGIN/END----- markers. RSA private keys in PKCS#1 PEM contain (n, e, d, p, q, dp, dq, qinv) for efficient CRT decryption. The openssl rsa -text verification step is non-negotiable: if exiftool grabbed the wrong field or you decoded the wrong hex span, openssl will say so before you waste time on a broken decryption.

  3. Step 3Decrypt the flag
    Try PKCS#1 v1.5 first (the openssl default). If the output looks like garbage and the encryption script imports OAEP or specifies padding=PKCS1_OAEP, retry with -oaep.
    bash
    # PKCS#1 v1.5 (default):
    bash
    openssl rsautl -decrypt -inkey private.pem -in flag.enc -out flag.txt
    bash
    # OAEP (if the encryption script uses PKCS1_OAEP):
    bash
    openssl rsautl -decrypt -oaep -inkey private.pem -in flag.enc -out flag.txt
    bash
    cat flag.txt
    Learn more

    PKCS#1 v1.5 vs OAEP. Both are RSA padding schemes. PKCS#1 v1.5 is the legacy default and what openssl rsautl -decrypt assumes. OAEP (Optimal Asymmetric Encryption Padding) is the modern recommendation: same RSA primitive, but with a randomized hash-based padding that is provably secure against chosen-ciphertext attacks. They are not interchangeable - decrypting an OAEP ciphertext with PKCS#1 v1.5 padding yields random bytes (or a padding error). If you see PKCS1_OAEP in any provided Python script, add -oaep to the openssl call.

    The challenge combines steganography (hiding the key in EXIF) with RSA decryption (using it). The actual vulnerability isn't in the math; it's that the "private" key wasn't private. This mirrors real incidents where developers commit private keys to public repos or embed them in shipped binaries. See the RSA Attacks for CTF guide for the broader catalog of RSA-specific bugs (small e, common modulus, Wiener, etc.).

Alternate Solution

If you prefer not to bounce through OpenSSL, use the RSA Calculator on this site. Pull the four inputs out of the recovered PEM with openssl rsa -text -noout -in private.pem and the ciphertext from flag.enc:

  • p, q: the two prime factors of the modulus, used to reconstruct the private exponent.
  • e: the public exponent (almost always 65537).
  • c: the ciphertext as an integer (read flag.enc as bytes and convert big-endian).

The calculator computes n = p*q, phi = (p-1)(q-1), d = e^-1 mod phi, then m = c^d mod n. Decode m back to bytes to read the flag.

Flag

picoCTF{...}

The RSA private key is hidden in the EXIF metadata (Comment field) of the image as a hex-encoded PEM string.

Want more picoCTF 2026 writeups?

Browse picoCTF 2026Full library

Useful tools for Cryptography

Related reading

What to try next