Skip to main content
picoCTF Solutions
picoCTF 2026CryptographyEasy

StegoRSA picoCTF 2026 Solution

Retrieve an RSA private key carelessly hidden in image file metadata, then use it to decrypt the flag.

Published: March 20, 2026Updated: September 22, 2026

Description

A message has been encrypted using RSA. The public key is gone... but someone might have been careless with the private key. Can you recover it and decrypt the message? Download the flag.enc and image.jpg.

Setup

Download flag.enc
Download flag.enc and image.jpg.
Examine the image metadata - something may be hidden there.
bash
exiftool image.jpg
bash
strings image.jpg

Solution

Want to try it yourself first?

The guided walkthrough reveals hints one step at a time.

Walk me through it
  1. Step 1Find the private key in the image EXIF metadata
    Observation
    There is an image.jpg next to the encrypted file, and a hint that someone was careless with the private key. It is not in a separate file; look in the image's EXIF metadata.
    Run exiftool on image.jpg. The Comment field contains a large hex string. This hex string, when decoded, is the RSA private key PEM file.
    bash
    exiftool image.jpg
    bash
    # The Comment field contains a large hex string
    bash
    exiftool -b -Comment image.jpg > key.hex

    Expected output

    Comment                         : 2d2d2d2d2d424547494e2050524956415445204b45592d2d2d2d2d0a4d494945766749424144414e42676b71686b6947397730424151454641415343424b67... (3408 hex characters)
    What didn't work first

    Tried: Run strings image.jpg and grep the output for the key instead of using exiftool.

    strings pulls printable sequences out of raw bytes without parsing EXIF, so you get hundreds of unrelated strings from JFIF headers and color profiles, and the hex comment can fragment across line boundaries. exiftool reads the tag directly and returns the field intact, which is what you want to pipe into a file.

    Tried: Use exiftool image.jpg without the -b flag and copy the Comment value from terminal output.

    Without -b, exiftool prints the field as a labelled line: the tag name 'Comment', column padding, then the value and a newline. The value itself is not truncated, but the label and padding come with it, so redirecting that into a file gives you text that is not pure hex and unhexlify rejects it. -b prints the raw field value and nothing else.

    Learn more

    Why `-b` matters here. By default exiftool prints the tag name, column padding and a trailing newline around the value, so a multi-kilobyte hex string arrives wrapped in text that is not hex. -b (binary mode) bypasses print conversion and dumps the raw field value alone, which is exactly what you want when piping into a file or hex decoder.

    EXIF metadata was originally designed for camera settings (shutter, aperture, GPS, timestamp), but the spec includes free-form text fields like Comment, UserComment, ImageDescription, and Artist that can hold arbitrary data. Hiding a key here is light steganography: the image renders normally; only metadata inspection reveals the payload.

    Real-world organizations strip metadata before publishing images (mat2, ImageMagick's -strip) precisely because camera model, GPS, and software version leak operational details. See the Steganography Tools guide for a broader EXIF/strings/binwalk workflow.

  2. Step 2Decode the hex string to the PEM private key
    Observation
    The Comment field holds a long run of hex digits rather than text, which is how binary PEM data gets into a plain-text EXIF field. Unhexlify it back into a key file.
    Use Python's binascii.unhexlify() to convert the hex comment into the private key PEM bytes, then decode to ASCII and save as private.pem. In CyberChef use 'From Hex' with delimiter set to 'None'.
    python
    python3 -c "
import binascii
hex_key = open('key.hex').read().strip()
pem = binascii.unhexlify(hex_key).decode()
open('private.pem', 'w').write(pem)
print('Key written to private.pem')
"
    bash
    openssl pkey -text -noout -in private.pem | head
    What didn't work first

    Tried: Use CyberChef 'From Hex' with 'Auto' delimiter to decode the hex comment.

    From Hex on Auto expects separated pairs, and exiftool gives you a continuous string with no delimiters, so CyberChef reads the whole blob as one token and returns garbage. Set the delimiter to None, or use Python's unhexlify, which handles run-together hex natively.

    Tried: Open key.hex in a text editor and manually copy the content into openssl pkcs8 -inform PEM to check it.

    At this point the file is still hex, not PEM, and openssl reports no start line because it is looking for the BEGIN header. Convert it first, then verify the resulting key file.

    Learn more

    What hex-encoded PEM looks like. A real PEM file is plain ASCII:

    -----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0...
...
-----END PRIVATE KEY-----

    Hex-encoded, every byte becomes two ASCII hex digits, so the dashes, newlines, and base64 body all show up as a long string of [0-9a-f]. The file starts with 2d2d2d2d2d424547494e (which is "-----BEGIN" in hex). That pattern is the giveaway: if you see a hex blob whose first bytes decode to -----BEGIN, run unhexlify and you have a PEM.

    PEM is base64-encoded DER wrapped in -----BEGIN/END----- markers. The key here is a 2048-bit RSA key in PKCS#8 format (indicated by the -----BEGIN PRIVATE KEY----- header, as opposed to the PKCS#1 -----BEGIN RSA PRIVATE KEY----- form). The openssl pkey -text verification step is non-negotiable: if exiftool grabbed the wrong field or you decoded the wrong hex span, openssl will say so before you waste time on a broken decryption.

  3. Step 3Decrypt the flag
    Observation
    With the key recovered and flag.enc in hand, the decryption is ordinary RSA. The padding mode has to match whatever encrypted it, and here that is the PKCS#1 v1.5 default.
    PKCS#1 v1.5, the openssl default, is the right mode here: the 256-byte flag.enc decrypts straight to the flag. Note that rsautl is deprecated in OpenSSL 3.x, so pkeyutl is the current spelling; both work. Only if a challenge's encryption script imports PKCS1_OAEP do you switch the padding mode.
    bash
    # PKCS#1 v1.5 (the default, and what this challenge used):
    bash
    openssl pkeyutl -decrypt -inkey private.pem -in flag.enc -out flag.txt
    bash
    # Same thing with the deprecated pre-3.0 spelling:
    bash
    openssl rsautl -decrypt -inkey private.pem -in flag.enc -out flag.txt
    bash
    # OAEP, for a challenge whose script imports PKCS1_OAEP (fails here):
    bash
    openssl pkeyutl -decrypt -inkey private.pem -in flag.enc -pkeyopt rsa_padding_mode:oaep
    bash
    cat flag.txt
    What didn't work first

    Tried: Add -oaep (or -pkeyopt rsa_padding_mode:oaep) on the assumption that modern RSA challenges always use OAEP.

    This ciphertext was produced with PKCS#1 v1.5, so the OAEP unpadding check rejects it: openssl prints 'RSA_padding_check_PKCS1_OAEP_mgf1:oaep decoding error' and writes nothing. Padding is not a preference, it has to match the encryptor. Try the default first and only switch if it fails.

    Tried: Force raw RSA with -pkeyopt rsa_padding_mode:none to 'see the real plaintext'.

    That strips no framing at all, so you get the full 256-byte padded block: a 0x00 0x02 header, random non-zero filler, a 0x00 separator, and only then the flag at the tail. It is readable if you squint, but the plain default (v1.5) removes the padding for you. Note pkeyutl does NOT default to raw: with no -pkeyopt it uses PKCS#1 v1.5, exactly like the deprecated rsautl.

    Learn more

    PKCS#1 v1.5 vs OAEP. Both are RSA padding schemes. PKCS#1 v1.5 is the legacy default and what openssl rsautl -decrypt assumes. OAEP (Optimal Asymmetric Encryption Padding) is the modern recommendation: same RSA primitive, but with a randomized hash-based padding that is provably secure against chosen-ciphertext attacks. They are not interchangeable in either direction: this challenge's ciphertext is v1.5, and asking for OAEP gets you oaep decoding error rather than a flag. If you see PKCS1_OAEP in a provided Python script, then add -pkeyopt rsa_padding_mode:oaep (or the older -oaep flag on rsautl) to the openssl call.

    The challenge combines steganography (hiding the key in EXIF) with RSA decryption (using it). The actual vulnerability isn't in the math; it's that the "private" key wasn't private. This mirrors real incidents where developers commit private keys to public repos or embed them in shipped binaries. See the RSA Attacks for CTF guide for the broader catalog of RSA-specific bugs (small e, common modulus, Wiener, etc.).

Interactive tools
  • StegallDrop any file and Stegall runs every applicable steg technique in parallel: LSB sweeps, bit planes, spectrograms, polyglot carving, metadata, whitespace decode, and a 6-layer base/ROT/XOR/zlib cascade. Recursively unpacks results and surfaces flag matches.
  • Hex ViewerView text or raw hex bytes as a xxd-style hex dump with byte offset, hex columns, and ASCII sidebar. Highlights printable characters and null bytes.
  • Strings ExtractorPull printable text from any binary, library, or image. ASCII and UTF-16 detection, configurable minimum length, flag-like highlight, no command line needed.
Alternate Solution

If you prefer not to bounce through OpenSSL, use the RSA Calculator on this site. Pull the four inputs out of the recovered PEM with openssl pkey -text -noout -in private.pem and the ciphertext from flag.enc:

  • p, q: the two prime factors of the modulus, used to reconstruct the private exponent.
  • e: the public exponent (almost always 65537).
  • c: the ciphertext as an integer (read flag.enc as bytes and convert big-endian).

The calculator computes n = p*q, phi = (p-1)(q-1), d = e^-1 mod phi, then m = c^d mod n. Decode m back to bytes to read the flag.

Flag

Reveal flag

picoCTF{rs4_k3y_1n_1mg_...}

Run exiftool on the image to find the 3408-character hex-encoded RSA private key in the JPEG Comment field. Decode hex to a 2048-bit PKCS#8 PEM with CyberChef or Python, then decrypt flag.enc with 'openssl pkeyutl -decrypt -inkey private.pem -in flag.enc' (PKCS#1 v1.5, the default). The flag is shown abbreviated on this page; work the steps above to recover the full value.

Key takeaway

RSA is only as secure as the private key is private. The math is unbroken, and a key leaked through EXIF metadata, a public commit, or a misconfigured file server ends it anyway. EXIF fields take free-form text while the image renders identically to the eye, which makes them a natural hiding place for attackers and challenge authors alike. Incident responders find credentials in image metadata, binary strings, and git history constantly, which is why organizations strip metadata before publishing and rotate anything that ever reached source control.

Related reading

Useful tools for Cryptography

Where to go next

Try these next