Description
A message has been encrypted using RSA. The public key is gone... but someone might have been careless with the private key. Can you recover it and decrypt the message? Download the flag.enc and image.jpg .
Setup
Download flag.enc and image.jpg.
Examine the image metadata - something may be hidden there.
exiftool image.jpgstrings image.jpgSolution
Want to try it yourself first?
The guided walkthrough reveals hints one step at a time.
Step 1
Find the private key in the image EXIF metadataObservationI noticed the challenge provided an image.jpg alongside the encrypted file and hinted that someone was careless with the private key, which suggested the key was not stored in a separate file but embedded in the image itself via EXIF metadata fields like Comment.Run exiftool on image.jpg. The Comment field contains a large hex string. This hex string, when decoded, is the RSA private key PEM file.bashexiftool image.jpgbash# The Comment field contains a large hex stringbashexiftool -b -Comment image.jpg > key.hexExpected output
picoCTF{RS4_k3y_1n_1m4g3_...}What didn't work first
Tried: Run strings image.jpg and grep the output for the key instead of using exiftool.
strings extracts printable ASCII sequences from raw bytes but does not parse EXIF structure, so it dumps hundreds of unrelated strings (JFIF headers, color profiles, embedded text) and the hex comment can get fragmented or split across line-length boundaries. exiftool reads the EXIF IFD tag directly and returns the full field value intact, which is what you need to pipe cleanly into key.hex.
Tried: Use exiftool image.jpg without the -b flag and copy the Comment value from terminal output.
Without -b, exiftool applies its print conversion and wraps long values at the terminal width, silently truncating the hex string mid-character. Pasting that truncated output into a hex decoder produces an incomplete or misaligned byte sequence, causing unhexlify to raise a ValueError or produce a truncated PEM that openssl rejects with 'no start line'.
Learn more
Why `-b` matters here. By default exiftool wraps long values for terminal display and applies print conversions, so a multi-kilobyte hex string in the Comment field gets visually truncated and you grab the wrong slice.
-b(binary mode) bypasses print conversion and dumps the raw field value, which is exactly what you want when piping into a file or hex decoder.EXIF metadata was originally designed for camera settings (shutter, aperture, GPS, timestamp), but the spec includes free-form text fields like Comment, UserComment, ImageDescription, and Artist that can hold arbitrary data. Hiding a key here is light steganography: the image renders normally; only metadata inspection reveals the payload.
Real-world organizations strip metadata before publishing images (
mat2, ImageMagick's-strip) precisely because camera model, GPS, and software version leak operational details. See the Steganography Tools guide for a broader EXIF/strings/binwalk workflow.Step 2
Decode the hex string to the PEM private keyObservationI noticed the Comment field from exiftool contained a long string of hex digits rather than readable text, and hex-encoding is the standard way to store binary PEM data in a plain-text EXIF field, which suggested running unhexlify to recover the actual private.pem file.Use Python's binascii.unhexlify() to convert the hex comment into the private key PEM bytes, then decode to ASCII and save as private.pem. In CyberChef use 'From Hex' with delimiter set to 'None'.pythonpython3 -c " import binascii hex_key = open('key.hex').read().strip() pem = binascii.unhexlify(hex_key).decode() open('private.pem', 'w').write(pem) print('Key written to private.pem') "bashopenssl pkey -text -noout -in private.pem | headWhat didn't work first
Tried: Use CyberChef 'From Hex' with 'Auto' delimiter to decode the hex comment.
'From Hex' with Auto delimiter expects space- or colon-separated hex pairs (e.g. '2d 2d 2d'), while the raw exiftool output is a continuous hex string with no delimiters. CyberChef treats the whole blob as a single token and produces garbage output. The correct CyberChef operation for a continuous hex string is 'From Hex' with delimiter set to 'None', or the Python binascii.unhexlify path which handles run-together hex natively.
Tried: Open key.hex in a text editor and manually copy the content into openssl pkcs8 -inform PEM to check it.
key.hex is still a raw hex string at this point, not a PEM file. Feeding hex text to openssl pkcs8 -inform PEM gives 'no start line' because openssl is looking for '-----BEGIN'. You must run the unhexlify conversion step first to produce private.pem, then run the openssl verification against that file.
Learn more
What hex-encoded PEM looks like. A real PEM file is plain ASCII:
-----BEGIN PRIVATE KEY----- MIIEvQIBADANBgkqhkiG9w0... ... -----END PRIVATE KEY-----
Hex-encoded, every byte becomes two ASCII hex digits, so the dashes, newlines, and base64 body all show up as a long string of
[0-9a-f]. The file starts with2d2d2d2d2d424547494e(which is "-----BEGIN" in hex). That pattern is the giveaway: if you see a hex blob whose first bytes decode to-----BEGIN, rununhexlifyand you have a PEM.PEM is base64-encoded DER wrapped in
-----BEGIN/END-----markers. The key here is PKCS#8 format (indicated by the-----BEGIN PRIVATE KEY-----header, as opposed to the PKCS#1-----BEGIN RSA PRIVATE KEY-----form). Theopenssl pkey -textverification step is non-negotiable: if exiftool grabbed the wrong field or you decoded the wrong hex span, openssl will say so before you waste time on a broken decryption.Step 3
Decrypt the flagObservationI noticed that with the recovered private.pem in hand and flag.enc provided by the challenge, the final step was a straightforward RSA decryption, but the hint about OAEP in the encryption script meant the padding mode had to be matched exactly or openssl would produce garbled output.Try PKCS#1 v1.5 first (the openssl default). If the output looks like garbage and the encryption script importsOAEPor specifiespadding=PKCS1_OAEP, retry with-oaep.bash# PKCS#1 v1.5 (default):bashopenssl rsautl -decrypt -inkey private.pem -in flag.enc -out flag.txtbash# OAEP (if the encryption script uses PKCS1_OAEP):bashopenssl rsautl -decrypt -oaep -inkey private.pem -in flag.enc -out flag.txtbashcat flag.txtWhat didn't work first
Tried: Run openssl rsautl -decrypt without the -oaep flag when the encryption script used PKCS1_OAEP padding.
openssl rsautl defaults to PKCS#1 v1.5 padding. When the ciphertext was produced with OAEP, the padding bytes in the decrypted block do not match the v1.5 structure, so openssl either returns 'RSA_padding_check_PKCS1_type_2: padding check failed' or silently outputs garbled bytes. Adding -oaep tells openssl to use the OAEP unpadding path, which matches how the message was originally padded.
Tried: Use openssl pkeyutl instead of openssl rsautl without specifying the decryption padding option.
openssl pkeyutl -decrypt with no padding option uses raw RSA (no padding at all), which strips neither PKCS#1 v1.5 nor OAEP framing and passes the raw decrypted integer bytes directly - producing a large binary blob instead of the flag string. For pkeyutl you must add -pkeyopt rsa_padding_mode:oaep (or :pkcs1) explicitly, unlike rsautl where -oaep is a simple flag.
Learn more
PKCS#1 v1.5 vs OAEP. Both are RSA padding schemes. PKCS#1 v1.5 is the legacy default and what
openssl rsautl -decryptassumes. OAEP (Optimal Asymmetric Encryption Padding) is the modern recommendation: same RSA primitive, but with a randomized hash-based padding that is provably secure against chosen-ciphertext attacks. They are not interchangeable - decrypting an OAEP ciphertext with PKCS#1 v1.5 padding yields random bytes (or a padding error). If you seePKCS1_OAEPin any provided Python script, add-oaepto the openssl call.The challenge combines steganography (hiding the key in EXIF) with RSA decryption (using it). The actual vulnerability isn't in the math; it's that the "private" key wasn't private. This mirrors real incidents where developers commit private keys to public repos or embed them in shipped binaries. See the RSA Attacks for CTF guide for the broader catalog of RSA-specific bugs (small e, common modulus, Wiener, etc.).
Interactive tools
- StegallDrop any file and Stegall runs every applicable steg technique in parallel: LSB sweeps, bit planes, spectrograms, polyglot carving, metadata, whitespace decode, and a 6-layer base/ROT/XOR/zlib cascade. Recursively unpacks results and surfaces flag matches.
- Hex ViewerView text or raw hex bytes as a xxd-style hex dump with byte offset, hex columns, and ASCII sidebar. Highlights printable characters and null bytes.
- Strings ExtractorPull printable text from any binary, library, or image. ASCII and UTF-16 detection, configurable minimum length, flag-like highlight, no command line needed.
Alternate Solution
If you prefer not to bounce through OpenSSL, use the RSA Calculator on this site. Pull the four inputs out of the recovered PEM with openssl pkey -text -noout -in private.pem and the ciphertext from flag.enc:
- p, q: the two prime factors of the modulus, used to reconstruct the private exponent.
- e: the public exponent (almost always 65537).
- c: the ciphertext as an integer (read
flag.encas bytes and convert big-endian).
The calculator computes n = p*q, phi = (p-1)(q-1), d = e^-1 mod phi, then m = c^d mod n. Decode m back to bytes to read the flag.
Flag
Reveal flag
picoCTF{RS4_k3y_1n_1m4g3_...}
Run exiftool on the image to find the hex-encoded RSA private key in the Comment field. Decode hex to PEM with CyberChef or Python, then decrypt flag.enc with 'openssl rsautl -decrypt -inkey private.pem -in flag.enc'.