AES for CTF: Read the Ciphertext, Not the Math

The bytes are talking. You just have to look.

Here is a hexdump of a ciphertext from a picoCTF challenge. You don't have the key. You don't even know what algorithm produced it.

$ xxd body.txt.enc | head -8
00000000: 41 a8 7f c2 03 5e 9b 11 d4 65 22 b0 c7 fe 3a 99
00000010: 41 a8 7f c2 03 5e 9b 11 d4 65 22 b0 c7 fe 3a 99
00000020: 41 a8 7f c2 03 5e 9b 11 d4 65 22 b0 c7 fe 3a 99
00000030: 8c 47 ee 2d 91 50 ab f3 16 e8 33 d9 7c 4a 0b 5b
00000040: 41 a8 7f c2 03 5e 9b 11 d4 65 22 b0 c7 fe 3a 99
00000050: 8c 47 ee 2d 91 50 ab f3 16 e8 33 d9 7c 4a 0b 5b

Look at the first three rows. Identical, byte for byte, sixteen at a time. Row five matches row one. Row six matches row four.

Whatever the encryption scheme is, it sent the same ciphertext for the same plaintext, three times in a row, then again two rows later. That is not a property of any cipher you would actually want. It is the bug, and the bug just told you which mode this is, what the plaintext probably looks like, and how to attack it. You can finish the challenge before you read a line of the source.

This post covers the three AES modes you actually meet in picoCTF crypto (ECB, CBC, CTR), the fingerprint each one leaves in the ciphertext, and the one move you make once you have spotted it. Each mode gets a real picoCTF challenge to solve and a production CVE (Common Vulnerabilities and Exposures identifier) where the same bug shipped to a real system. They really are the same bug.

The first time someone showed me that move I felt cheated by every textbook that opened with finite fields and S-box derivations. None of it mattered. The cipher was fine. The mode was the bug.

AES in one minute, no math

AES (Advanced Encryption Standard) is a 128-bit block cipher: a function that turns a fixed-size 16-byte input into a 16-byte output, parameterized by a key. FIPS 197, the federal standard that specifies AES, calls it "a family of permutations of blocks that is parameterized by the key." In plain English: same key plus same input always produces the same output. That is the entire property the modes are trying to hide, and the entire property all the bugs leak.

A challenge ciphertext is almost never one block. The cipher only handles 16 bytes at a time, so encrypting longer messages requires a mode of operation: a recipe that decides what to do with the second block, the third block, and so on. Modes are the part where the engineer can actually screw up.

The three modes you will meet in picoCTF, in roughly the order they appear:

Three modes. Three bugs. That's the map.

ECB: the mode that tells you what's identical

The whole story is in the hexdump from the opener. ECB (Electronic Codebook) encrypts every 16-byte block independently under the same key. There is no chaining, no IV (initialization vector), no nonce (number used once). Identical plaintext blocks produce identical ciphertext blocks, every time, forever.

NIST (the US National Institute of Standards and Technology) called this out, almost forty years before the picoCTF authors did. SP 800-38A, Section 6.1: any given plaintext block always gets encrypted to the same ciphertext block. If this property is undesirable in a particular application, the ECB mode should not be used. The applications where it is desirable to leak the plaintext equality structure are, in practice, none of them.

The fingerprint detector is four lines of Python:

from collections import Counter
blocks = [ct[i:i+16] for i in range(0, len(ct), 16)]
dupes = [b for b, n in Counter(blocks).items() if n > 1]
if dupes: print(f'ECB. {len(dupes)} block(s) repeat.')

If that script prints anything, the cipher is ECB or something close enough that you can attack it the same way. picoCTF 2019's AES-ABC hands you a BMP image whose pixel rows were encrypted with AES-ECB. Large monochrome regions in the image become long runs of identical 16-byte blocks. Stitching the original 54-byte BMP header onto the front of the file makes the flag legible without decrypting anything.

The same shape made it to production. In October 2013, Adobe lost 153 million account records: passwords had been "encrypted" with 3DES in ECB mode under a single shared key (instead of hashed), so two users with the same password produced the same ciphertext, and helpfully-stored password hints leaked the rest by frequency analysis. NIST's SP 800-38A, which discourages ECB on anything longer than one block, had been published in 2001.

ECB is broken in the way the penguin is broken: visibly, embarrassingly, on a t-shirt.

A second ECB shape is worth knowing. picoCTF 2026's Timestamped Secrets uses AES-ECB with a key derived from the Unix timestamp at encryption time. The fingerprint is no longer about repeated blocks; it is about a key search space the engineer thought was 2¹²⁸(the full AES-128 keyspace) and is actually about ten thousand timestamps (one per second across a few-hour window). AES did its job. The key derivation didn't.

CBC: the mode you can puppet without the key

CBC (Cipher Block Chaining) was supposed to fix ECB by making blocks depend on each other. It does. The dependency is exactly the thing the attacker needs.

The decryption rule, from NIST SP 800-38A, Section 6.2:

P[0] = D_K(C[0]) XOR IV
P[i] = D_K(C[i]) XOR C[i-1]   for i >= 1

Stop and read that for a second. Plaintext block i is the AES decryption of ciphertext block i, XORed with ciphertext block i-1. The previous ciphertext block is right there in the formula, on the attacker's side of the wire, completely unauthenticated.

Flip a bit in C[i-1]. The XOR flips the same bit in P[i]. The price is that P[i-1] decrypts to garbage, because D_K(C[i-1])is now the decryption of a block AES never encrypted. The attacker doesn't care: trade one corrupted block of plaintext for surgical control over the next one. No key, no oracle, no math beyond XOR.

picoCTF 2021's More Cookies ships exactly this gadget. The login server stores the user record (something like isAdmin=0;username=user) as an AES-CBC ciphertext in a cookie. The same server reads the cookie back, decrypts it, and trusts whatever it sees. Find the byte position of the 0, XOR the corresponding byte of the previous ciphertext block with ord('0') ^ ord('1') = 1, send the cookie back, get admin.

import base64
ct = bytearray(base64.b64decode(cookie))
ct[target_pos] ^= ord('0') ^ ord('1')   # 0x01
print(base64.b64encode(bytes(ct)).decode())

The same gadget shipped on a much wider stage. POODLE (CVE-2014-3566, disclosed by Möller, Duong, and Kotowicz at Google) decrypted SSL 3.0 cookies one byte per ~256 requests by exploiting CBC's malleability and SSLv3's "MAC-then-encrypt" construction (where the server checks the MAC, the Message Authentication Code that detects tampering, only after decrypting; the attacker can flip ciphertext bytes and learn things from how the server fails). The ASP.NET padding oracle (CVE-2010-3332) used the same primitive plus verbose error messages to decrypt and forge __VIEWSTATE tokens and exfiltrate web.config. The bit-flip is older than most of the people reading this and it still ships.

The fingerprint to look for: any system that takes ciphertext from the user, decrypts it, and acts on the result without checking a separate authentication tag. If you can put bytes into a CBC ciphertext and the server treats the result as authoritative, the bit-flip is going to land. (Yes, CBC with a random IV and a separate HMAC over the ciphertext, the keyed-MAC variant that authenticates the bytes before decryption, is fine. No, you should not be reaching for it in 2026; reach for an AEAD construction instead.)

CTR: the mode where the nonce is the entire game

CTR (Counter mode) doesn't really use the cipher to encrypt your data. It uses the cipher to encrypt a counter, and then XORs the result with the plaintext. AES-CTR is, structurally, a stream cipher built out of AES.

KS[i] = E_K(nonce || counter[i])     # the keystream block
C[i]  = P[i] XOR KS[i]                # encrypt
P[i]  = C[i] XOR KS[i]                # decrypt is the same

There is exactly one rule. The pair (key, nonce) must never repeat. NIST SP 800-38A, Section 6.5, says the counter blocks "must be distinct" across all messages encrypted under one key. That sentence is doing a lot of work.

The math when the rule is broken is one line. Encrypt two messages with the same (key, nonce):

C1 = P1 XOR KS
C2 = P2 XOR KS
C1 XOR C2 = P1 XOR P2     # the keystream cancels

XOR the two ciphertexts and the keystream falls out. Whatever is left is the XOR of the two plaintexts, which is enough to crib-drag (slide guessed plaintext fragments along the XORed result, looking for English to fall out) on either one if the messages share any structure (and CTF flags always share picoCTF{). This is the two-time pad attack, the same flaw that made one-time pads insecure on reuse, the same flaw that broke WEP, the same flaw that turns up every other CTF crypto round.

AES-CTR is fine. The thing that isn't fine is that nobody can keep a counter unique across two processes.

picoCTF 2025's Tap Into Hash is the textbook version: a homemade scheme XORs every 16-byte block of the flag with the same SHA-256 hash, which is to say it reuses one block of keystream forever. Reading the script, computing the hash, XORing it back gets the flag. ChaCha-Slide is the AEAD cousin (Authenticated Encryption with Associated Data: ChaCha20-Poly1305, in this case) with a reused nonce. You can recover the keystream via the same XOR-cancellation move and (separately, with more math) forge valid Poly1305 tags. AES-CTR has the identical XOR-cancellation behavior; the reasoning is portable.

Real systems hit this rake constantly. WEP, the original Wi-Fi encryption standard, shipped with a 24-bit IV in 1999 (see the Berkeley ISAAC FAQ). The IV collided within hours on a busy access point, and Fluhrer, Mantin, and Shamir's 2001 paper turned the resulting keystream collisions plus RC4's key-schedule biases into full key recovery. Twenty-five years on, the failure shape hasn't changed.

The mode is part of the bug, not all of it

A mode without a MAC is malleable. A mode with a poorly-placed MAC is malleable in different ways. A mode with a well-placed MAC and a misused IV can leak a key directly. The mode is the obvious diagnostic, but a few cousin bugs sit one layer over and they all show up in CTFs.

Padding oracles attack CBC, but the bug is not in CBC. It is in the server's habit of telling the attacker whether a decrypted plaintext had valid PKCS#7 padding (the standard way to round a message up to a whole number of 16-byte blocks). RFC 5652 defines the padding rule: append k bytes of value k. The validity check leaks one bit per request, and one bit per request adds up to byte-by-byte plaintext recovery. POODLE used a variant of this. So did CVE-2010-3332. So does every "submit modified ciphertext, get padding error or decryption error" CTF challenge.

IV-as-key. If the IV is reused across messages with the same key, CBC degenerates. The first plaintext block of every message is XORed with the same constant D_K(C[0]) XOR IV, which means two ciphertexts encrypted under the same key and IV reveal the XOR of their first plaintext blocks. The same crib-drag move that breaks CTR breaks the first block of CBC. ECB is the limiting case of this bug, where the "IV" is implicitly all zeros for every block.

AEAD modes (Authenticated Encryption with Associated Data: GCM, CCM, GCM-SIV). AES-GCM is CTR plus a polynomial MAC, and it has the same nonce-reuse cliff as CTR, only worse: NIST SP 800-38D, Section 8, calls a single nonce repetition "almost as important as the secrecy of the key." Böck, Zauner, and Devlin (2016) ran an internet-wide scan and found 184 HTTPS servers reusing GCM nonces in the wild, including financial institutions and a credit-card processor. AES-GCM-SIV exists because engineers cannot be trusted with a counter, which is to say all of us.

The cheat sheet

GCM is CTR plus a tag. The nonce-reuse cliff is identical in shape and worse in consequence; the auth key is recoverable too. Böck-Zauner-Devlin (2016) is the canonical reference if you meet AES-GCM in a CTF.

Where to start

AES-ABC if you have never seen ECB. More Cookies for the CBC bit-flip. Tap Into Hash for keystream reuse. The picoCTF crypto category leans on these three shapes; once you have solved one of each, the rest are pattern-matching exercises with new wrapping paper.

The thing to do, when you next open a crypto challenge, is read the bytes before the source. The repeats, the absent IV, the duplicate nonce, the Base64-decoded cookie that's exactly a multiple of sixteen: those are the bug reporting itself. The source code only confirms what the ciphertext already told you. I have lost more time than I'd like to admit reading hundred-line Python encryption scripts before checking whether the ciphertext repeats every 16 bytes.

For the math-heavy half of CTF crypto (RSA, lattice, discrete log), see the RSA Attacks for CTF post. For the tooling, the Pwntools for CTF post. For the cookie machinery behind More Cookies, the Cookies and JWTs in CTF post. If you are starting from zero on picoCTF, the Beginner's Guide sets up the rest.

Open a challenge. Find the AES import. Find the mode. Look up the one bug for that mode. That is the whole game.