Reversing Custom Ciphers for CTF: Breaking Homebrew Encryption

What counts as a custom cipher

A custom cipher is any encryption scheme the challenge author wrote themselves instead of calling a library. You'll recognize the genre immediately: a Python file named encrypt.py, forty lines of loops and modular arithmetic, and an output file full of numbers or base64. No AES, no RSA, just somebody's idea of scrambling.

These challenges frustrate people because the standard cryptography playbook doesn't apply. There's no published attack on a cipher that has existed for six weeks and has a total user base of one. Running a cipher identifier returns nothing useful, because identification tools match against known ciphers and this one is known to nobody.

Here's the reframe that makes them easy. Custom ciphers in CTF are almost never novel mathematics. They're short pipelines built from a handful of reversible primitives: add a constant, multiply mod something, XOR against a repeating key, permute the alphabet, chain each byte into the next. The author's creativity went into combining the primitives, not inventing one. So the job isn't cryptanalysis. The job is classification and inversion: figure out which primitives are in the pipeline, then run the pipeline backwards.

That distinction matters more than it sounds. Cryptanalysis is hard and open-ended. Inversion is mechanical. Every minute you spend trying to be clever about a homebrew cipher is a minute you could have spent asking it questions.

The three-probe test: what kind of cipher is this?

Most guides teach you to identify a cipher by staring at its output. That works for classical ciphers, where a Vigenere ciphertext has a recognizable index of coincidence and a rail fence has a recognizable shape. It fails on custom ciphers by construction, because the whole point is that nobody has seen this one before.

The chosen-plaintext probe works instead. If you have the encryption function, whether as a Python file you can import or a remote service you can send data to, you can ask it direct questions. Three of them settle the structure.

Probe 1: is it deterministic?

Encrypt the same plaintext twice. If the ciphertexts differ, there's a random nonce, an initialization vector, or a timestamp in the mix. That's not necessarily bad news: a weak nonce is often the intended vulnerability, and a nonce derived from the clock is usually brute-forceable over a few thousand candidate seconds.

Probe 2: does length change?

Encrypt inputs of length 1, 15, 16, 17, and 32. If ciphertext length tracks plaintext length exactly, you have a stream-like or substitution cipher and each output position corresponds to an input position. If output length jumps to the next multiple of 8 or 16, you have a block cipher with padding, and the block size just announced itself.

Probe 3: how far does one flipped byte travel?

This is the one that does the real work. Take a fixed baseline plaintext, flip exactly one byte, encrypt, and record which output positions changed. Repeat for every input position. The shape of the resulting diffusion map tells you the cipher family outright.

#!/usr/bin/env python3
"""Classify an unknown cipher by chosen-plaintext probing."""
 
def enc(data: bytes) -> bytes:
    # Replace this with the challenge's encryption call.
    # For a local encrypt.py:   from encrypt import encrypt; return encrypt(data)
    # For a remote service:    io.sendline(data.hex().encode()); return bytes.fromhex(io.recvline().strip().decode())
    raise NotImplementedError
 
 
def probe(n: int = 32) -> None:
    base = bytes(n)
 
    # Probe 1: determinism
    print('deterministic:', enc(base) == enc(base))
 
    # Probe 2: length behaviour
    for size in (1, 15, 16, 17, 32):
        print(f'len(pt)={size:3d} -> len(ct)={len(enc(bytes(size))):3d}')
 
    # Probe 3: diffusion map
    c0 = enc(base)
    for i in range(n):
        pt = bytearray(base)
        pt[i] ^= 0xFF
        c1 = enc(bytes(pt))
        changed = [j for j in range(min(len(c0), len(c1))) if c0[j] != c1[j]]
        print(f'flip byte {i:2d} -> {len(changed):3d} output bytes changed, first at {changed[0] if changed else None}')
 
 
if __name__ == '__main__':
    probe()

Read the output of probe 3 against this table:

Byte-independent ciphers fall in one loop

If flipping input byte i only ever changes output byte i, then the cipher is a function of one byte at a time. It might be a Caesar shift, an affine map, an XOR with a constant, a scrambled alphabet, or something nobody has a name for. It does not matter. A function from one byte to one byte has exactly 256 possible inputs, so you can tabulate the entire thing.

# Build the full forward table, then invert it.
forward = {i: enc(bytes([i]))[0] for i in range(256)}
reverse = {v: k for k, v in forward.items()}
 
if len(reverse) < len(forward):
    print('warning: not injective, some plaintexts collide')
 
plain = bytes(reverse[b] for b in ciphertext)
print(plain)

Two wrinkles show up often enough to plan for. The first: the map may depend on position as well as value, so byte i is transformed by a rule that includes iitself. The diffusion map still shows one-to-one locality, but a single table won't invert it. Build one table per position instead.

# Position-dependent substitution: one table per index.
n = len(ciphertext)
tables = []
for pos in range(n):
    table = {}
    for byte in range(256):
        probe_pt = bytearray(n)
        probe_pt[pos] = byte
        table[enc(bytes(probe_pt))[pos]] = byte
    tables.append(table)
 
plain = bytes(tables[i][c] for i, c in enumerate(ciphertext))

That costs 256 * nencryption calls, which is instant locally and worth measuring first against a remote service. For a 50-byte flag it's 12,800 round trips, so batch them or find the pattern by hand.

The second wrinkle: the map may not be injective. If two plaintext bytes encrypt to the same ciphertext byte, information is genuinely gone and you'll need context to resolve the ambiguity. Flags are ASCII and follow a known format, so filter candidate decryptions by printability and the picoCTF{ prefix.

rotation and new-caesar both live in this family. So does reverse-cipher, where the transform is a short chain of per-byte operations that reads almost the same backwards as forwards.

Arithmetic and modular ciphers: invert the algebra

The most common homebrew scheme in CTF is some flavor of c = (a * p + b) mod m. Authors like it because it looks mathematical, and it is, but it's also the single most invertible thing in cryptography.

To reverse a multiplication modulo m, you need the modular inverse of a, which exists exactly when gcd(a, m) = 1. Python 3.8 and later give it to you directly with pow(a, -1, m).

# Affine cipher over bytes: c = (a * p + b) mod 256
def affine_decrypt(ct: bytes, a: int, b: int, m: int = 256) -> bytes:
    a_inv = pow(a, -1, m)  # raises ValueError if gcd(a, m) != 1
    return bytes(((a_inv * (c - b)) % m) for c in ct)
 
 
# If a and b are unknown, brute-force them. There are only 128 valid
# multipliers mod 256 (the odd ones) and 256 offsets.
from math import gcd
 
for a in range(1, 256):
    if gcd(a, 256) != 1:
        continue
    for b in range(256):
        guess = affine_decrypt(ciphertext, a, b)
        if b'picoCTF{' in guess:
            print(a, b, guess)

Watch the modulus. When authors work over the printable alphabet they often use mod 26 or mod 95 instead of mod 256, and the decryption silently produces garbage if you guess wrong. If the ciphertext bytes all sit inside a narrow range, that range is telling you the modulus.

A larger version of the same idea appears when the cipher chunks the plaintext into integers and does arithmetic on big numbers. Look for the chunk size, convert back with int.to_bytes, and check whether the operation is invertible over the integers before assuming you need anything cleverer.

# Plaintext chunked into big integers, then transformed.
def chunks_to_bytes(values: list[int], size: int) -> bytes:
    out = b''
    for v in values:
        out += v.to_bytes(size, 'big')
    return out.lstrip(b'\x00')
 
 
# Try both endiannesses and a few chunk sizes when the size isn't stated.
for size in (2, 4, 8, 16):
    for order in ('big', 'little'):
        try:
            guess = b''.join(v.to_bytes(size, order) for v in values)
        except OverflowError:
            continue
        if b'pico' in guess:
            print(size, order, guess)

custom-encryption is the canonical example: a Diffie-Hellman-flavored key exchange wrapped around what turns out to be a simple multiply-and-XOR pipeline. The key exchange is theater. The cipher underneath inverts in four lines. c3 and rolling-my-own reward the same skepticism.

Stateful ciphers: unwind them left to right

If flipping byte i corrupts everything from position i onward, each output depends on the outputs before it. The cipher carries state. The two common shapes are chained XOR and a keystream generator whose state advances per byte.

Chaining is the easier one. A scheme like c[i] = p[i] ^ c[i-1] ^ k looks tangled but unwinds in a single forward pass, because by the time you need c[i-1]you already have it: it's ciphertext, which you were given.

# Forward:  c[i] = p[i] ^ c[i-1] ^ k        (c[-1] = iv)
# Inverse:  p[i] = c[i] ^ c[i-1] ^ k
def unchain(ct: bytes, k: int, iv: int = 0) -> bytes:
    out = bytearray()
    prev = iv
    for c in ct:
        out.append(c ^ prev ^ k)
        prev = c
    return bytes(out)
 
 
# k and iv are one byte each, so 65,536 combinations is nothing.
for k in range(256):
    for iv in range(256):
        guess = unchain(ciphertext, k, iv)
        if guess.startswith(b'picoCTF{'):
            print(k, iv, guess)

Keystream generators need a different move. If the cipher is c[i] = p[i] ^ ks[i] for some generated keystream, and you can recover any plaintext you know, you get the keystream for free by XORing it back out. The flag format hands you eight known bytes before you start.

# Known-plaintext recovery of a keystream.
known = b'picoCTF{'
keystream_prefix = bytes(c ^ p for c, p in zip(ciphertext, known))
print(keystream_prefix.hex())
 
# If the generator repeats with period n, the whole thing unravels.
def find_period(ks: bytes, max_period: int = 64) -> int | None:
    for n in range(1, max_period + 1):
        if all(ks[i] == ks[i % n] for i in range(len(ks))):
            return n
    return None

The most interesting stateful family is the linear feedback shift register, a keystream generator that produces the next bit as a fixed XOR of selected previous bits. LFSRs feel cryptographic and are catastrophically weak: because every output bit is a linear function of the initial state, seeing 2n output bits from an n-bit register is enough to solve for the whole thing.

# Berlekamp-Massey over GF(2): recover the shortest LFSR from a bit sequence.
def berlekamp_massey(bits: list[int]) -> list[int]:
    n = len(bits)
    c = [1] + [0] * n   # current connection polynomial
    b = [1] + [0] * n   # last one that caused a length change
    length, m = 0, -1
 
    for i in range(n):
        d = bits[i]
        for j in range(1, length + 1):
            d ^= c[j] & bits[i - j]
        if d:
            t = c[:]
            for j in range(n - (i - m)):
                c[j + i - m] ^= b[j]
            if 2 * length <= i:
                length, m, b = i + 1 - length, i, t
 
    return c[1:length + 1]   # the tap positions
 
 
# Feed it the keystream bits you recovered from known plaintext.
bits = [(byte >> (7 - k)) & 1 for byte in keystream_prefix for k in range(8)]
taps = berlekamp_massey(bits)
print('register length:', len(taps), 'taps:', taps)

Once you have the taps and any n consecutive output bits, you can run the register forward to generate as much keystream as the ciphertext needs, or run it backwards to recover the seed. shift-registers is exactly this attack. not-true and clouds also reward treating the generator as a linear system rather than a black box.

When the encryption is really an encoding

A surprising share of custom cipher challenges aren't encryption at all. They're an encoding with unusual parameters: base64 with a shuffled alphabet, base-N with a handmade digit set, or a chain of standard encodings stacked deep enough to look scrambled.

The tell is the character set. Count distinct characters in the ciphertext. Sixteen means hex, thirty-two or thirty-six means a base-32 variant, sixty-four with padding means base64, eighty-five means ascii85. If the count matches a standard base but the characters are in the wrong order, you have a substituted alphabet and one known plaintext recovers the mapping.

from collections import Counter
 
alphabet = sorted(set(ciphertext_text))
print(len(alphabet), ''.join(alphabet))
print(Counter(ciphertext_text).most_common(10))
 
 
# Custom base64 alphabet: map the custom set onto the standard one.
import base64
 
STANDARD = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'
CUSTOM   = '...the challenge alphabet, in its own order...'
 
table = str.maketrans(CUSTOM, STANDARD)
print(base64.b64decode(ciphertext_text.translate(table)))

For stacked encodings, work outside in and let a chain tool do the bookkeeping. Recipe Chain applies decoding steps in sequence so you can peel layers without rewriting a script each time, and the ROT tool sweeps all 25 rotations at once when a layer turns out to be a plain shift.

multicode and hidden-cipher-1 both reward checking for an encoding before assuming encryption.

Homemade block modes: attack the mode, not the cipher

When the diffusion map shows changes confined to a 16-byte window, the author built a block cipher, or more often wrapped a real one in a mode they invented. Do not attack the round function. Attack the mode.

The classic weakness is deterministic block reuse. If two plaintext blocks are identical and their ciphertext blocks are identical, the mode is ECB-shaped and leaks structure. You can detect this without knowing anything about the underlying cipher.

def repeated_blocks(ct: bytes, size: int = 16) -> int:
    blocks = [ct[i:i + size] for i in range(0, len(ct), size)]
    return len(blocks) - len(set(blocks))
 
 
print('duplicate blocks:', repeated_blocks(ciphertext))
 
# A byte-at-a-time ECB decryption oracle, when you control a prefix.
# Align the unknown byte against a block boundary, then brute-force it.
known = b''
block = 16
for i in range(block * 4):
    pad = b'A' * (block - 1 - (len(known) % block))
    target_block = (len(pad) + len(known)) // block
    want = enc(pad)[target_block * block:(target_block + 1) * block]
    for guess in range(256):
        probe_ct = enc(pad + known + bytes([guess]))
        if probe_ct[target_block * block:(target_block + 1) * block] == want:
            known += bytes([guess])
            break
    else:
        break
print(known)

The other frequent weakness is a mode that chains blocks with a reversible combiner. If the author XORs each ciphertext block into the next, or adds blocks together mod 2^128, the chaining is invertible on its own and you can strip it without touching the block cipher at all. aes-abc is the textbook case: real AES, a homemade chaining mode, and the mode is additive, so the chain peels off with modular subtraction and the underlying blocks turn out to be recoverable.

black-cobra-pepper and play-nice sit in the same territory, where the structure around the cipher is the vulnerability.

Reading the source: a checklist that finds the weakness fast

Most of these challenges hand you encrypt.py. Reading it well is a skill of its own, and the trick is to read it in a deliberate order rather than top to bottom.

1. Find where the flag enters. Search for the file read or the variable that holds plaintext. Everything before it is setup you can often ignore.
2. Find every line that touches plaintext bytes. That short list is the actual cipher. Key exchanges, hashing of unrelated values, and elaborate class hierarchies frequently touch nothing.
3. Check the key space. If the key is derived from random.seed(int(time.time())), from a four-digit PIN, or from a short password, the cipher does not matter. Brute-force the seed.
4. Check each operation for invertibility. XOR and addition always invert. Multiplication inverts when it is coprime to the modulus. A hash, a truncation, or an and with a mask destroys information and tells you where the intended attack must live.
5. Check the loop bounds. Off-by-one errors that leave the first or last byte unencrypted are common, and one plaintext byte is often enough to unravel a chained scheme.
6. Write the inverse function directly under the original. Line for line, in reverse order. Do not try to hold the inversion in your head.

guess-my-cheese-part-1 and ricochet both reward the key-space check before any cryptanalysis. college-rowing-team hinges on a seed you can enumerate.

The decision workflow

Put together, the whole approach is a short decision tree. Follow it in order and you will almost never waste time on the wrong attack.

Quick reference

Inversion cheatsheet

Tools on this site

Sanity checks before you give up

• Did you try both endiannesses when converting integers back to bytes?
• Is the modulus 256, or is it 26, 95, or 128 because the author worked in ASCII?
• Does the first or last byte behave differently from the rest?
• Is the key actually shorter than it looks, so the keystream repeats within the ciphertext?
• Did you check whether the ciphertext decodes as an encoding before assuming a key exists?

picoCTF custom cipher challenges

Every writeup below involves a hand-rolled cipher. Work them in roughly this order, since the earlier ones use one primitive and the later ones stack several.

The first time I worked one of these I spent two hours trying to find a published attack on a cipher that existed only inside a 60-line file, which in hindsight is a bit like searching for a Wikipedia article about your neighbor. The cipher had no name because it had no history. Once you accept that, the work gets much simpler: stop asking what it is, and start asking what it does to a byte you control.

A homebrew cipher has no literature, no known attacks, and no name. It also has an author who had to make it reversible, which means you can reverse it too.