XOR for CTF: Breaking XOR Encryption Without the Key

XOR is the most common homemade cipher in CTF, and it always breaks

When a challenge author wants encryption that looks scary but is quick to write, they reach for XOR. It shows up as a "custom encryption" function in a Python script, as an obfuscated string table in a binary, as a Java for loop over a char array, and as the guts of every stream cipher. If you learn one primitive in CTF cryptography before any of the named algorithms, learn this one, because XOR is where the beginner-to-intermediate difficulty band actually lives.

The good news is that XOR is symmetric and structureless. It hides data only as well as the key hides, and CTF keys are short, reused, or recoverable from a single guessed word. Here is the entire decision tree:

XOR does not have a weakness. XOR is the weakness. It is a reversible mask, and every CTF attack below is just a different way of learning the mask.

Three properties, and every attack falls out of them

XOR compares two bits and returns 1 when they differ. That is the whole operation. Three consequences drive everything else:

• It is its own inverse. (a ^ k) ^ k == a. Encryption and decryption are the same function, which is why a challenge that gives you an encrypt() and no decrypt() has already given you both.
• Anything XORed with itself is zero. a ^ a == 0, and a ^ 0 == a. This is what makes reused keystreams fatal: XOR two ciphertexts encrypted under the same key and the key cancels out, leaving the XOR of two plaintexts.
• It is bytewise and position-independent. Byte 5 of the plaintext only ever touches byte 5 of the keystream. There is no diffusion, no avalanche, no mixing across positions. Every byte is an independent tiny puzzle, which is why a repeating key of length 7 is really seven separate single-byte problems.

The contrast with a real cipher is measurable. NIST SP 800-38A specifies AES modes over a 128-bit block, and AES is designed so that flipping one input bit changes about half of the 128 output bits. XOR changes exactly one. That single number is the whole difference between a cipher and a mask, and it is why MITRE files homemade XOR "encryption" under CWE-326, inadequate encryption strength.

If ciphers as inverse transformations is new territory, the classical ciphers guide covers the frequency-analysis mindset this post assumes, and the cryptography roadmap places XOR in the wider learning order.

Single-byte XOR: brute force plus a scoring function

A single-byte key has 256 possibilities. You do not need to be clever, you need to be able to recognize the right answer automatically, because eyeballing 256 candidate plaintexts is how people miss the flag sitting at key 0x5A.

The naive score is "count printable ASCII bytes." It works, but it ties often. The better score weights English letter frequency, and the best score for CTF is simplest of all: look for the flag format.

import string
 
ct = bytes.fromhex("3b2c2f2a3a2d38...")
 
def score(b: bytes) -> float:
    # ETAOIN-weighted printability score
    common = b"etaoinshrdlu ETAOIN"
    printable = sum(c in bytes(string.printable, "ascii") for c in b)
    freq = sum(c in common for c in b)
    return printable + 2 * freq
 
best = sorted(
    ((score(bytes(c ^ k for c in ct)), k, bytes(c ^ k for c in ct))
     for k in range(256)),
    reverse=True,
)
 
for s, k, pt in best[:5]:
    print(hex(k), s, pt[:60])

for k in range(256):
    pt = bytes(c ^ k for c in ct)
    if b"picoCTF{" in pt:
        print(k, pt)

The same loop is the standard opener on any high-entropy blob that survived your encoding triage. If the encodings pass found nothing and the bytes are not a known file format, a 256-key sweep costs one second and is worth trying before anything sophisticated.

Known plaintext: the flag format hands you the key

This is the attack that beginners skip and that solves more picoCTF challenges than any other XOR technique. You know something about the plaintext before you start: it begins with picoCTF{. Nine known bytes. XOR them against the first nine bytes of ciphertext and you are literally reading the key off the screen.

known = b"picoCTF{"
key_material = bytes(c ^ p for c, p in zip(ct, known))
print(key_material)   # e.g. b'secretsecre' -> key is 'secret'

Read the recovered bytes and look for the repeat. If the output is b'keykeykey', the key is key. If the output is random-looking, either the key is at least as long as your crib (a one-time pad, see the keystream reuse section) or the plaintext does not start where you assumed.

A masked file is worth calling out specifically. If a forensics challenge gives you bytes that almost look like a PNG, XOR the first eight ciphertext bytes against the known PNG signature. The key drops out, and the whole file decrypts.

Repeating-key XOR: find the length, then split into columns

A repeating key (sometimes called Vigenere XOR) cycles a short key over a long message. Breaking it is two steps, and the second step is just the single-byte attack you already have.

Step 1: find the key length. The idea is older than computers: Kasiski published the repeated-substring method for periodic ciphers in 1863, and Friedman's index of coincidence followed in 1922. The modern XOR version scores candidate lengths by normalized Hamming distance. Split the ciphertext into blocks of the candidate length, measure the average bit difference between blocks, and pick the length with the smallest distance. Same-key columns share statistics, so the true length scores lowest.

def hamming(a: bytes, b: bytes) -> int:
    return sum(bin(x ^ y).count("1") for x, y in zip(a, b))
 
def best_keysize(ct: bytes, lo=2, hi=40):
    scores = []
    for size in range(lo, hi + 1):
        blocks = [ct[i * size:(i + 1) * size] for i in range(4)]
        pairs = [(blocks[i], blocks[j]) for i in range(4) for j in range(i + 1, 4)]
        avg = sum(hamming(a, b) for a, b in pairs) / len(pairs) / size
        scores.append((avg, size))
    return sorted(scores)[:3]

Step 2: transpose and solve each column. Once you believe the key is n bytes, byte 0, byte n, byte 2n and so on were all encrypted with the same key byte. Gather them into one string and run the 256-key brute force on it. Do that n times and you have the whole key.

size = 7   # from step 1
key = bytes(
    max(range(256), key=lambda k: score(bytes(c ^ k for c in ct[i::size])))
    for i in range(size)
)
pt = bytes(c ^ key[i % size] for i, c in enumerate(ct))
print(key, pt[:80])

Note that step 1 is optional whenever you have a crib. Known plaintext gives you the key bytes and their period in one shot, so try that first and fall back to Hamming distance only when you know nothing about the plaintext.

Keystream reuse: two messages, one pad, no key needed

A one-time pad, where the key is as long as the message and truly random, is unbreakable, and that is not a figure of speech: Shannon proved it in "Communication Theory of Secrecy Systems" (1949), showing the ciphertext gives an attacker zero information about the plaintext. The proof holds only under three conditions: the key is truly random, at least as long as the message, and used exactly once. Used twice, it collapses. Given c1 = p1 ^ k and c2 = p2 ^ k, the attacker computes c1 ^ c2 = p1 ^ p2. The key is gone, and what remains is two English messages XORed together, which is solvable by hand.

The technique is crib dragging: guess a common word, XOR it at every offset of p1 ^ p2, and look for a result that is also readable English. When it is, you have just recovered a fragment of the other message, which you then extend by guessing the rest of its word and dragging again.

One ASCII coincidence makes the first step almost free. RFC 20 places space at 0x20 and puts the uppercase and lowercase alphabets exactly 0x20 apart, so XORing a letter with a space simply flips its case. In p1 ^ p2, any position where one message has a space shows up as a readable letter with the wrong case, and positions where both have a space are 0x00. Highlighting the bytes in the ranges 0x41-0x5A and 0x61-0x7A maps out the word boundaries of both messages before you guess a single crib.

This is a historical attack, not a theoretical one. Soviet operators reused one-time pad pages under wartime production pressure, and the resulting depth let US and British cryptanalysts read roughly 3,000 messages over the VENONA program, which ran from 1943 to 1980. A provably unbreakable cipher lost to a key used twice.

xored = bytes(a ^ b for a, b in zip(c1, c2))
crib = b" the "
for off in range(len(xored) - len(crib)):
    guess = bytes(x ^ c for x, c in zip(xored[off:], crib))
    if all(32 <= g < 127 for g in guess):
        print(off, guess)

This is exactly the failure mode behind RC4 and ChaCha20 nonce reuse, so treat this section as the bridge into the stream ciphers guide, which covers the modern versions of the same mistake.

XOR inside binaries: obfuscated strings and check functions

Half the XOR you meet in CTF is not in a crypto challenge at all. It is in reverse engineering, where authors mask a hardcoded flag so that strings returns nothing useful. The pattern is always recognizable:

// C, as Ghidra will show it
for (i = 0; i < 0x20; i++) {
    local_38[i] = enc_data[i] ^ 0x42;
}
 
// Java, as jd-gui will show it
for (int i = 0; i < passBytes.length; i++)
    passBytes[i] = (byte)(passBytes[i] ^ myBytes[i % myBytes.length]);

Three ways to beat it, in increasing order of effort:

• Read the key and reimplement. The key is a constant in the disassembly or an array in the decompiled source. Copy the encrypted bytes and the key into Python and XOR them yourself. The Ghidra guide covers extracting a byte array from a data section.
• Let the program do it and read memory. Break after the XOR loop in GDB and dump the buffer. No reimplementation, no transcription errors. See the GDB guide for breakpoints and x/s.
• Invert a comparison. When the binary XORs your input and compares it to a constant, you do not need to understand anything: the expected input is constant ^ key, because XOR is its own inverse.

The Java flavor of this shows up throughout the vault-door series, covered in the Java reverse engineering guide, and the general "reimplement the author's homemade scheme" workflow is in the custom cipher reversing guide.

The toolkit: pwnlib.util.fiddling and a five-line helper

You can write every attack above from scratch, but pwntools ships the primitives, and they handle length mismatch and type coercion for you. Per the pwnlib.util.fiddling documentation, xor accepts any number of arguments, cycles the shorter operands to the length of the longest by default, and coerces int, str, and bytes alike, which removes exactly the three bugs a hand-rolled loop introduces. Pass cut='min' when you want truncation instead of cycling.

from pwn import xor, xor_pair
 
xor(b"hello", 0x42)              # single-byte mask
xor(b"hello world", b"key")     # repeating key, cycled automatically
xor(ct, b"picoCTF{")             # crib against a ciphertext prefix
 
# no pwntools available? the whole primitive in one line:
def x(a, b):
    return bytes(i ^ b[j % len(b)] for j, i in enumerate(a))

For interactive work, the on-site recipe chain has XOR and XOR-brute-force operations you can chain after a from-hex or from-base64 step, which is faster than scripting when you are still guessing at the encoding layers. The Python for CTF guide covers the bytes-versus-str handling that trips up most first attempts.

Where to practice, easiest first

These picoCTF challenges cover the full range: a self-inverse trick, a homemade cipher, a reused keystream, and XOR inside a compiled binary. Each one is solvable with only the ideas on this page.

• picoCTF 2019 vault-door-6 is the gentlest possible start: a Java check that XORs the password with 0x55. The key is sitting in the decompiled source, so you invert the comparison rather than cracking anything.
• picoMini XtraORdinary applies several known values an unknown number of times. Because XOR is self-inverse, only the parity of each application matters, which collapses the search space to a handful of combinations you can enumerate.
• picoCTF 2025 Tap into Hash reuses one keystream segment across every block. A known plaintext at that position reveals the keystream bytes outright, and those bytes then decrypt all the other blocks. This is the reuse section made concrete.
• picoCTF 2024 Custom encryption derives its XOR key through a toy key exchange whose private values are visible in the provided source. Rebuild the shared key, then invert the encryption function, and note how little of it is real cryptography.
• picoCTF 2020 Mini OTP Implementation validates a long hex key through a custom jumble function, then XOR-decrypts the flag with it. Breaking at the comparison recovers the key a character at a time, which is far cheaper than inverting the transform.
• picoCTF 2026 Secure Password Database mixes its input with XOR inside a homemade hash. You do not have to reverse the algorithm at all: break after the mixing loop and read the computed value straight out of a register.

Quick reference

XOR is not a cipher you break. It is a mask you learn. Once you can recognize the four shapes (one byte, known plaintext, repeating key, reused pad), the entire homemade -encryption tier of CTF stops being cryptography and becomes bookkeeping. From here, stream ciphers generalizes the reuse attack, and the cryptography roadmap shows what to learn next.

Sources and further reading

XOR is the rare CTF topic where the primary literature is both short and decisive.

• Shannon, "Communication Theory of Secrecy Systems" (1949) for the perfect-secrecy proof and its three conditions, and NSA's VENONA release for what happens when one of them is broken.
• NIST SP 800-38A for what a real block cipher mode does that XOR does not, and CWE-326 for the weakness class homemade XOR falls into.
• RFC 20 for the ASCII layout behind the space-XOR case-flip trick used in crib dragging.
• pwnlib.util.fiddling and Python binary sequence types for the implementation details.