Classical Ciphers for CTF: Caesar, Vigenère, and Substitution

The whole game is ten seconds of recognition

I once watched a teammate spend twenty minutes hand-decoding a string that turned out to be ROT13. Letter by letter, counting on his fingers, muttering "c, d, e, f..." while a one-line command sat three keystrokes away. The string started with cvpbPGS. That prefix is ROT13 for picoCTF, and once you have seen it once you never un-see it. He just hadn't seen it yet.

That is the entire lesson of classical-cipher challenges, and it is worth saying out loud before we touch a single algorithm: the decryption is never the hard part. Every cipher on this page breaks in one click with a tool that already exists, half of them inside this site. The hard part is the ten seconds where you look at a blob of garbage text and correctly name which cipher made it. Name it right and you are done. Name it wrong and you will spend twenty minutes counting on your fingers.

So this is not a cryptography lecture. It is a guide to building one reflex: reading the tella cipher leaves in its output. A Caesar cipher, a Vigenère cipher, a substitution cipher, and a rail fence all scramble text, but each one scrambles it in a way that leaves a fingerprint. Learn the four fingerprints and the picoCTF crypto challenges in the "General Skills" and "Cryptography" tracks stop being puzzles and start being a checklist. If you are brand new to all of this, start with the picoCTF beginner's guide and come back.

The challenge is not "can you decrypt this." The challenge is "can you tell, on sight, which thing this is." Tools do the rest.

The recognition reflex: four questions in order

When a challenge hands you a wall of text, do not start guessing shifts. Run it through four questions, top to bottom. The first "yes" you hit usually names the cipher. This is the flow chart that lives in my head, and after a dozen challenges it runs in about the time it takes to read the ciphertext.

That is the spine of everything below. The rest of this post is just walking each branch and showing the picoCTF challenge that lives there.

Caesar and ROT: 26 keys, brute every one

The Caesar cipher shifts each letter a fixed number of positions down the alphabet, wrapping Z back to A. Julius Caesar reportedly used a shift of 3 for military messages. ROT13 is the same idea with a shift of 13, and because 13 is exactly half of 26, applying it twice gets you back where you started. ROT13 is its own inverse, which is why tr can encode and decode with one command.

Here is the tell, and it is the most reliable one in all of CTF crypto: a Caesar cipher has only 26 possible keys. That is not a key space, that is a short list. You never deduce the shift, you print all 26 and read the one that makes sense.

ct = "WKLV LV D WHVW"
for k in range(26):
    out = "".join(
        chr((ord(c) - 65 - k) % 26 + 65) if c.isalpha() else c
        for c in ct.upper()
    )
    print(k, out)
# the row that reads as English is your shift

That loop solves Caesar (2019) and rotation (2023) outright, and you do not even have to know the shift in advance. When you do recognize the cipher as ROT13 specifically, like the cvpbPGS prefix in 13 (2019) and the misleadingly named Mod 26 (2021) (which is just ROT13 with modular-arithmetic flavor text), the whole thing collapses to one line of shell:

echo "cvpbPGS{...}" | tr 'A-Za-z' 'N-ZA-Mn-za-m'

You do not have to write the loop at all if you do not want to. The site's ROT / Caesar tool shows all 26 rotations at once, which is genuinely faster than typing the Python. I still keep the loop in my head because it generalizes: the day a challenge uses a 25-letter or 16-letter alphabet, you edit one number instead of hunting for a new tool.

When the alphabet lies: New Caesar and custom bases

Question 1 of the reflex had a sharp edge to it: is the alphabet still A-Z? When the answer is no, a challenge that looks like a Caesar cipher can hide a second layer. New Caesar (2021) is the canonical trap. Its ciphertext uses only the letters a through p, sixteen of them, and that restricted alphabet is the entire clue.

Under the hood, New Caesar splits each plaintext byte into two 4-bit halves (nibbles), maps each nibble to one of sixteen letters, then applies a Caesar shift on that 16-letter alphabet. So before you brute-force anything, validate your assumption about the alphabet. One line tells you whether you understand the cipher at all:

print(sorted(set(ciphertext)))
# expect: ['a','b','c',...,'p']  -> 16-letter alphabet, key space is 16
# anything outside a..p means your model of the cipher is wrong

The payoff: a 16-letter alphabet has only 16 possible shifts, so the brute force is smaller than a normal Caesar, not bigger. Layering an encoding on top of a weak cipher did not add security; it shrank the key space and left a fingerprint in the alphabet. That is the lesson worth carrying forward into real crypto: independent weak layers stay weak.

A restricted alphabet is a confession. Sixteen letters means base-16. Dots and dashes mean Morse. Digits one through twenty-six mean A equals one. The character set tells you what you are looking at before you decode a single symbol.

Vigenère: when the same letter won't sit still

The Vigenère cipher is a Caesar cipher that changes its shift on every letter, driven by a repeating keyword. For a key of length n, position i is shifted by the (i mod n)-th key letter. The consequence is the whole point: the same plaintext letter encodes to different ciphertext letters depending on where it sits. That defeats the frequency analysis that flattens a plain substitution cipher, which is why Vigenère went unbroken for nearly three hundred years and earned the nickname le chiffre indéchiffrable, the indecipherable cipher.

The tell is question 3 of the reflex. If you stare at the ciphertext and notice that there is no consistent letter-to-letter mapping, that the same symbol clearly stands for different things in different places, you are looking at something polyalphabetic, and Vigenère is the one you will meet in a CTF. From there, it splits into two completely different problems depending on one thing: do you have the key?

Most picoCTF Vigenère challenges give you the key, so the work really is "paste into a tool." But it is worth knowing how the unkeyed break works, because it is one of the prettiest ideas in classical cryptanalysis. Charles Babbage cracked it privately around 1854 and Friedrich Kasiski published the method in 1863: repeated chunks of plaintext that happen to line up with the same part of the repeating key produce repeated ciphertext. Measure the gaps between those repeats, take their common factors, and you have recovered the key length without ever knowing the key. Once you know the length, the unbreakable cipher dissolves into a handful of ordinary Caesar shifts.

Substitution: let frequency do the work

A monoalphabetic substitution cipher swaps each letter for a fixed but arbitrary other letter. No shift pattern, no math, just a scrambled alphabet. There are 26 factorial possible keys, which is a number with twenty-six digits, so brute force is out. And yet these are some of the easiest challenges on the board, because the cipher leaves the biggest fingerprint of all: it preserves the structure of the language underneath.

The most common letter in the ciphertext almost certainly stands for 'e'. The most common three-letter word is almost certainly "the." A single-letter word is "a" or "I." Those constraints stack up fast, and a solver that scores candidate decryptions against English letter and word statistics, then climbs toward better scores, converges in seconds. That is exactly what quipqiup does, and it is the intended solution for substitution1 and substitution2.

# substitution cipher workflow
1. Paste the ciphertext into quipqiup.com (keep the spaces and punctuation).
2. It hill-climbs against English n-gram stats and returns ranked guesses.
3. Read the top result. If the prose is gibberish but the flag region looks
   plausible, it's a wrong key that got lucky -- pick the candidate where
   BOTH the body text and the flag read as clean English.

When word boundaries are stripped out and the automated solver stalls, you fall back to doing it by hand: build a frequency table, pencil in 'e' and 't' and 'a' for the top letters, look for the "the" pattern, and grow the mapping outward from there. The frequency analysis tool builds that table for you. This is the same letter-frequency idea that breaks a Caesar cipher, just applied to an arbitrary mapping instead of a fixed shift, which is why I think of Caesar and substitution as two ends of the same skill.

Rail fence, Morse, and the non-letter crowd

Back to question 2, the one beginners skip. If you run a frequency count and the letters look exactly like normal English, 'e' and 't' on top, but the text still reads as nonsense, then nothing was substituted. The letters are all correct. They are just in the wrong order. That is a transposition cipher, and in picoCTF it is almost always a rail fence.

The rail fence writes your message in a zigzag across a number of horizontal "rails," then reads it off row by row. Rail Fence (2022) tells you it used 4 rails, so the only knob left is the offset. There is a one-click operation for it:

# CyberChef: Rail Fence Cipher Decode
#   Key    = 4   (the rail count, given by the challenge)
#   Offset = 0
# or use the site's Rail Fence tool with the same parameters

The frequency-matches-English-but-reads-as-garbage tell is what stops you wasting time on substitution solvers that will never converge, because there is nothing for them to solve. Use the rail fence tool and move on.

That leaves the non-letter crowd from question 1, the schemes that do not even pretend to be A-Z. These barely qualify as ciphers, they are really encodings, but they show up constantly so you should recognize them instantly:

• Dots and dashes are Morse code. Morse Code (2022) drops straight into the Morse decoder.
• Numbers 1 through 26are A1Z26: map 1 to A, 2 to B, and so on. The picoGym "The Numbers" challenge spells out its answer this way.
• A-Z reversed (A becomes Z, B becomes Y) is Atbash, a self-inverse substitution worth knowing because it hides inside steganography challenges. The Atbash tool flips it back.
• Stacked layers (binary, then base64, then ROT13, then Atbash) are a multi-encoding puzzle, not a single cipher. Peel them one at a time. The CTF encodings guide covers that whole ladder.

If you genuinely cannot tell what you are staring at, paste it into the cipher identifier. It runs the same tells we just walked, the alphabet check and the frequency profile, and gives you a ranked guess. Think of it as the reflex, automated, for the days your own pattern-matching comes up blank.

Quick reference: the tell, the key space, the tool

Print this table on the inside of your skull. The middle column is the whole post: the tell is what you train your eye to catch, and everything to the right of it is mechanical.

Notice the column nobody put in the table: a "difficulty" column. There isn't one, because once you have named the cipher there is no difficulty left. That is the thing I wish someone had told me before I watched my teammate count to thirteen on his fingers. Spend your effort on recognition, not on decryption, and classical crypto becomes the fastest points on the board.

From here, the natural next steps: graduate from fixed shifts to stream ciphers, watch frequency analysis grow up into the attacks on RSA, and pick up the Python patterns that turn every brute-force loop above into a five-line script you can reuse forever. The ciphers got harder over the centuries. The reflex stayed exactly the same: look at the output, name the thing, reach for the tool.

Sources and further reading

• picoCTF writeups on this site: Caesar, 13, Mod 26, New Caesar, Vigenere, substitution1, Rail Fence, rotation.
• quipqiup, the automated cryptogram solver, and the Kasiski examination for recovering Vigenère key length.
• CyberChef and dCode for one-click decoders of every cipher above.