Play Nice picoCTF 2021 Solution

Description

1. Step 1

   Connect and capture the key and ciphertext

   Observation

   I noticed the challenge description mentioned a 6x6 key alphabet with 36 characters (a-z plus 0-9), which is the defining characteristic of the extended Playfair cipher variant and suggested connecting first to capture the random key grid and ciphertext for decryption.

   Connect to the server. It will display a 6x6 Polybius-style key alphabet (36 characters - a-z and 0-9) and a ciphertext. Copy both for the next step.

   bash

   nc mercury.picoctf.net 6057

   Copied!

   What didn't work first

   Tried: Treating the 6x6 grid as a standard 5x5 Playfair cipher and merging I/J

   The standard 5x5 Playfair tool will reject or silently mangle the 36-character alphabet because it only expects 25 unique letters. The 6x6 variant keeps I and J separate and adds digits 0-9, so the modular arithmetic is mod 6 not mod 5, and any tool or formula built for 5x5 produces garbage output.

   Tried: Feeding the key alphabet to dCode with spaces or newlines still in it

   dCode's Playfair tool expects the alphabet as a flat string with no separators. If you paste the server output with row breaks preserved, it will either error or miscount positions, shifting every digraph lookup by the number of whitespace characters it encounters. Concatenate all six rows into a single 36-character string before pasting.

   Learn more

   The Playfair cipher is a digraph substitution cipher invented by Charles Wheatstone in 1854 and popularized by Lord Playfair. Unlike simple substitution ciphers that replace individual characters, Playfair encrypts pairs of letters (digraphs) at a time, making simple frequency analysis less effective. It was used by the British military in World War I and by the Australians in World War II.

   The standard Playfair cipher uses a 5x5 grid containing the 25 letters of the alphabet (I and J are combined). This challenge uses a 6x6 variant that accommodates all 26 letters plus 10 digits (0-9), for 36 total characters. The key grid is filled with a keyword first (removing duplicates), then the remaining characters in order. The extended grid supports encrypting alphanumeric messages without losing digit information.

   The server generates a fresh random key grid and ciphertext each connection, so the solution cannot be hardcoded - it must automate Playfair decryption using the received key. This is a common CTF pattern: the server presents a random challenge that must be solved programmatically, rewarding a correct implementation of the cryptographic algorithm rather than manual solving.

2. Step 2

   Decrypt with dCode Playfair decoder

   Observation

   I noticed the 6x6 key grid and ciphertext matched the exact input format expected by the Playfair cipher tool on dCode.fr, which suggested using that decoder with the 6x6 variant selected and the alphabet pasted as a flat 36-character string to recover the plaintext.

   Navigate to dCode.fr's Playfair cipher tool. Select the 6x6 grid variant. Paste the alphabet as a single 36-character string with no spaces and no line breaks (concatenate all six rows). Enter the ciphertext and submit; the result is the plaintext.

   Learn more

   Playfair decryption rules (for a 6x6 grid) operate on character pairs from the ciphertext:

   • Same row: replace each character with the one to its left (wrapping around)
   • Same column: replace each character with the one above it (wrapping around)
   • Rectangle: replace each character with the one in the same row but the other character's column

   During encryption, if a pair contains the same character, an "X" (or "0" in 6x6 variants) is inserted between them. The plaintext may have these fillers removed during decryption to recover the original message. The message is also padded to an even length if needed.

   dCode.fr is a comprehensive online cipher reference with implementations of hundreds of historical and modern ciphers. For one-off manual decryption, it is the fastest path. However, since the server key changes each session, an automated Python implementation is needed for reliable flag retrieval - implement the grid lookup and three decryption rules in a pwntools script that parses the server output, decrypts, and sends the result.

3. Step 3

   Submit the plaintext

   Observation

   I noticed the server generates a fresh random key each connection, which ruled out manual one-off decryption and suggested automating the Playfair grid lookup and three-rule decryption in a pwntools script that parses the server output and sends the plaintext back in one session.

   Send the decrypted plaintext back as a single line. The server compares strictly; submit only if the result looks like recognizable English or starts with a known prefix. Use pwntools: p.sendline(plaintext.encode()) then p.recvall() captures the flag.

   python

   python3 - <<'EOF'
   from pwn import remote
   
   ALPHA_ROWS = 6  # 6x6 = 36 chars
   
   def make_grid(alpha):
       return [list(alpha[i*ALPHA_ROWS:(i+1)*ALPHA_ROWS]) for i in range(ALPHA_ROWS)]
   
   def find(grid, ch):
       for r, row in enumerate(grid):
           if ch in row:
               return r, row.index(ch)
       raise ValueError(ch)
   
   def decrypt(ct, alpha):
       grid = make_grid(alpha)
       out = []
       for i in range(0, len(ct), 2):
           a, b = ct[i], ct[i+1]
           ra, ca = find(grid, a)
           rb, cb = find(grid, b)
           if ra == rb:                              # same row -> shift left
               out += [grid[ra][(ca - 1) % ALPHA_ROWS],
                       grid[rb][(cb - 1) % ALPHA_ROWS]]
           elif ca == cb:                            # same col -> shift up
               out += [grid[(ra - 1) % ALPHA_ROWS][ca],
                       grid[(rb - 1) % ALPHA_ROWS][cb]]
           else:                                     # rectangle -> swap cols
               out += [grid[ra][cb], grid[rb][ca]]
       return ''.join(out)
   
   p = remote('mercury.picoctf.net', 6057)
   # Adjust recvuntil targets to match the actual server prompts
   alpha = p.recvline_contains(b'alphabet').decode().split()[-1].strip()
   ct    = p.recvline_contains(b'iphertext').decode().split()[-1].strip()
   pt = decrypt(ct, alpha)
   print('plaintext:', pt)
   p.sendline(pt.encode())
   print(p.recvall(timeout=3).decode())
   EOF

   Copied!

   Expected output

   picoCTF{2e71b99f...}

   What didn't work first

   Tried: Using ALPHA_ROWS = 5 instead of 6 in the make_grid and modular arithmetic

   With ALPHA_ROWS set to 5, the grid slices each row to only 5 characters and wraps positions mod 5, but the actual alphabet is 36 characters wide across 6 columns. Every row and column lookup returns a character from the wrong cell, producing a completely incorrect decryption that the server rejects. Change ALPHA_ROWS to 6 throughout so that slicing, find(), and the modular shifts all match the real grid dimensions.

   Tried: Parsing the server output with p.recvline() blindly instead of recvline_contains on the correct label

   The server prints several lines of explanation before showing the alphabet and ciphertext. Calling plain p.recvline() in sequence will assign a banner or empty line to the alpha variable, making the string length wrong and crashing find() with a ValueError on the first missing character. Using recvline_contains with b'alphabet' and b'iphertext' anchors each parse to the correct output line regardless of how many preamble lines the server emits.

   Learn more

   The function above implements all three Playfair decryption rules for an arbitrary 6x6 alphabet (~30 lines of cipher logic plus a pwntools wrapper). For a 36-character alphabet, modular arithmetic is mod 6 across each row/column instead of mod 5 in the classical 5x5 variant. Inserting separators (X or 0) is an encryption-side concern; decryption simply applies the inverse rules and you trim any padding letters by eye after seeing the recovered plaintext.

   The pwntools tube API is useful here: p.recvuntil(b'Ciphertext:') reads until the label appears, p.recvline() reads the ciphertext, and p.sendline(plaintext) submits the answer. p.recvall() captures the server's response including the flag.

   Output validation. Before submitting, glance at the decrypted string - if it does not look like English or does not start with a known prefix from the challenge prompt, your grid parse is off. Submitting wrong text repeatedly may be rate-limited by the server.

Flag

The flag is returned by the server only after submitting the correct plaintext - each session uses a fresh key.