Custom encryption picoCTF 2024 Solution

Description

1. Step 1

   Rebuild the shared key

   Observation

   I noticed that custom_encryption.py's test() routine contains explicit print('a:', a) and print('b:', b) calls, which suggested the private Diffie-Hellman exponents were exposed and I could reconstruct the shared key directly via generator(g, x, p) without solving any discrete log.

   custom_encryption.py prints a and b during test() (look for the print('a:', a) and print('b:', b) calls). Plug them into generator(g, x, p) to recover the same shared key that encrypt() used.

   Learn more

   Diffie-Hellman key exchange lets two parties agree on a shared secret over an insecure channel. The generator(g, x, p) function computes g^x mod p. When two parties compute g^a mod p and g^b mod p and swap results, each raises the received value to their own private exponent and lands on the same shared secret g^(ab) mod p.

   In this challenge a, b, g, and p are all printed by the script (search for print('a:', a) and the matching b line), so the "key exchange" is trivially reversible. Real DH keeps a and b private; only g^a mod p and g^b mod p ever go on the wire. Its security rests on the discrete logarithm problem.

   Modern DH shows up in TLS handshakes, the Signal protocol, and WireGuard. The elliptic-curve variant (ECDH) gives equivalent security with much smaller keys and is the default in modern systems.

2. Step 2

   Invert encrypt()

   Observation

   I noticed the encrypt() function multiplied each character ordinal by key * 311 to produce the cipher list entries, which suggested the decryption step was straightforward integer division by that same factor to recover the intermediate semi_cipher values.

   Write a decrypt() that integer-divides each cipher entry by key * 311 with //. This yields the "semi_cipher" string prior to the dynamic XOR stage.

   bash

   semi_cipher = [c // (key * 311) for c in cipher]

   Copied!

   What didn't work first

   Tried: Use regular division (/) instead of integer division (//) when inverting the multiplication

   Python's / operator returns a float, so each entry in semi_cipher becomes something like 112.0 instead of 112. When you pass that float to chr() downstream it raises a TypeError: 'float' object cannot be interpreted as an integer. The fix is // which performs floor division and returns an int, making chr() happy.

   Tried: Guess that the magic constant 311 is a red herring and try dividing by just the key alone

   Without the factor of 311, every divided value is off by a factor of 311, producing character ordinals well above 127 that either raise OverflowError in chr() or produce garbage Unicode. The constant 311 is hard-coded inside encrypt() on the multiplication line and must be included in the divisor as key * 311.

   Learn more

   The encryption multiplies each character's ordinal by key * 311. Decryption divides. Use Python's integer division operator // here, not /: regular division returns a float, which then breaks the chr() call downstream because character codes have to be integers.

   The value 311 is hard-coded in the encryption function. Custom ciphers often add "complexity" through multiplication by a magic constant, but this provides zero security once the script is in your hands. Security through obscurity is not a primitive.

   The cipher is layered: DH key exchange, then multiplication, then XOR. Decryption undoes each layer in reverse: derive the key, divide out key * 311, then reverse the XOR. This compositional pattern is exactly how block-cipher modes and AEAD schemes are reasoned about.

3. Step 3

   Reverse dynamic_xor_encrypt

   Observation

   I noticed dynamic_xor_encrypt() first reverses the plaintext with [::-1] and then XORs each character against the repeating key 'trudeau', which suggested that decryption must apply the same forward-cycling XOR to semi_cipher and then reverse the result to restore the original order.

   Write a dynamic_xor_decrypt that applies the same forward-cycling repeating-key XOR (using text_key[i % key_length]) to the semi_cipher string, then reverses the result with [::-1]. The key walk direction is unchanged from encrypt; the only difference is where the reversal happens. Applying it to semi_cipher with the key "trudeau" reveals the flag.

   python

   python3 solver.py  # uses decrypt + dynamic_xor_decrypt

   Copied!

   Expected output

   picoCTF{custom_d2cr0pt6d_751a...}

   What didn't work first

   Tried: Skip reversing the string at the end and just return the XOR result directly

   The encrypt() function reverses the plaintext with [::-1] before XORing, so the XOR was applied to the reversed string. If you XOR semi_cipher with the repeating key but forget to reverse the result afterward, you get the flag characters in reverse order - the output starts with } instead of picoCTF{. You must apply [::-1] after the XOR to undo the reversal that happened before the XOR.

   Tried: Try the XOR key in reverse (walking text_key backward) instead of cycling forward through it

   The encrypt function cycles text_key forward with text_key[i % key_length] regardless of the plaintext reversal. Because XOR is its own inverse, you need to use the same forward-cycling key direction on the semi_cipher - reversing the key walk produces mismatched key bytes at every position and yields garbled output instead of readable ASCII.

   Learn more

   XOR encryption with a repeating key is a simple stream cipher. Each character of plaintext is XORed with the corresponding character of the key, cycling if the key is shorter. XOR is its own inverse: if cipher = plain XOR key, then plain = cipher XOR key.

   The script's text_key variable holds the repeating key bytes, derived from the literal string "trudeau". The encrypt function reverses the plaintext string first (plaintext[::-1]) and then XORs each character with text_key[i % key_length], cycling forward. Decryption applies the same forward-cycling key XOR to the intermediate string, then reverses the result with [::-1]. There is no chained state; each position XORs independently with text_key[i % key_length].

   The key "trudeau" is a weak key: a short dictionary word. Real stream ciphers (RC4, ChaCha20) use much longer pseudorandom key streams. A repeating ASCII key is trivially broken by frequency analysis once the key length is guessed (Kasiski examination, index of coincidence).

Flag

The decrypted semi_cipher plus the reversed XOR routine yields the flag above. If the output doesn't start with picoCTF{, recheck the DH key recovery and confirm you used // (integer division) when undoing key * 311.