Custom encryption picoCTF 2024 Solution

Description

1. Step 1Rebuild the shared key

   Observation

   The test() routine in custom_encryption.py prints a and b outright, so the private Diffie-Hellman exponents are handed to you. Rebuild the shared key with generator(g, x, p) and skip the discrete log entirely.

   custom_encryption.py prints a and b during test() (look for the print('a:', a) and print('b:', b) calls). Plug them into generator(g, x, p) to recover the same shared key that encrypt() used.

   Learn more

   Diffie-Hellman key exchange lets two parties agree on a shared secret over an insecure channel. The generator(g, x, p) function computes g^x mod p. When two parties compute g^a mod p and g^b mod p and swap results, each raises the received value to their own private exponent and lands on the same shared secret g^(ab) mod p.

   In this challenge a, b, g, and p are all printed by the script (search for print('a:', a) and the matching b line), so the "key exchange" is trivially reversible. Real DH keeps a and b private; only g^a mod p and g^b mod p ever go on the wire. Its security rests on the discrete logarithm problem.

   Modern DH shows up in TLS handshakes, the Signal protocol, and WireGuard. The elliptic-curve variant (ECDH) gives equivalent security with much smaller keys and is the default in modern systems.

2. Step 2Invert encrypt()

   Observation

   encrypt() multiplies each character ordinal by key * 311 to build the cipher list. Divide by the same factor to get the intermediate semi_cipher values back.

   Write a decrypt() that integer-divides each cipher entry by key * 311 with //. This yields the "semi_cipher" string prior to the dynamic XOR stage.

   bash

   semi_cipher = [c // (key * 311) for c in cipher]

   Copied!

   What didn't work first

   Tried: Use regular division (/) instead of integer division (//) when inverting the multiplication

   Python's / returns a float, so each entry lands as 112.0 rather than 112, and chr() then refuses it with a TypeError. Use // for floor division and you get an int.

   Tried: Guess that the magic constant 311 is a red herring and try dividing by just the key alone

   Leave out the 311 and every value comes back 311 times too large, giving ordinals far above 127 that either overflow chr() or decode to garbage Unicode. The constant is hard-coded on the multiplication line in encrypt(), so it belongs in the divisor too.

   Learn more

   The encryption multiplies each character's ordinal by key * 311. Decryption divides. Use Python's integer division operator // here, not /: regular division returns a float, which then breaks the chr() call downstream because character codes have to be integers.

   The value 311 is hard-coded in the encryption function. Custom ciphers often add "complexity" through multiplication by a magic constant, but this provides zero security once the script is in your hands. Security through obscurity is not a primitive.

   The cipher is layered: DH key exchange, then multiplication, then XOR. Decryption undoes each layer in reverse: derive the key, divide out key * 311, then reverse the XOR. This compositional pattern is exactly how block-cipher modes and AEAD schemes are reasoned about.

3. Step 3Reverse dynamic_xor_encrypt

   Observation

   dynamic_xor_encrypt() reverses the plaintext first, then XORs each character against the repeating key 'trudeau'. So XOR semi_cipher the same way, forward through the key, and reverse the result at the end.

   Write a dynamic_xor_decrypt that applies the same forward-cycling repeating-key XOR (using text_key[i % key_length]) to the semi_cipher string, then reverses the result with [::-1]. The key walk direction is unchanged from encrypt; the only difference is where the reversal happens. Applying it to semi_cipher with the key "trudeau" reveals the flag.

   python

   python3 solver.py  # uses decrypt + dynamic_xor_decrypt

   Copied!

   Expected output

   picoCTF{custom_d2cr0pt6d_751a...}

   What didn't work first

   Tried: Skip reversing the string at the end and just return the XOR result directly

   encrypt() reverses the plaintext before XORing, so the XOR ran against the reversed string. Forget to reverse afterwards and the flag comes out backwards, starting with } instead of picoCTF{. The reversal goes after the XOR, undoing the one that happened before it.

   Tried: Try the XOR key in reverse (walking text_key backward) instead of cycling forward through it

   encrypt() cycles the key forward with i % key_length no matter what the plaintext reversal did. XOR is its own inverse, so walk the key forward here too. Walk it backwards and every position gets the wrong key byte.

   Learn more

   XOR encryption with a repeating key is a simple stream cipher. Each character of plaintext is XORed with the corresponding character of the key, cycling if the key is shorter. XOR is its own inverse: if cipher = plain XOR key, then plain = cipher XOR key.

   The script's text_key variable holds the repeating key bytes, derived from the literal string "trudeau". The encrypt function reverses the plaintext string first (plaintext[::-1]) and then XORs each character with text_key[i % key_length], cycling forward. Decryption applies the same forward-cycling key XOR to the intermediate string, then reverses the result with [::-1]. There is no chained state; each position XORs independently with text_key[i % key_length].

   The key "trudeau" is a weak key: a short dictionary word. Real stream ciphers (RC4, ChaCha20) use much longer pseudorandom key streams. A repeating ASCII key is trivially broken by frequency analysis once the key length is guessed (Kasiski examination, index of coincidence).

Flag

The decrypted semi_cipher plus the reversed XOR routine yields the flag above. If the output doesn't start with picoCTF{, recheck the DH key recovery and confirm you used // (integer division) when undoing key * 311.