Custom encryption picoCTF 2024 Solution

Description

1. Step 1Rebuild the shared key
   custom_encryption.py prints a and b during test() (look for the print('a:', a) and print('b:', b) calls). Plug them into generator(g, x, p) to recover the same shared key that encrypt() used.
   Learn more

   Diffie-Hellman key exchange lets two parties agree on a shared secret over an insecure channel. The generator(g, x, p) function computes g^x mod p. When two parties compute g^a mod p and g^b mod p and swap results, each raises the received value to their own private exponent and lands on the same shared secret g^(ab) mod p.

   In this challenge a, b, g, and p are all printed by the script (search for print('a:', a) and the matching b line), so the "key exchange" is trivially reversible. Real DH keeps a and b private; only g^a mod p and g^b mod p ever go on the wire. Its security rests on the discrete logarithm problem.

   Modern DH shows up in TLS handshakes, the Signal protocol, and WireGuard. The elliptic-curve variant (ECDH) gives equivalent security with much smaller keys and is the default in modern systems.

2. Step 2Invert encrypt()
   Write a decrypt() that integer-divides each cipher entry by key * 311 with //. This yields the "semi_cipher" string prior to the dynamic XOR stage.
   bash

   semi_cipher = [c // (key * 311) for c in cipher]

   Copied!
   Learn more

   The encryption multiplies each character's ordinal by key * 311. Decryption divides. Use Python's integer division operator // here, not /: regular division returns a float, which then breaks the chr() call downstream because character codes have to be integers.

   The value 311 is hard-coded in the encryption function. Custom ciphers often add "complexity" through multiplication by a magic constant, but this provides zero security once the script is in your hands. Security through obscurity is not a primitive.

   The cipher is layered: DH key exchange, then multiplication, then XOR. Decryption undoes each layer in reverse: derive the key, divide out key * 311, then reverse the XOR. This compositional pattern is exactly how block-cipher modes and AEAD schemes are reasoned about.

3. Step 3Reverse dynamic_xor_encrypt
   Create dynamic_xor_decrypt that walks text_key in the opposite order from the encrypt path. Applying it to semi_cipher with the key "trudeau" reveals the flag.
   python

   python3 solver.py  # uses decrypt + dynamic_xor_decrypt

   Copied!
   Learn more

   XOR encryption with a repeating key is a simple stream cipher. Each character of plaintext is XORed with the corresponding character of the key, cycling if the key is shorter. XOR is its own inverse: if cipher = plain XOR key, then plain = cipher XOR key.

   The script's text_key variable holds the repeating key bytes, derived from the literal string "trudeau". The encrypt function walks text_key forward and applies a running XOR; "reverse" here means walking the same key bytes in the opposite direction (and using the previous decrypted byte as the running state instead of the previous plaintext byte). Each XOR is its own inverse, so applying the operations in opposite sequence undoes the chain.

   The key "trudeau" is a weak key: a short dictionary word. Real stream ciphers (RC4, ChaCha20) use much longer pseudorandom key streams. A repeating ASCII key is trivially broken by frequency analysis once the key length is guessed (Kasiski examination, index of coincidence).

Alternate Solution

Related guides

Flag

The decrypted semi_cipher plus the reversed XOR routine yields the flag above. If the output doesn't start with picoCTF{, recheck the DH key recovery and confirm you used // (integer division) when undoing key * 311.