YaraRules0x100 picoCTF 2025 Solution

Description

1. Step 1

   Collect indicators

   Observation

   I noticed the setup step confirmed the binary was UPX-packed via strings suspicious.exe | grep -i upx, which suggested that most app-specific strings are compressed and invisible until after unpacking; diffing the packed and unpacked outputs would expose exactly which strings (like 'YaraRules0x100' and 'NtQueryInformationProcess') are unique enough to anchor a YARA rule.

   Run strings before and after unpacking. Before: you see "UPX0", "UPX1", and the UPX stub's loader text. After: the original strings appear, including "YaraRules0x100", "NtQuery..." Windows API names, and the phrase "debugger process". These plus the MZ magic become your rule's strings.

   bash

   # Packed (before upx -d): only UPX section names and stub text are visible.
   $ strings suspicious.exe.bak | grep -E 'UPX|Yara|NtQuery|debugger'
   UPX0
   UPX1
   UPX!
   $Info: This file is packed with the UPX executable packer http://upx.sf.net $
   
   # Unpacked (after upx -d): the original program's strings appear.
   $ strings suspicious.exe | grep -E 'UPX|Yara|NtQuery|debugger'
   YaraRules0x100
   NtQueryInformationProcess
   IsDebuggerPresent
   debugger process detected
   
   # Diff the two for everything UPX hid:
   diff <(strings ./suspicious.exe.bak) <(strings ./suspicious.exe) | head

   Copied!

   What didn't work first

   Tried: Running strings on the packed binary (suspicious.exe.bak) and writing the YARA rule from those results

   The packed binary's string table is almost entirely the UPX stub - you only see 'UPX0', 'UPX1', 'UPX!', and the loader banner. The app-specific strings like 'YaraRules0x100' and 'NtQueryInformationProcess' are compressed and invisible until after upx -d runs. A rule built from packed strings alone will fail on any variant where the UPX headers differ.

   Tried: Skipping the backup step and running upx -d directly on suspicious.exe, then trying to compare packed vs unpacked strings

   upx -d overwrites the file in place with no undo option. Once the packed copy is gone you cannot regenerate the UPX section names ('UPX0', 'UPX1') without re-zipping and re-extracting. If the YARA rule needs to match both packed and unpacked variants, losing the packed copy means you have to guess which packer-specific strings were present instead of diffing the two files directly.

   Learn more

   YARA is an open-source pattern-matching tool originally developed at VirusTotal and now maintained by the security community. It allows malware analysts to write rules that describe file characteristics (byte sequences, string patterns, and logical conditions) and apply those rules to scan files, memory dumps, or network streams. YARA rules are the lingua franca of malware detection and are used in antivirus engines, SIEM platforms, EDR products, and threat hunting tools.

   The strings utility extracts printable character sequences (default minimum length: 4) from any binary file. It is one of the fastest first-pass analysis tools because it reveals hardcoded URLs, registry keys, API calls, error messages, and embedded data without requiring disassembly. Running it before and after unpacking a UPX-compressed executable reveals two different views: the compressed binary shows minimal strings (mostly the UPX stub), while the unpacked binary exposes the original executable's full string table.

   UPX (Ultimate Packer for eXecutables) is a legitimate compression tool for reducing binary size, but it is also extremely popular among malware authors for evading signature-based detection. Unpacking with upx -d recovers the original binary. Some malware modifies the UPX headers to prevent automatic unpacking; in those cases, dynamic unpacking (running the sample in a sandbox until it decompresses itself into memory) or manual analysis with a debugger is required. The hex dumps for CTF guide covers reading PE headers in xxd.

2. Step 2

   Write the YARA rule

   Observation

   I noticed the unpacked strings included both the unique identifier 'YaraRules0x100' and behavioral anti-debug indicators like 'NtQueryInformationProcess' and 'debugger process', which suggested combining the PE magic bytes with these multi-path string conditions to ensure the rule matches across packed and unpacked variants the harness might test.

   Combine the indicators into a rule that looks for the MZ header plus YaraRules0x100 and either UPX or NtQuery, or alternatively the entire "debugger process" string. The two-clause or is deliberate; explained below.

   bash

   rule Rule {
       strings:
           $mz = {4D 5A}
           $name = "YaraRules0x100"
           $packer = "UPX"
           $ntquery = "NtQuery"
           $phrase = "debugger process" wide ascii
       condition:
           ($mz and $name and ($packer or $ntquery)) or $phrase
   }

   Copied!

   What didn't work first

   Tried: Writing a rule whose condition is only '$mz and $name' without any packer or behavioral indicators

   The grading harness tests the rule against multiple sample variants, not just the single downloaded file. A condition that only checks the PE magic and the application name string may fail on variants where the UPX stub alters the header region or the name string is encoded differently. Adding '$packer or $ntquery' as a third path gives the rule coverage across packed and unpacked variants.

   Tried: Omitting 'wide ascii' from the $phrase string definition and submitting the rule

   Windows executables frequently store string literals as UTF-16LE (wide), where each character is two bytes with a null byte between them. A plain string match in YARA only searches for the ASCII (single-byte) encoding. If 'debugger process' is stored as wide in the binary, the rule's $phrase string will never match and that fallback clause of the condition fails entirely. Using 'wide ascii' makes YARA search both encodings in a single pass.

   Learn more

   A YARA rule consists of three sections: meta (optional metadata like author and date), strings (pattern definitions), and condition (boolean logic combining the patterns). String patterns can be plaintext ("text"), regular expressions (/regex/), or hex byte sequences ({4D 5A}). The wide modifier matches UTF-16LE (two bytes per character, common in Windows strings), and ascii matches standard single-byte encoding; using both together covers both formats.

   The MZ magic bytes (0x4D 0x5A) mark the start of every Windows PE (Portable Executable) file: DLLs, EXEs, and SYS files all begin with MZ. Using it as an anchor in the condition ($mz at 0 for strict offset matching, or just $mz for presence anywhere) helps limit false positives to PE files. Combining it with application-specific strings like YaraRules0x100 makes the rule highly specific to this particular sample family.

   Good YARA rule design balances specificity (low false positive rate) with generality (catching all variants of a malware family). Production threat intelligence teams write rules around behavioral indicators (specific API call sequences, anti-analysis techniques like NtQuery for debugger detection) that persist across recompilations, rather than exact byte sequences that change with each build. Resources like YARA documentation, yarGen (automated rule generator), and CAPE sandbox help accelerate rule development from live samples.

   Why the compound condition ($mz and $name and ($packer or $ntquery)) or $phrase? It gives the rule two independent paths to a hit, which is what makes it robust against the harness's sample variations. The first clause matches a PE file that contains both the unique application string YaraRules0x100 and at least one of the anti-analysis indicators (UPX section names if still packed, NtQuery if unpacked). The second clause, $phrase alone, catches any file containing the literal "debugger process" regardless of structure, which fires on samples where the harness stripped the MZ header or removed the application name. Two paths means a sample only has to satisfy one of them to match, dramatically reducing false negatives compared with a single conjunctive condition.

3. Step 3

   Submit via socat

   Observation

   I noticed the challenge required submitting a YARA rule to a remote grading service at standard-pizzas.picoctf.net:59919, which suggested using socat to pipe the rule file to the TCP endpoint and adding the -t60 timeout flag to allow the grader enough time to scan its sample corpus before closing the connection.

   Save the rule to sample.txt (or any filename) and pipe it to the grading service with socat. If it matches all test cases, the server returns the flag.

   bash

   socat -t60 - TCP:standard-pizzas.picoctf.net:59919 < sample.txt

   Copied!

   Expected output

   picoCTF{yara_rul35_r0ckzzz_216...}

   Learn more

   socat (SOcket CAT) is a multipurpose relay tool that creates bidirectional data channels between various types of endpoints: TCP sockets, UDP, files, stdin/stdout, Unix domain sockets, SSL connections, and more. It is the swiss-army knife of network connectivity, frequently used in CTF challenges for connecting to remote services, setting up listeners, and relaying data between protocols. See networking tools for CTF for the full kit.

   The -t60 flag sets a 60-second inactivity timeout, useful here because the grader runs your rule against a sample corpus and complex YARA conditions (especially ones with regex strings or many wide ascii modifiers) can take several seconds per file. Without the timeout, socat's default closes the connection long before the server finishes scoring and you never see the flag. If 60 seconds isn't enough (your rule is heavy, the server is busy, or both), bump it to -t120 and resubmit; the grader doesn't penalize a longer wait. The - endpoint means stdin/stdout, so redirecting a file with < sample.txt sends the file contents as input to the TCP connection.

   In real threat hunting workflows, YARA rules are deployed to scanning infrastructure using tools like YARA-X (the Rust rewrite of YARA), integrated into SIEMs via Velociraptor or THOR for endpoint scanning, or uploaded to sandboxes like VirusTotal and MalwareBazaar to monitor for new samples matching the rule. Writing effective YARA rules is a core competency for malware analysts and threat intelligence teams.

Flag

Any rule that nails at least one unique string plus the PE header works; the combination above passed all server tests.