YaraRules0x100 picoCTF 2025 Solution

Description

1. Step 1Collect indicators

   Observation

   The setup already confirmed the binary is UPX-packed, so most of its own strings are compressed and invisible until you unpack it. Diff the packed and unpacked string dumps and the distinctive ones, such as the application name and NtQueryInformationProcess, stand out as anchors for a rule.

   Run strings before and after unpacking. Before: you see "UPX0", "UPX1", and the UPX stub's loader text. After: the original strings appear, including "YaraRules0x100", "NtQuery..." Windows API names, and the phrase "debugger process". These plus the MZ magic become your rule's strings.

   bash

   # Packed (before upx -d): only UPX section names and stub text are visible.
   $ strings suspicious.exe.bak | grep -E 'UPX|Yara|NtQuery|debugger'
   UPX0
   UPX1
   UPX!
   $Info: This file is packed with the UPX executable packer http://upx.sf.net $
   
   # Unpacked (after upx -d): the original program's strings appear.
   $ strings suspicious.exe | grep -E 'UPX|Yara|NtQuery|debugger'
   YaraRules0x100
   NtQueryInformationProcess
   IsDebuggerPresent
   debugger process detected
   
   # Diff the two for everything UPX hid:
   diff <(strings ./suspicious.exe.bak) <(strings ./suspicious.exe) | head

   Copied!

   What didn't work first

   Tried: Running strings on the packed binary (suspicious.exe.bak) and writing the YARA rule from those results

   While packed, the string table is almost entirely the UPX stub: the section names, the magic, the loader banner. The application's own strings stay compressed until you unpack. A rule built only from the packed view fails on any variant whose UPX headers differ.

   Tried: Skipping the backup step and running upx -d directly on suspicious.exe, then trying to compare packed vs unpacked strings

   upx -d overwrites in place with no undo. Lose the packed copy and you cannot recover the UPX section names without re-extracting the download. Since the rule needs to match both variants, that leaves you guessing at packer strings instead of diffing the two files.

   Learn more

   YARA is an open-source pattern-matching tool originally developed at VirusTotal and now maintained by the security community. It allows malware analysts to write rules that describe file characteristics (byte sequences, string patterns, and logical conditions) and apply those rules to scan files, memory dumps, or network streams. YARA rules are the lingua franca of malware detection and are used in antivirus engines, SIEM platforms, EDR products, and threat hunting tools.

   The strings utility extracts printable character sequences (default minimum length: 4) from any binary file. It is one of the fastest first-pass analysis tools because it reveals hardcoded URLs, registry keys, API calls, error messages, and embedded data without requiring disassembly. Running it before and after unpacking a UPX-compressed executable reveals two different views: the compressed binary shows minimal strings (mostly the UPX stub), while the unpacked binary exposes the original executable's full string table.

   UPX (Ultimate Packer for eXecutables) is a legitimate compression tool for reducing binary size, but it is also extremely popular among malware authors for evading signature-based detection. Unpacking with upx -d recovers the original binary. Some malware modifies the UPX headers to prevent automatic unpacking; in those cases, dynamic unpacking (running the sample in a sandbox until it decompresses itself into memory) or manual analysis with a debugger is required. The hex dumps for CTF guide covers reading PE headers in xxd.

2. Step 2Write the YARA rule

   Observation

   The unpacked strings carry both a unique identifier and behavioral anti-debug indicators. Combine the PE magic bytes with several string paths so the rule matches whichever variant the harness tests.

   Combine the indicators into a rule that looks for the MZ header plus YaraRules0x100 and either UPX or NtQuery, or alternatively the entire "debugger process" string. The two-clause or is deliberate; explained below.

   bash

   rule Rule {
       strings:
           $mz = {4D 5A}
           $name = "YaraRules0x100"
           $packer = "UPX"
           $ntquery = "NtQuery"
           $phrase = "debugger process" wide ascii
       condition:
           ($mz and $name and ($packer or $ntquery)) or $phrase
   }

   Copied!

   What didn't work first

   Tried: Writing a rule whose condition is only '$mz and $name' without any packer or behavioral indicators

   The grader tests against several sample variants, not just your download. A condition resting on the PE magic and one name string fails where the UPX stub alters the header or the name is encoded differently. A third path covering the packer artifact or the API name spans both variants.

   Tried: Omitting 'wide ascii' from the $phrase string definition and submitting the rule

   Windows executables often store literals as UTF-16, two bytes per character with a null between them. A plain YARA string searches only the single-byte encoding, so a wide-stored phrase never matches and that clause of the condition dies quietly. Marking the string wide ascii searches both encodings in one pass.

   Learn more

   A YARA rule consists of three sections: meta (optional metadata like author and date), strings (pattern definitions), and condition (boolean logic combining the patterns). String patterns can be plaintext ("text"), regular expressions (/regex/), or hex byte sequences ({4D 5A}). The wide modifier matches UTF-16LE (two bytes per character, common in Windows strings), and ascii matches standard single-byte encoding; using both together covers both formats.

   The MZ magic bytes (0x4D 0x5A) mark the start of every Windows PE (Portable Executable) file: DLLs, EXEs, and SYS files all begin with MZ. Using it as an anchor in the condition ($mz at 0 for strict offset matching, or just $mz for presence anywhere) helps limit false positives to PE files. Combining it with application-specific strings like YaraRules0x100 makes the rule highly specific to this particular sample family.

   Good YARA rule design balances specificity (low false positive rate) with generality (catching all variants of a malware family). Production threat intelligence teams write rules around behavioral indicators (specific API call sequences, anti-analysis techniques like NtQuery for debugger detection) that persist across recompilations, rather than exact byte sequences that change with each build. Resources like YARA documentation, yarGen (automated rule generator), and CAPE sandbox help accelerate rule development from live samples.

   Why the compound condition ($mz and $name and ($packer or $ntquery)) or $phrase? It gives the rule two independent paths to a hit, which is what makes it robust against the harness's sample variations. The first clause matches a PE file that contains both the unique application string YaraRules0x100 and at least one of the anti-analysis indicators (UPX section names if still packed, NtQuery if unpacked). The second clause, $phrase alone, catches any file containing the literal "debugger process" regardless of structure, which fires on samples where the harness stripped the MZ header or removed the application name. Two paths means a sample only has to satisfy one of them to match, dramatically reducing false negatives compared with a single conjunctive condition.

3. Step 3Submit via socat

   Observation

   The rule goes to a remote grading service over TCP. Pipe the file with socat, and raise the timeout so the grader can finish scanning its sample corpus before the connection closes.

   Save the rule to sample.txt (or any filename) and pipe it to the grading service with socat. If it matches all test cases, the server returns the flag.

   bash

   socat -t60 - TCP:standard-pizzas.picoctf.net:<PORT_FROM_INSTANCE> < sample.txt

   Copied!

   Expected output

   picoCTF{yara_rul35_r0ckzzz_216...}

   Learn more

   socat (SOcket CAT) is a multipurpose relay tool that creates bidirectional data channels between various types of endpoints: TCP sockets, UDP, files, stdin/stdout, Unix domain sockets, SSL connections, and more. It is the swiss-army knife of network connectivity, frequently used in CTF challenges for connecting to remote services, setting up listeners, and relaying data between protocols. See networking tools for CTF for the full kit.

   The -t60 flag sets a 60-second inactivity timeout, useful here because the grader runs your rule against a sample corpus and complex YARA conditions (especially ones with regex strings or many wide ascii modifiers) can take several seconds per file. Without the timeout, socat's default closes the connection long before the server finishes scoring and you never see the flag. If 60 seconds isn't enough (your rule is heavy, the server is busy, or both), bump it to -t120 and resubmit; the grader doesn't penalize a longer wait. The - endpoint means stdin/stdout, so redirecting a file with < sample.txt sends the file contents as input to the TCP connection.

   In real threat hunting workflows, YARA rules are deployed to scanning infrastructure using tools like YARA-X (the Rust rewrite of YARA), integrated into SIEMs via Velociraptor or THOR for endpoint scanning, or uploaded to sandboxes like VirusTotal and MalwareBazaar to monitor for new samples matching the rule. Writing effective YARA rules is a core competency for malware analysts and threat intelligence teams.

Flag

Any rule that nails at least one unique string plus the PE header works; the combination above passed all server tests.