Document Forensics for CTF | picoCTF Solutions

Introduction

Two commands solve most document challenges, and neither of them is strings:

# Any .docx, .xlsx, .pptx, .pptm: it is a ZIP file
unzip -o suspicious.docx -d doc/ && grep -ria picoctf doc/
 
# Any PDF: decompress every stream, then read it as text
qpdf --qdf --object-streams=disable in.pdf out.pdf && grep -a picoCTF out.pdf

The reason strings fails on both is the same in both cases. Modern document formats compress their content, so the text you are looking for is not stored as text. A Word document is a ZIP archive of XML; a PDF stores its page content in Flate-compressed streams. Run strings on either and you get font names, a producer string, and nothing else. That is the result that convinces people the flag is not in the file.

Document forensics is not about finding hidden data. It is about decompressing the file into the form the application sees, and then reading it.

I find this category quietly satisfying because the bugs are real and they are everywhere. Failed redaction in a court filing, a spreadsheet whose deleted rows are still in the tracked-changes log, a slide deck with a hidden slide from the previous client. The CTF versions are not simulations of a professional problem. They are the professional problem, scaled down.

Start from the file you were given

You want	Read
Commands to run right now	Sixty-second triage then Quick reference
A PDF that looks empty	How a PDF is built and why redaction fails
To understand the formats	Top to bottom. Roughly twenty minutes

Sixty-second triage

Always start by finding out what you actually have, because the extension lies more often in this category than anywhere else. A .doc can be a ZIP, a .pdf can be an image with a PDF wrapper, and a .docx with a renamed extension is still a ZIP underneath.

file mystery.doc
xxd mystery.doc | head -2
 
#  50 4b 03 04         PK..    -> ZIP, so OOXML (docx/xlsx/pptx) or a plain archive
#  d0 cf 11 e0 a1 b1   ....    -> OLE2 compound file, the legacy .doc/.xls/.ppt
#  25 50 44 46 2d      %PDF-   -> PDF
#  7b 5c 72 74 66      {\rtf   -> RTF, which is plain text and often overlooked
 
# Then, before anything clever:
exiftool mystery.doc          # author, producer, timestamps, sometimes the flag
binwalk -e mystery.doc        # appended archives and embedded files

Magic bytes	Format	First move
50 4b 03 04	OOXML or ZIP	unzip -o f -d out/ && grep -ria picoctf out/
d0 cf 11 e0	OLE2 compound file	oleid f / olevba f
25 50 44 46	PDF	qpdf --qdf --object-streams=disable in out
7b 5c 72 74	RTF	grep -o '\\objdata' f / rtfobj f

Tip: If you have no tools installed and a browser, the file magic identifier takes a drag and drop and tells you what the header says, and the hex viewer shows you the bytes. The general skill of reading headers is in file carving and magic bytes.

Office files are ZIP archives

Since Office 2007, every .docx, .xlsx, .pptx and their macro-enabled cousins have been ZIP archives of XML, specified publicly as ECMA-376. That single fact is worth more than any tool, because it means every standard archive utility on earth can take one apart.

unzip -l report.docx
 
  [Content_Types].xml            what each part is
  _rels/.rels                    the root relationships
  docProps/core.xml              author, title, revision, timestamps
  docProps/app.xml               word counts, template, application name
  word/document.xml              the actual text
  word/_rels/document.xml.rels   external and internal targets
  word/comments.xml              review comments, if any
  word/media/image1.png          every embedded picture, uncompressed
  word/embeddings/oleObject1.bin embedded files, often a whole other document

The media/ directory is the first place to look and the most commonly missed, because an image inside a document is stored as an ordinary file. That means every image forensics technique applies to it directly: run exiftool on it, check it for appended data, run it through a steganography workflow. A picture pasted into a slide deck carries its own metadata into the archive, and nobody ever strips it.

# Unpack and look at everything at once
mkdir -p x && unzip -q -o deck.pptx -d x && find x -type f | sort
 
# Search the XML, not the archive
grep -ria 'picoctf' x/
 
# Pretty-print the XML so it is readable by a human
xmllint --format x/word/document.xml | less
 
# Any embedded image is a file in its own right
exiftool x/word/media/* | head -40

Where things hide in OOXML

Once the archive is open, there are six places a flag realistically lives, and they are worth checking in this order because that is roughly the order of how often they pay off.

Hiding place	What to look at	Why it survives
A file that is not part of the document	Anything in the archive listing that no relationship references	The editor never renders it, so nobody notices it is there
Hidden slides	The slide parts themselves: `ppt/slides/slideN.xml` whose root `p:sld` element carries `show="0"`	Hidden slides still ship in the file, they just do not present
Tracked changes	`w:del` and `w:ins` elements in `word/document.xml`	Deleted text is kept so the deletion can be undone
Comments and notes	`word/comments.xml`, `ppt/notesSlides/`	Speaker notes are not shown to the audience or in a print
Embedded objects	`word/embeddings/`, usually OLE2 blobs	A whole second document lives inside the first
Custom XML parts	`customXml/` and unreferenced entries in `[Content_Types].xml`	Applications write data here that no UI ever displays

MacroHard WeakEdge is the first row. It hands you a PowerPoint file with a macro-enabled extension, which points every instinct at the macros, and the flag is in a file sitting in a subdirectory of the archive that the presentation never references. The intended path is simply to list the archive and notice the entry that does not belong.

# Find the parts nothing references: list every file, then every rel target
unzip -q -o deck.pptm -d x
find x -type f | sed 's|^x/||' | sort > /tmp/present
grep -rhoE 'Target="[^"]+"' x/ | sed 's/Target="//; s/"//' \
  | sed 's|^\.\./||; s|^/||' | sort -u > /tmp/referenced
comm -23 /tmp/present /tmp/referenced      # candidates worth opening

Note: The comparison above is approximate: relationship targets are relative to the part that declares them, so a few false positives are normal. That is fine. You are looking for one obviously odd filename in a list of twenty, not building a validator.

The legacy formats and macros

A file starting d0 cf 11 e0 is an OLE2 compound file, which Microsoft documents as [MS-CFB]. It is a whole filesystem inside a file, with directories and streams, which is why unzip does nothing to it. The tool for this is Philippe Lagadec's oletools, and it is worth installing before you need it.

pip install oletools
 
oleid   suspicious.doc     # a summary: macros? encrypted? external links?
oledir  suspicious.doc     # every directory entry and stream in the file
olemeta suspicious.doc     # the metadata streams
olevba  suspicious.doc     # extract and deobfuscate VBA source
oleobj  suspicious.doc     # pull out embedded objects
rtfobj  suspicious.rtf     # embedded objects out of an RTF

Macros are the reason these formats have a reputation, and in a CTF a macro is usually an obfuscated string builder rather than real malware. olevba already understands the common obfuscations and prints a deobfuscated view, which handles most of it. When it does not, the VBA is doing arithmetic on character codes and you can transcribe it into Python in a couple of minutes.

' The VBA you will actually meet
s = ""
For i = 1 To Len(k)
    s = s & Chr(Asc(Mid(k, i, 1)) Xor 42)
Next
 
# The transcription
print(''.join(chr(ord(c) ^ 42) for c in k))

Warning: Macro-enabled extensions (.docm, .pptm, .xlsm) are OOXML, not OLE2, so they unzip normally and the VBA lives in a single OLE2 blob inside the archive at word/vbaProject.bin or ppt/vbaProject.bin. Pull that out and hand it to oletools. And do not run the document to see what the macro does. If you need to watch it run, use a throwaway virtual machine with no network, which is the same discipline described in setting up a CTF environment.

How a PDF is actually built

PDF has a reputation for being opaque and it does not deserve it. A PDF is a text file with compressed blobs in it, and the structure is simple enough to read by eye once somebody shows you the four pieces.

%PDF-1.7                     header, the version
 
1 0 obj                      an object: number, generation, content
<< /Type /Page /Contents 2 0 R >>
endobj
 
2 0 obj                      a stream object: a dictionary plus bytes
<< /Length 512 /Filter /FlateDecode >>
stream
...compressed page content...
endstream
endobj
 
xref                         a table of byte offsets for every object
trailer << /Root 5 0 R >>
startxref
%%EOF                        end of this revision

Two properties of that structure are what make PDF forensics work. First, page text lives inside /FlateDecode streams, which is zlib compression, which is why grep finds nothing until you decompress. Second, a PDF can be updated incrementally: an editor appends new objects and a new cross-reference table rather than rewriting the file, so an edited PDF contains its own previous versions. More than one %%EOF means more than one revision is present.

# Decompress everything into readable form. This is the whole trick.
qpdf --qdf --object-streams=disable in.pdf readable.pdf
grep -a picoCTF readable.pdf
 
# How many revisions does this file contain?
grep -abo '%%EOF' in.pdf
 
# The text layer, in reading order
pdftotext -layout in.pdf - | less
 
# Every embedded image, as its own file
pdfimages -all in.pdf img
 
# Embedded files and fonts
mutool extract in.pdf
 
# Structure summary and risk indicators. These two are single-file scripts from
# Didier Stevens, downloaded rather than installed from a package.
python3 pdfid.py in.pdf         # /JS /OpenAction /EmbeddedFile counts
python3 pdf-parser.py -a in.pdf # object statistics

Why redaction fails

Redaction gone wrong is a two-minute challenge with a large lesson behind it. The document has black bars over the sensitive text. The text is still there.

This happens because drawing a rectangle and deleting text are different operations, and the tool most people reach for does the first one. A PDF page is a sequence of drawing instructions: show this text at this position, fill this rectangle in black. Adding the rectangle does not remove the text instruction that came before it. The text is still in the content stream, in reading order, and any extractor will hand it to you.

pdftotext -layout redacted.pdf -    # the black bars are not in the text layer
 
# If the text layer is empty, the page may be an image. Then:
pdfimages -all redacted.pdf page && tesseract page-000.png -

Redaction method	Does it work?	How to defeat it
Black rectangle over text	No	pdftotext
White text on white background	No	Select all and copy, or pdftotext
Text deleted, file saved incrementally	No	Read the earlier revision before the last %%EOF
Page rasterised to an image	Mostly	Nothing, unless the original text layer was kept alongside
Content removed and file rewritten in full	Yes	Check metadata and attachments anyway

The incremental-update row is the one worth carrying out of this guide. If a PDF has three %%EOF markers, it has three versions of itself inside it, and the first one may predate whatever the author removed. Truncating the file just after an earlier %%EOF gives you a valid PDF of that revision.

# Where do the revisions end?
grep -abo '%%EOF' doc.pdf
#   1024:%%EOF     <- first revision ends here
#   8891:%%EOF
 
# Carve the first revision out and open it
head -c 1029 doc.pdf > rev1.pdf     # offset of %%EOF + 5
pdftotext rev1.pdf -

Every redaction failure is the same mistake: hiding something from the renderer instead of removing it from the file.

What I still find hard to believe is how well this keeps working outside of CTFs. It is not an obscure bug in an old viewer. It is what happens when somebody draws a black box in the tool they already had open, and the file dutifully records both the box and the text underneath it, because recording both is what a page description language is for.

Invisible text and zero-width tricks

The other half of this category is text that is present and rendered, but not visible. Word documents give an author three separate ways to do that, and all three leave the text in word/document.xml where grep finds it instantly.

<!-- Marked hidden. Word simply does not draw it. -->
<w:rPr><w:vanish/></w:rPr>
 
<!-- White on white -->
<w:rPr><w:color w:val="FFFFFF"/></w:rPr>
 
<!-- One point type. w:sz is in half-points, so 2 means 1pt. -->
<w:rPr><w:sz w:val="2"/></w:rPr>
 
# All three are found the same way
grep -oE '<w:(vanish|color w:val="FFFFFF"|sz w:val="[1-4]")' x/word/document.xml

Then there is the encoding version, where the message is not hidden by formatting but carried by characters that render as nothing. Zero-width space, zero-width non-joiner, and the byte-order mark all occupy no visual space, so a run of them between ordinary words is a binary channel: one character means zero, the other means one.

# Count the invisible code points, then decode the two that dominate.
import collections, sys
 
text = open(sys.argv[1], encoding='utf-8').read()
blank = [c for c in text if c.isspace() or ord(c) in (0x200b, 0x200c, 0x200d, 0xfeff)]
counts = collections.Counter(blank)
print(counts.most_common(5))          # two symbols dominating = a one-bit channel
 
(zero, _), (one, _) = counts.most_common(2)
bits = ''.join('0' if c == zero else '1' for c in blank if c in (zero, one))
 
# You cannot know which symbol means zero, so print whichever polarity is readable.
for b in (bits, bits.translate(str.maketrans('01', '10'))):
    out = ''.join(chr(int(b[i:i+8], 2)) for i in range(0, len(b) - 7, 8))
    if out.isprintable():
        print(out)

WhitePages is the pure form of this: a text file that renders as blank space and is actually two distinct whitespace characters encoding a binary message. The technique is the same whether the carrier is a bare text file, a Word document, or a source file where tabs and spaces have been swapped, and it appears again in steganography techniques. Do not confuse it with the Whitespace programming language, which uses three symbols rather than two and has to be executed rather than decoded; esoteric languages in CTF covers how to tell those two apart from a character count.

Tip: When a document renders as visibly empty, the fastest check is a character-frequency count rather than a hex dump. Two unusual code points in a roughly even split is a binary encoding; one code point repeated is padding; a spread of many is probably just a document. Ten seconds, and it tells you which of the three you have.

The metadata layer

Every format in this guide carries metadata, and unlike the content it is usually uncompressed and always ignored. Run exiftool on the document itself before you unpack anything.

exiftool report.docx
#   Creator, Last Modified By, Revision Number, Total Edit Time,
#   Company, Template, Application, Create Date, Modify Date
 
exiftool paper.pdf
#   Producer and Creator name the exact software and version,
#   which dates the file and sometimes names the machine
 
# The raw XMP packet, which exiftool summarises but does not always show whole
strings -a paper.pdf | sed -n '/<x:xmpmeta/,/<\/x:xmpmeta>/p'

Three fields earn their keep repeatedly. Revision Number in an Office document counts saves, so a value of 1 on a document that claims a long history is a contradiction worth chasing. Total Edit Time is minutes of real editing and is trivially checked against a claimed timeline. And Producer in a PDF names the exact library that wrote the file, which tells you whether it was generated by a scanner, a word processor, or a script, and therefore what other layers you should expect to find.

Metadata forensics has its own guide, since the same field families show up on images and media: image metadata and EXIF forensics, and the metadata viewer handles a document drag-and-drop in the browser when you are on a machine without exiftool.

picoCTF challenges

Challenge	The document trick	Command that solves it
Redaction gone wrong	Text covered by a drawn rectangle, never removed	pdftotext doc.pdf -
MacroHard WeakEdge	A file inside the pptm archive that no slide references	unzip -o f.pptm -d x && find x -type f
WhitePages	Two invisible whitespace characters encoding bits	Count code points, decode as binary
Big Zip	The archive skill on its own: search inside, not around	grep -r picoCTF .

The rest of the category ladder is in the forensics roadmap, and documents pulled out of a disk image or a packet capture are handled in disk forensics and Wireshark and pcap analysis.

Quick reference

# Always first
file f && xxd f | head -2 && exiftool f
 
# OOXML (docx, xlsx, pptx, docm, pptm, xlsm)
unzip -o f -d x/ && find x -type f | sort
grep -ria picoctf x/
xmllint --format x/word/document.xml | less
exiftool x/*/media/*                       # embedded images carry their own metadata
grep -l 'show="0"' x/ppt/slides/*.xml     # hidden slides
grep -oE '<w:(vanish|del |ins )' x/word/document.xml
 
# Legacy OLE2 (doc, xls, ppt) and macros
oleid f && oledir f && olevba f && oleobj f
rtfobj f                                    # RTF embedded objects
 
# PDF
qpdf --qdf --object-streams=disable in.pdf out.pdf && grep -a picoCTF out.pdf
pdftotext -layout in.pdf -
pdfimages -all in.pdf img
grep -abo '%%EOF' in.pdf                    # more than one means older revisions
mutool extract in.pdf                       # embedded files and fonts
 
# Anything that renders as blank
python3 -c "import collections,sys;
print(collections.Counter(open(sys.argv[1],encoding='utf-8').read()))" f

Related reading: file carving and magic bytes for headers and embedded files, EXIF and metadata forensics for the metadata layer, steganography techniques for the images you pulled out of the archive, archive password cracking when the embedded ZIP wants a password, and the forensics roadmap for where this sits in the category.

Sources and further reading

Both document families are publicly specified, which is why the tooling is so good. Where a tool is the reference, the tool is cited.

ECMA-376 is the Office Open XML specification, and the part worth knowing is that it exists at all. The formats are open, which is why unzip is a legitimate forensics tool rather than a hack.
The PDF 1.7 specification documents the object model, stream filters, and incremental updates. Section 7.5.6 on incremental updates is the one that explains why deleted content survives in a saved file, and it is three pages.
[MS-CFB] for the legacy compound file format, which is genuinely a small filesystem and explains why the tooling for .doc looks nothing like the tooling for .docx.
oletools is the reference implementation for everything OLE2, and its documentation doubles as a tour of where data can live in those files. qpdf is the same for PDF, and its --qdf mode is documented as being for exactly the purpose it is used for here.
ExifTool reads metadata from every format above, and its tag documentation is the best available cross-reference for which fields each format actually stores.

Document Forensics for CTF: PDFs, Office Files, and Everything strings Misses