CTF•逆向工程

CTF Reverse Engineering

Quick reference for RE challenges. For detailed techniques, see supporting files. Don't know where to start, or stuck? Open methodology.md first — it has the triage decision tree, the L0–L6 escalation ladder, and a stuck-recovery table. Ready-to-run scripts (angr solver, Frida hook pack, Ghidra bulk export, Unicorn harness, XOR toolkit) live in scripts/.

Prerequisites

Python packages (all platforms):

pip install frida-tools angr qiling uncompyle6 capstone lief z3-solver unicorn
# For Python 3.9+ bytecode: build pycdc from source
git clone https://github.com/zrax/pycdc && cd pycdc && cmake . && make

Linux (apt):

apt install gdb radare2 binutils strace ltrace apktool upx

macOS (Homebrew):

brew install gdb radare2 binutils apktool upx ghidra

radare2 plugins:

r2pm -ci r2ghidra   # Native Ghidra decompiler for radare2

Manual install:

• pwndbg — Linux: GitHub, macOS: brew install pwndbg/tap/pwndbg-gdb

Additional Resources

• methodology.md - Systematic playbook: Stage-0 triage table (observation → conclusion → next move), symptom routing tree, escalation ladder L0 quick-wins → L1 static → L2 dynamic hooks → L3 symbolic → L4 emulation → L5 side channels → L6 manual deobfuscation, stuck-recovery table, verification checklist, competition time-boxing
• scripts/ - Ready-to-run templates: angr_solve.py (generic stdin flag-checker solver, stdout-string or address mode), frida_hooks.js (comparison hook pack with hexdump + optional backtrace), ghidra_export.py (headless decompile-all to .c for grep/LLM triage), unicorn_harness.py (x86-64/AArch64 ELF function harness with input/output buffers), xor_solve.py (single-byte brute, known-plaintext keystream, repeating-key period search), const_scan.py (crypto magic-constant scanner, both endians)
• crypto-constants.md - Identify crypto before reversing it: fingerprint table (TEA golden ratio, MD5/SHA init+T constants, AES S-box/T-tables, Blowfish pi P-array, RC5/RC6, Murmur/FNV, Base32 alphabets, CRC32, DES S-boxes), decompiled shapes of standard crypto (TEA rounds, RC4 KSA/PRGA, AES rounds, bignum modexp), key/IV/mode extraction recipe, custom-table rule (always dump from binary)
• vm-analysis.md - Custom VM playbook: recognizing the dispatch shape, the six steps (dispatch → VM state struct → instruction format → semantics table → Python disassembler → trace-first shortcut), VM archetypes table (stack/register/accumulator/encrypted-opcode/self-modifying/one-opcode/VM-in-VM), solve strategies (invert, Z3, hybrid trace+lift), pitfalls (computed jumps, decoy handlers, early-exit side channels)
• z3-cookbook.md - Transcribing decompiled checks to Z3: C→Z3 translation dictionary (wrapping arithmetic, LShR vs arithmetic shift, UDiv/URem vs signed, casts via Extract/ZeroExt, rotations, lookup tables via If-chains), recipes (per-byte transforms, checksum-vs-dumped-target, unrolled loops, mixed widths), seven gotchas (printability-first, uniqueness re-check, wrong-transcription validation), when NOT to use Z3
• decompiler-literacy.md - Reading Hex-Rays/Ghidra output: type vocabulary (_DWORD, LOBYTE/BYTE1, qmemcpy), compiler idioms that masquerade as logic (magic-number division, SIMD compare, sign-bit tricks, goto-from-switch), the three readability moves (rename, retype, restructure data into structs), where decompilers lie (mid-function entry, custom calling conventions, self-modifying code, stack-frame errors, Thumb confusion), x86-64/AArch64/MIPS register quick tables, backwards reading workflow for check functions
• tools.md - Static analysis tools (GDB, Ghidra, radare2, IDA, Binary Ninja, dogbolt.org, RISC-V with Capstone, Unicorn emulation, Python bytecode, WASM, Android APK, .NET, packed binaries)
• tools-dynamic.md - Dynamic analysis tools: Frida (hooking, anti-debug bypass, memory scanning, Android/iOS), angr symbolic execution (path exploration, constraints, CFG), lldb (macOS/LLVM debugger), x64dbg (Windows)
• tools-emulation.md - Emulation frameworks and side-channel tooling: Qiling (cross-platform OS-level emulation), Triton (DSE), Intel Pin instruction-counting + genetic algorithm side channel, opcode-only trace reconstruction, LD_PRELOAD time freeze and memcmp side-channel for byte-by-byte bruteforce
• tools-advanced.md - Advanced tools (Part 1): VMProtect/Themida analysis, binary diffing (BinDiff, Diaphora), deobfuscation frameworks (D-810, GOOMBA, Miasm), Qiling framework, Triton DSE, Manticore, Rizin/Cutter, RetDec, custom VM bytecode lifting to LLVM IR
• tools-advanced-2.md - Advanced tools (Part 2): advanced GDB (Python scripting, brute-force, conditional breakpoints, watchpoints, reverse debugging with rr, pwndbg/GEF), advanced Ghidra scripting, patching (Binary Ninja API, LIEF), GDB constraint extraction + ILP solver (BackdoorCTF 2017), GDB position-encoded input zero flag monitoring (EKOPARTY 2017), LD_PRELOAD execute-only binary dump (BackdoorCTF 2017), PEDA current_inst bit-by-bit flag scraper (CONFidence CTF 2019 Teaser)
• anti-analysis.md - Anti-analysis taxonomy: Linux anti-debug (ptrace, /proc, timing, signals, direct syscalls), Windows anti-debug (PEB, NtQueryInformationProcess, heap flags, TLS callbacks, HW/SW breakpoint detection, exception-based, thread hiding), anti-VM/sandbox (CPUID, MAC, timing, artifacts, resources), anti-DBI (Frida detection/bypass), code integrity/self-hashing, anti-disassembly (opaque predicates, junk bytes), MBA identification/simplification, comprehensive bypass strategies
• anti-analysis-ctf.md - CTF writeup techniques: SIGILL handler for execution mode switching (Hack.lu 2015), SIGFPE signal handler side-channel via strace counting (PlaidCTF 2017), instruction trace inversion with Keystone and Unicorn (MeePwn 2017), call-less function chaining via stack frame manipulation (THC 2018), parent-patched child binary dump via process_vm_writev (Google CTF Quals 2018)
• anti-analysis-macos.md - macOS-specific anti-analysis: ptrace(PT_DENY_ATTACH) detection and four bypasses (binary patch byte table for arm64/x86_64, lldb thread return, Frida replace, DYLD interposer), sysctl P_TRACED scrubbing, hardened runtime / library validation / DYLD env stripping with codesign re-signing recipe (get-task-allow), lldb PIE recipes
• patterns.md - Foundational binary patterns: custom VMs, anti-debugging, nanomites, self-modifying code, XOR ciphers, mixed-mode stagers, LLVM obfuscation, S-box/keystream, SECCOMP/BPF, exception handlers, memory dumps, byte-wise transforms, x86-64 gotchas, custom mangle reversing, position-based transforms, hex-encoded string comparison, signal-based binary exploration
• patterns-runtime.md - Runtime patching and oracle techniques: malware anti-analysis bypass, multi-stage shellcode loaders, timing side-channel attacks, multi-thread anti-debug with decoy + signal handler MBA (ApoorvCTF 2026), INT3 patch + coredump brute-force oracle (Pwn2Win 2016), signal handler chain + LD_PRELOAD oracle (Nuit du Hack 2016), printf format string VM decompilation to Z3 (SECCON 2017), quadtree recursive image format parser (Google CTF Quals 2018)
• patterns-ctf.md - Competition-specific patterns (Part 1): hidden emulator opcodes, LD_PRELOAD key extraction, SPN static extraction, image XOR smoothness, byte-at-a-time cipher, mathematical convergence bitmap, Windows PE XOR bitmap OCR, two-stage RC4+VM loaders, GBA ROM meet-in-the-middle, Sprague-Grundy game theory, kernel module maze solving, multi-threaded VM channels, backdoored shared library detection via string diffing, custom binfmt kernel module with RC4 flat binaries, hash-resolved imports / no-import ransomware, ELF section header corruption for anti-analysis
• patterns-ctf-2.md - Competition-specific patterns (Part 2): multi-layer self-decrypting brute-force, embedded ZIP+XOR license, stack string deobfuscation, prefix hash brute-force, CVP/LLL lattice for integer validation, decision tree function obfuscation, GF(2^8) Gaussian elimination, ROP chain obfuscation analysis (ROPfuscation)
• patterns-ctf-3.md - Competition-specific patterns (Part 3): Z3 single-line Python circuit, sliding window popcount, keyboard LED Morse code via ioctl, C++ destructor-hidden validation, syscall side-effect memory corruption, MFC dialog event handlers, VM sequential key-chain brute-force, Burrows-Wheeler transform inversion, OpenType font ligature exploitation, GLSL shader VM with self-modifying code, instruction counter as cryptographic state, batch crackme automation via objdump, fork+pipe+dead branch anti-analysis, TensorFlow DNN inversion via sigmoid layer inversion, BPF filter analysis via kernel JIT to x64 assembly
• languages.md - Language-specific: Python bytecode & opcode remapping, Python version-specific bytecode, Pyarmor static unpack, DOS stubs, Unity IL2CPP, HarmonyOS HAP/ABC, Brainfuck/esolangs (+ BF character-by-character static analysis, BF side-channel read count oracle, BF comparison idiom detection), UEFI, transpilation to C, code coverage side-channel, OPAL functional reversing, non-bijective substitution, FRACTRAN program inversion
• languages-platforms.md - Platform/framework-specific: Roblox place file analysis, Godot game asset extraction, Rust serde_json schema recovery, Android JNI RegisterNatives obfuscation, Android DEX runtime bytecode patching via /proc/self/maps, Android native .so loading bypass via new project, Frida Firebase Cloud Functions bypass, Verilog/hardware RE, prefix-by-prefix hash reversal, Ruby/Perl polyglot constraint satisfaction, Electron ASAR extraction + native binary analysis, Node.js npm runtime introspection
• languages-compiled.md - Go binary reversing (GoReSym, goroutines, memory layout, channel ops, embed.FS, Go binary UUID patching for C2 enumeration), Rust binary reversing (demangling, Option/Result, Vec, panic strings), Swift binary reversing (demangling, protocol witness tables), Kotlin/JVM (coroutine state machines), Haskell GHC CMM intermediate language for recursive structure analysis, C++ (vtable reconstruction, RTTI, STL patterns), Nim & Zig (identification markers, NimMain entry chain, length-prefixed strings/slices, Zig panic-string anchoring, optional/error-union value semantics)
• platforms.md - Platform-specific RE: macOS/iOS (Mach-O, code signing, Objective-C runtime, Swift, dyld, jailbreak bypass), embedded/IoT firmware (binwalk, UART/JTAG/SPI extraction, ARM/MIPS, RTOS), kernel drivers (Linux .ko, eBPF, Windows .sys), game engines (Unreal Engine, Unity, anti-cheat, Lua), automotive CAN bus
• platforms-hardware.md - Hardware and advanced architecture RE: HD44780 LCD controller GPIO reconstruction, RISC-V advanced (custom extensions, privileged modes, debugging), ARM64/AArch64 reversing and exploitation (calling convention, ROP gadgets, qemu-aarch64-static emulation)
• field-notes.md - Quick reference notes: binary types, anti-debugging bypass, specialized patterns, CTF case notes

---

When to Pivot

• If you already understand the binary and now need heap, ROP, or kernel exploitation, switch to /ctf-pwn.
• If the challenge is really about recovering deleted files, PCAP data, or disk artifacts, switch to /ctf-forensics.
• If the target is a web app and you are only reversing a small client-side helper script, switch to /ctf-web.
• If the binary implements a machine learning model and the challenge is about model attacks or adversarial inputs, switch to /ctf-ai-ml.
• If the reversed binary's core logic is a cryptographic algorithm or math problem, switch to /ctf-crypto.
• If the binary is a real malware sample with C2, packing, or evasion behavior, switch to /ctf-malware.
• If the challenge is a toy VM, encoding puzzle, or pyjail rather than a real binary, switch to /ctf-misc.

Problem-Solving Workflow

Full version with entry/exit criteria per level: methodology.md.

1. Start with strings extraction - many easy challenges have plaintext flags
2. Try ltrace/strace - dynamic analysis often reveals flags without reversing
3. Try Frida hooking - hook strcmp/memcmp to capture expected values without reversing
4. Try angr - symbolic execution solves many flag-checkers automatically
5. Try Qiling - emulate foreign-arch binaries or bypass heavy anti-debug without artifacts
6. Map control flow before modifying execution
7. Automate manual processes via scripting (r2pipe, Frida, angr, Python)
8. Validate assumptions by comparing decompiler outputs (dogbolt.org for side-by-side)
9. Climb, don't dig - if a level stalls, note what you know and escalate to the next technique instead of grinding one decompile for hours

Quick Wins (Try First!)

# Plaintext flag extraction
strings binary | grep -E "flag\{|CTF\{|pico"
strings binary | grep -iE "flag|secret|password"
rabin2 -z binary | grep -i "flag"

# Dynamic analysis - often captures flag directly
ltrace ./binary
strace -f -s 500 ./binary

# Hex dump search
xxd binary | grep -i flag

# Run with test inputs
./binary AAAA
echo "test" | ./binary

Initial Analysis

file binary           # Type, architecture
checksec --file=binary # Security features (for pwn)
chmod +x binary       # Make executable

Memory Dumping Strategy

Key insight: Let the program compute the answer, then dump it. Break at final comparison (b *main+OFFSET), enter any input of correct length, then x/s $rsi to dump computed flag.

Decoy Flag Detection

Pattern: Multiple fake targets before real check. Look for multiple comparison targets in sequence with different success messages. Set breakpoint at FINAL comparison, not earlier ones.

GDB PIE Debugging

PIE binaries randomize base address. Use relative breakpoints:

gdb ./binary
start                    # Forces PIE base resolution
b *main+0xca            # Relative to main
run

Comparison Direction (Critical!)

Two patterns: (1) transform(flag) == stored_target — reverse the transform. (2) transform(stored_target) == flag — flag IS the transformed data, just apply transform to stored target.

Common Encryption Patterns

• XOR with single byte - try all 256 values
• XOR with known plaintext (flag{, CTF{)
• RC4 with hardcoded key
• Custom permutation + XOR
• XOR with position index (^ i or ^ (i & 0xff)) layered with a repeating key

Quick Tool Reference

# Radare2
r2 -d ./binary     # Debug mode
aaa                # Analyze
afl                # List functions
pdf @ main         # Disassemble main

# Ghidra (headless)
analyzeHeadless project/ tmp -import binary -postScript script.py

# IDA
ida64 binary       # Open in IDA64

Deep-Dive Notes

Use field-notes.md after the first round of triage when you know what kind of target you have.

• Target formats: Python bytecode, WASM, Android, Flutter, .NET, UPX, Tauri
• Technique notes: anti-debug bypass, VM analysis, x86-64 gotchas, iterative solvers, Unicorn, timing side channels
• Platform notes: Godot, Roblox, macOS/iOS, embedded firmware, kernel drivers, game engines, Swift, Kotlin, Go, Rust, D
• Case notes: modern CTF-specific reversing patterns and older classic challenge patterns