AsFuzzer is an assembler testing tool that detects potential bugs in assemblers. Firstly, AsFuzzer infers the grammar of assembly instructions that are accepted by analyzing the error messages generated by a given assembler. Then, AsFuzzer performs differential testing to identify bugs in these assemblers.
We tested AsFuzzer with four real-world assemblers: Clang (version 16.0.0), GNU
assembler (GAS, version 2.41), Intel's assembler (ICC, version 2021.8.0), and
Microsoft macro assembler (MASM, version 14.37.32824.0). To reproduce the
evaluation results in our paper 'AsFuzzer: Differential Testing of Assemblers
with Error-Driven Grammar Inference,' we placed the evaluated assemblers in the
./bin/ directory.
AsFuzzer currently works on Linux and we tested on Ubuntu 20.04. To run AsFuzzer, Python3 is required. Additionally, Wine v3.0 is needed to run MASM, is needed to run MASM, a Windows-based assembler, and libncurses5 and gcc are needed to run Clang and ICC.
$ sudo apt update
$ sudo apt install python3 wine64 libncurses5 gcc
You can download our project using the following command:
$ git clone https://github.com/SoftSec-KAIST/AsFuzzer.git
If you encounter git-lfs: command not found error, please install Git Large
File Storage, a git command line extension for handling
large files. This is necessary because one of the assemblers (Clang) exceeds
100 MB in size.
AsFuzzer requires the grammar rules of given assemblers. So you should run the
Inferrer module first. Currently, AsFuzzer supports arm, aarch64, riscv,
mips, x86, and x86-64 architectures. You can select an architecture using
the following command:
$ python3 inferrer.py [architecture]
For example, to infer the grammar rules of x86 assemblers, you can run:
$ python3 inferrer.py x86
The Inferrer module will execute the assemblers that support the specified architecture. For x86 and x86-64 architectures, it runs Clang, GCC, ICC, and MASM. For other architectures, it runs Clang and GCC.
The inferred grammars will be located in the ./asfuzzer_data/ folder.
asfuzzer_data
|-- clang
| `-- x86
| |-- ...
| `-- template
| |- asm
| `- db # DB folder that stores grammar rules
| |- aaa.db # DB file for Opcode #1
| |- aad.db # DB file for Opcode #2
| ...
|-- gas
|-- icc
`-- masm
After completing the inference process, you can move on to running the Fuzzer module. To do this, execute the Fuzzer module with the following command:
$ python3 fuzzer.py [architecture]
You can stop the fuzzer module by pressing Ctrl+C. Additionally, the fuzzer module includes a timeout option (--timeout) to specify the duration of the run. The following example sets a 1-hour timeout.
$ python3 fuzzer.py x86 --timeout 3600
By default, the Fuzzer module generates 20 instructions per iteration. To change this configuration, you can use the --N option as follows:
$ python3 fuzzer.py x86 --N 10
The fuzzing output will be located in the ./diff/ folder.
diff/x86
|-- diff # The result of consistency checking b/w input & output of assembler
| |-- clang # Target Assembler
| | |-- temp # Temp directory
| | `-- bindiff # Bugs that AsFuzzer Found
| | |-- *.s # Assembly Code
| | `-- *.txt # Disassembly Code
| |-- gas
| |-- icc
| `-- masm
`-- same # The result of differential testing b/w two assemblers
`-- clang_gas # Target assemblers
| |-- clang # Assembler #1
| | |-- temp
| | `-- bindiff
| `-- gas # Aseembler #2
| |-- temp
| `-- bindiff
|-- clang_icc
|-- clang_masm
|-- gas_icc
|-- gas_masm
`-- icc_masm
We provide triage.py script for merging error reports and categorizing them
according to opcode. You can run it using the following command:
$ python3 triage.py [architecture]
To triage the fuzzing result of the x86 assemblers, you can run the script as follows:
$ python3 triage.py x86
The triage.py script categorizes assembly files that trigger assembly bugs
based on their opcode. For errors related to consistency checking, it compiles
a single assembly code for each opcode. Then, it compares the disassembly
output with the assembly code and records the results in the diff.txt file.
For errors related to differential testing, it creates one assembly code per
opcode, compiles it using two assemblers, and compares the disassembly outputs,
logging the results in diff.txt.
The results will be saved in the ./triage/ folder.
triage/intel
|-- clang # The result of consistency checking b/w input & output of assembler
| |-- enqcmd # opcode #1
| | |-- diff.txt # diff file
| | ...
| |-- enqcmds # opcode #2
| ...
|-- gas
|-- icc
|-- masm
|-- clang_gas # The result of differential testing b/w two assemblers
| |-- clang # Assembler #1
| | |-- call # opcode #1
| | | |-- diff.txt # diff file
| | | ...
| | ...
| |-- gas # Assembler #2
| | |-- call # opcode #1
| | | |-- diff.txt # diff file
| | | ...
| | ...
...
You can find the file diff results in the diff.txt file located within each
opcode folder:
$ cat triage/x86/clang/enqcmd/diff.txt
enqcmd AX, [EAX+1] | 0: enqcmd ax,[bx+si+0x1]
enqcmd BX, ZMMWORD PTR [1] | 7: enqcmd bx,[di]
enqcmd BX, [1] | d: add DWORD PTR [eax],eax
enqcmd BX, [EAX] | f: add BYTE PTR [eax],al
...
$ cat triage/x86/clang_gas/clang/call/diff.txt
0: call ax 0: call ax
3: call 0x5 | 3: call DWORD PTR ds:0x1
...
You can use Docker image to try out AsFuzzer.
$ docker build --tag asfuzzer:demo .
We have provided our artifact to reproduce our evaluation.
We conducted our experiments on a desktop machine with 16 Intel i9-11900 cores, using Ubuntu 20.04 containers. Each running instance of the inferrer module was allocated one CPU core and 16GB of memory.
You can use a Docker image to run the inferrer module. The inference process for x86 and x86-64 will take approximately 5-6 hours each.
$ workdir=$(pwd)
$ docker run --rm --memory 16g --cpus 1 -v "$workdir/asfuzzer_data:/asfuzzer/asfuzzer_data" asfuzzer:demo sh -c "cd /asfuzzer; python3 inferrer.py x86 "
$ docker run --rm --memory 16g --cpus 1 -v "$workdir/asfuzzer_data:/asfuzzer/asfuzzer_data" asfuzzer:demo sh -c "cd /asfuzzer; python3 inferrer.py x86-64 "
$ docker run --rm --memory 16g --cpus 1 -v "$workdir/asfuzzer_data:/asfuzzer/asfuzzer_data" asfuzzer:demo sh -c "cd /asfuzzer; python3 inferrer.py arm "
$ docker run --rm --memory 16g --cpus 1 -v "$workdir/asfuzzer_data:/asfuzzer/asfuzzer_data" asfuzzer:demo sh -c "cd /asfuzzer; python3 inferrer.py aarch64 "
$ docker run --rm --memory 16g --cpus 1 -v "$workdir/asfuzzer_data:/asfuzzer/asfuzzer_data" asfuzzer:demo sh -c "cd /asfuzzer; python3 inferrer.py mips "
$ docker run --rm --memory 16g --cpus 1 -v "$workdir/asfuzzer_data:/asfuzzer/asfuzzer_data" asfuzzer:demo sh -c "cd /asfuzzer; python3 inferrer.py riscv "
The inferrer module will execute each assembler multiple times to infer its grammar rules. Once the inference is complete, the total time taken will be displayed.
The gen.py script is provided to generate test cases. Run it with the
following command:
$ python3 gen.py [architecture] [assembler] [num_of_test_cases] [save_file]
For example, to generate 1 million random assembly instructions, use this command:
$ python3 gen.py x86 clang 1000000 x86_clang.txt
$ python3 gen.py x86 gas 1000000 x86_gas.txt
$ python3 gen.py riscv clang 1000000 riscv_clang.txt
$ python3 gen.py riscv gas 1000000 riscv_gas.txt
(Optional) You can use a Docker image to run the gen.py script as follows:
$ workdir=$(pwd)
$ docker run --rm -v "$workdir/asfuzzer_data:/asfuzzer/asfuzzer_data" asfuzzer:demo sh -c "cd /asfuzzer; python3 gen.py x86 clang 1000000 x86_clang.txt
You can run the Fuzzer module to find bugs. To run the Fuzzer module for six hours, use the following command:
$ python3 fuzzer.py --timeout 21600 x86
$ python3 fuzzer.py --timeout 21600 x86-64
$ python3 fuzzer.py --timeout 21600 arm
$ python3 fuzzer.py --timeout 21600 aarch64
$ python3 fuzzer.py --timeout 21600 mips
$ python3 fuzzer.py --timeout 21600 riscv
(Optional) You can use the --N option to change the number of instances. However, make sure to clean up the diff folder before running the Fuzzer module again:
$ python3 fuzzer.py --timeout 21600 x86 --N 1
$ mv diff diff_N1
$ python3 fuzzer.py --timeout 21600 x86 --N 5
$ mv diff diff_N5
$ python3 fuzzer.py --timeout 21600 x86 --N 10
$ mv diff diff_N10
$ python3 fuzzer.py --timeout 21600 x86 --N 50
$ mv diff diff_N50
(Optional) If you want to use a Docker image, run it with the followig command:
$ workdir=$(pwd)
$ docker run --rm -v "$workdir/asfuzzer_data:/asfuzzer/asfuzzer_data" -v "$workdir/diff:/asfuzzer/diff" asfuzzer:demo sh -c "cd /asfuzzer; python3 fuzzer.py x86 --timeout 21600"
To create triage files, follow these steps:
$ python3 triage.py x86
$ python3 triage.py x86-64
$ python3 triage.py arm
$ python3 triage.py aarch64
$ python3 triage.py mips
$ python3 triage.py riscv
(Optional) If you want to use a Docker image, run it with the followig command:
$ workdir=$(pwd)
$ docker run --rm -v "$workdir/asfuzzer_data:/asfuzzer/asfuzzer_data" -v "$workdir/diff:/asfuzzer/diff" asfuzzer:demo sh -c "cd /asfuzzer; python3 triage.py x86 "
The triage results can be found in the ./triage/ folder.
triage
|-- aarch64
|-- arm
|-- mips
|-- riscv
|-- x86-64
`-- x86
| |-- clang
| | |-- enqcmd
| | | |-- diff.txt # The result of consistency check
| | | ...
| | ...
| |-- gas
| `-- clang_gas
| |-- clang
| | |-- enqcmd
| | | `-- diff.txt # The result of differential testing
| |-- gas
| | |-- enqcmd
| | | `-- diff.txt # The result of differential testing
We manually inspected error reports generated by AsFuzzer and grouped them into six major categories. (C1) There were cases where the assembler silently changed a register into another one. (C2) Assemblers often misinterpret a register/immediate/memory operand as a label. (C3) Assemblers often ignore pointer directives, thereby producing instructions that have wrong memory access sizes. (C4) Assemblers sometimes accept or produce an incorrect number of operands. (C5) There were several cases from Clang where it generates invalid binary code for valid instructions. (C6) There were several cases from Clang where it accepts a wrong instruction and silently produces nothing.
The detected bugs are organized in the ./found_bug/ folder. You can view
these bugs in this folder.
found_bugs/
|-- clang # Assembler #1
| |-- arm.s
| |-- mips.s
| |-- x86-64.s
| `-- x86.s
|-- gas # Assembler #2
| |-- aarch64.s
| |-- arm.s
| |-- mips.s
| |-- riscv.s
| |-- x86-64.s
| `-- x86.s
|-- icc # Assembler #3
| |-- x86-64.s
| `-- x86.s
`-- masm # Assembler #4
|-- x86-64.asm
`-- x86.asm
You can find the errors for each assembler in the ./found_bugs/[assembler]
folder. Each folder includes multiple assembly files containing assembly
instructions categorized by bug type. Additionally, the assembly files describe
the compilation and disassembly process.
$ cd found_bugs/clang
$ ls
arm.s mips.s x86-64.s x86.s
$ cat x86.s
###########################################################
# Total: 60 (C1:3, C2:24, C3:18, C4:0, C5:15, C6:0)
# Compile: ../../bin/clang -m32 -c x86.s -o x86.o
# disassembly: ../../bin/objdump -d -M intel --no-show-raw-insn x86.o
###########################################################
.intel_syntax noprefix
C1:
# C1. Using wrong register(s) (3)
enqcmd SP, ZMMWORD PTR [EAX+1] # enqcmd sp,[bx+si+0x1]
enqcmds SP, ZMMWORD PTR [EAX] # enqcmds sp,[bx+si]
movdir64b SP, ZMMWORD PTR [ECX] # movdir64b sp,[bx+di]
...
$ ../../bin/clang -m32 -c x86.s -o x86.o
$ ../../bin/objdump -d -M intel x86.o
00000000 <C1>:
0: enqcmd sp,[bx+si+0x1]
7: enqcmds sp,[bx+si]
d: movdir64b sp,[bx+di]
...
You can reproduce case #1, where four different assemblers produce different machine code, as described in Section 4.5.
$ cd found_bugs/case_study
$ /bin/bash build1.sh
$ objdump -d case1_clang.o -M intel | grep lar
0: 4d 0f 02 dc lar r11,r12w
$ objdump -d case1_gas.o -M intel | grep lar
0: 45 0f 02 dc lar r11d,r12w
$ objdump -d case1_icc.o -M intel | grep lar
0: 45 0f 02 dc lar r11d,r12w
$ objdump -d case1_masm.o -M intel | grep lar
0: 4d 0f 02 dc lar r11,r12w
You can also reproduce case #2, where Clang emits no machine code for the given instructions.
$ cat case2.s
dsb [R3,#1]
dmb [R8]
isb [R1,#1]
$ /bin/bash build2.sh
$ objdump -d case2.o
case2.o: file format elf32-littlearm
If you plan to use AsFuzzer in your own research, please consider citing our paper:
@INPROCEEDINGS{kim:issta:2024,
author = {Hyungseok Kim and Soomin Kim and Jungwoo Lee and Sang Kil Cha},
title = {AsFuzzer: Differential Testing of Assemblers with Error-Driven Grammar Inference},
booktitle = {Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis},
year = 2024
}