This is a project to make a fuzzing platform for the Hack assembly language from the Nand2Tetris (http://nand2tetris.org/) course. A fuzzer generates vast quantities of random (valid or invalid) code in order to test program robustness. It is especially good at detecting inputs that crash a program. However, as we are provided a reference Hack assembler we can also use fuzzing to test program correctness.
- Largely automated fuzzing platform
- Excellent coverage of Hack assembly structures (90%+ in one pass)
- Uses predefined and randomly generated labels
- Error logging
- Multiprocessor support
- Can additionally check for assembler error handling
Download the Nand2Tetris Software Suite to grab the reference assembler.
Python 3.6: (http://python.org)
Nand2Tetris Software Suite: (http://nand2tetris.org/software.php)
(Ideally, you will want to install this into a virtual Python environment with venv)
Download the Nand2Tetris software suite (this guide assumes it is in the ~/Downloads directory).
Clone the repo.
git clone https://github.com/achermannagement/assemfuzz.git
cd assemfuzz
Download the prerequisite python packages.
pip install -rrequirements.txt
Use the setup script.
python setup.py install
Run
python setup.py prepare --path ~/Downloads/nand2tetris.zip
to in order to make the testing directories.
Move your assembler into the mine/ directory.
Update RUN_STRING in myprogram.py to run your assembler
You should also update the my_cond function in myprogram.py so it extracts the line number from your error messages OR make it None if your program does not return a line number in its error messages.
Run the program
python assemfuzz.py
If your assembler produces the same output as the reference assembler for the fuzzed file, you will get a message saying tests passed.
This only runs a single assembly file against your assembler though so you might want to run more to have more confidence in your assembler, run
python assemfuzz.py -n 1000
to have a thousand passes done.
If your assembler output does not match the reference compiler, the fuzzing will halt and a note is made in the error log. The offending fuzzing file should remain in the mine directory so you can investigate the issue manually.
All configuration for assemfuzz is through arguments.
You can run the program with the --help flag in order to see what options are available.
If you want to run most tests, then you will need to provide an assembler, put it into your mine/ directory. If you lack one of your own, you can use the reference assembler from the Nand2Tetris software pack by updating myprogram.py like so:
PATH_TO_ASSEMBLER = os.path.join("mine", "HackAssembler.jar")
RUN_STRING = "java -jar {} {}"
becomes
PATH_TO_ASSEMBLER = os.path.join("mine", "Assembler.sh")
RUN_STRING = "./{} {}"
and
return int(err.split()[4][:-1])
becomes
from assemfuzz.common import their_cond
return their_cond(err)
PyTest and Tox are used to run tests in the setup file.
First run the prepare command to extract the software suite.
python setup.py prepare --path ~/Downloads/nand2tetris.zip --test 1
Run the tests with the path to software suite as an argument.
python -m tox
You can also run pytest directly.
python -m pytest
An argument can be provided in the setup script to enable coverage testing.
python -m pytest --cov
Pylint is used to measure code quality in this project. You can run it with
python -m pytest --pylint
See TODO.md
assemfuzz is not envisioned to become a large project but if you are keen to help feel free to fork this project or report an issue for bugs or desired features
- tox - Testing automation
- pytest - Testing framework
- setuptools - Packaging framework
- pytest-cov - pytest coverage testing
- pytest-pylint - pytest pylint plugin
- Joshua Achermann - Initial work - achermannagement
See also the list of contributors who participated in this project.
This project is licensed under the GPL License - see the LICENSE file for details