Skip to content

Latest commit

 

History

History
472 lines (351 loc) · 17.4 KB

CONTRIBUTING.md

File metadata and controls

472 lines (351 loc) · 17.4 KB

Table of Contents

Contributing to tt-metal

Thank you for your interest in this project.

If you are interested in making a contribution, then please familiarize yourself with our technical contribution standards as set forth in this guide.

Next, please request appropriate write permissions by opening an issue for GitHub permissions.

All contributions require:

  • an issue
    • Your issue should be filed under an appropriate project. Please file a feature support request or bug report under Issues to get help with finding an appropriate project to get a maintainer's attention.
  • a pull request (PR).
    • Your PR must be approved by appropriate reviewers.

Furthermore, all PRs must follow the contribution standards.

Machine setup

Hugepages setup

Hugepages is required to both run and develop on the Metalium project.

If you ever need to re-enable Hugepages, you can try the script we homemade for this:

sudo python3 infra/machine_setup/scripts/setup_hugepages.py enable

Then to check if Hugepages is enabled:

python3 infra/machine_setup/scripts/setup_hugepages.py check

Developing tt-metal

Currently, the most convenient way to develop is to do so on our cloud machines. They have prerequisite dependencies, model files, and other settings set up for users.

Please refer to the README for source installation and environment setup instructions, then please read the the Getting Started page.

Setting up Git

We use # as a special character to denote issue numbers in our commit messages. Please change your comment character in your Git to not conflict with this:

git config core.commentchar ">"

Setting logger level

In order to get debug level log messages, set the environment variable TT_METAL_LOGGER_LEVEL=Debug.

For example,

TT_METAL_LOGGER_LEVEL=Debug ./build/test/tt_metal/test_add_two_ints

Building and viewing the documentation locally

  1. First, ensure that you have built the project and activated the Python environment, along with any required PYTHONPATH variables.

  2. Build the HTML documentation.

cd docs
make clean
make html

You can optionally build and view the ttnn sweeps results with:

make ttnn_sweeps/check_directory
make ttnn_sweeps

then turn on the server to view.

make server

You can customize the port by using the PORT=<port> environment variable. If you're using a customer-facing cloud machine, please disregard this point.

  1. Navigate to the docs page.

Navigate your web browser to http://<ip address>:<port>, where <ip address> is the IP address of the machine on which you launched the web server. For example: http://10.250.37.37:4242, for port 4242.

If you forwarded your port, navigate to http://localhost:8888.

  1. If you make changes, you may need to check spelling errors.

We use the spell-checker, Aspell, to ensure we don't sneak in some typos in our documentation. This is enforced by static-checks on github workflows as well.

To check if your updated docs pass this check you can run,

$ cd ${TT_METAL_HOME} && ./docs/spellcheck.sh

If there are errors in this check you will see an exit code non-zero.

To update the documentation for spelling errors or any out-of-dictionary words you can run,

$ cd ${TT_METAL_HOME} && ./docs/spellcheck.sh update

Commit your changes and the personal dictionary, at docs/aspell-dictionary.pws, that is changed.

Cleaning the dev environment with make nuke

Normally, make clean only clears out build artifacts. It does not delete the built Python dev environment stored at build/python_env/.

To delete absolutely everything including the Python environment, use make nuke.

Tests in tt-metal

Ensure you're in a developer Python environment with necessary environment variables set as documentating in the developing section.

This includes the environment variables, Python dev environment etc.

All developers are responsible for ensuring that post-commit regressions pass upon any submission to the project. We will cover how to run these regressions both locally and on CI. Failure to ensure these tests pass will constitute a major regression and will likely mean reverting your commits.

Running post-commit regressions

You must run post-commit regressions before you commit something.

These regressions will also run after every pushed commit to the GitHub repo.

make build
make tests
./tests/scripts/run_tests.sh --tt-arch $ARCH_NAME --pipeline-type post_commit

If changes affect tensor or tt_dnn libraries, run this suite of pytests which tests tensor APIs and tt_dnn ops. These are also tested in post commit.

pytest tests/python_api_testing/unit_testing/ -vvv
pytest tests/python_api_testing/sweep_tests/pytests/ -vvv

If you would like to run the post-commit tests on GitHub Actions, please refer to using CI for development.

Adding post-commit tests

Make sure to add post-commit tests in the at the lowest two levels of the tests directory to make sure tests are executed on the workflows.

New shell scripts added above the lowest two levels may not be executed on the post-commit workflows!

Running model performance tests

After building the repo and activating the dev environment with the appropriate environment variables, you have two options for running performance regressions on model tests.

If you are using a machine with virtual machine specs, please use

./tests/scripts/run_tests.sh --tt-arch $ARCH_NAME --pipeline-type models_performance_virtual_machine

If you are using a machine with bare metal machine specs, please use

./tests/scripts/run_tests.sh --tt-arch $ARCH_NAME --pipeline-type models_performance_bare_metal

Running C++ Integration Tests (Legacy)

We have a legacy suite of C++ integration tests that are built like standalone executables. This section goes over how to generally run such tests if there's a specific one you'd like to run.

  1. Build the API integration tests using the make command,
make tests
  1. Run the test binaries from the path ${TT_METAL_HOME}/build/test/tt_metal

Running Googletest (gtest) C++ tests

The new fangled way we run our tests is with Googletest. The way we generally structure our tests with this framework is to bundle it into a single executable.

You can use --gtest_filter_test to filter out the specific test you'd like. For example, to build and run the CommonFixture.DRAMLoopbackSingleCore on fast dispatch, you can

  1. Build the unit tests:
    make tests
    
  2. Run the test:
    ./build/test/tt_metal/unit_tests_fast_dispatch --gtest_filter="CommonFixture.DRAMLoopbackSingleCore"
    

On slow dispatch, to run another specific test, the equivalent would be:

  1. Build the unit tests as you would above.
  2. Run with the slow dispatch mode:
    export TT_METAL_SLOW_DISPATCH_MODE=1
    ./build/test/tt_metal/unit_tests/fast_dispatch --gtest_filter_test="BasicFixture.TestL1BuffersAllocatedTopDown"
    

We have split our tests into the two dispatch modes for less pollution of state between the two. We would like to eventually enable switching between the two modes easily.

Running Python integration tests

We use pytest to run our Python-based tests. This is the general procedure for running such tests.

  1. Run the specific test point with pytest tool, e.g.
    $ pytest tests/tt_eager/python_api_testing/sweep_tests/pytests/tt_dnn/test_composite.py
    
  2. If you have any issues with import paths for python libraries include the following environment variable,
    $ export PYTHONPATH=${PYTHONPATH}:${TT_METAL_HOME}
    

Debugging tips

  • To print within a kernel the following must be added:
    • In the C++ python binding API file: #include "tt_metal/impl/debug/dprint_server.hpp"
    • In the same file, before launching your kernel : tt_start_debug_print_server(<device>, {<pci slot>}, {{<physical core coordinates>}});
    • Note for core 0,0 it is 1,1
    • You can get the physical core given a logical core with the following call: device->worker_core_from_logical_core(<logical_core>);
    • In the kernel: #include "debug/dprint.h"
    • To print in the kernel : DPRINT << <variable to print> << ENDL();
  • To use GDB to debug the C++ python binding itself:
    • Build with debug symbols make build CONFIG=debug
    • Ensure the python file you wish to debug, is standalone and has a main function
    • Run gdb --args python <python file>
    • You can add breakpoints for future loaded libraries
  • To log the compile time arguments passed with -D during the kernel build phase:
    • Run with Watcher enabled. Please refer to the Watcher documentation
    • Files with the kernel configurations will be automatically generated. For example: built/0/kernels/kernel_args.csv
  • To examine the compile time arguments of a kernel:
    • Within your kernel, assign the arguments to constexpr like this: constexpr uint32_t in1_mcast_sender_noc_y = get_compile_time_arg_val(0);
    • Run dump-constexprs.py script on the generated ELF file. E.g. python tt_metal/tools/dump-consts.py built/0/kernels/command_queue_producer/1129845549852061924/brisc/brisc.elf --function kernel_main

Contribution standards

File structure and formats

  • Every source file must have the appropriate SPDX header at the top following the Linux conventions for C++ source files, RST files, ASM files, and scripts. For Python files, we are to use this convention:

    # SPDX-FileCopyrightText: © 2023 Tenstorrent Inc.
    
    # SPDX-License-Identifier: Apache-2.0
    

    For C++ header files, we will treat them as C++ source files and use this convention:

    // SPDX-FileCopyrightText: © 2023 Tenstorrent Inc.
    //
    // SPDX-License-Identifier: Apache-2.0
    

CI/CD Principles

  • Revert commits on main which fail post-commit tests immediately.
  • There shall be a periodic discussion among the technical leads of this project concerning:
    • Certain codeowners and project-specific members review current tests in post-commit.
    • Certain codeowners and project-specific members decide whether to remove/add any current tests in post-commit as project priorities change on an ongoing basis.
    • Certain codeowners and project-specific members decide if we need to change owners or add more as project priorities change on an ongoing basis.
    • Communication channels for these decisions and meetings shall be kept internal to Tenstorrent with the intent of having such discussions in the open later.
  • Non-post-commit pipelines will not necessarily mean we have to revert the breaking commit, however any broken pipelines will be considered a priority bug fix.
  • The responsibility of identifying, announcing status-tracking, and escalating broken non-post-commit pipelines will be the responsibility of codeowners whose tests are in the said non-post-commit pipeline.
    • In the case of the model performance test pipeline, there are codeowners for such tests. However, it is the collective responsibility of all developers to ensure that we do not regress this pipeline.

Using CI/CD for development

  • There are some automated checks upon opening a PR. These checks are part, but not all, of the post-commit test suite. They must pass, but are not enough to ensure your PR will not be reverted.

  • To run any CI pipeline on GitHub Actions, please navigate to the actions page.

    Next, you can navigate to any pipeline on the left side of the view. For example, you can run the entire post-commit CI suite by clicking on on the link to all post-commit workflows, clicking "Run workflow", selecting your branch, and pressing "Run workflow".

    Dropdown menu of all post-commit workflows and Run Workflow button

    You can see the status of your CI run by clicking on the specific run you dispatched.

    We have a sizeable number of workflows, so don't forget to press "Show more workflows...".

  • Unfortunately, we currently do not do automatic checks of all required workflows upon opening a PR. There are various reasons for this, such as limited machine resources. This means that developer and reviewer discretion is still the most important factor in ensuring PRs are merged successfully and without CI failure.

Documentation

  • Any API changes must be accompanied with appropriate documentation changes.

Git rules and guidelines

  • Any commit message must be accompanied with an appropriate GitHub issue number with a colon and following message. The message must start with an imperative verb and descripton of what was done. Preferably a reason is included. Ex.

    #41: Fix data format error in Gelu op.
    
  • The following is not allowed in commit messages:

    • Commit messages which state that a code review or comments are being addressed. You must explicitly state what you are doing in each commit even if it's just cosmetic.
  • If you are working on a branch and would like to skip the Git commit hooks, you may delete the git_hooks Makefile directive in /module.mk before your first build. However, you are responsible for making sure your final submission follows the contribution guidelines. Failure to do so constitutes a violation of these contribution guidelines.

  • Merge commits are not allowed in our main branch. We enforce a linear history.

Code reviews

  • A PR must be opened for any code change with the following criteria:
    • Be approved, by a maintaining team member and any codeowners whose modules are relevant for the PR.
    • Pass any required post-commit pipelines rebased on the latest main. These pipelines will generally, but not always, be defined in .github/workflows/all-post-commit-workflows.yaml.
    • Pass any acceptance criteria mandated in the original issue.
    • Pass any testing criteria mandated by codeowners whose modules are relevant for the PR.
  • Avoid opening/re-opening/push new commits to PRs before you're ready for review and start running pipelines. This is because we don't want to clog our pipelines with unnecessary runs that developers may know will fail anyways.

New feature and design specifications

  • New or changing features require the following accompanying documentation:
    • An architectural change plan approved by maintaining team members.
    • A design plan with associated GitHub project/large containing issue. with sub-issues for proper documentation of project slices.
    • An appropriate test plan with issues.

Release flows

  • Any release must be externally-available artifacts generated by a workflow on a protected branch.

Logging, assertions, and exceptions

  • Use Loguru for Python logging.
  • Use Tenstorrent logger for C++ logging.

Hardware troubleshooting

Resetting an accelerator board

If a Tenstorrent chip seems to hang and/or is producing unexpected behaviour, you may try a software reset of the board.

For Grayskull: tt-smi -tr all

For Wormhole: tt-smi -wr all wait

If the software reset does not work, unfortunately you will have to power cycle the board. This usually means rebooting the host of a board.