From 57b47af1edc71301e5456c3323b97bb066e09191 Mon Sep 17 00:00:00 2001 From: Marius Poke Date: Thu, 22 Dec 2022 17:54:45 +0100 Subject: [PATCH] Move info about testing from README to testing.md (#616) * Update README.md * move testing info to testing.md --- README.md | 104 +--------------------------------------------- docs/testing.md | 107 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 108 insertions(+), 103 deletions(-) create mode 100644 docs/testing.md diff --git a/README.md b/README.md index b7623b137e..039216e805 100644 --- a/README.md +++ b/README.md @@ -45,109 +45,7 @@ Inspect the [Makefile](./Makefile) if curious. ## Testing -### Unit Tests - -Unit tests are useful for simple standalone functionality, and CRUD operations. Unit tests should use golang's standard testing package, and be defined in files formatted as ```_test.go``` in the same directory as the file being tested, following standard conventions. - -[Mocked external keepers](./testutil/keeper/mocks.go) (implemented with [gomock](https://github.com/golang/mock)) are available for testing code that briefly interacts with external modules, but still only a single function/method relevant to ccv, and a single chain. Ie. do not use mocked external keepers to test the integration of the ccv module with external modules, or integration between consumer and provider. - -### End to End (e2e) Tests - -[e2e-tests](./tests/e2e/) utilize the [IBC Testing Package](https://github.com/cosmos/ibc-go/tree/main/testing), and test functionality that is wider in scope than a unit test, but still able to be validated in-memory. Ie. code where advancing blocks would be useful, simulated handshakes, simulated packet relays, etc. - -To run e2e tests against your own consumer/provider implementations, use [instance_test.go](./tests/e2e/instance_test.go) as an example. All you'll need to do is make sure your applications implement the necessary interfaces defined in [interfaces.go](./testutil/e2e/interfaces.go), pattern match [specific_setup.go](./testutil/ibc_testing/specific_setup.go), then pass in the appropriate types and parameters to the suite, as is done in `instance_test.go` for the dummy provider/consumer implementations. - -### Differential Tests (WIP) - -Similar to e2e tests, but they compare the system state to an expected state generated from a model implementation. - -### Integration Tests - -[Integration tests](./tests/integration/) run true consumer and provider chain binaries within a docker container and are relevant to the highest level of functionality. Integration tests use queries/transactions invoked from CLI to drive and validate the code. - -### Running Tests -Tests can be run using `make`: - -```bash -# run unit, e2e, diff, and integration tests -make test - -# run unit and e2e tests - prefer this for local development -make test-short - -# run difference tests -make test-diff - -# run integration tests -make test-integration - -# equivalent to make test with caching disabled -make test-no-cache -``` - -Alternatively you can run tests using `go test`: -```bash -# run all unit, e2e, and diff tests using go -go test ./... -# run all unit, e2e, and diff tests with verbose output -go test -v ./.. -# run all unit, e2e, and diff tests with coverage stats -go test -coverpkg=./x/... -coverprofile=coverage.out ./... -# run a single unit test -go test -run path/to/package -# example: run a single unit test -go test -run TestSlashAcks ./x/ccv/provider/keeper -# run a single e2e test -go test -run / ./... -# example: run a single e2e test -go test -run TestProviderTestSuite/TestPacketRoundtrip ./... -# run all integration tests -go run ./tests/integration/... -# run all integration tests with a local cosmos sdk -go run ./tests/integration/... --local-sdk-path "/Users/bob/Documents/cosmos-sdk/" -# run golang native fuzz tests (https://go.dev/doc/tutorial/fuzz) -go test -fuzz= -# run verbose integration tests -go run ./tests/integration/... --local-sdk-path "/Users/bob/Documents/cosmos-sdk/" --verbose -``` - -### Linters and Static Analysis - -Several analyzers are used on the code including [CodeQL](https://codeql.github.com/), [SonarCloud](https://sonarcloud.io/), [golangci-lint](https://golangci-lint.run/) and [gosec](https://github.com/securego/gosec). Some of these are run on github when committing to PRs ect, but some tools are also applicable locally, and are built into golang. - -```bash -# gofmt to format and simplify code (https://pkg.go.dev/cmd/gofmt) -gofmt -w -s -e . -# go vet to search for suspicious code (https://pkg.go.dev/cmd/vet) -go vet ./... -``` - -Some useful tools are included in the repository using [pre-commit](https://pre-commit.com/hooks.html). pre-commit lets you run developer tools either on every git commit, or manually with `pre-commit run --all-files`. See the [config](./.pre-commit-config.yaml) for details. In this repo the hooks are not installed to git, as that can be cumbersome, but it is still possible to benefit from them. - -```bash -## Prerequisites - -# pre-commit -brew install pre-commit -# goimports (https://pkg.go.dev/golang.org/x/tools/cmd/goimports) -go install golang.org/x/tools/cmd/goimports@latest -# gocyclo (https://github.com/fzipp/gocyclo) -go install github.com/fzipp/gocyclo/cmd/gocyclo@latest -# go-critic https://github.com/go-critic/go-critic -go install github.com/go-critic/go-critic/cmd/gocritic@latest - -## Run the tools - -pre-commit run --all-files -``` - -### Debugging - -If using VSCode, see [vscode-go/wiki/debugging](https://github.com/golang/vscode-go/wiki/debugging) to debug unit tests or go binaries. - -### More - -More instructions will be added soon, in time for the testnet. +See [testing docs](./docs/testing.md). ## Learn more diff --git a/docs/testing.md b/docs/testing.md new file mode 100644 index 0000000000..558566b21e --- /dev/null +++ b/docs/testing.md @@ -0,0 +1,107 @@ +# Testing + +To increase confidence in the correctness of the Interchain Security code we consider various testing approaches. + +## Unit Tests + +Unit tests are useful for simple standalone functionality, and CRUD operations. Unit tests should use golang's standard testing package, and be defined in files formatted as ```_test.go``` in the same directory as the file being tested, following standard conventions. + +[Mocked external keepers](./testutil/keeper/mocks.go) (implemented with [gomock](https://github.com/golang/mock)) are available for testing code that briefly interacts with external modules, but still only a single function/method relevant to ccv, and a single chain. Ie. do not use mocked external keepers to test the integration of the ccv module with external modules, or integration between consumer and provider. + +## End to End (e2e) Tests + +[e2e-tests](./tests/e2e/) utilize the [IBC Testing Package](https://github.com/cosmos/ibc-go/tree/main/testing), and test functionality that is wider in scope than a unit test, but still able to be validated in-memory. Ie. code where advancing blocks would be useful, simulated handshakes, simulated packet relays, etc. + +To run e2e tests against your own consumer/provider implementations, use [instance_test.go](./tests/e2e/instance_test.go) as an example. All you'll need to do is make sure your applications implement the necessary interfaces defined in [interfaces.go](./testutil/e2e/interfaces.go), pattern match [specific_setup.go](./testutil/ibc_testing/specific_setup.go), then pass in the appropriate types and parameters to the suite, as is done in `instance_test.go` for the dummy provider/consumer implementations. + +## Differential Tests (WIP) + +Similar to e2e tests, but they compare the system state to an expected state generated from a model implementation. + +## Integration Tests + +[Integration tests](./tests/integration/) run true consumer and provider chain binaries within a docker container and are relevant to the highest level of functionality. Integration tests use queries/transactions invoked from CLI to drive and validate the code. + +## Running Tests +Tests can be run using `make`: + +```bash +# run unit, e2e, diff, and integration tests +make test + +# run unit and e2e tests - prefer this for local development +make test-short + +# run difference tests +make test-diff + +# run integration tests +make test-integration + +# equivalent to make test with caching disabled +make test-no-cache +``` + +Alternatively you can run tests using `go test`: +```bash +# run all unit, e2e, and diff tests using go +go test ./... +# run all unit, e2e, and diff tests with verbose output +go test -v ./.. +# run all unit, e2e, and diff tests with coverage stats +go test -coverpkg=./x/... -coverprofile=coverage.out ./... +# run a single unit test +go test -run path/to/package +# example: run a single unit test +go test -run TestSlashAcks ./x/ccv/provider/keeper +# run a single e2e test +go test -run / ./... +# example: run a single e2e test +go test -run TestProviderTestSuite/TestPacketRoundtrip ./... +# run all integration tests +go run ./tests/integration/... +# run all integration tests with a local cosmos sdk +go run ./tests/integration/... --local-sdk-path "/Users/bob/Documents/cosmos-sdk/" +# run golang native fuzz tests (https://go.dev/doc/tutorial/fuzz) +go test -fuzz= +# run verbose integration tests +go run ./tests/integration/... --local-sdk-path "/Users/bob/Documents/cosmos-sdk/" --verbose +``` + +## Linters and Static Analysis + +Several analyzers are used on the code including [CodeQL](https://codeql.github.com/), [SonarCloud](https://sonarcloud.io/), [golangci-lint](https://golangci-lint.run/) and [gosec](https://github.com/securego/gosec). Some of these are run on github when committing to PRs ect, but some tools are also applicable locally, and are built into golang. + +```bash +# gofmt to format and simplify code (https://pkg.go.dev/cmd/gofmt) +gofmt -w -s -e . +# go vet to search for suspicious code (https://pkg.go.dev/cmd/vet) +go vet ./... +``` + +Some useful tools are included in the repository using [pre-commit](https://pre-commit.com/hooks.html). pre-commit lets you run developer tools either on every git commit, or manually with `pre-commit run --all-files`. See the [config](./.pre-commit-config.yaml) for details. In this repo the hooks are not installed to git, as that can be cumbersome, but it is still possible to benefit from them. + +```bash +## Prerequisites + +# pre-commit +brew install pre-commit +# goimports (https://pkg.go.dev/golang.org/x/tools/cmd/goimports) +go install golang.org/x/tools/cmd/goimports@latest +# gocyclo (https://github.com/fzipp/gocyclo) +go install github.com/fzipp/gocyclo/cmd/gocyclo@latest +# go-critic https://github.com/go-critic/go-critic +go install github.com/go-critic/go-critic/cmd/gocritic@latest + +## Run the tools + +pre-commit run --all-files +``` + +## Debugging + +If using VSCode, see [vscode-go/wiki/debugging](https://github.com/golang/vscode-go/wiki/debugging) to debug unit tests or go binaries. + +## More + +More instructions will be added soon, in time for the testnet. \ No newline at end of file