diff --git a/EVALUATION.md b/EVALUATION.md index b505ae8..ffbbb15 100644 --- a/EVALUATION.md +++ b/EVALUATION.md @@ -7,7 +7,7 @@ We assume that you have already done the installation steps in the [README.md](. When you run the container, you can download repositories of benchmarks and set the configuration: ```bash -# Inside the image; +# Inside the dokcer container; cd ~ cd configuration python download_repo.py @@ -22,6 +22,7 @@ If you can see `~/typebugs`, `~/bugsinpy`, and `~/excepy` directories, the downl It is necessary to preprocess the [homeassistant-core](https://github.com/home-assistant/core) project before running all tools: ```bash +# Inside the dokcer container; cd ~ python run/change_core_async.py ``` @@ -33,6 +34,7 @@ Then, you are ready to run Pyinder and other tools! You can run all tools with all projects by following the instructions: ```bash +# Inside the dokcer container; cd ~ cd run python pyinder_run.py @@ -75,6 +77,7 @@ All tools generate other warnings than type errors, so you need to filter out th At first, you have to change result log file to json file of each tool: ```bash +# Inside the dokcer container; cd ~ cd run # Change the result log file to json file @@ -89,6 +92,7 @@ It makes json files named `result_.json` in the `~/result/` directory Then, you can filter out the type errors from the results: ```bash +# Inside the dokcer container; cd ~ cd run python filter_error.py @@ -99,6 +103,7 @@ You can see the filtered results in the `~/result/` directory (e.g., Before checking the results, you have to run cloc to check the per kloc results: ```bash +# Inside the dokcer container; cd ~ cd eval python cloc.py @@ -107,6 +112,7 @@ python cloc.py Then, you can see the results by following these steps: ```bash +# Inside the dokcer container; cd ~ cd eval python check_alarm.py # show the number of alarms by each tool diff --git a/Pyinder/README.md b/Pyinder/README.md new file mode 100644 index 0000000..2d5b42d --- /dev/null +++ b/Pyinder/README.md @@ -0,0 +1,22 @@ +### Structure + +```bash +Pyinder +├── client +│ ├── pyre.py # Command line interface for Pyinder +│ ├── ... +├── source +│ ├── analysis # Analysis module for Pyinder +│ │ ├── ... +│ ├── ... +│ ├── command +│ │ ├── mineCommand.ml # Start point of Pyinder +│ │ ├── ... +│ └── ... +├── ... +``` + +We provide a brief description of the core parts of Pyinder. +`pyre.py` is the command line interface for Pyinder. +`mineCommand.ml` is the start point of Pyinder to run the analysis. +`analysis` contains the analysis module for Pyinder. diff --git a/README.md b/README.md index c164b12..b7b9a13 100644 --- a/README.md +++ b/README.md @@ -59,7 +59,7 @@ We recommend setting the memory reservation to 32GB for the container to fully r You can build Pyinder by following the instructions: ```bash -# Inside the image; +# Inside the dokcer container; cd ~ cd Pyinder/source make @@ -80,7 +80,7 @@ You can skip instructions for other tools if you only want to run Pyinder. At first, clone the project and prepare the project for the analysis: ```bash -# Inside the image; +# Inside the dokcer container; cd ~ cd configuration python download_repo.py --project luigi @@ -90,6 +90,8 @@ python setting_config.py --project luigi Next, run the tools with the luigi project: ```bash +# Inside the dokcer container; + # It will take about 30 minutes # to run all tools with the luigi project. cd ~ @@ -120,6 +122,7 @@ The result of each tool is stored in the `~/result///r Next, post-process the results to collect the type errors and check the results: ```bash +# Inside the dokcer container; cd ~ cd run @@ -163,6 +166,7 @@ The command `python cloc.py` makes the `~/cloc` directory that contains the resu Finally, you can see the results by following these steps: ```bash +# Inside the dokcer container; cd ~ cd eval python check_alarm.py -p luigi # show the number of alarms by each tool diff --git a/configuration/README.md b/configuration/README.md new file mode 100644 index 0000000..e82fad9 --- /dev/null +++ b/configuration/README.md @@ -0,0 +1,32 @@ +### Structure + +```bash +configuration +├── bugsinpy_repo.json # GitHub repositories and their commit id for bugsinpy +├── config # Configuration files for all benchmark projects +│ ├── bugsinpy # Configuration files for bugsinpy +│ │ ├── ansible-1 +│ │ │ ├── .pyre_configuration # Configuration file for Pyinder and pyre +│ │ │ ├── mypy.ini # Configuration file for mypy +│ │ │ ├── pyrightconfig.json # Configuration file for pyright +│ │ │ └── pytype.cfg # Configuration file for pytype +│ │ ├── ansible-2 +│ │ │ ├── ... +│ │ ├── ... +| ├── excepy # Configuration files for excepy +│ │ ├── ... +│ ├── pyinder # Configuration files for Pyinder +│ │ ├── ... +├── download_repo.py # The script to download repositories +├── excepy_repo.json # GitHub repositories and their commit id for excepy +├── setting_config.py # The script to set the configuration for evaluation +└── typebugs_repo.json # GitHub repositories and their commit id for typebugs +``` + +### Options + +`download_repo.py` and `setting_config.py` have the option **`--project`**. +The option **`--project`** denotes the project name to analyze. +| Option | Description | +|:------:|:------------| +| `--project` | Set the project name to analyze. | diff --git a/eval/README.md b/eval/README.md new file mode 100644 index 0000000..ca5d052 --- /dev/null +++ b/eval/README.md @@ -0,0 +1,18 @@ +### Structure + +```bash +eval +├── check_alarm.py # show the number of alarms by each tool +├── check_correct.py # show the number of detecting type errors by each tool +├── check_time.py # show the time taken by each tool +├── cloc.py # run cloc to check the per kloc results +└── draw_venn.py # draw the venn diagram of the results +``` + +### Options + +`check_alarm.py`, `check_correct.py`, and `check_time.py` have the option **`-p`** or **`--project`**. +The option **`-p`** or **`--project`** denotes the project name to analyze. +| Option | Description | +|:------:|:------------| +| `-p` or `--project` | Set the project name to analyze. | diff --git a/run/README.md b/run/README.md new file mode 100644 index 0000000..08b76eb --- /dev/null +++ b/run/README.md @@ -0,0 +1,29 @@ +### Structure + +```bash +run +├── change_core_async.py # change the async keyword for old version of core project +├── filter_error.py # extract only the type errors from the results +├── mypy_change_json.py # change the result log file to json file for mypy +├── mypy_run.py # run mypy with the project +├── pyinder_change_json.py # change the result log file to json file for Pyinder +├── pyinder_run.py # run Pyinder with the project +├── pyre_change_json.py # change the result log file to json file for pyre +├── pyre_run.py # run pyre with the project +├── pyright_run.py # run pyright with the project +├── pytype_change_json.py # change the result log file to json file for pytype +├── pytype_run.py # run pytype with the project +├── run.sh # script to run Pyinder (please use `pyinder_run.py` instead) +├── run_mypy.sh # script to run mypy (please use `mypy_run.py` instead) +├── run_pyre.sh # script to run pyre (please use `pyre_run.py` instead) +├── run_pyright.sh # script to run pyright (please use `pyright_run.py` instead) +└── run_pytype.sh # script to run pytype (please use `pytype_run.py` instead) +``` + +### Options + +`mypy_run.py`, `pyinder_run.py`, `pyre_run.py`, and `pytype_run.py` have the option **`-p`** or **`--project`**. +The option **`-p`** or **`--project`** denotes the project name to analyze. +| Option | Description | +|:------:|:------------| +| `-p` or `--project` | Set the project name to analyze. |