Issue with extracting whitebox tools

[joeshuttleworth]

I am trying to run the extended demo and running into and issue with the extraction of whitebox.

I have created a new env with python 3.10 and used pip install swmmanywhere [doc]

It is giving me this error message repeatedly:

Downloading WhiteboxTools from https://www.whiteboxgeo.com/WBT_Darwin/WhiteboxTools_darwin_amd64.zip
Failed to extract /var/folders/f4/6cfrn3rs0v137s8kxpcr4k340000gn/T/tmpfe58c9sp/my_first_swmm/bbox_1/download/whiteboxtools_binaries.zip. Redownloading...
Downloading WhiteboxTools from https://www.whiteboxgeo.com/WBT_Darwin/WhiteboxTools_darwin_amd64.zip
Failed to extract /var/folders/f4/6cfrn3rs0v137s8kxpcr4k340000gn/T/tmpfe58c9sp/my_first_swmm/bbox_1/download/whiteboxtools_binaries.zip. Redownloading...
Downloading WhiteboxTools from https://www.whiteboxgeo.com/WBT_Darwin/WhiteboxTools_darwin_amd64.zip
Failed to extract /var/folders/f4/6cfrn3rs0v137s8kxpcr4k340000gn/T/tmpfe58c9sp/my_first_swmm/bbox_1/download/whiteboxtools_binaries.zip.

Which carries on repeatedly. Any ideas on how to solve? Probably a very basic question

[barneydobson]

Hi Joe, thanks for posting here - I reran the tests here to ensure that it's not a server issue (doesn't seem to be).

Probably worth running it from the command line to verify it's not to do with your interactive python:

conda create -n sa python=3.10
conda activate sa
conda install swmmanywhere
pip install swmmanywhere[doc]
python /path/to/extended_demo.py

The extended demo runs everything in a temporary directory - this shouldn't be an issue but it can make debugging hard - try removing temp_dir = .. and replacing base_dir = ... with `base_dir = Path(r"/path/to/somewhere/on/your/machine").

[barneydobson]

Should also work with a normal python venv if you don't have conda: python -m venv sa (etc)

[cheginit]

Also, make sure you're using pywbt>=0.2.7.

[joeshuttleworth]

I'm following your new instructions Barney - but now getting this error, where it's saying the bounding box is too large:

2024/12/05 13:42:37 | Model number: 1
2024/12/05 13:42:37 | Loading and setting parameters.
2024/12/05 13:42:37 | Running downloads.
2024/12/05 13:42:37 | downloading elevation to /Users/josephshuttleworth/Documents/my_swmm_project/my_first_swmm4/bbox_1/download/elevation.tif
2024/12/05 13:42:42 | downloading buildings to /Users/josephshuttleworth/Documents/my_swmm_project/my_first_swmm4/bbox_1/download/building.geoparquet
2024/12/05 13:43:23 | downloading network to /Users/josephshuttleworth/Documents/my_swmm_project/my_first_swmm4/bbox_1/download/street.parquet
/Users/josephshuttleworth/anaconda3/envs/sb/lib/python3.10/site-packages/osmnx/_overpass.py:265: UserWarning: This area is 7,607 times your configured Overpass max query area size. It will automatically be divided up into multiple sub-queries accordingly. This may take a long time.

I'm running the code in the example still. Any ideas?

[barneydobson]

That's just a warning - it shouldn't be causing an error and eventually running?

Are the tests running, see instructions from #351 (below)?

conda create -n sa python=3.10
conda activate sa
git clone https://github.com/ImperialCollegeLondon/SWMManywhere.git
cd SWMManywhere
pip install -e .[dev,doc]
pytest

[joeshuttleworth]

I've followed the steps above, I'm encountering a FileNotFoundError in pytest while running the extended_demo.py script from the pywbt package. The error occurs when pywbt attempts to set execute permissions on the whitebox_tools executable, but the file is not found at the expected path. This prevents the script from executing successfully.

Error Message:

_______________ ERROR collecting docs/notebooks/extended_demo.py _______________
/opt/anaconda3/envs/sh/lib/python3.10/site-packages/pywbt/pywbt.py:74: in _extract_wbt
exec_path.chmod(exec_path.stat().st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)
/opt/anaconda3/envs/sh/lib/python3.10/pathlib.py:1097: in stat
return self._accessor.stat(self, follow_symlinks=follow_symlinks)
E FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/f4/6cfrn3rs0v137s8kxpcr4k340000gn/T/tmpf83a6nbp/my_first_swmm/bbox_1/download/tmp3pvh6tnb/WBT/whitebox_tools'

Expected Behavior:

The script should successfully locate the whitebox_tools executable, set the necessary execute permissions, and proceed with its operations without encountering errors.

Actual Behavior:

The script fails with a FileNotFoundError, indicating that the whitebox_tools executable does not exist at the specified temporary directory path.

Environment:

Operating System: macOS (Darwin)
Python Version: 3.10.15

[barneydobson]

Thanks for posting that... hmm I am not sure the extended demo actually runs in our tests on macos - I will set up a test to see

[barneydobson]

@joeshuttleworth it didn't reproduce in a generic mac environment, from your terminal could you call sw_vers to see the specific macos verison? I guess it isn't to do with that but just to be sure.

[barneydobson]

I think if we can't reproduce it we can at least add a workaround that you can provide the location of the WBT.exe, so you can download it in a stable way and manually provide it via the config file. This is also a useful feature independent of this issue

[joeshuttleworth]

extended_demo.py now works for me. The issue was the whitebox tools zip file not being able to be un-zipped. So I downloaded it locally, passed it via config and then updated preprocessing and main swmmanywhere .py files.

[barneydobson]

Thanks @joeshuttleworth - sounds like you made a feature! - do you want to make a branch to push your changes to and we can implement them via a pull request?