Use async io where possible to improve runtime performance

[netomi]

This PR fixes #162 .

Things that this PR will modify:

• use aiohttp whereever appropriate to improve performance
• cleanups of existing code where appropriate (e.g. get_pypi_data_from_purl which returned duplicate data for each package)

[TG1999]

@netomi thanks! please check the errors in the CI and also do not forget to regen the tests just in case dependencies in tests have been updated https://github.com/nexB/python-inspector#testing

[netomi]

There are a couple of things that I still need to fix, but the failing CI runs are unrelated to this PR afaict. They also exist for other PRs and look like that the setup is broken.

[TG1999]

@netomi please rebase your PR with latest main, tests are fixed in this PR #165

[netomi]

ok so there are some unit test failure that I need to address. Most of them are from the fact that python-inspector returned duplicate package entries in its result up to now, which I fixed, so the expected test results need to be adjusted to take this change into account.

[netomi]

I could fix most tests (weird thing is that 4 tests fail locally when run in pycharm, but work on the command line so that might be a pycharm problem), there are some weird test failures on macos and windows only. Do you have any idea about that?

[netomi]

still need to take a look at etc/scripts if the there also need to be adjustments after the changes.

[pombredanne]

@netomi BTW, do not bother with the etc/scripts ... these come from the shared skeleton repo at https://github.com/nexB/skeleton and should not be of concern for you.

[sschuberth]

Can this be moved forward, please? 😬

[TG1999]

@sschuberth tests are falling, tests needs to be regen

[sschuberth]

@sschuberth tests are falling, tests needs to be regen

Would you have time to do this @netomi?

[netomi]

hmm I can take a look, but I am quite discouraged from contributing to a repo that has to constantly rebuilding its tests and they are failing most of the time when creating a PR. You still need to inspect if its related to your changes or the test data is just out of date.

[pombredanne]

hmm I can take a look, but I am quite discouraged from contributing to a repo that has to constantly rebuilding its tests and they are failing most of the time when creating a PR. You still need to inspect if its related to your changes or the test data is just out of date.

@netomi I agree this is not great. What do you think we could do as an alternative?

[netomi]

please dont actually call live servers during tests but rather mock their responses with data you might have retrieved from the server at one point in time. I understand that is a bit harder to do, but ultimately results in much more stable and reliable tests imho.

PyGitHub has a nice system for their tests where they replay data:

https://github.com/PyGithub/PyGithub

[sschuberth]

At least one of the errors seems to be

ImportError: cannot import name 'cached_property' from 'functools' (/opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/functools.py)

[netomi]

no idea where this error is coming from. This decorator is not used anywhere in the code. Probably from a dependency? This error btw also occurs for the default branch, so its not related to this PR afaict.

[sschuberth]

This error btw also occurs for the default branch, so its not related to this PR afaict.

@TG1999 or @chinyeungli, can you assist here by fixing the test baseline in main to move things forward?