fetch missing SSL certificates

[milahu]

fix #4469

based on danilobellini/aia#3

[CLAassistant]

All committers have signed the CLA.

[pep8speaks]

Hello @milahu! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

• In the file src/pyload/core/network/http/aia_retry_wrap_download.py:

Line 121:89: E501 line too long (96 > 88 characters)

We recommend to use black to automatically correct these files.

Comment last updated at 2024-06-16 09:25:18 UTC

[milahu]

mmkay. works for me

tests

• tests/test_ssl_certs.py based on aia/test/test.py
• https://ddownload.com/ urls

missing root certs are rare, so this is not-yet implemented in the webui
when pyload finds a missing root cert, the cert is saved to
~/.pyload2/cache/untrusted-root-certs/
to trust that root cert, it must be moved to
~/.pyload2/settings/trusted-root-certs/
and pyload must be restarted

todo: remove debug prints

this error is probably not caused by my changes
the request succeeds on retry

  File "pyload/core/network/http/http_request.py", line 236, in set_request_context
    self.c.setopt(pycurl.URL, url)
pycurl.error: cannot invoke setopt() - perform() is currently running

[2024-06-05 18:51:15]  DEBUG               pyload  ADDON AntiCaptcha: LOAD URL https://api.anti-captcha.com/createTask | get={} | post={"clientKey": "xxx", "softId": 976, "task": xxx | ref=True | cookies=True | just_header=False | decode=True | multipart=False | redirect=True | req=None
Exception in thread Thread-46:
Traceback (most recent call last):
  File "/nix/store/k3701zl6gmx3la7y4dnflcvf3xfy88kh-python3-3.11.9/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/nix/store/qi6lvjl6i1hiwp9r5fbinnia0nwj3rap-pyload-0.5.0b3.dev82-unstable/lib/python3.11/site-packages/pyload/core/threads/addon_thread.py", line 61, in run
    self.f(*self.args, **self.kwargs)
  File "/nix/store/qi6lvjl6i1hiwp9r5fbinnia0nwj3rap-pyload-0.5.0b3.dev82-unstable/lib/python3.11/site-packages/pyload/plugins/addons/AntiCaptcha.py", line 96, in _process_captcha
    api_data = self.api_request(
               ^^^^^^^^^^^^^^^^^
  File "/nix/store/qi6lvjl6i1hiwp9r5fbinnia0nwj3rap-pyload-0.5.0b3.dev82-unstable/lib/python3.11/site-packages/pyload/plugins/addons/AntiCaptcha.py", line 65, in api_request
    json_data = self.load(self.API_URL + method, post=json.dumps(post))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qi6lvjl6i1hiwp9r5fbinnia0nwj3rap-pyload-0.5.0b3.dev82-unstable/lib/python3.11/site-packages/pyload/plugins/base/plugin.py", line 217, in load
    html = req.load(
           ^^^^^^^^^
  File "/nix/store/qi6lvjl6i1hiwp9r5fbinnia0nwj3rap-pyload-0.5.0b3.dev82-unstable/lib/python3.11/site-packages/pyload/core/network/browser.py", line 128, in load
    return self.http.load(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qi6lvjl6i1hiwp9r5fbinnia0nwj3rap-pyload-0.5.0b3.dev82-unstable/lib/python3.11/site-packages/pyload/core/network/http/aia_retry_wrap_download.py", line 27, in download_wrapper_aia_retry
    return download(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qi6lvjl6i1hiwp9r5fbinnia0nwj3rap-pyload-0.5.0b3.dev82-unstable/lib/python3.11/site-packages/pyload/core/network/http/http_request.py", line 307, in load
    self.set_request_context(url, get, post, referer, cookies, multipart, decode)
  File "/nix/store/qi6lvjl6i1hiwp9r5fbinnia0nwj3rap-pyload-0.5.0b3.dev82-unstable/lib/python3.11/site-packages/pyload/core/network/http/http_request.py", line 236, in set_request_context
    self.c.setopt(pycurl.URL, url)
pycurl.error: cannot invoke setopt() - perform() is currently running

[vuolter]

Hi @milahu thanks for the pr!
I see some commits not related to the ssl issue you targeted, could you pls check them all again? thx.

[milahu]

done. should i squash this into one commit?

there are still some debug prints
which should be removed or converted to log.debug calls ...
feel free to help ^^
currently im more worried about fixing #3899

[lukastribus]

I don't think pyload should do AIA chaising at all, not even Firefox does this.

That's just a lot of complexity to workaround broken servers that imho doesn't belong in pyload.

Also not a fan of bash scripts downloading python code from github within the pyload codebase:

wget -N https://github.com/milahu/python-aia/raw/use-cryptography-module/aia.py

If there are dependencies that need to be included, they are pythonic ways to do so. A bash script downloading a py file from a personal github account is not it.

[milahu]

the whole point of pyload is to work around "server bugs"
if all servers were perfect, we could just download files via http / ftp / torrent / rsync

aia.py would be added to pyload, because the aia package is outdated
see also danilobellini/aia#3

[lukastribus]

What you are trying to achieve is understood. But adding new dependencies needs to be balanced with maintainability, also see:

#4462 (comment)

[milahu]

can you please stop coming up with bullshit "reasons"?
my code works, there is nothing to "maintain"

it could be refactored to be more pretty
but ugly code is better than no code

[lukastribus]

Maintainers will decide, one way or the other.