Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Breaks with Chromium 115, Python 3.11, fanficfare 3.19.0 and the proxy from master #18

Open
mcepl opened this issue Aug 16, 2023 · 1 comment
Open

Breaks with Chromium 115, Python 3.11, fanficfare 3.19.0 and the proxy from master #18

mcepl opened this issue Aug 16, 2023 · 1 comment

Comments

@mcepl
Copy link

mcepl commented Aug 16, 2023 

sedmikraska~/r/fanfictionnet_ff_proxy (master)$ python3.11 chrome_content.py --verbose --write-log  --chrome-path /usr/bin/chromium-browser --log-filename /tmp/fanfiction_proxy_log.txt
2023-08-16 21:40:57.598 CEST INFO main fanfictionnet_ff_proxy version 0.5.8 by Nicolas SAPA <[email protected]>
2023-08-16 21:40:57.599 CEST INFO main This Alpha software is licensed under CECILL-2.1
2023-08-16 21:40:57.600 CEST INFO main Running on Linux-5.14.21-150500.55.19-default-x86_64-with-glibc2.31
2023-08-16 21:40:57.686 CEST DEBUG main ChromeVersionFinder returned 115 for /usr/bin/chromium-browser
2023-08-16 21:40:57.687 CEST DEBUG ProxiedBrowser(init) Forcing binary path to /usr/bin/chromium-browser
2023-08-16 21:40:57.689 CEST DEBUG undetected_chromedriver.patcher getting release number from /LATEST_RELEASE_114
2023-08-16 21:40:57.765 CEST DEBUG undetected_chromedriver.patcher downloading from https://chromedriver.storage.googleapis.com/114.0.5735.90/chromedriver_linux64.zip
2023-08-16 21:41:11.303 CEST DEBUG undetected_chromedriver.patcher unzipping /tmp/tmpm52c3eoz
2023-08-16 21:41:11.466 CEST INFO undetected_chromedriver.patcher patching driver executable /home/matej/.local/share/undetected_chromedriver/undetected_chromedriver
2023-08-16 21:41:11.483 CEST DEBUG undetected_chromedriver.patcher found block:
b'{window.cdc_adoQpoasnfa76pfcZLmcfl_Array = window.Array;window.cdc_adoQpoasnfa76pfcZLmcfl_Object = window.Object;window.cdc_adoQpoasnfa76pfcZLmcfl_Promise = window.Promise;window.cdc_adoQpoasnfa76pfcZLmcfl_Proxy = window.Proxy;window.cdc_adoQpoasnfa76pfcZLmcfl_Symbol = window.Symbol;}'
replacing with:
b'{console.log("undetected chromedriver 1337!")}                                                                                                                                                                                                                                               '
2023-08-16 21:41:11.486 CEST DEBUG undetected_chromedriver.patcher patching took us 0.02 seconds
2023-08-16 21:41:11.494 CEST DEBUG selenium.webdriver.common.service Started executable: `/home/matej/.local/share/undetected_chromedriver/undetected_chromedriver` in a child process with pid: 22990
2023-08-16 21:41:11.996 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session {"capabilities": {"firstMatch": [{}], "alwaysMatch": {"browserName": "chrome", "pageLoadStrategy": "normal", "goog:chromeOptions": {"extensions": [], "binary": "/usr/bin/chromium-browser", "args": ["--no-first-run --no-service-autorun --password-store=basic", "--remote-debugging-host=127.0.0.1", "--remote-debugging-port=37477", "--user-data-dir=/tmp/tmp9dndvg1o", "--lang=cs-CZ", "--no-default-browser-check", "--no-first-run", "--no-sandbox", "--test-type", "--window-size=1920,1080", "--start-maximized", "--no-sandbox", "--log-level=0"], "debuggerAddress": "127.0.0.1:37477"}}}}
2023-08-16 21:41:11.996 CEST DEBUG urllib3.connectionpool Starting new HTTP connection (1): localhost:35903
2023-08-16 21:41:14.969 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session HTTP/1.1" 200 0
2023-08-16 21:41:14.969 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":{"capabilities":{"acceptInsecureCerts":false,"browserName":"chrome","browserVersion":"115.0.5790.170","chrome":{"chromedriverVersion":"114.0.5735.90 (386bc09e8f4f2e025eddae123f36f6263096ae49-refs/branch-heads/5735@{#1052})"},"goog:chromeOptions":{"debuggerAddress":"127.0.0.1:37477"},"pageLoadStrategy":"normal","platformName":"","proxy":{},"setWindowRect":false,"strictFileInteractability":false,"timeouts":{"implicit":0,"pageLoad":300000,"script":30000},"unhandledPromptBehavior":"dismiss and notify","webauthn:extension:credBlob":true,"webauthn:extension:largeBlob":true,"webauthn:extension:minPinLength":true,"webauthn:extension:prf":true,"webauthn:virtualAuthenticators":true},"sessionId":"8c8ed1f3a720b4b0cf6ee41895d58afb"}} | headers=HTTPHeaderDict({'Content-Length': '739', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:14.969 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:14.969 CEST INFO ProxiedBrowser(init) chromedriver version 114.0.5735.90 (386bc09e8f4f2e025eddae123f36f6263096ae49-refs/branch-heads/5735@{#1052}) running as pid 22990 driving Chrome version 115.0.5790.170 running as pid 22989 with undetected_chromedriver 3.5.0
2023-08-16 21:41:14.970 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session/8c8ed1f3a720b4b0cf6ee41895d58afb/url {"url": "chrome://version"}
2023-08-16 21:41:15.247 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session/8c8ed1f3a720b4b0cf6ee41895d58afb/url HTTP/1.1" 200 0
2023-08-16 21:41:15.247 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:15.248 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:15.248 CEST INFO ProxiedBrowser(init) Trying to load existing cookie...
2023-08-16 21:41:15.248 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session/8c8ed1f3a720b4b0cf6ee41895d58afb/url {"url": "http://www.fanfiction.net"}
2023-08-16 21:41:20.992 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session/8c8ed1f3a720b4b0cf6ee41895d58afb/url HTTP/1.1" 200 0
2023-08-16 21:41:20.993 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:20.993 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:20.993 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session/8c8ed1f3a720b4b0cf6ee41895d58afb/cookie {"cookie": {"domain": ".fanfiction.net", "expiry": 1721776020, "httpOnly": false, "name": "__gpi", "path": "/", "sameSite": "Lax", "secure": false, "value": "UID=00000c7c8a1569c3:T=1688080020:RT=1689611326:S=ALNI_MYE67Qi8lv-_tS0dHsGY39tRETfIQ"}}
2023-08-16 21:41:20.999 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session/8c8ed1f3a720b4b0cf6ee41895d58afb/cookie HTTP/1.1" 200 0
2023-08-16 21:41:20.999 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:21.000 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:21.000 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session/8c8ed1f3a720b4b0cf6ee41895d58afb/url {"url": "http://www.fanfiction.net"}
2023-08-16 21:41:34.339 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session/8c8ed1f3a720b4b0cf6ee41895d58afb/url HTTP/1.1" 200 0
2023-08-16 21:41:34.339 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:34.340 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:34.340 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session/8c8ed1f3a720b4b0cf6ee41895d58afb/cookie {"cookie": {"domain": ".fanfiction.net", "expiry": 1721776020, "httpOnly": false, "name": "__gads", "path": "/", "sameSite": "Lax", "secure": false, "value": "ID=d275f01438cd380f-22da69f236e2003a:T=1688080020:RT=1689611326:S=ALNI_MZ-nzyOgy6_TL0DIiivWxJAMSoe0w"}}
2023-08-16 21:41:34.360 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session/8c8ed1f3a720b4b0cf6ee41895d58afb/cookie HTTP/1.1" 200 0
2023-08-16 21:41:34.361 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:34.361 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:34.362 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session/8c8ed1f3a720b4b0cf6ee41895d58afb/url {"url": "http://www.fanfiction.net"}
2023-08-16 21:41:38.367 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session/8c8ed1f3a720b4b0cf6ee41895d58afb/url HTTP/1.1" 200 0
2023-08-16 21:41:38.367 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:38.367 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:38.368 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session/8c8ed1f3a720b4b0cf6ee41895d58afb/cookie {"cookie": {"domain": ".fanfiction.net", "expiry": 1689613032, "httpOnly": true, "name": "__cf_bm", "path": "/", "sameSite": "Strict", "secure": true, "value": "fwarHa7Giu_dvfcvIQX9J2oyboHFQl_wC0zoryJ99ho-1689611232-0-AePdLYYDOJP2r5BVFlGcuqBx5LLK/d1dUKMTdeRVuliVLeXuupE503A6bfZ/BPJtMas53mlrjYzTousy1REl3fxQHCbbXiagJ2WLju6utxWX"}}
2023-08-16 21:41:38.374 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session/8c8ed1f3a720b4b0cf6ee41895d58afb/cookie HTTP/1.1" 200 0
2023-08-16 21:41:38.374 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:38.374 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:38.374 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session/8c8ed1f3a720b4b0cf6ee41895d58afb/url {"url": "http://www.fanfiction.net"}
2023-08-16 21:41:44.329 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session/8c8ed1f3a720b4b0cf6ee41895d58afb/url HTTP/1.1" 200 0
2023-08-16 21:41:44.330 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:44.330 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:44.330 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session/8c8ed1f3a720b4b0cf6ee41895d58afb/cookie {"cookie": {"domain": ".fanfiction.net", "expiry": 1721146351, "httpOnly": true, "name": "cf_clearance", "path": "/", "sameSite": "Strict", "secure": true, "value": "Vnd68anJN93GI.murJ5JvnmdYQevT1r69dNr5vl.ZcU-1689610349-0-160"}}
2023-08-16 21:41:44.336 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session/8c8ed1f3a720b4b0cf6ee41895d58afb/cookie HTTP/1.1" 200 0
2023-08-16 21:41:44.336 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:44.336 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:44.336 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session/8c8ed1f3a720b4b0cf6ee41895d58afb/url {"url": "http://www.fanfiction.net"}
2023-08-16 21:41:50.826 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session/8c8ed1f3a720b4b0cf6ee41895d58afb/url HTTP/1.1" 200 0
2023-08-16 21:41:50.826 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:50.827 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:50.827 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session/8c8ed1f3a720b4b0cf6ee41895d58afb/cookie {"cookie": {"domain": "www.fanfiction.net", "expiry": 1689613949, "httpOnly": false, "name": "cf_chl_2", "path": "/", "sameSite": "Lax", "secure": false, "value": "30a437ac0c75762"}}
2023-08-16 21:41:50.833 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session/8c8ed1f3a720b4b0cf6ee41895d58afb/cookie HTTP/1.1" 200 0
2023-08-16 21:41:50.833 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:50.834 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:50.834 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session/8c8ed1f3a720b4b0cf6ee41895d58afb/url {"url": "http://www.fanfiction.net"}
2023-08-16 21:41:57.225 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session/8c8ed1f3a720b4b0cf6ee41895d58afb/url HTTP/1.1" 200 0
2023-08-16 21:41:57.225 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:57.225 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:57.225 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session/8c8ed1f3a720b4b0cf6ee41895d58afb/cookie {"cookie": {"domain": "www.fanfiction.net", "expiry": 1720984497, "httpOnly": false, "name": "cookies", "path": "/", "sameSite": "Lax", "secure": true, "value": "yes"}}
2023-08-16 21:41:57.232 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session/8c8ed1f3a720b4b0cf6ee41895d58afb/cookie HTTP/1.1" 200 0
2023-08-16 21:41:57.232 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:57.232 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:57.232 CEST DEBUG ProxiedBrowser(init) Added 6 cookie(s)
2023-08-16 21:41:57.232 CEST INFO main Chrome is initialized & ready to works!
2023-08-16 21:41:57.232 CEST INFO main Listening on 127.0.0.1:8888

and on the other terminal:

sedmikraska~/K/f/n/Pride_and_Prejudice$ fanficfare -u A\ sunrise\ and\ a\ conversation-ffnet_12620565.epub
Updating A sunrise and a conversation-ffnet_12620565.epub, URL: https://www.fanfiction.net/s/12620565/1/
Traceback (most recent call last):
  File "/usr/bin/fanficfare", line 11, in <module>
    load_entry_point('FanFicFare==3.19.0', 'console_scripts', 'fanficfare')()
  File "/usr/lib/python3.6/site-packages/fanficfare/cli.py", line 310, in main
    passed_personalini)
  File "/usr/lib/python3.6/site-packages/fanficfare/cli.py", line 423, in do_download
    adapter.getStoryMetadataOnly()
  File "/usr/lib/python3.6/site-packages/fanficfare/adapters/base_adapter.py", line 297, in getStoryMetadataOnly
    self.doExtractChapterUrlsAndMetadata(get_cover=get_cover)
  File "/usr/lib/python3.6/site-packages/fanficfare/adapters/adapter_fanfictionnet.py", line 111, in doExtractChapterUrlsAndMetadata
    raise e
  File "/usr/lib/python3.6/site-packages/fanficfare/adapters/adapter_fanfictionnet.py", line 104, in doExtractChapterUrlsAndMetadata
    data = self._fetchUrl(url)
  File "/usr/lib/python3.6/site-packages/fanficfare/adapters/adapter_fanfictionnet.py", line 85, in _fetchUrl
    usecache=usecache)
  File "/usr/lib/python3.6/site-packages/fanficfare/configurable.py", line 1338, in _fetchUrl
    referer=referer)[0]
  File "/usr/lib/python3.6/site-packages/fanficfare/configurable.py", line 1327, in _fetchUrlOpened
    referer=referer)
  File "/usr/lib/python3.6/site-packages/fanficfare/configurable.py", line 1228, in _fetchUrlOpened
    raise(excpt)
  File "/usr/lib/python3.6/site-packages/fanficfare/configurable.py", line 1208, in _fetchUrlOpened
    referer=referer)
  File "/usr/lib/python3.6/site-packages/fanficfare/configurable.py", line 1167, in _fetchUrlRawOpened
    float(self.getConfig('connect_timeout',30.0)))
  File "/usr/lib64/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/lib64/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/lib64/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib64/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
sedmikraska~/K/f/n/Pride_and_Prejudice$

fanfiction_proxy_log.txt
@nsapa
Copy link
Owner

nsapa commented Aug 16, 2023 •
edited
Loading

I going to be on holiday next week: I will works on the open issues (minimal version of undetectedchromedriver, making a release with binary for Windows, checking what is happing on MacOS, ...)

Look like FFF didn't use nsapa_proxy in your screendump.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mcepl @nsapa