[nodriver] Handler not firing all events registered with fetch.RequestPaused event (repro code provided)

[epicwhale]

I am trying to use the Fetch CDP domain, and have set up a very basic example of adding a handler for cdp.fetch.RequestPaused, and for now I am just responding with a fetch.continue_frequest (https://chromedevtools.github.io/devtools-protocol/tot/Fetch/#method-continueRequest) command whenever the request is paused.

from nodriver import start, cdp, loop

browser = None
tab = None

async def main():
    global tab, browser
    browser = await start()

    tab = browser.main_tab
    tab.add_handler(cdp.fetch.RequestPaused, requested_paused_handler) # This statement also enables the domain, so all requests will be paused if you use this handler!

    url = "https://nowsecure.nl"
    tab = await browser.get(url)
    print("PAGE LOADED") # <--- CODE NEVER REACHED


async def requested_paused_handler(event: cdp.fetch.RequestPaused):
    r = event.request
    s = f"{r.method} {r.url}"
    print(f" >>> ------------\n{s}\n >>> ------------") # <--- FIRED JUST ONCE

    global browser, tab
    await tab.send(cdp.fetch.continue_request(request_id=event.request_id))


if __name__ == "__main__":
    loop().run_until_complete(main())

Oddly, the handler is called once for the main page loads html, but the next page resource (in this case a javascript file network request) never reaches this handler, so it stays in a hung state. So the browser tab remains in a spinning stage, and the code flow does not proceed either in python... Screenshot of Network (see the Status for api.js as Pending) and CDP Protocol in Dev Tools.

To diagnose this further, I turned on debug logging to see if the websocket is actually receiving the second "Fetch.requestPaused" message, for the the screenshot above, and turns out it is (see log entries below).. but the second message is somehow never processed by nodriver, and sent to the callback.

2024-08-03 20:22:17.135 | DEBUG    | websockets.legacy.protocol:read_frame:1172 - < TEXT '{"method":"Fetch.requestPaused","params":{"requ...7DD5920EAC5C5CC48068"}}' [806 bytes]
...
...
...
 >>> ------------
GET https://nowsecure.nl/
 >>> ------------
2024-08-03 20:22:17.137 | DEBUG    | websockets.legacy.protocol:write_frame_sync:1178 - > TEXT '{"method": "Fetch.continueRequest", "params": {...ion-job-1.0"}, "id": 2}' [93 bytes]
2024-08-03 20:22:17.137 | DEBUG    | nodriver.core.browser:_handle_target_update:191 - target #1 has changed: 
attached: False => True

2024-08-03 20:22:17.138 | DEBUG    | websockets.legacy.protocol:read_frame:1172 - < TEXT '{"id":2,"result":{}}' [20 bytes]
2024-08-03 20:22:17.255 | DEBUG    | websockets.legacy.protocol:read_frame:1172 - < TEXT '{"id":0,"result":{"frameId":"B7CC82929E5EC68FE0...7DD5920EAC5C5CC48068"}}' [110 bytes]
2024-08-03 20:22:17.255 | DEBUG    | websockets.legacy.protocol:read_frame:1172 - < TEXT '{"method":"Target.targetInfoChanged","params":{...3B3775B475B3D8CF981"}}}' [362 bytes]
2024-08-03 20:22:17.257 | DEBUG    | nodriver.core.browser:_handle_target_update:191 - target #1 has changed: 
attached: True => False

2024-08-03 20:22:17.259 | DEBUG    | websockets.legacy.protocol:read_frame:1172 - < TEXT '{"method":"Target.targetInfoChanged","params":{...3B3775B475B3D8CF981"}}}' [270 bytes]
2024-08-03 20:22:17.261 | DEBUG    | websockets.legacy.protocol:read_frame:1172 - < TEXT '{"method":"Target.targetDestroyed","params":{"t...EEFF586848A5E6DCBF87"}}' [92 bytes]
2024-08-03 20:22:17.262 | DEBUG    | nodriver.core.browser:_handle_target_update:191 - target #0 has changed: 
title: New Tab => nowsecure.nl

url: chrome://newtab/ => https://nowsecure.nl/

2024-08-03 20:22:17.262 | DEBUG    | websockets.legacy.protocol:read_frame:1172 - < TEXT '{"method":"Fetch.requestPaused","params":{"requ..."networkId":"99851.2"}}' [677 bytes]

        **^^ this Fetch.requestPaused is the one we are interested in, but never reaches the handler callback**

Is this working as intended, or what am I doing incorrectly?

Attached full logs for reference.
github_bug_log.txt

[lunden23]

I am having the exact same issue as you by the way. Have not found a solution so far.

[ultrafunkamsterdam]

It will be there next update. Then you could create handler with 1 or 2 params, of which the second will be tab/connection.
Also tab gets a feed_cdp method, which can be used for responding (it will wrap it in ensure future, but makes it more concise.
Like

def myhandler(event, tab):
    ... Code....
    tab.feed_cdp( 
          cdp.fetch.continue_request(
                  event.request_id)
         )

[ultrafunkamsterdam]

@ultrafunkamsterdam oddly, it works without enabling Fetch domain for me. Even your code works for me even when I don't explicitly enable the Fetch domain? I thought when you add a handler using add_handler(..), it also enables the Domain for that event in the socket send message flow?

and it would indeed be great to have access to the tab connection in the event callback! (a temporary workaround i've figured is by just wrapping it in a class based function)

Hmm I thought fetch needed to be enabled explicitly, but then I'm wrong

Ive tried even using a function you could pass it , but it is hacky

def myhandler(event):
    tab = myhandler.get_tab()
    print(event, tab)

myhandler.get_tab = lambda: tab


tab.add_handler(....., myhandler)

[ultrafunkamsterdam]

Implemented (now in 0.36 )

[lunden23]

Thanks for all the help! I had a couple of days off and came back to realize that I already had an event handler for Paused Requests to handle proxy authentication and clearing the tab handler dictionary didn't actually detach the event handler, so I had two of them which was the cause for my issues. Removing the tab for which the proxy authentication handler was attached to and then creating a new tab resolved my issues and the request interception works fine now.

Thanks again everyone for the support!

[therealpurplemana]

It will be there next update. Then you could create handler with 1 or 2 params, of which the second will be tab/connection. Also tab gets a feed_cdp method, which can be used for responding (it will wrap it in ensure future, but makes it more concise. Like

def myhandler(event, tab):
    ... Code....
    tab.feed_cdp( 
          cdp.fetch.continue_request(
                  event.request_id)
         )

Is this already implemented?