Cookies not propagated to requests.Session

[jamielennox]

I'm having problems setting cookies using the requests_mock header= option. A cookie makes it as far as the requests.Response object, but does not reach the requests.Session I used to make it. I've tried several variants of the Set-Cookie header and believe I have it formatted correctly.

The following code demonstrates it:

import requests, requests_mock

with requests_mock.Mocker() as m:
m.register_uri('GET', 'http://github.com', text='data', headers={
"Set-Cookie": "food=pie; Max-Age=3600; Version=1"
})
sess = requests.Session()
resp = sess.get("http://github.com")
for c in resp.cookies:
print("resp cookie: %s" % c)
for c in sess.cookies:
print("sess cookie: %s" % c)

Expected result: 'food=pie' cookie listed in both resp and sess cookies. Running the code outside of the "with" does this.

Actual result: 'food=pie' cookie listed in only resp and is not sent back to the server in further requests.

Launchpad Details: #LP1550980 Alex - 2016-02-28 19:09:46 +0000

[jamielennox]

You're right, unfortunately this is a harder problem to fix than i would expect.

When requests parses the cookies off the response object and onto the session it doesn't read it the cookiejar on the response object, it reads the headers from the httplib response object that urllib3 provides "for debugging purposes". I'm not sure if this is to provide seperate handling for session cookies, it just seems to be old code.

There are a couple of places i fake this interface because all we need are the headers and it generally works, and I can fix your sample case pretty easily because it is in cookie header format. However i'm stuck with how to handle the case where a user provides a cookiejar object. To make this work i would need to convert the cookiejar back into Set-Cookie headers so that requests could parse them again. There's no obvious interface to do this in python - cookiejars aren't supposed to work like that.

The solution then is to just dump the cookie objects from the jar and construct the headers myself. This might be the only way but i'd really prefer not to maintain that/learn that much about cookies if i can avoid it.

Ideally It would be nice if requests fixed there interface here but that doesn't solve the problem for existing versions.

I'll keep looking, maybe Ian can lend some advice here?

Launchpad Details: #LPC Jamie Lennox - 2016-02-28 23:52:59 +0000

[jamielennox]

The problem seems still present today. Is the bug due to python-requests? If it is, maybe the bug could be forwarded to them?

Launchpad Details: #LPC Pablo Pizza - 2018-01-31 17:54:04 +0000

[Lx]

I'm bumping into this issue now as well, and I'm keen to see this issue resolved. Can I do anything to help?

The unit test I'm writing mocks the creation of a cookie, which the tested code expects to find in the session. As described above, the cookies are successfully coming back in the response object but not propagating to the session object. As a result I'm prevented from writing a proper test.

[Lx]

I noted that this issue wasn't listed on the Known Issues page of the docs, so I've submitted a PR for its addition.

[jamielennox]

Merged #99 - sorry it took a while.

Ideally i'd still love to solve this but i'm still not sure what correct behaviour looks like. I'm open to any suggestions/discussion.

[Lx]

Thank you.

I guess the best outcome would be for the requests library to properly read the cookie jar, but as you’ve mentioned, this doesn’t resolve the issue for existing versions of the library.

The next best solution seems to be explicitly constructing cookie headers within the requests-mock library from the cookie jar, as you’ve mentioned. It’s more work but it guarantees resolution of the issue for all requests-mock users regardless of the underlying requests library version.

[jaraco]

This issue is affecting me also. Here's the repro I created:

"""
Run this file with:

pip-run -q -- -m pytest file.py
"""

__requires__ = ['requests', 'requests_mock', 'pytest']


import requests


def test_session(requests_mock):
    requests_mock.register_uri('GET', '/', cookies={'foo': 'bar'})
    session = requests.Session()
    resp = session.get('http://anything/')
    assert resp.cookies == {'foo': 'bar'}
    assert session.cookies == {'foo': 'bar'}

Running it as instructed produces:

 draft $ pip-run -q -- -m pytest file.py
======================================================================= test session starts ========================================================================
platform darwin -- Python 3.11.0, pytest-7.2.0, pluggy-1.0.0
rootdir: /Users/jaraco/draft
plugins: requests-mock-1.10.0
collected 1 item                                                                                                                                                   

file.py F                                                                                                                                                    [100%]

============================================================================= FAILURES =============================================================================
___________________________________________________________________________ test_session ___________________________________________________________________________

requests_mock = <requests_mock.mocker.Mocker object at 0x1052f6210>

    def test_session(requests_mock):
        requests_mock.register_uri('GET', '/', cookies={'foo': 'bar'})
        session = requests.Session()
        resp = session.get('http://anything/')
        assert resp.cookies == {'foo': 'bar'}
>       assert session.cookies == {'foo': 'bar'}
E       AssertionError: assert <RequestsCookieJar[]> == {'foo': 'bar'}
E         Use -v to get more diff

file.py:12: AssertionError
===================================================================== short test summary info ======================================================================
FAILED file.py::test_session - AssertionError: assert <RequestsCookieJar[]> == {'foo': 'bar'}
======================================================================== 1 failed in 0.03s =========================================================================

And since the session object doesn't get its cookies set, it doesn't accurately mimic the behavior against a real server.

I see there is a pending PR to fix the issue that's stalled.

Is there anything I can do to help get a fix released?

[jaraco]

I can confirm that applying the pending PR does fix the issue:

 draft $ cat file.py
__requires__ = [
    'requests',
    'requests_mock@git+https://github.com/jaraco/requests-mock@bugfix-17',
    'pytest',
]


import requests


def test_session(requests_mock):
    requests_mock.register_uri('GET', '/', cookies={'foo': 'bar'})
    session = requests.Session()
    resp = session.get('http://anything/')
    assert resp.cookies == {'foo': 'bar'}
    assert session.cookies == {'foo': 'bar'}
 draft $ py -3.10 -m pip-run -q -- -m pytest file.py
======================================================================= test session starts ========================================================================
platform darwin -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0
rootdir: /Users/jaraco/draft
plugins: requests-mock-1.10.1.dev1
collected 1 item                                                                                                                                                   

file.py .                                                                                                                                                    [100%]

======================================================================== 1 passed in 0.01s =========================================================================

[jaraco]

Hmm. Or maybe only partially fixes the issue. The current patch doesn't work properly if more than one cookie is passed in:

 draft $ cat two-cookies.py
__requires__ = [
    'requests',
    'requests_mock@git+https://github.com/jaraco/requests-mock@bugfix-17',
    'pytest',
]


import requests


def test_session(requests_mock):
    requests_mock.register_uri('GET', '/', cookies={'foo': 'bar', 'bing': 'baz'})
    session = requests.Session()
    resp = session.get('http://anything/')
    assert resp.cookies == {'foo': 'bar', 'bing': 'baz'}
    assert session.cookies == {'foo': 'bar', 'bing': 'baz'}
 draft $ py -3.11 -m pip-run -- -m pytest two-cookies.py -v
================================================================ test session starts =================================================================
platform darwin -- Python 3.11.1, pytest-7.2.0, pluggy-1.0.0 -- /opt/homebrew/Cellar/[email protected]/3.11.1/bin/python3.11
cachedir: .pytest_cache
rootdir: /Users/jaraco/draft
plugins: requests-mock-1.10.1.dev1, anyio-3.6.2
collected 1 item                                                                                                                                     

two-cookies.py::test_session FAILED                                                                                                            [100%]

====================================================================== FAILURES ======================================================================
____________________________________________________________________ test_session ____________________________________________________________________

requests_mock = <requests_mock.mocker.Mocker object at 0x1045aac10>

    def test_session(requests_mock):
        requests_mock.register_uri('GET', '/', cookies={'foo': 'bar', 'bing': 'baz'})
        session = requests.Session()
        resp = session.get('http://anything/')
        assert resp.cookies == {'foo': 'bar', 'bing': 'baz'}
>       assert session.cookies == {'foo': 'bar', 'bing': 'baz'}
E       AssertionError: assert <RequestsCook...c2109=False)]> == {'bing': 'baz', 'foo': 'bar'}
E         Full diff:
E         - {'bing': 'baz', 'foo': 'bar'}
E         + <RequestsCookieJar[Cookie(version=0, name='foo', value='bar', port=None, port_specified=False, domain='anything.local', domain_specified=False, domain_initial_dot=False, path='/', path_specified=False, secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={'bing': 'baz'}, rfc2109=False)]>

two-cookies.py:16: AssertionError
============================================================== short test summary info ===============================================================
FAILED two-cookies.py::test_session - AssertionError: assert <RequestsCook...c2109=False)]> == {'bing': 'baz', 'foo': 'bar'}
================================================================= 1 failed in 0.03s ==================================================================

As you can see from the error, the bing=baz cookie is getting bundled into the rest of the foo cookie. I believe that's incorrect.