SQLAlchemy 1.4.x: RootTransaction object cannot serve as stash storage for original method references

[jayaddison]

The 1.4 major release of sqlalchemy seems to introduce some interface changes that pytest-flask-sqlalchemy 1.0.2 has trouble handling.

In particular one case is that pytest-flask-sqlalchemy will attempt to store a transaction's original rollback method in a temporary / ephemeral force_rollback property, but the current sqlalchemy 1.4.0 release seems incompatible with this -- I think because the RootTransaction class defines an explicit list of slots available on the object.

This issue might be a one-off, or it could be part of a larger category of sqlalchemy 1.4 compatibility concerns.

[jtfidje]

Here's one I just had to deal with.

Whenever I use the db_session fixture this error is thrown:

AttributeError: 'RootTransaction' object has no attribute 'force_rollback'

... /site-packages/pytest_flask_sqlalchemy/fixtures.py:43: AttributeError

I've had to roll-back to SQLAlchemy v.1.3.24

[jace]

Is anyone working on a fix? pytest-flask-sqlalchemy is the last blocker for upgrading to SQLAlchemy 1.4 for me.

[itajaja]

@jace we quite successfully dropped this package and we wrote our own fixture. this is the code, you might need to adapt it slightly for your needs

@pytest.fixture(autouse=True)
def enable_transactional_tests(db, connection):
    transaction = connection.begin()
    db.session = db.create_scoped_session(options={"bind": connection, "binds": {}})
    db.session.begin_nested()

    # for handling tests that actually call "session.rollback()"
    # https://docs.sqlalchemy.org/en/13/orm/session_transaction.html#joining-a-session-into-an-external-transaction-such-as-for-test-suites
    @event.listens_for(db.session, "after_transaction_end")
    def restart_savepoint(session, transaction_in):
        if transaction_in.nested and not transaction_in._parent.nested:
            session.expire_all()
            session.begin_nested()

    yield

    db.session.close()
    transaction.rollback()

where the db fixture is the db and the connection fixture is

@pytest.fixture(scope="session")
def connection(db):
    return db.engine.connect()

[GuillaumeDesforges]

pytest-flask-sqlalchemy is the last blocker for upgrading to SQLAlchemy 1.4 for me

Deal breaker for me as well.

[jayaddison]

@jace we quite successfully dropped this package and we wrote our own fixture. this is the code, you might need to adapt it slightly for your needs

@itajaja That snippet looks pretty great. Perhaps this is verging off-topic a little, but do you have any plans to wrap that into a pytest plugin or library? (or would you be supportive of others doing that?)

[itajaja]

@jayaddison absolutely, feel free to package it up! it's small enough that we just put it in our internal utilities library, but make it an OSS package if you want :)

[jayaddison]

Thanks @itajaja. I'm puzzling over this a little bit. One of the features that pytest-flask-sqlalchemy has is that it can mock database engines/sessions/sessionmakers for the entire application under test.

In other words, even if the test-covered application code runs some of its' own database modifications, pytest-flask-sqlalchemy wraps those and is designed to guarantee a safe rollback of them after the unit test completes.

Currently I think your code will rollback any database modifications made using the db fixture while enable_transactional_tests is also active -- but maybe not modifications made within the application codebase (that is unaware of the fixtures). I could well be mistaken.

[itajaja]

not sure I follow, could you give me a more concrete example as to when the pytest-flask-sqlalchemy approach would work while this would fail? Of course enable_transactional_tests needs to be called first, because it needs to open the transaction. but I think that this was the case with pytest-flask-sqlalchemy and also for how we structured the test that's a pretty fair assumption, since when you run pytest the first things that are executed are the fixture.

[jayaddison]

@itajaja Sure - see openculinary/backend@61bf1ae and the associated branch for an example of me testing this out.

On that branch I've been running the tests with py.test tests -x to stop at the first failure, and it currently includes this error in the output:

>       assert earliest_crawl.url == '//example.org/A'
E       AttributeError: 'NoneType' object has no attribute 'url'

Here's the relevant test code, for reference: https://github.com/openculinary/backend/blob/61bf1ae6f40e4fbd3468f27b7d208e278886806a/tests/models/test_url.py#L43-L69

From some debugging, it seems like the mocked db_session fixture isn't used by the flask web application -- so, for example, this query does not use the fixture.

(in comparison, pytest-flask-sqlalchemy does wrap database sessions created within application code)

[itajaja]

what if instead of using create_db, you from reciperadar import db and use that db? that's what we do in our code

[jayaddison]

@itajaja 🤦 Of course, yep - that would make a lot more sense, thank you. That has most of the tests passing, and the one remaining failure does not appear database-related. Great. I'll start to look into what's required (and/or possible) in terms of packaging the fixture up as a Python package soon, and I'll include credit and authorship details for the code.

[itajaja]

great!!

[jayaddison]

sqlalchemy 1.4 support was added in #51, so I think this can be resolved 🎉