-
Notifications
You must be signed in to change notification settings - Fork 20
Fix unclosed asyncpg connections causing SQLAlchemy pool warnings (#65) #88
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
…nsorBlock#65) - Apply recommended fix to get_async_db(): use explicit session management instead of async context manager - Remove redundant session.close() in get_db_session() after async with block - Fix synchronous commit() calls on async sessions in test files - Ensures proper cleanup of database connections and prevents pool exhaustion
Hey @Dokujaa , thanks for the fix PR. I think the following fixes are very solid:
However, I don't fully get the fix to function It seems that we did have a redundant code by utilizing the context manager and manually close the session in the final block as well. The part of "use explicit session management instead of async context manager" looks good to me. But I still don't know why the previous implementation caused an error there. |
Hey @lingtonglu. Thanks for reviewing my PR. I mainly went with this approach per @wilsonccccc's earlier comment on issue #65 What was mainly causing this issue was that the original block used async with (this should auto-close) and also calls session.close() in the finally block. This was closing the session twice. Pairing this with the sync commit calls caused some connections to remain checked out until the garbage collector. Now, background tasks should create their own short-lived sessions. This should make it so that connections are returned deterministically, which fixes the pool warnings. If I missed something, or you don't believe this implementation is proper, feel free to let me know. I'll make the changes! |
Hey @Dokujaa , the fix to not closing the session twice is a valid one. But I still don't think that's the root cause of the warning message. And that "sync commit" calls are in the test script. All the production code is using the async call. @wilsonccccc , do you have any suggestion on this? |
|
||
# Async dependency | ||
async def get_async_db(): | ||
async with AsyncSessionLocal() as session: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I personally prefer "async with" is because:
- The session is automatically closed when exiting the context (whether normally or due to an exception)
- If an exception occurs, the session is still properly closed
- Guarantees that database connections are returned to the pool
- Prevents connection leaks even if code fails
compare with "session = " statement, it requires:
- Manual cleanup required to call await session.close()
- If an exception occurs before close(), the session might leak
- Requires try/finally blocks everywhere
try: | ||
yield session | ||
finally: | ||
await session.close() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think most likely the issue is a race Condition: The get_async_db() dependency was trying to close a session while a commit operation was still in progress.
- A request used
get_user_by_api_key()
dependency - The dependency called
await db.commit()
to update the API key timestamp - The session was in a "committing" state
- The finally block in
get_async_db()
tried to callawait session.close()
- SQLAlchemy threw IllegalStateChangeError because you can't close a session while it's committing
async with AsyncSessionLocal() as session: | ||
try: | ||
yield session | ||
finally: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The async with AsyncSessionLocal() as session
: context manager already handles the session cleanup automatically. The finally block is redundant and could potentially cause issues. I think the proper fix should be just simply remove the finally
block, and add a catch block to catch the try: yield session, which need to await session.rollback().
try:
yield session
except Exception:
try:
await session.rollback()
except Exception:
pass
raise
Let me know if this is sufficient. I think swapping to await db.commit() is better, but let me know.
should fix Issue (#65)