Skip to content

Commit 2f69531

Browse files
authored
Break the database.users import cycle with utils.subscription and honor the dev goals limit (#11621)
# Problem Two reliability defects, rebuilt from #9966 and #9977, which had gone stale against current main. Both bugs are still present on `origin/main` today; I re-verified each against the current code before rebuilding. **1. `database/users.py` and `utils/subscription.py` form an import cycle.** `backend/database/users.py` imported back into `utils.subscription` at module scope: ```python from models.other import Person from utils.subscription import get_default_basic_subscription import logging ``` while `backend/utils/subscription.py` does `import database.users as users_db` at module scope. Whichever module is imported first decides whether the pair works. Importing `utils.subscription` first raises: ``` ImportError: cannot import name 'get_default_basic_subscription' from partially initialized module 'utils.subscription' (most likely due to a circular import) ``` Importing `database.users` first happens to succeed, which is why this surfaces intermittently as unrelated files change what gets imported first. The database -> utils edge is also the reverse of the documented backend layering (database/ -> utils/ -> routers/). **2. `GET /v1/dev/user/goals` ignores its documented `limit` when `include_inactive=true`.** The route clamps `limit` and then drops it on one branch: ```python # oversized limit cannot stream the whole collection. Mirrors the GET /v3/memories hardening. limit = max(1, min(limit, 1000)) if include_inactive: goals = goals_db.get_all_goals(uid, include_inactive=True) else: goals = goals_db.get_user_goals(uid, limit=limit) ``` `get_all_goals` had no limit parameter and streams the whole goals collection, so the clamp is dead code on that branch and the endpoint returns every goal the user has ever created, ignoring its own documented "**limit**: Maximum number of goals to return". # Reachability / failure scenario - Import cycle: any entry point that imports `utils.subscription` before `database.users` fails at import time with the error above. Reproduced with a fresh interpreter running `import utils.subscription` from `backend/`. - Goals limit: `GET /v1/dev/user/goals?include_inactive=true&limit=10` from any developer API consumer returns the full collection and reads every goal document in Firestore. The path that lets it through is `routers/developer.py::get_goals` -> `database/goals.py::get_all_goals`. # Fix **Import cycle.** The `get_default_basic_subscription` import in `database/users.py` moves from module scope to its call sites: inside `get_user_subscription`, and at the top of `get_user_valid_subscription`. Since #9966 was written, main added a `provision=False` branch to `get_user_valid_subscription` that also calls `get_default_basic_subscription`, earlier in the function than the old fallback site, so the function-level import sits above the first call site and covers both. Each deferred import carries a comment recording why it must not be folded back to the top. **Goals limit.** `get_all_goals` accepts an opt-in `limit` applied after the in-Python newest-first sort. The bound is deliberately not pushed into the Firestore query: `order_by('created_at')` excludes documents that lack the field entirely, and legacy or manually created goals can lack `created_at`, so a query-level order+limit would silently drop them. Goals without `created_at` coerce to `datetime.min` and sort last, so the bounded page is the newest goals with dateless legacy goals appearing only once dated goals run out, and the response honours the documented limit. Deliberately not changed: every other `get_all_goals` caller (the goal-by-id lookup, `routers/goals.py`, and the MCP goal reads) omits `limit` and keeps its fetch-everything behavior; bounding those is a behavior change with no bug attached. # Tests - `backend/tests/unit/test_subscription_import_cycle.py`: runs `import utils.subscription` (and separately `import database.users`) first in a fresh interpreter and asserts both succeed standalone. A fresh interpreter is the only honest seam here: once pytest has imported either module, the order is decided for the whole session, so an in-process assertion would pass either way. - `backend/tests/unit/test_dev_goals_limit_include_inactive.py`: router-level tests assert the clamp is delegated (captured kwargs for limit 5, ceiling clamp to 1000, and the active-only branch unchanged); database-level tests assert the bounded page is the newest goals, that a legacy goal without `created_at` is retained and sorts last instead of being dropped, that dated goals fill the page first, and that callers omitting `limit` keep the full unordered fetch with nothing pushed into the query. # Verification All commands run from `backend/` on Windows (Git Bash), `PYTHONUTF8=1`. Prove-fail. Reverted the three product files to `origin/main` (`git diff origin/main -- <files>` printed nothing, byte-identical), then ran both test files: ``` FAILED tests/unit/test_subscription_import_cycle.py::test_utils_subscription_imports_standalone AssertionError: utils.subscription is not importable on its own: ImportError: cannot import name 'get_default_basic_subscription' from partially initialized module 'utils.subscription' (most likely due to a circular import) FAILED tests/unit/test_dev_goals_limit_include_inactive.py::test_get_all_goals_bounds_the_query_when_limit_is_given TypeError: get_all_goals() got an unexpected keyword argument 'limit' 5 failed, 3 passed ``` The three passes on the unfixed tree are the behavior-preservation guards (importing `database.users` first, the active-only branch, and the unbounded default), which must pass on both sides. Restored the fix: `8 passed`. Static and contract gate: ``` black --line-length 120 --skip-string-normalization --check <5 files> -> 5 files would be left unchanged python -m pyright -p pyrightconfig.json database/goals.py -> 0 errors (users.py and routers/developer.py are in the pyright exclude list) python scripts/check_module_stub_pollution.py -> 898 test files, 0 violations python scripts/scan_async_blockers.py --dirs routers utils -> 0 findings in fail scope python scripts/scan_import_time_side_effects.py -> 804 files, 0 violations python scripts/check_conversation_lifecycle_writes.py -> passed ``` Workflow contracts: the touched files match three high-risk workflows (goals, users, developer). Ran all 13 unit and service test files listed by those workflows: `391 passed, 2 failed`, and the sync workflow file separately: `77 passed, 10 failed`. All 12 failures reproduce identically on a clean unmodified `origin/main` checkout in the same batch order (`tests/services/users/test_account_deletion.py` pair passes standalone on both trees), so they are pre-existing local-environment and test-ordering issues, not caused by this change. `testing/e2e/test_account_deletion_cloud_tasks.py` needs live services and errors locally on both trees. `scripts/pr-preflight --suggest`: Product invariants affected: none. No `fix:` commits, so no failure-class declaration is required. Full `scripts/pr-preflight --pr-body-file` run: every selected manifest check passes (including backend-route-policy-baseline, backend-async-blockers, backend-import-purity, backend-workflow-contracts, the line-count ratchet with the declarations below) except `desktop-backend-candidate-probe-fixtures`, which fails on this Windows machine with "gemini_proxy: total response deadline is unsupported on this runner". The files that check exercises are byte-identical to `origin/main` in this branch (the diff is five backend files), so it is a local runner-capability issue of the same class as the known Windows-only `detect_platform` false failure; CI runs on Linux. # Impact - The import cycle breaks any standalone consumer of `utils.subscription` and makes backend import order fragile: whether it bites depends on which module a given entry point or test session happens to import first. Developer-facing reliability, low direct user impact. - The goals endpoint bug affects developer API consumers who pass `include_inactive=true`: response size and Firestore read cost grow with the user's total historical goal count instead of respecting the documented bound. Narrow trigger, real cost on goal-heavy accounts. Line-Count-Exception: backend/database/users.py | 2131 -> 2140 | Breaking the utils.subscription import cycle moves one module-level import to deferred call sites with comments recording why each must stay deferred so the import is not folded back to the top and silently re-broken. Line-Count-Exception: backend/routers/developer.py | 2107 -> 2111 | The four added lines are the comment recording why the goals bound is applied after the in-Python sort rather than pushed into a Firestore order_by that would drop legacy goals lacking created_at.
2 parents aaa7b5c + d4cb5f8 commit 2f69531

5 files changed

Lines changed: 269 additions & 3 deletions

File tree

backend/database/goals.py

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -337,15 +337,28 @@ def get_all_goals(
337337
uid: str,
338338
include_inactive: bool = False,
339339
*,
340+
limit: Optional[int] = None,
340341
firestore_client: Any = None,
341342
) -> List[Dict[str, Any]]:
343+
"""Fetch a user's goals, newest first.
344+
345+
``limit`` bounds the returned page after the in-Python newest-first sort. It is opt-in:
346+
every existing caller omits it and keeps the full-list behaviour they rely on.
347+
348+
The bound is deliberately NOT pushed into the Firestore query: ``order_by('created_at')``
349+
excludes documents that lack the field entirely, and legacy or manually created goals can
350+
lack ``created_at`` — a query-level order+limit would silently drop them. Instead the full
351+
stream is sorted here (goals without ``created_at`` coerce to ``datetime.min`` and sort
352+
last) and the page is sliced, so the bound caps the response payload while dateless legacy
353+
goals still appear once dated goals run out.
354+
"""
342355
collection = _get_db(firestore_client).collection(users_collection).document(uid).collection(goals_collection)
343356
query = collection if include_inactive else collection.where(filter=FieldFilter('is_active', '==', True))
344357
goals = [normalize_goal_storage(_goal_dict(doc), goal_id=doc.id) for doc in query.stream()]
345358
if not include_inactive:
346359
goals = [goal for goal in goals if goal['is_active']]
347360
goals.sort(key=_goal_created_at_sort_key, reverse=True)
348-
return goals
361+
return goals if limit is None else goals[:limit]
349362

350363

351364
def create_goal(

backend/database/users.py

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,6 @@
2929
SubscriptionStatus,
3030
)
3131
from models.other import Person
32-
from utils.subscription import get_default_basic_subscription
3332
import logging
3433

3534
logger = logging.getLogger(__name__)
@@ -1547,6 +1546,11 @@ def subscription_payload(_snapshot: object) -> dict:
15471546
return subscription
15481547

15491548
# If subscription doesn't exist for the user, create and return a default free plan.
1549+
# Imported here, not at module scope: utils.subscription imports this module, so a
1550+
# module-level import makes utils.subscription unimportable on its own (see
1551+
# get_user_valid_subscription, which defers the same import for the same reason).
1552+
from utils.subscription import get_default_basic_subscription
1553+
15501554
default_subscription = get_default_basic_subscription()
15511555
# Strip dynamic fields before storing
15521556
sub_to_store = default_subscription.model_dump()
@@ -1678,6 +1682,11 @@ def get_user_valid_subscription(
16781682
quota must use that mode against the customer Firestore so a miss cannot
16791683
stamp ``plan: basic`` onto a paying user.
16801684
"""
1685+
# Imported here, not at module scope: utils.subscription imports this module, and a
1686+
# module-level import back into it forms a cycle that makes utils.subscription raise
1687+
# ImportError whenever it is the first of the pair to be imported.
1688+
from utils.subscription import get_default_basic_subscription
1689+
16811690
if provision:
16821691
subscription = get_user_subscription(uid, firestore_client=firestore_client)
16831692
else:

backend/routers/developer.py

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1942,7 +1942,11 @@ def get_goals(
19421942
# oversized limit cannot stream the whole collection. Mirrors the GET /v3/memories hardening.
19431943
limit = max(1, min(limit, 1000))
19441944
if include_inactive:
1945-
goals = goals_db.get_all_goals(uid, include_inactive=True)
1945+
# Pass the clamp down so the response honours the documented limit. The bound is
1946+
# applied after the in-Python newest-first sort rather than at the query, because a
1947+
# Firestore order_by('created_at') would silently exclude legacy goals that lack the
1948+
# field; see get_all_goals.
1949+
goals = goals_db.get_all_goals(uid, include_inactive=True, limit=limit)
19461950
else:
19471951
goals = goals_db.get_user_goals(uid, limit=limit)
19481952

Lines changed: 178 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,178 @@
1+
"""Regression test: GET /v1/dev/user/goals must honour `limit` when include_inactive=true.
2+
3+
routers.developer.get_goals clamps `limit` to [1, 1000] and then branches. The default branch
4+
calls goals_db.get_user_goals(uid, limit=limit), which is bounded. The include_inactive=True
5+
branch called goals_db.get_all_goals(uid, include_inactive=True), which had no limit parameter
6+
and streamed the whole goals collection, so the clamp was dead code on that branch and the
7+
endpoint returned every goal the user had ever created -- ignoring its own documented
8+
"**limit**: Maximum number of goals to return".
9+
10+
The clamp is now passed down and applied after the in-Python newest-first sort. The bound is
11+
deliberately not pushed into the Firestore query: order_by('created_at') excludes documents
12+
missing the field, and legacy goals can lack created_at, so a query-level bound would silently
13+
drop them. These tests assert the bounded page is the NEWEST goals, that dateless legacy goals
14+
survive bounding (sorting last), and that the bounded path is taken by the route.
15+
16+
get_all_goals stays fetch-everything by default for its other callers
17+
(/v1/dev/user/goals/{goal_id}, routers/goals.py::get_all_goals, and the MCP goal reads); the
18+
limit is opt-in and only this route passes it.
19+
"""
20+
21+
import pytest
22+
23+
import database.goals as goals_db_module
24+
import routers.developer as developer
25+
26+
27+
def _fake_goals(count):
28+
return [{'id': f'g{index}', 'title': f'goal {index}'} for index in range(count)]
29+
30+
31+
# --- router: the clamp reaches the helper -----------------------------------------------
32+
33+
34+
def test_include_inactive_passes_the_clamp_down_to_the_query(monkeypatch):
35+
captured = {}
36+
37+
def get_all_goals(uid, include_inactive=False, *, limit=None):
38+
captured.update(uid=uid, include_inactive=include_inactive, limit=limit)
39+
return _fake_goals(min(limit, 25))
40+
41+
monkeypatch.setattr(developer.goals_db, 'get_all_goals', get_all_goals)
42+
43+
result = developer.get_goals(uid='u1', limit=5, include_inactive=True)
44+
45+
# The bound is delegated, not applied after the fact.
46+
assert captured == {'uid': 'u1', 'include_inactive': True, 'limit': 5}
47+
assert len(result) == 5
48+
49+
50+
def test_include_inactive_delegates_the_clamp_ceiling(monkeypatch):
51+
captured = {}
52+
53+
def get_all_goals(uid, include_inactive=False, *, limit=None):
54+
captured.update(limit=limit)
55+
return _fake_goals(limit)
56+
57+
monkeypatch.setattr(developer.goals_db, 'get_all_goals', get_all_goals)
58+
59+
result = developer.get_goals(uid='u1', limit=99999, include_inactive=True)
60+
61+
assert captured == {'limit': 1000}
62+
assert len(result) == 1000
63+
64+
65+
def test_active_only_branch_still_delegates_the_limit(monkeypatch):
66+
captured = {}
67+
68+
def get_user_goals(uid, limit):
69+
captured.update(uid=uid, limit=limit)
70+
return _fake_goals(3)
71+
72+
monkeypatch.setattr(developer.goals_db, 'get_user_goals', get_user_goals)
73+
74+
result = developer.get_goals(uid='u1', limit=3, include_inactive=False)
75+
76+
assert captured == {'uid': 'u1', 'limit': 3}
77+
assert len(result) == 3
78+
79+
80+
# --- database: the page is the newest goals and legacy goals survive ---------------------
81+
82+
83+
class _FakeDoc:
84+
def __init__(self, doc_id, payload):
85+
self.id = doc_id
86+
self._payload = payload
87+
88+
def to_dict(self):
89+
return dict(self._payload)
90+
91+
92+
class _FakeCollection:
93+
"""Streams the given docs; records whether a query-level order/limit was pushed down
94+
(it must NOT be — that is the legacy-goal-dropping shape this fix avoids)."""
95+
96+
def __init__(self, docs, calls):
97+
self._docs = docs
98+
self.calls = calls
99+
100+
def collection(self, _name):
101+
return self
102+
103+
def document(self, _name):
104+
return self
105+
106+
def where(self, **kwargs):
107+
self.calls.append(('where', kwargs))
108+
return self
109+
110+
def order_by(self, field, direction=None):
111+
self.calls.append(('order_by', field, direction))
112+
return self
113+
114+
def limit(self, count):
115+
self.calls.append(('limit', count))
116+
return self
117+
118+
def stream(self):
119+
return iter(self._docs)
120+
121+
122+
def _docs(count):
123+
from datetime import datetime, timedelta, timezone
124+
125+
base = datetime(2026, 1, 1, tzinfo=timezone.utc)
126+
return [
127+
_FakeDoc(
128+
f'g{index}',
129+
{'created_at': base + timedelta(days=index), 'is_active': True, 'status': 'background'},
130+
)
131+
for index in range(count)
132+
]
133+
134+
135+
def test_get_all_goals_bounded_page_is_the_newest_goals():
136+
calls = []
137+
client = _FakeCollection(_docs(50), calls)
138+
139+
result = goals_db_module.get_all_goals('u1', include_inactive=True, limit=5, firestore_client=client)
140+
141+
# Newest first: the docs were streamed oldest-first, so the page must be the LAST five
142+
# ids in reverse creation order — proving the sort ran before the slice.
143+
assert [g['id'] for g in result] == ['g49', 'g48', 'g47', 'g46', 'g45']
144+
# And the bound was never pushed into the query, where order_by('created_at') would
145+
# exclude legacy goals lacking the field.
146+
assert not any(call[0] in ('order_by', 'limit') for call in calls)
147+
148+
149+
def test_get_all_goals_keeps_legacy_goals_without_created_at():
150+
calls = []
151+
dated = _docs(3)
152+
legacy = _FakeDoc('legacy', {'is_active': True, 'status': 'background'}) # no created_at
153+
client = _FakeCollection(dated + [legacy], calls)
154+
155+
result = goals_db_module.get_all_goals('u1', include_inactive=True, limit=10, firestore_client=client)
156+
157+
# The dateless legacy goal is retained (a query-level order_by would have dropped it)
158+
# and sorts deterministically last.
159+
assert [g['id'] for g in result] == ['g2', 'g1', 'g0', 'legacy']
160+
161+
162+
def test_get_all_goals_bounds_still_apply_over_legacy_goals():
163+
client = _FakeCollection(_docs(2) + [_FakeDoc('legacy', {'is_active': True, 'status': 'background'})], [])
164+
165+
result = goals_db_module.get_all_goals('u1', include_inactive=True, limit=2, firestore_client=client)
166+
167+
# Dated goals fill the page first; the dateless one only appears when room remains.
168+
assert [g['id'] for g in result] == ['g1', 'g0']
169+
170+
171+
def test_get_all_goals_stays_unbounded_for_existing_callers():
172+
calls = []
173+
client = _FakeCollection(_docs(50), calls)
174+
175+
result = goals_db_module.get_all_goals('u1', include_inactive=True, firestore_client=client)
176+
177+
assert not any(call[0] in ('limit', 'order_by') for call in calls)
178+
assert len(result) == 50
Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
"""Regression: utils.subscription must be importable on its own.
2+
3+
utils/subscription.py imports database.users, and database/users.py imported
4+
get_default_basic_subscription back from utils.subscription at module scope. That pair formed a
5+
cycle, so whichever of the two was imported first decided whether it worked: importing
6+
utils.subscription first raised
7+
8+
ImportError: cannot import name 'get_default_basic_subscription' from partially initialized
9+
module 'utils.subscription' (most likely due to a circular import)
10+
11+
while importing database.users first happened to succeed. The order-dependence is the bug, which
12+
is why it surfaces intermittently as unrelated test files change what gets imported first. The
13+
database -> utils edge is also the reverse of the documented backend layering
14+
(database/ -> utils/ -> routers/ -> main.py).
15+
16+
A fresh interpreter is the only honest seam here: once pytest has imported either module, the
17+
order is already decided for the whole session, so an in-process assertion would pass either way.
18+
"""
19+
20+
import os
21+
import subprocess
22+
import sys
23+
from pathlib import Path
24+
25+
import pytest
26+
27+
# Marked slow: each test boots a fresh interpreter that imports the full backend
28+
# module graph, which per tests/README.md belongs outside the PR fast lane
29+
# (`not integration and not slow`). A fresh interpreter is still the only honest
30+
# seam for import-order bugs, so the cost is the test's point, not an accident.
31+
32+
BACKEND_ROOT = Path(__file__).resolve().parents[2]
33+
34+
35+
def _import_first_in_fresh_interpreter(module: str) -> subprocess.CompletedProcess:
36+
env = dict(os.environ)
37+
env.setdefault("ENCRYPTION_SECRET", "test_secret_for_ci_only_0123456789")
38+
env.setdefault("OPENAI_API_KEY", "sk-fake")
39+
env.setdefault("PINECONE_API_KEY", "fake")
40+
return subprocess.run(
41+
[sys.executable, "-c", f"import {module}"],
42+
cwd=str(BACKEND_ROOT),
43+
env=env,
44+
capture_output=True,
45+
text=True,
46+
timeout=180,
47+
)
48+
49+
50+
@pytest.mark.slow
51+
def test_utils_subscription_imports_standalone():
52+
result = _import_first_in_fresh_interpreter("utils.subscription")
53+
54+
assert result.returncode == 0, f"utils.subscription is not importable on its own:\n{result.stderr}"
55+
56+
57+
@pytest.mark.slow
58+
def test_database_users_imports_standalone():
59+
# The other side of the pair must keep working too.
60+
result = _import_first_in_fresh_interpreter("database.users")
61+
62+
assert result.returncode == 0, f"database.users is not importable on its own:\n{result.stderr}"

0 commit comments

Comments
 (0)