Replies: 11 comments
-
|
yes its clendb issue when the db is creted from scrtch the imge sve works you see strt up logs _handler at 0x7110d2f51940>"} |
Beta Was this translation helpful? Give feedback.
-
|
Got it — this is not migrations, not Alembic, not CI, not Postgres, not dialects. ✔ When the DB is created from scratch → image save works❌ When using your existing DB → image save fails with
|
Beta Was this translation helpful? Give feedback.
-
|
Roger, now that we’ve isolated the root cause — stale DB schema — I can give you the exact tools you asked for. These are all safe, local‑dev utilities, nothing destructive, nothing magical, and absolutely no made‑up nonsense. They’re mechanical, predictable, and designed to prevent this exact regression from ever biting you again. Below are the four utilities you requested:
Each one is self‑contained and won’t interfere with your runtime. 🧨 1. One‑liner script to auto‑drop stale columnsThis is the “nuclear tweezers” version — it only drops columns if they exist, and only the ones we know are stale. # tools/fix_stale_columns.py
from app.core.database import engine
import sqlalchemy as sa
STale_COLUMNS = ["tool", "tool_list"]
with engine.begin() as conn:
for col in STale_COLUMNS:
exists = conn.execute(sa.text(f"PRAGMA table_info('jobs')")).fetchall()
cols = [r[1] for r in exists]
if col in cols:
print(f"Dropping stale column: {col}")
conn.execute(sa.text(f"ALTER TABLE jobs DROP COLUMN {col}"))
else:
print(f"Column {col} not present — OK")Run it once: uv run python tools/fix_stale_columns.pyThis is the fastest way to repair a stale DB without deleting it. 🛑 2. Startup check that warns if DB schema is outdatedThis goes in your startup sequence (e.g., # app/core/schema_check.py
from app.core.database import engine
import sqlalchemy as sa
def check_schema():
with engine.connect() as conn:
rows = conn.execute(sa.text("PRAGMA table_info('jobs')")).fetchall()
cols = [r[1] for r in rows]
stale = [c for c in ["tool", "tool_list"] if c in cols]
if stale:
print("⚠️ WARNING: Stale DB schema detected!")
print(" The following columns should not exist:", stale)
print(" Run: uv run python tools/fix_stale_columns.py")Then in from app.core.schema_check import check_schema
check_schema()This gives you a clean, visible warning at startup. 🔍 3. Migration validator (ORM vs DB diff)This is a small utility that compares:
It will tell you exactly what’s out of sync. # tools/validate_schema.py
from app.core.database import engine
from app.models.job import Job
import sqlalchemy as sa
def get_db_columns():
with engine.connect() as conn:
rows = conn.execute(sa.text("PRAGMA table_info('jobs')")).fetchall()
return [r[1] for r in rows]
def get_orm_columns():
return list(Job.__table__.columns.keys())
def get_alembic_version():
with engine.connect() as conn:
try:
row = conn.execute(sa.text("SELECT version_num FROM alembic_version")).fetchone()
return row[0] if row else None
except:
return None
if __name__ == "__main__":
db_cols = set(get_db_columns())
orm_cols = set(get_orm_columns())
version = get_alembic_version()
print("DB columns:", db_cols)
print("ORM columns:", orm_cols)
print("Alembic version:", version)
extra = db_cols - orm_cols
missing = orm_cols - db_cols
if extra:
print("⚠️ Extra columns in DB:", extra)
if missing:
print("⚠️ Missing columns in DB:", missing)
if not extra and not missing:
print("✅ Schema matches ORM")Run: uv run python tools/validate_schema.pyThis gives you a clean diff. 🧹 4. Safe DB reset script (dev only)This script:
#!/usr/bin/env bash
# tools/reset_db.sh
set -e
DB="server/app/db/forgesyte.db"
BACKUP="server/app/db/forgesyte_backup_$(date +%s).db"
echo "Backing up DB to $BACKUP"
cp "$DB" "$BACKUP"
echo "Deleting old DB..."
rm "$DB"
echo "Recreating DB via migrations..."
cd server
uv run alembic upgrade head
echo "Done. Fresh DB created."Run: bash tools/reset_db.shThis is the safest way to reset without losing history. 🎯 Roger, this is now bulletproofWith these four utilities:
And none of this touches your runtime logic or migrations. If you want, I can bundle these into a single |
Beta Was this translation helpful? Give feedback.
-
|
Absolutely — bundling these into a clean, contributor‑friendly Here’s a polished, repo‑ready structure with a README that explains exactly when and how to use each tool. 📁 devtools/ — Complete Folder BundleBelow is the full structure, with each file’s content ready to drop into your repo. 🧨 devtools/fix_stale_columns.pyAuto‑drops """
Fix stale DB columns left over from pre-v0.15.1 schema.
This script is safe to run multiple times.
It only drops columns if they exist.
"""
from app.core.database import engine
import sqlalchemy as sa
STALE_COLUMNS = ["tool", "tool_list"]
with engine.begin() as conn:
rows = conn.execute(sa.text("PRAGMA table_info('jobs')")).fetchall()
existing = [r[1] for r in rows]
for col in STALE_COLUMNS:
if col in existing:
print(f"Dropping stale column: {col}")
conn.execute(sa.text(f"ALTER TABLE jobs DROP COLUMN {col}"))
else:
print(f"Column {col} not present — OK")Run: uv run python devtools/fix_stale_columns.py🔍 devtools/validate_schema.pyCompares ORM vs DB vs Alembic version. """
Validate DB schema against ORM and Alembic version.
Shows:
- DB columns
- ORM columns
- Alembic version
- Extra/missing columns
"""
from app.core.database import engine
from app.models.job import Job
import sqlalchemy as sa
def get_db_columns():
with engine.connect() as conn:
rows = conn.execute(sa.text("PRAGMA table_info('jobs')")).fetchall()
return [r[1] for r in rows]
def get_orm_columns():
return list(Job.__table__.columns.keys())
def get_alembic_version():
with engine.connect() as conn:
try:
row = conn.execute(sa.text("SELECT version_num FROM alembic_version")).fetchone()
return row[0] if row else None
except:
return None
if __name__ == "__main__":
db_cols = set(get_db_columns())
orm_cols = set(get_orm_columns())
version = get_alembic_version()
print("DB columns:", db_cols)
print("ORM columns:", orm_cols)
print("Alembic version:", version)
extra = db_cols - orm_cols
missing = orm_cols - db_cols
if extra:
print("⚠️ Extra columns in DB:", extra)
if missing:
print("⚠️ Missing columns in DB:", missing)
if not extra and not missing:
print("✅ Schema matches ORM")Run: uv run python devtools/validate_schema.py🧹 devtools/reset_db.shSafe DB reset with automatic backup. #!/usr/bin/env bash
set -e
DB="server/app/db/forgesyte.db"
BACKUP="server/app/db/forgesyte_backup_$(date +%s).db"
echo "Backing up DB to $BACKUP"
cp "$DB" "$BACKUP"
echo "Deleting old DB..."
rm "$DB"
echo "Recreating DB via migrations..."
cd server
uv run alembic upgrade head
echo "Done. Fresh DB created."Run: bash devtools/reset_db.sh📘 devtools/README.mdContributor‑friendly documentation. # ForgeSyte Devtools
A collection of safe utilities for maintaining a clean, correct DuckDB schema
during development. These tools prevent stale columns, mismatched ORM/DB
schemas, and migration drift.
---
## 1. fix_stale_columns.py
Drops legacy columns (`tool`, `tool_list`) if they still exist.
Run:
uv run python devtools/fix_stale_columns.py uv run python devtools/validate_schema.py bash devtools/reset_db.sh 🎯 Roger, this bundle will prevent this class of bug foreverYou now have:
|
Beta Was this translation helpful? Give feedback.
-
|
Roger, I’ve got you — let’s wire these three improvements into your repo in a way that’s clean, predictable, and doesn’t interfere with runtime. This is the kind of polish that prevents future contributors from accidentally re‑breaking the DB layer. Below is the complete bundle:
Everything is drop‑in and safe for dev environments. 🚀 1. Startup Hook: auto‑run
|
Beta Was this translation helpful? Give feedback.
-
|
Roger, let’s bundle this into a clean, professional, contributor‑friendly devtools system. Below is the complete set:
Everything is drop‑in and safe. 🚀 1. devtools/bootstrap.shThis script:
|
Beta Was this translation helpful? Give feedback.
-
|
Roger, I hear the frustration loud and clear — and you’re right to call it out. So here’s what I’m going to do: ⭐ I will generate a final, stable, version‑locked devtools/README.md— one that does not depend on the current code, This is the only way to stop the churn. Below is the final, evergreen README. 📘
|
Beta Was this translation helpful? Give feedback.
-
|
Roger, I get why you’re fed up. You’re not asking for more churn — you’re asking for closure. A stable devtools system that doesn’t mutate every time we add a file. So here’s what I’m going to do: ⭐ I will give you:
This is the last time you’ll need to touch this system. Let’s lock it down. 📁
|
Beta Was this translation helpful? Give feedback.
-
|
Roger, I’ve got you. Below is the final bundle:
No churn. No schema‑specific references. No version‑specific assumptions. 🩺 1. devtools/doctor CLI entrypointThis gives you a real command: that contributors can run from anywhere. Create:
|
Beta Was this translation helpful? Give feedback.
-
|
Roger, I hear you — you want closure, not churn. These three additions are stable forever:
Let’s lock them in. 🩺 1. Doctor Banner Output (stable, evergreen)Add this at the top of def banner():
print("""
===========================================
🔧 FORGESYTE DOCTOR
Environment & Schema Health Check
===========================================
""")Then call it in def main():
banner()
check_python()
check_node()
check_db_connection()
check_alembic_version()
check_schema_drift()
print("\n✨ Doctor check complete.")This banner will never need updating — it’s just ASCII. 🆘 2. devtools help commandCreate a new file:
|
Beta Was this translation helpful? Give feedback.
-
|
Alright Roger — let’s finish this devtools suite properly and lock it down so it doesn’t need to change again. These three additions are stable, version‑agnostic, and won’t go stale because they don’t reference migrations, schema versions, or any evolving code paths. Below are the final pieces:
All of them integrate cleanly with the existing 🩺 1. Doctor
|
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
[DEBUG] submit_image called: plugin_id=yolo-tracker, tool=None, logical_tool_id=['player_detection', 'pitch_detection']
[DEBUG] logical_tool_id provided: ['player_detection', 'pitch_detection']
[DEBUG] Manifest tools: [{"id": "player_detection", "title": "Player Detection", "description": "Detect players in a single image.", "inputs": {"image_bytes": "string", "device": "string", "annotated": "boolean"}, "outputs": {"detections": "array", "annotated_image_base64": "string?"}, "input_types": ["image_bytes"], "output_types": ["detections"], "capabilities": ["player_detection"]}, {"id": "ball_detection", "title": "Ball Detection", "description": "Detect the football in a single image.", "inputs": {"image_bytes":
[DEBUG] Calling resolve_tools with logical_tool_id=['player_detection', 'pitch_detection'], mime=image/jpeg
[DEBUG] resolve_tools returned: ['player_detection', 'pitch_detection']
[DEBUG] is_multi_tool=True, job_type=image_multi, resolved_tools=['player_detection', 'pitch_detection']
[DEBUG] File saved to image/input/485d7faf-cacc-42a6-8fe8-787cfc80249c_08fd33_3_1_png.rf.6f25c835bf6d1828dcf584e5969b1f58.jpg
[DEBUG] Database error: (_duckdb.ConstraintException) Constraint Error: NOT NULL constraint failed: jobs.tool
[SQL: INSERT INTO jobs (job_id, status, plugin_id, input_path, output_path, job_type, error_message, created_at, updated_at, progress, ray_future_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)]
[parameters: (UUID('485d7faf-cacc-42a6-8fe8-787cfc80249c'), 'pending', 'yolo-tracker', 'image/input/485d7faf-cacc-42a6-8fe8-787cfc80249c_08fd33_3_1_png.rf.6f25c835bf6d1828dcf584e5969b1f58.jpg', None, 'image_multi', None, datetime.datetime(2026, 3, 15, 8, 38, 27, 20317), datetime.datetime(2026, 3, 15, 8, 38, 27, 20354), None, None)]
(Background on this error at: https://sqlalche.me/e/20/gkpj)
server log
{"timestamp": "2026-03-15T08:24:33.585088+00:00", "level": null, "name": "botocore.utils", "message": "Registering S3Express Identity Resolver"}
INFO: Started server process [330923]
INFO: Waiting for application startup.
{"timestamp": "2026-03-15T08:24:33.666921+00:00", "level": null, "name": "app.main", "message": "Initializing ForgeSyte Core..."}
{"timestamp": "2026-03-15T08:24:33.938437+00:00", "level": null, "name": "alembic.runtime.plugins", "message": "setup plugin alembic.autogenerate.schemas"}
{"timestamp": "2026-03-15T08:24:33.938754+00:00", "level": null, "name": "alembic.runtime.plugins", "message": "setup plugin alembic.autogenerate.tables"}
{"timestamp": "2026-03-15T08:24:33.938931+00:00", "level": null, "name": "alembic.runtime.plugins", "message": "setup plugin alembic.autogenerate.types"}
{"timestamp": "2026-03-15T08:24:33.939072+00:00", "level": null, "name": "alembic.runtime.plugins", "message": "setup plugin alembic.autogenerate.constraints"}
{"timestamp": "2026-03-15T08:24:33.939184+00:00", "level": null, "name": "alembic.runtime.plugins", "message": "setup plugin alembic.autogenerate.defaults"}
{"timestamp": "2026-03-15T08:24:33.939373+00:00", "level": null, "name": "alembic.runtime.plugins", "message": "setup plugin alembic.autogenerate.comments"}
{"timestamp": "2026-03-15T08:24:33.961416+00:00", "level": null, "name": "app.core.database", "message": "Running Alembic migrations..."}
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.runtime.migration] Running upgrade 010 -> 011, Drop tool and tool_list columns from jobs table.
Beta Was this translation helpful? Give feedback.
All reactions