perf(deploy): raise production crawler concurrency to 2 and validate capacity

[absolutepraya]

Goal

Increase production crawl throughput by changing the VPS runtime setting CRAWLER_NUM_WORKERS from 1 to 2, then validate that the current production worker/container/browser setup handles the extra concurrency safely.

Context

PR #36 improves URL-ingestion efficiency but intentionally does not change crawler concurrency because the effective production value lives in the VPS .env, not in repository configuration.

This is an operator/deployment follow-up, not a code-default change.

Related: #36

Scope

• Inspect the current production .env and effective worker configuration on the VPS without exposing secrets.
• Record baseline crawl latency, queue depth/throughput, worker memory, CPU, Browserless/Chrome resource use, and relevant failure/retry rates.
• Set CRAWLER_NUM_WORKERS=2 in the production VPS environment.
• Restart/redeploy only the necessary worker service using the existing production deployment workflow.
• Verify both concurrent crawler slots are actually active.
• Exercise multiple URL crawls, including normal pages and representative heavier pages.
• Watch for OOMs, parser failures, browser/context leaks, timeouts, SQLite contention, queue instability, or elevated retry/error rates.
• Keep the repository default unchanged unless later evidence supports making 2 the general default.

Acceptance criteria

• Production effective config reports CRAWLER_NUM_WORKERS=2.
• Two concurrent crawls complete successfully under representative load.
• Worker/container memory remains within safe limits with no OOM/restart loop.
• Browserless/Chrome remains stable under the increased concurrency.
• No material increase in crawl failures, retries, timeouts, or DB contention is observed.
• Before/after throughput and latency observations are recorded.
• If capacity is insufficient, the setting is returned to 1 and the observed bottleneck is documented for follow-up.

Notes

Do not commit or print VPS .env contents or secrets. This issue should be performed when authenticated VPS access is available.

[absolutepraya]

Audit 2026-08-22: valid and live-confirmed. A read-only production inspection found karakeep-workers-1 effective at CRAWLER_NUM_WORKERS=1; the repository default is also 1. The issue remains an operator-capacity experiment, not a repository code change or current outage. Severity:low reflects a throughput improvement with no demonstrated user-facing failure. Keep the baseline, observation window, rollback, and no-secret handling requirements before changing the VPS setting.