[ENH] Add local support for schema

[jairad26]

Description of changes

Summarize the changes made by this PR.

• Improvements & Bug fixes
  • This PR adds support for schema in sqlite sysdb, correctly reconciling with schema, legacy metadata, and supporting configuration updates. It also adds support for passing schema via bindings, to allow for local chroma support. It also updates cli usage of to allow copying of schema
• New functionality
  • ...

Test plan

How are these changes tested?

expanded schema e2e tests to ensure bindings and single node all work as intended

• [ x] Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Migration plan

Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?

Observability plan

What is the plan to instrument and monitor this change?

Documentation Changes

Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs section?

[jairad26]

• [ENH] recognize and flush new metadata keys to schema on local compaction #5728
• [ENH] Add local support for schema #5714 👈 (View in Graphite)
• [ENH] Embed query strings in search api #5599
• [BUG] Default create path with no config or schema does not populate default ef in schema #5726
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[propel-code-bot]

[BestPractice]

Error handling inconsistency: The schema deserialization logic handles empty strings and "null" values differently across the codebase. In the SQLite implementation, it checks for both conditions, but other parts of the code may not handle these edge cases consistently.

match first_row.get::<Option<&str>, _>(7) {
    Some(json_str) if !json_str.trim().is_empty() && json_str.trim() != "null" => {
        // This logic should be centralized in a helper function
        // to ensure consistency across all schema deserialization points
    }
    // ...
}

Consider creating a centralized deserialize_schema_from_db helper function to ensure consistent handling of these edge cases.

Context for Agents

[**BestPractice**]

Error handling inconsistency: The schema deserialization logic handles empty strings and "null" values differently across the codebase. In the SQLite implementation, it checks for both conditions, but other parts of the code may not handle these edge cases consistently.

```rust
match first_row.get::<Option<&str>, _>(7) {
    Some(json_str) if !json_str.trim().is_empty() && json_str.trim() != "null" => {
        // This logic should be centralized in a helper function
        // to ensure consistency across all schema deserialization points
    }
    // ...
}
```

Consider creating a centralized `deserialize_schema_from_db` helper function to ensure consistent handling of these edge cases.

File: rust/sysdb/src/sqlite.rs
Line: 844

[propel-code-bot]

[BestPractice]

Potential null database migration issue: The code checks if applied_migrations.len() > source_migrations.len() and returns empty migrations, but this could mask real migration problems. If the database has more applied migrations than source migrations, this suggests a version mismatch or corrupted migration state that should be explicitly handled.

Suggested Change

Suggested change

	// Check if database has more applied migrations than available source migrations
	if applied_migrations.len() > source_migrations.len() {
	return Ok(vec![]);
	// Check if database has more applied migrations than available source migrations
	if applied_migrations.len() > source_migrations.len() {
	return Err(SqliteError::MigrationVersionMismatch(
	format!(
	"Database has {} applied migrations but only {} source migrations available",
	applied_migrations.len(),
	source_migrations.len()
	)
	));
	}

⚡ Committable suggestion

Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.

Context for Agents

[**BestPractice**]

Potential null database migration issue: The code checks `if applied_migrations.len() > source_migrations.len()` and returns empty migrations, but this could mask real migration problems. If the database has more applied migrations than source migrations, this suggests a version mismatch or corrupted migration state that should be explicitly handled.

<details>
<summary>Suggested Change</summary>

```suggestion
// Check if database has more applied migrations than available source migrations
if applied_migrations.len() > source_migrations.len() {
    return Err(SqliteError::MigrationVersionMismatch(
        format!(
            "Database has {} applied migrations but only {} source migrations available", 
            applied_migrations.len(), 
            source_migrations.len()
        )
    ));
}
```

⚡ **Committable suggestion**

Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.

</details>

File: rust/sqlite/src/db.rs
Line: 159

[jairad26]

@sanketkedia is it fine to not reconcile on the writer? i believe it should come through frontend, so it should reconcile

[sanketkedia]

Looking at the code, I think we need this here. The reconcile of schema with config happens in the handler for BackfillMessage. And then we need to reconcile this with legacy metadata here so this seems correct and necessary

[sanketkedia]

Ideally, I'd have liked all the three reconciles to happen at one place but that's not what it is now even with collection config

[jairad26]

i meant reconcile schema and config. yes the legacy fallback is needed everywhere

[jairad26]

use validate_schema from validators.rs in bindings.rs

[jairad26]

nvm

[sanketkedia]

shouldn't you reconcile with legacy metadata here before getting the space?

[sanketkedia]

In general, I feel like we should remove the blanket reconcile at the top and reconcile here in various places for readability

[jairad26]

added

[sanketkedia]

In general, I feel like the reconciliation logic is all over the place. But that's from before (collection config) so ok for now. But ideally once you've read from sysdb, you should assume that it has schema set and properly reconciled with both collection config and legacy metadata and just use it downstream