New Schema Refactor (Dev Migrations Required)

[magiccodingman]

Overview

This proposal introduces a significant refactor to the Magic.IndexDB library, focusing on simplifying database schema management, improving ease of use, and removing unnecessary manual setup. The primary change involves eliminating the need for explicit database initialization by leveraging attribute-based metadata, thus reducing developer overhead and improving usability.

Current System

Currently, Magic.IndexDB requires developers to explicitly register and manage database connections using the MagicDbFactory. The following methods are used to open and interact with databases:

ValueTask<IndexedDbManager> OpenAsync(DbStore dbStore, bool force = false, CancellationToken cancellationToken = default);
IndexedDbManager Get(string dbName);
ValueTask<IndexedDbManager> GetRegisteredAsync(string dbName, CancellationToken cancellationToken = default);

Developers then use the IndexedDbManager to query and manipulate records. Example usage:

var manager = await _MagicDb.GetRegisteredAsync("client");

// Get all records
allPeople = (await manager.GetAllAsync<Person>()).ToList();

// Where statement with filtering
var WhereExample = (await manager.Where<Person>(x => x.Name.StartsWith("c", StringComparison.OrdinalIgnoreCase)
    || x.Name.StartsWith("l", StringComparison.OrdinalIgnoreCase)
    || x.Name.StartsWith("j", StringComparison.OrdinalIgnoreCase) && x._Age > 35
    || x.Name.Contains("bo", StringComparison.OrdinalIgnoreCase))
    .OrderBy(x => x._Id)
    .Skip(1)
    .ToListAsync());

This approach requires developers to explicitly open databases and associate models with the correct database manager. However, this information is already available in model attributes:

[MagicTable("Person", "client")]
public class Person

Where the MagicTable attribute includes:

public MagicTableAttribute(string schemaName, string databaseName)

Since the system already knows which database a model belongs to, requiring manual database initialization is redundant and adds unnecessary complexity.

Proposed Changes

1. Automatic Database Initialization on First Use

Instead of requiring explicit database registration, databases will automatically open when a model is accessed. This eliminates the need for developers to manually manage database connections.

• Before: Developers must manually register and open databases.
• After: Databases open automatically when accessed, based on model attributes.

2. Removing Explicit Database Manager Assignment

Currently, multiple managers must be instantiated for different databases, and developers must remember which manager corresponds to which models. This proposal eliminates that requirement, as the system can infer database assignments from attributes.

3. Simplified Database Closing Mechanism

While databases will open automatically, developers should retain control over closing them. This proposal introduces new methods:

// Close a specific database
IndexedDBManager.CloseDatabase("client");

// Close all open databases
IndexedDBManager.CloseAllDatabases();

// Close databases based on model types
IndexedDBManager.CloseDatabases(typeof(Person), typeof(Order));

This provides a streamlined approach to resource management while removing unnecessary manual steps.

4. Refactoring Startup Configuration

Currently, database registration in Program.cs looks like this:

builder.Services.AddBlazorDB(options =>
{
    options.Name = DbNames.Client;
    options.Version = 1;
    options.StoreSchemas = SchemaHelper.GetAllSchemas(DbNames.Client);    
});

The proposed refactor eliminates options.Name and options.StoreSchemas, as this information is redundant given the attribute-based schema system. The new approach would be:

builder.Services.AddBlazorDB({
    options.Version = 1;
});

The system will infer database names and schemas at runtime.

5. Removing DbStore.StoreSchemas

Previously, DbStore required explicit schema definitions:

public class DbStore
{
    public string Name { get; set; }
    public int Version { get; set; }

    public List<StoreSchema> StoreSchemas { get; set; }
}

This proposal removes StoreSchemas entirely, as Magic.IndexDB can determine schema details automatically.

6. Introducing Dynamic Runtime Table & Database Creation

To allow for dynamic, runtime-created tables and databases, a special runtime model will be introduced. This model will allow developers to:

• Define properties dynamically at runtime.
• Set primary keys and indexes dynamically.
• Maintain strongly typed structures even for dynamically created tables.

This ensures that even runtime-created databases maintain the benefits of strong typing and schema validation.

Summary of Benefits

Feature	Current System	Proposed System
Database Opening	Manual registration	Automatic on first use
Database Manager Association	Must track and assign manually	Fully inferred from attributes
Database Closing	Not easily managed	Explicit close methods provided
Startup Configuration	Requires explicit schemas	Completely inferred
Schema Definitions	Must be manually provided	Automatically determined
Dynamic Tables	Not well supported	Fully supported with structured runtime models

Migration Guide

Changes to Database Initialization

Before:

var manager = await _MagicDb.GetRegisteredAsync("client");
var people = await manager.GetAllAsync<Person>();

After:

var people = await MagicDb.GetAllAsync<Person>();

Changes to Closing Databases

Before:

var manager = await _MagicDb.GetRegisteredAsync("client");
manager.Close();

After:

var manager = await _MagicDb.GetRegisteredAsync();
manager.CloseDatabase("client");

Changes to Startup Configuration

Before:

builder.Services.AddBlazorDB(options =>
{
    options.Name = "client";
    options.Version = 1;
    options.StoreSchemas = SchemaHelper.GetAllSchemas("client");
});

After:

builder.Services.AddBlazorDB({
    options.Version = 1;
});

Conclusion

This refactor simplifies database usage, eliminates redundant setup, and improves developer experience by making database management seamless. While a migration is required, it is a one-time change that provides long-term benefits by reducing boilerplate and enforcing best practices.

Feedback from the community is encouraged before implementation!

[yueyinqiu]

I believe for a static database whose schema won't change at runtime, there is no reason to reject these changes.

except the codes in Changes to Closing Databases section. I think you just made a mistake here. Perhaps it's closer to:

_MagicDb.CloseDatabase<Person>();

---

And are there any specific examples about the dynamic runtime tables? Will they use another attribute other than the [MagicTable]? And to access them, maybe we still need an extra parameter to indicate database/store for each function?

[magiccodingman]

Yes that's more the idea I was thinkin @yueyinqiu Also I was considering some more radical changes as well potentially. So consider this scenario:

_MagicDb.Where<Person>(x => x.Name == "John").Where<Person>(x => x.Age <= 30 && x.Age >= 18).Skip(4);

Now when I created this library, I wanted to more replicate repository LINQ to SQL because I like it more, plus the normal EF Core syntax is harder to replicate in a non relational database. But That syntax I showed you, it can happen, especially in deferred execution scenarios. So multiple where statements are not uncommon.

But what if you wanted to start with a skip? Where do you put that? The answer is you can't.

Additionally what if those where statements type of T weren't identical? What would happen? I am not sure and I'm sure that'd cause failure but you shouldn't be allowed to do that in the first place. Plus it's kind of annoying I had to write it like that over and over anyways. Especially for complex deferred execution scenarios.

So I'm thinking of removing "Where<T>" and replacing it with just "Where".

As there should instead be something like, "Repository<T>" or maybe, "Table<T>". I'll have to think of what syntax best fits this scenario. But this would allow not just cleaner code, but less code. It would also build better syntax to prevent errors:

_MagicDb.Repository<Person>().OrderBy(x => x.Id)
.Where(x => x.Name == "John")
.Where(x => x.Age <= 30 && x.Age >= 18).Skip(4);

Or you could do:

var personRepository = _MagicDb.Repository<Person>().OrderBy(x => x.Id);

personRepository.Where(x => x.Name == "John");
personRepository.Where(x => x.Age <= 30 && x.Age >= 18).Skip(4);

List<Person> personList = await personRepository.ToListAsync()

Anyways hopefully this makes sense. But this shows more ways to break it apart by using repositories and more control as I could put an OrderBy, or Skip, or Take, or other attached to the repository instead of being forced to attach it to a Where statement first.

I also think this is just a lot cleaner. Major down side is that this will require each project to refactor. Though not the hardest refactor in the world. But I could also obsolete the current Where, add the repository, and just let people migrate over if that's a concern and not just instantly remove the code. On the other hand, all of my ideas would require a refactor so why not just make everyone refactor all at once? Hmm.

[yueyinqiu]

What about Query() which returns a IQueryable, like:

_MagicDb.Query<Person>().Where(...

// for dynamically created ones:
_MagicDb.Query<Person>("database", "store").Where(...

[magiccodingman]

@yueyinqiu That... I really like that. You're absolutely right. Repository is common in LINQ to SQL terminology, but that applies more to relational databases with rigid schema structures—something I'm actively simplifying.

In our case, we don’t truly have a "repository" in the traditional sense. The idea of Table was close, but it doesn’t fully fit either.

But Query<T>? That just makes sense.

I was actually just about to start implementing some of this, so this suggestion came at the perfect time. Great call! I'll go ahead and move forward with Query<T> as the primary approach.

[magiccodingman]

@yueyinqiu
Oh, and I forgot to address dynamically created models. I’ve been exploring solutions, but there's challenges. That said, I do have some interesting ideas to make this work seamlessly.

One approach I’m considering is introducing a specialized class (StoreSafeNoCache) with strict rules in the caching layer. This class would allow defining primary keys, database names, and table names at runtime, along with dynamic properties. The key distinction? Anything using this class (or linked to it) would automatically be exempt from caching.

Caching dynamic objects doesn’t make sense—what if their structure changes at runtime? The existing caching system is built for strictly-typed, immutable objects. I also realized that if the serializer encounters dynamic, object, or other ambiguous types, those too should be excluded from caching. That’s an edge case I hadn’t considered until now.

With this in mind, dynamic runtime tables in Magic.IndexDB would need new query initiators. Here’s how I see it working:

Query Initiator Overloads

1. Fully Dynamic Query (No Type)

_MagicDb.Query("database", "store") // Returns type of object, relies on runtime definitions

This allows full flexibility without requiring a predefined model.

2. Explicit Type, No MagicTable Attribute

_MagicDb.Query<Person>("database", "store") 

• Provides T = Person, but since Person lacks the MagicTable attribute, required info must be provided manually.
• Ensures proper deserialization into Person instead of returning a raw dynamic object.

I could also introduce considerations as well to use expression array parameters for you to define multiple properties as indexed, or not mapped, or different property names in IndexDB, or more. Thus allowing strict classes that don't have attributes to be flexible in our system.

3. Standard Typed Query (MagicTable Enabled)

_MagicDb.Query<Person>() 

• Requires Person to have the MagicTable attribute.
• If improperly attributed, an error is thrown at runtime.

Key Takeaways

1. Flexible Model Reuse – A model can now exist in multiple databases/stores.
2. Dynamic & Strict Models Coexist – You can mix strictly defined models with dynamic ones in the same system.
3. Attribute-Free Flexibility – If you don’t want to use MagicTable, override Option 2 enables explicit control.
4. Internal Handling for No-Cache Models – StoreSafeNoCache (internal-use) ensures runtime reevaluation for anything dynamic.
5. Future Caching Control – A potential future option could allow explicitly opting certain dynamic models into caching if structure remains stable.

This approach would enable both dynamic flexibility and strict type safety while preventing common pitfalls. Thoughts?

[yueyinqiu]

Oh I didn't even consider such complicated scenarios.

And I'd like to suggest another case between the overload 2 and 3, that is, the type has no [MagicTable] to indicate database and store, but has [MagicPrimaryKey] and etc. to indicate the store schema. (Mayeb we need a new attribute for it, or just reuse MagicTable)

[magiccodingman]

Yup haha, that's why when you said we should support dynamic object I broke out sweating a bit haha! But I think my proposed resolution resolves this perfectly. But I also had wayyy too much fun today building some pretty crazy stuff. I've been working too hard recently so I decided to chill and have fun with this project.

the MagicCodingMan/SyntaxRefactor branch is some insight into what's going on. Tons of changes today :) But I'm about to hop off for the night again. Was fun building a LINQ to IndexDB true translation layer.