Introducing a New Era for LINQ to IndexedDB (Large Scale Refactor) #64
Replies: 12 comments 7 replies
-
Pretty massive work has been done on my end today. Will likely drop a pretty massive update likely in the next 1-3 days. Extremely powerful cursor support has been added. No more in memory processing. I even figured out how to break Blazor itself and build real time yielded responses from the javascript layer! That feature has to become a more modular shared feature on a future blazor toolkit of mine because that is so OP. Tons of new abilities as well like NOT operations like NOT greater than, NotContains, and so much more! The memory performance on this new update will be absolutely massive. There will be nothing you shouldn't be able to efficiently query. Lots of new LINQ to IndexDB capabilities as well that I've found nobody in the entire space both in Blazor and IndexDB community in general who conceptualized something like this. This update will be so fun! |
Beta Was this translation helpful? Give feedback.
-
I just wrapped up 98% of my refactor, and I did a lot of research and I may have essentially built something that’s never existed before: true LINQ to IndexedDB. Not just for Blazor, but period. After a ton of research, I might be the first to do this, which is pretty cool! Blazor gets it first, but I’ll be sharing the universal layer with the JS community later. Dropping GetAllAsync SupportCurious what you both think on this. But I'm wanting to drop It’s unnecessary. The new system already does it better: IMagicQuery<Person> personQuery = manager.Query<Person>();
List<Person> people = await personQuery.ToListAsync(); This project is LINQ to IndexedDB, not LINQ-to-in-memory. LINQ to SQL doesn’t have a IMagicQuery<Person> personQuery = manager.Query<Person>();
List<Person> people = await personQuery.OrderByDescending(x => x._Age).ToListAsync(); Keeping Honestly, I’m set on fully removing it—it doesn’t fit the LINQ-like syntax, would be an unnecessary maintenance burden, and would likely become deprecated or buggy over time. That said, if anyone has strong counterpoints, I’m all ears. But I’ve already cut it from the refactor and just wanted to explain why. Sneak PeekSo the new LINQ to IndexDB will be firing on your behalf multiple extremely optimized queries. It utilizes a single dynamically created Cursor to catch any and all normally unsupported operations. You can utilize the As for the new The new system will perfectly be akin to LINQ to SQL, but with LINQ to IndexDB flavors. THings like Skip and Take operations are backwards, but that's because it's IndexDB. And I couldn't really "fix" that without breaking other logic that didn't make sense anymore in edge cases. So just like how LINQ to SQL follows different syntax and results than LINQ in memory. We as well have differences when in the LINQ to IndexDB mode. Everything is enforced by interfaces to keep you safe and knowledgeable of the operations you can use. And just a note that the IMagicQuery and IMagicCursor (aka using where or cursor) allows multi appending where statements for deferred execution identical to LINQ to SQL. Additionally the IMagicQuery will provide you with Cursor support as backups when it can't complete operations on your request with only indexes. Allowing a multi index and cursor environment for the most optimized search algorithms to be executed. Building dynamically created IndexDB optimized queries on your behalf. The C# layer also flattens and optimizes your nested OR statements allowing complex OR queries moving forward. Oh and another cool feature about the cursor metadata logic I developed. Not only is it efficient, but it perfectly replicates the ordering of database lookups. So results will always be returned as you wanted. The new LINQ to IndexDB system is a much larger and more optimized algorithm that truly translates your desires to the best and most optimized LINQ to IndexDB possible! It's hooked up for future compound key support too. 🛠 Basic Query ExampleIMagicQuery<Person> query = manager.Query<Person>();
// Fetch people older than 30, ordering by Age
IMagicQuery results = await query.Where(x => x.Age > 30)
.OrderBy(x => x.Age)
.Take(5)
.ToListAsync();
// Cursor-based query (allows unlimited operations but runs in-memory)
IMagicCursor cursorResults = await query.Cursor(x => x.Name.StartsWith("A"))
.OrderBy(x => x.Name)
.Take(3)
.Skip(2)
.OrderBy(x => x.Name)
.Take(3)
.ToListAsync(); The showcase of the cursor operation shows how you can utilize the Cursor to do more complex not normally allowed operations in IndexDB. Which you can't do that for the IMagicQuery as there's no way that indexed Queries can fire on your behalf if you did that. So using IMagicQuery is preferred as it'll use fallback cursors but you have way better odds of more optimized queries and performance if you utilize the IMagicQuery instead. But hey, why not make IMagicCursor? Why not just give the ability to unlock the power at your digression? 📌 Query Path Overview
🔹 Key Features
🔹 IndexedDB’s Reverse Take & SkipIndexedDB processes // Standard LINQ-style (Relational DB)
var result = await query.Take(3).Skip(2).ToListAsync(); // Items 3, 4, 5
// IndexedDB LINQ (Reversed Order)
var result = await query.Take(3).Skip(2).ToListAsync(); // Same result, different execution order
🚀 SummaryThis will be a pretty major update that'll unlock tons of functionality. I'm at the final stretch of optimizations and adding specific capabilities. As I've been having a lot of fun with this before I got to turn into "grandpa" mode and venture off on my other projects i need to focus on. But being the first to create a true LINQ to IndexDB translation layer was too fun not to finish. Anyways I've got a lot of unit tests working. Just have a few niche scenarios right now where I built really cool ways to allow operation indexDB support when not normally allowed, and it's still firing cursors instead because my new algorithm is falsely flagging it as invalid. Which would be correct under normal circumstances, but I've allowed the capability to exist so just have some polishing to do, but I've got all these unit tests and will finish roughly double them tomorrow. As I need to validate multiple versions of how this LINQ to IndexDB can fire: |
Beta Was this translation helpful? Give feedback.
-
Shoot. I should drop support for await query.Where(x => x.Id == 3).FirstOrDefault(); So I think I'm putting |
Beta Was this translation helpful? Give feedback.
-
oh and @yueyinqiu I've put a lot of thought into how to allow dynamic objects, creation, and runtime changes. Going to make a proposal to you after I launch this update to see if you like the direction. The serializer that was built is actually the biggest challenge. Caching cannot occur on dynamically created runtime objects, so I've built avenues for easy features we can build later to implement support. |
Beta Was this translation helpful? Give feedback.
-
Okay I figured it out. You'll be able to use the same model in different databases with the same or different schema's once this new update drops. I think I'll have full compound key support as well coming up. Having trouble with certain tests at the moment because it's actually working too fast. I can't get certain failures I'm trying to force. I'm pretty close to having compound keys working in the LINQ translation. But I'm also stuck because I have to refactor the factory since the new LINQ system doesn't need it at all anymore. Hoping to be done by tomorrow or the day after. |
Beta Was this translation helpful? Give feedback.
-
Just an update on the new changes coming up. Not only will this be pretty major, even how you utilize the library from the start will be changed. I've built in automated migration on your behalf, validation testing in debug startup for migrations. And a lot more. You can effectively just add indexes and combined queries to your hearts content without any issue moving forward as well. THe system will auto deploy, handle and clean up client side tables after migration automatically. Repository controlMoving forward there will be repository control. You must create this in your own project with the appended interface like so: public class IndexDbContext : IMagicRepository
{
public static readonly IndexedDbSet Client = new ("Client");
public static readonly IndexedDbSet Employee = new ("Employee");
public static readonly IndexedDbSet Animal = new ("Animal");
} There will be a bit more for migration control, but it's all going to be documented later. Goal is to think tables first, not database first. It's a really weird concept, but after a lot of testing, this is 100% how IndexedDB should work. But when initiating your tables to a C# class, instead of an appended MagicTable attribute on the class you moving forward must append an interface that enforces good practice like: public class Person : IMagicTable<DbSets>
{
public int GetVersion() => 1;
public string GetTableName() => "Person";
public IndexedDbSet GetDefaultDatabase() => IndexDbContext.Client;
public DbSets Databases { get; } = new();
public sealed class DbSets
{
public readonly IndexedDbSet Client = IndexDbContext.Client;
public readonly IndexedDbSet Employee = IndexDbContext.Employee;
} The old system from the original fork set versions really annoyingly and it was too code first mentality. This isn't code first, this isn't a normal database, Databases go to tables, not the other way around is what I kind of realized. So it may seem really non intuitive but I promise, it's a breeze and for the best! New capabilities@yueyinqiu This one is for you bud! // Targets the default database automatically if called without any parameters.
IMagicQuery<Person> clientDbQuery = await _MagicDb.Query<Person>();
// Choose an assigned database to Person with strictly typed enforced connected Db's.
IMagicQuery<Person> employeeDbQuery = await _MagicDb.Query<Person>(x => x.Databases.Client);
// Highly not suggested, but you're allowed to target databases not assigned to the Person table
IMagicQuery<Person> animalDbQuery = await _MagicDb.Query<Person>(IndexDbContext.Animal);
// DO NOT DO THIS! I only am allowing this for maximum flexibility but this is very dangerous.
IMagicQuery<Person> unassignedDbQuery = await _MagicDb.QueryOverride<Person>("DbName", "SchemaName"); You can assign your repository variables of databases and their names directly to tables. Hard typing the specifications of which databases connect to where. Discontinued use of strings when assigning to prevent mistakes or error. System allows you to default automatically to the default DB for simple setups. Then you can utilize hard typed controlled database names as you see in Utilizing the final overrides to specify the query is the Person table in the Animal database is dangerous, but allowed. And full Override support is allowed for very dynamic custom situations, but no validations, automatic deployment or any other support is built into those use cases. Override is considered terrible practice and should only be done under very weird circumstances. MigrationsI will release more on how migrations work later, but it's going to be really pro Blazor. We don't want to put too much in the C# level itself because Blazor WASM gives you absolutely zero choice but to ship every single library, reference, and thing to the front end. Nothing is hidden at all in Blazor. If you have years of migration scripts, that's going to be a major pain in the butt. Migrations will utilize the wwwroot to allow migration scripts to get a table version from 1 point to another be callable as needed. Thus making your front end wasm much cleaner and smaller. Only calling migration scripts when they're needed. Though the migrations will be also built into the repository to specify which versions are breaking points and what the path to your migration is, aka `./IndexDbMigrations/TableMigrations" and you can have it target multiple files or just one because it auto imports dynamically the method assigned to that migration. So you won't need a million migrations either. Plus it will recognize when a table update must occur but no column changes occurred. Thus no migrations needed. The system will blend a lot of abilities only C# Blazor can do. It's going to make Index DB really amazing I promise! I've been having way too much fun resolving all the legendary IndexedDB issues haha. UpdateMigrations are going to be even cooler than what I wrote. I'm so hype |
Beta Was this translation helpful? Give feedback.
-
@yueyinqiu Any suggestions on how to label this release? I'm obviously doing tons of testing, but I'm thinking maybe labeling the next launch as "beta" or "experimental"? Version 1.0.12 will be known as the 1.0 era and the last of that era. I'm thinking of labeling the next launch as 2.0 to clearly separate the difference. Also because the code changes are foundational and it's nearly unrecognizable. |
Beta Was this translation helpful? Give feedback.
-
New SyntaxJust showcasing the new syntax that's very likely going to be the finalized version. Just a showcase as I've been referring to it a lot in opened tickets and discussions. IMagicQuery<Person> personQuery = await _MagicDb.Query<Person>();
/*
full cursor queries for more allowed combinations of additions
to a query that can't be done in indexed query combinations.
By default you can't do take, skip, orderby, then take again or
anything after the skip because there's no way to make any
indexed query at that point. It's an IndexedDB limitation.
But the new Cursor system can still handle this. So the
ordering and additions are unlocked when utilizing
"Cursor" instead of "Where". It's meant to help the user
stay in constraints to build optimized queries. And
be aware when they're making a query that will be
fully in the cursor.
*/
var allTheZacks = await personQuery.Cursor(x => x.Name == "Zack").ToListAsync().OrderBy(x => x._Age)
.Take(10)
.Skip(2)
.OrderBy(x => x._Age
.Take(3));
/*
new Where statement allows significantly better queries
that translates your intent just like LINQ to SQL. Mixing
indexed queries, cursor queries, and combined key/indexed
queries on your behalf. No more memory loading, all occurs
as the truest form of queries.
*/
var multiChainedWheres = personQuery.Where(x => x._Age > 30)
.Where(x => x.TestInt == 9)
.OrderBy(x => x._Age)
.Take(3)
.Skip(2)
.ToListAsync();
// Replacement of the GetById as FirstOrDefault will be properly implemented.
var firstOrResultQuery = personQuery.FirstOrDefault(x => x.Id == 2);
/*
Asynchronous Enumerable return response. As queries
complete they're yielded back in real time. A custom
Blazor Yield has been created for this project to allow
true yielding per item. You can work on content as it's
coming back.
*/
await foreach (var person in personQuery.Where(x => x._Age > 30)
.OrderBy(x => x._Age)
.Take(50)
.AsAsyncEnumerable())
{
Console.WriteLine($"Name: {person.Name}, Age: {person._Age}");
} |
Beta Was this translation helpful? Give feedback.
-
@yueyinqiu Going to have more news on it soon, but I just finished the LINQ translation layer. I did some heavy research as well and it's pretty exciting. Because it seems this library isn't just about to transform, this library will be able to claim that we are The First True LINQ To IndexedDBYup, that's right! I built the first true LINQ to IndexedDB translation layer to ever exist! There's no library that can do what we're about to do. It'll be ultra powerful on Blazor because C# is OP, but I built a universal translation layer for other languages and frameworks where they can plug into the current system. No more API or Memory CallsWhat makes this different is every other library has utilized methods to replace API calls with LINQ like syntax. But what makes LINQ a thing isn't the syntax it's the translation. Standard LINQ will translate in memory operations to your intent. LINQ to SQL translates your intent to SQL. And we now officially translate full predicate intent the same way to LINQ to IndexedDB. I just passed 60+ unit tests and it's supporting significant multi query situations, utilizing a lot of optimization algorithms, and so much more. I'm going to need to write a lot of documentation on this because just like how LINQ doesn't have the same syntax as LINQ to SQL, this will also be true for LINQ to IndexedDB. There's nuances to working with IndexedDB that's different from SQL or memory operations. MigrationsI am going to finish up some documentation but then finish out my migration strategy that I believe will completely reform how IndexedDB migrations work moving forward. Making it all automated, easy, and scaffolded. And migrations between the scaffolded snapshots dynamically generate the migration code and still allow advanced configurability where necessary. Testing, Validation, and MoreI'll keep you updated, but if you have time to spare (no worries if not) I may need some support on the unit tests again. I'm sorry I keep breaking your unit tests, but things have changed so drastically that it wasn't really salvageable. There's still a lot that has gone untested with standard operations. Especially because this library will support now complex combined keys, combined indexes, and more. Lots of self validation built as well. The code will prevent your app from running if you try building a table for example that goes against IndexedDB rules of creating tables. It also has various ways of enforcing best practice queries as well. I got a little obsessed with this recently lol. But I just had a fascinating realization and I couldn't help myself. I had to do it! I wanted to make IndexedDB not suck! And it's about to get so much better! Unit TestsAs for the current unit test project. Once I get some documentation done, I'll make a post here. It'll be important to read documentation before moving forward. As you may have a test work now in one scenario, but fail in a seemingly identical test elsewhere, but that's due to how the translator layer works and which queries it builds and optimizes. So understanding how LINQ to IndexedDB works is important to working on this project moving forward. I'm excited to finally get this released! Update 1 - DocumentationThe primary ReadMe has been altered and points to the v1.0 era as separate legacy documentation: Update 2The basics of LINQ to IndexedDB documentation has been created and released. This'll be very important to read and digest. I'll be working on the fundamentals of LINQ to IndexedDB on how to build optimized queries and how the migrations work later. But this is a good starting point. The rest is in main so I may need some support with tests again to build more validation and start double checking the work I'm building. |
Beta Was this translation helpful? Give feedback.
-
I'm considering dropping an alpha release while I fix up the last of the migration code. I think I'll get the last of the functionality hooked up. Finish up some unit tests and I'll get the first release out. Gonna get the E2e unit tests back up and running as well. |
Beta Was this translation helpful? Give feedback.
-
Okay I may have gone a bit overboard, but I built an entire query engine.... I've also laid the groundwork to make it so I can turn cursor calls in the future into fully indexable queries via absolute black magic. I think I may have become a priest. I have performed I think the most holy of exorcisms on IndexedDB. If you take a look starting at the The amount of optimizations performed as well has been drastic. This project will hyper optimize your queries in ways that normally aren't even possible. This'll be an extremely exciting launch. I've been in contact with some vanilla JS and Typescript developers in another community I am well enough known in. They're aware of this project and will be assisting in reviewing the universal translation layer and the query engine. While I'm mostly hoping to enlist help in building a wrapper around the engine for other languages as well. I'm in the final stage of implementing the last of the new features I'm allowing. Like you can now perform: int[] myArray = { 38, 39 };
await personQuery.Where(x => myArray.Contains(x._Age) And vice versa, it all is handled for you. I'm thinking of implementing a x => x.DateOfBirth.Value.Year >= 2020 Which as of right now, effectively every operation is supported in the query engine. It's just finalizing the Blazor C# wrapper layer right now. This'll be a super fun launch! |
Beta Was this translation helpful? Give feedback.
-
This'll be an extremely feature rich release. |
Beta Was this translation helpful? Give feedback.
-
Introducing a New Era for LINQ to IndexedDB
We are on the cusp of a major transformation—true LINQ to IndexedDB is coming to life! Much like how LINQ to SQL evolved from an inefficient, often-criticized tool in its early days to a cornerstone of modern .NET development, LINQ to IndexedDB is taking its first real steps toward full optimization and scalability.
The Problem:
For too long, IndexedDB implementations in Blazor and JavaScript have been plagued by inefficiencies, requiring excessive memory usage and unnecessary client-side filtering. While Magic.IndexDB has made significant strides in abstracting IndexedDB’s complexities, it has been limited by its early-stage architecture.
But no more.
With the new LINQ to IndexedDB translation layer, we are ushering in a true translation system that efficiently maps C# LINQ expressions directly to the most optimal IndexedDB operations possible. While there is always room for further improvements, this is the first step toward making LINQ to IndexedDB as powerful as LINQ to SQL.
What's Changing?
1. Large-Scale Syntax Overhaul
A complete overhaul of LINQ-to-IndexedDB syntax is inbound, making queries more intuitive and aligned with traditional LINQ patterns.
Before:
After:
Where<T>(x => x.name)
, instead, queries now follow the explicit query instantiation pattern.New Query Initialization:
IQueriable<T>
, automatic execution, and so much more! Await the future release and documentation to see the full guide as I don't want to show too much until it's finalized.2. Automatic Database Management
3. A New Level of Optimization: No More Memory Overhead!
Previously, LINQ-to-IndexedDB suffered from significant performance issues because:
||
) operators in LINQ expressions were split into multiple separate queries.&&
) operators within OR conditions were also executed as distinct queries.What's New?
OR
Query Support in IndexedDB!Where Are We Now?
We are currently in our .NET 4.0-era of LINQ-to-IndexedDB: functional, but still early in its optimization journey. The first phase of refactoring began in v1.0.11, introducing cursor-based queries. But now, with the full LINQ translation layer rewrite, we’re moving into a new era of performance, scalability, and reliability.
This isn’t just about making queries work. This is about making LINQ to IndexedDB something developers will love and rely on—without exploding their browser’s memory.
How Can You Get Involved?
1. Join the Discussion!
We have an active discussion about these changes happening now: GitHub Discussion #60
2. Stay Updated with Documentation Changes
Our current LINQ to IndexedDB documentation is here: Current Docs
##The Future of LINQ to IndexedDB**
Blazor developers, let’s redefine IndexedDB together. Let’s make LINQ to IndexedDB not just possible, but incredible. 🚀
Beta Was this translation helpful? Give feedback.
All reactions