Questions on upcoming Hail-backend release of Seqr

[tommyli]

Hi Hana,

We're always keen to keep our Seqr instance updated with Broad's. Your team do such great work and we always look forward to seeing new and exciting features every time we merge. I'm aware of some major changes coming with using Hail query as the variant backend store and I've got a few questions on it.

1. Will both Hail-backend and Elasticsearch be supported in the long run or will Hail-backend replace Elasticsearch entirely? I.e. merging codebase with latest trunk/master will require us move to Hail-backend?
2. Is there a migration path from Elasticsearch to Hail-backend? We've kept all of our Hail matrix tables in our instance of Seqr as part of the pipeline-runner Seqr import. Will they be compatible (i.e. the same queryable shape I suppose) with the new Seqr Hail-backend? Or is the only upgrade path is to reload everything?
3. Any documentation (even very rough ones) on how the new Hail-backend works? E.g. how does it query multiple families when they're loaded into separate matrix tables? I presume the motivation for this change was motivated by performance, scalability and/or costs? How have these metrics improved?
4. Any other breaking changes local Seqr installs should be aware of? Our codebase is probably a few months behind.

[hanars]

Hi Tommy

At a high level, we have not had any external communication about the hail backend because we do not have the bandwidth to make it available yet, and we intend to continue to support the existing seqr infrastructure until at least 2025. We plan to communicate out more details about support and migration for local seqr installations plans in the first quarter of 2024. I will answer your questions as best I can for now, but be aware that plans may change and these answers may not be final. We do not recommend making any changes to your seqr instance to utilize the new backend at this time

1. In the long run the hail backend will fully replace elasticsearch. This will not happen until 2025
2. The new loading pipeline will accept either VCFs or those matrix tables as inputs, but the intermediate matrix tables output from the v2 pipeline are not themselves the new v3 database. To migrate to the new backend, you will need to reload all your data using the new loading pipeline.
3. There is currently no documentation because this backend is not ready yet. There will be documentation available before we release this backend to the rest of the seqr community. A primary motivation is scalability, and our data size has currently dropped from 13Tb in elasticsearch to <500Gb in the new backend, with optimizations that mean that adding new projects/ data to seqr will no longer be a linear size increase
4. We maintain a changelog that is intended to include all major breaking changes and the dates of their release