Release 1.0.0#184
Open
FusRoman wants to merge 40 commits into
Open
Conversation
* add architecture for min-cost flow algorithm * add architecture for min-cost flow algorithm * graph construction done and little bit optimized, must had more tests * add tests for flow graph * first version of the graph inter night resolution * add new crates : fink-fat-engine, refactoring of the rust codebase into a new pure rust crates; seeds done * add score in engine * add modifs from the engine * add edges and associated generation method * add night ingestion for graph struct * optimise edge generation and graph structure * add component computation, preparation before adding solvers * add incremental DSU for fast connected component resolution with local recomputing for deactivated edges * move edge_id into its dedicated file * add csr adjacency to speed up some computation * add components statistics and move components in its own folder * add solver manager, solver plan generation and the trivial solver, add also the struct managing the track hypothesis * add min_cost_flow solver and refactor trivial solver * add min_cost_flow solve call in run_plan method of the solver_manager * add ztf alert loader in the eval crate * add ground truth trajectory id with the alert store * add lib.rs, add one example wich show how to load ztf alerts into an alert store * add seed generation et metrics computation in preparation of the seeding evaluation * add tests for the metrics module * add display for metrics, add an example of the metric display, add the missing test file of metrics * add plot file for seeding plots * first results on seeding perf evaluation * add posthoc diagnotics on pairs generation seeding * add one plot * change max-sep parameter for a max angular speed parameter in the seeding of fink-fat-engine. add a first seeding evaluation reports on the seeding parameters * change the --only-truth option for an enum with three mode: oracle, fink and all allowing to load a subset of alerts depending of the context * refine the alert loading filters * add a new evalutation binary looking for the seeding performances over multiple nights * add plots for the multi-night evaluation * add multi-nights report * refactoring of the engine config, allow to load the config from file or environment variable * add possibility to specify units in the fink-fat-engine config file * refactor and improve binaries * rename optimizer file * add display for seed_node * add missing file for display seed_node * improve photometry handling * a lot of work on fink_fat_eval * add comments for graph_metrics * add a new oracle seed generation function * add function to compute score_edges * improve ScoreEdge documentation * add a frozen version of the seeds and edges * pipeline for generating frozen_pairs and evaluating score * implemnt optimisation pipeline * add logs * fix report path * add cli argument for optuna optimization * refactor and improve edge generation in the engine * graph edge generation improved and optimized, need to disable some big part of the code related to solver * fix config, remove duplicated entries * add diagnostics edge generation * improve edge diagnostic pipeline * improve performances of score_edges * improve fink-fat-engine performances * continue work on trying to optimize scoring * start to work on ml inference for the engine * add edge classifier project * big work on enabling ml classifier for false edge filtering * remove agent.md * implement onnx inference, remove useless features, add a fink_fat global struct for states * implement ml prediction and topk pruning for edges * make ml filtering and topk pruning optional for edge generation * add negative log likelihood score for graph solver * add parallel edge generation with batch using rayon, add thread_local and rayon dependencies, thread_local used to share ml model across threads safely * update and improve edge documentation, remove unused edge files * remove seddnode_ids * remove bincode unmaintained deps, remove unused code, prepare for on-disk storage * remove alertid and introduce lifetime for seeds and alerts * working on disk persistence * Persistence work in progress (#175) * working on disk persistence * work on the persistence for the data structure across night execution * add graph owned struct for disk persistence * add envelope for disk persistence * add edge journal persistence, refactor disk persistence, almost finished * add dedicated error for persistence, add a with_borrowed for pipeline execution * restore trivial solver, stil refactoring to do and restoring mcf and blob_breaker * move a function * small refactor * refactor components * refactor components * add node degree in and out in the graph struct * change the design and logics of the trivial solver, should be better know but still need to be tested * improve components and trivial solver * remove unused attributes * add doc and refactor solver name * improve config, improve doc, fix config test * add track_id, rename mjd_tt type * refactor type name, add track conversion to outfit struct for orbit estimation * add tests for envelope and improve docs * big work on fink-fat pipeline, made ingest_alert, build_seed and build_edge * fix night window issues * fix pairingmode contains * [WIP] solve stage run * Refactor from lifetime to index (#176) * refactor alert and alert_store * move seeds * restore ids for alerts in seeds * restore ids for seeds in edge * fix benches * add tests for union-find and seed_index * remove one wrong test * add tests for connected components * add tests for the boundedbeam solver * add synthetic alert generation for tests, add integration test for stage IngestAlert and BuildSeeds * add integration tests for build_edge stage * improve doc for alerts * update doc for edge features, add mathematics formula rendered through katek * clean whole project * improve code and docs * add integration tests for the solver stage run, refactor integration tests * add fit orbit stage and associated integration tests * add orbit and track persistence on disk within parquet files * add persistence stage implementation * add persistence test and refactor integration test * add compaction config entry for graph and integration test for graph compaction * remove duplicated deps * add ml model test file, change test to use it * clean workspace deps, fix license file * apply fmt, fix cargo.toml * fix cargo.toml * rewrite ci, no more python * rewrite ci, no more python * fix ci test, number of thread should be 1 to avoid race condition on the ephemeris file written by outfit * Issue/178/cli 1.0 (#179) * add support for file compression, add progress bar as a cli option * run cargo fmt * clippy fixes * fix doc * fix ci for coverage report * fix non positive log score, add tests for log score, make edge new return a result instead of assert * fix error message * fix edge_builder progress bar, fix manifest test * fix multiple seed building over night of the build_edge stage for single night mode * improve progress bar of loading data stage * optimize save edge compaction * optimize loading and saving by doing async and concurrency * add solver and fit orbit stage to the command line * add new cost function for edges * add new cost function for edges * precision on doc * change pre-commit to trigger workspace doc * add logging * add logging and tests * add new config parameters to control edge generation * fix chi2 position blow-up, replace gnomonic residuals for spherical residuals * add data volume logs * deactivate edges after successfull fit orbit, add unit and integration tests for that * fix a huge bug regarding edge generation * optimize memory consumption for edges * remove the usage of pairingwindow and add a ingest_night attribute in the runtime_state instead (#181) * big refactor of the fink-fat-eval crate (#183) * big refactor of the fink-fat-eval crate * update gitignore * enabling seed parallel construction * add seeding evaluation * add plots for seeding * fix an overflow error bug * add edges stats * move dev profile in the workspace cargo toml * improve edge generation * forgotten files * optimize edge generation * add tests, add solver evaluation * add an edge features exporter * fix docs and missing system dep * clean edge ml prediction project, add a ml model evaluation in rust * add rust ml eval plots * add optuna optimization * add fink-fat-eval readme * add evaluation compilation profile note in readme * add a new edges generation benchmark * change ml behavior in the pipeline, ml is not a ranking anymore, it is now a filtering * add engine readme, remove units file, put content in lib.rs * add main doc page * refactor fink fat main * skip code coverage for fink-fat-eval * update agents and skills * improve doc * update readme and make coverage ony on ci * add licence badge * unified cargo metadata across workspace * add version for fink-fat-engine dep * update changelog * add integration tests for fink-fat cli
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #184 +/- ##
===========================================
+ Coverage 54.71% 91.30% +36.58%
===========================================
Files 24 68 +44
Lines 2659 13963 +11304
===========================================
+ Hits 1455 12749 +11294
- Misses 1204 1214 +10
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.