feat: improve block selection in stacks-inspect
#6735
Conversation
Improves the UX by merging `validate-naka-block` and `validate-block,` improves the lookup time by optimizing the queries, and improves the output for a better experience. Adds `--early-exit` flag to exit on first error instead of completing all blocks.
Codecov Report❌ Patch coverage is
❌ Your project check has failed because the head coverage (40.72%) is below the target coverage (80.00%). You can increase the head coverage or adjust the target coverage.
Additional details and impacted files@@ Coverage Diff @@
## develop #6735 +/- ##
============================================
- Coverage 70.05% 40.72% -29.33%
============================================
Files 578 578
Lines 358373 358445 +72
============================================
- Hits 251064 145985 -105079
- Misses 107309 212460 +105151
... and 470 files with indirect coverage changes Continue to review full report in Codecov by Sentry.
🚀 New features to boost your workflow:
|
| fn collect_block_entries_for_selection( | ||
| db_path: &str, | ||
| selection: &BlockSelection, | ||
| chainstate: &StacksChainState, | ||
| ) -> Vec<BlockScanEntry> { |
There was a problem hiding this comment.
sanity check: isn't this method currently returning 2n blocks when called with BlockSelection::Last and BlockSelection::First? seems like it will get the first N and last N blocks from both tables
There was a problem hiding this comment.
You're right. I decided to just remove first, since that doesn't really seem like a useful scenario any way, and then fixed last.
contrib/stacks-inspect/src/lib.rs
Outdated
| panic!("Failed to open staging blocks DB at {staging_blocks_db_path}: {e}"); | ||
| }); | ||
| let sql = format!( | ||
| "SELECT index_block_hash, consensus_hash, anchored_block_hash, height FROM staging_blocks {clause}" |
There was a problem hiding this comment.
nit: since we are only consuming index_block_hash, maybe we can simplity the query as SELECT index_block_hash FROM staging_blocks {clause}
There was a problem hiding this comment.
Yeah, I had that there originally because I needed other fields when I was also reusing the new structure in other places, but then ended up deleting that code. I'll delete these now.
contrib/stacks-inspect/src/lib.rs
Outdated
| for (i, index_block_hash) in index_block_hashes.iter().enumerate() { | ||
| if i % 100 == 0 { | ||
| println!("Checked {i}..."); | ||
| let sql = format!("SELECT index_block_hash, height FROM nakamoto_staging_blocks {clause}"); |
There was a problem hiding this comment.
nit: since we are only consuming index_block_hash, maybe we can simplity the query as SELECT index_block_hash FROM nakamoto_staging_blocks {clause}
contrib/stacks-inspect/src/lib.rs
Outdated
| let mut seen = HashSet::new(); | ||
| let mut entries = Vec::new(); |
There was a problem hiding this comment.
suuuuper nit: These two variables hold over 5 million index_block_hash values each when the function is called with BlockSelection::All, stored as 64 bytes hex strings. ~320 MB for each variable (plus additional overhead from the data structures). It's not a concern right now, but we could reduce memory usage by about half if we move the
let index_block_hash = StacksBlockId::from_hex(&index_block_hash_hex).unwrap();processing inside this function and avoid keeping the hex strings around.
There was a problem hiding this comment.
as always, feel free to skip the changes for the nits
There was a problem hiding this comment.
This is a good point. Also, a good reason to just remove the seen tracking altogether.
Remove the `seen` tracking since it doesn't really gain us much and uses a lot of space when processing many blocks.
3 participants
Improves the UX by merging
validate-naka-blockandvalidate-block,improves the lookup time by optimizing the queries, and improves the output for a better experience. Adds--early-exitflag to exit on first error instead of completing all blocks.