-
Notifications
You must be signed in to change notification settings - Fork 873
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Parquet Modular decryption support #6637
Open
rok
wants to merge
70
commits into
apache:main
Choose a base branch
from
rok:decryption-basics-fork
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
70 commits
Select commit
Hold shift + click to select a range
9f240bd
first commit
ggershinsky 3c1ca4f
Use ParquetMetaDataReader
rok a1bf0ea
Fix CI
rok d752073
test
rok 8e2e118
save progress
rok be10eb3
work
rok 46910b2
Review feedback
rok e8e5df2
page decompression issue
rok 8501cf9
add update_aad
rok 1a8fd94
Change encrypt and decrypt to return Results
adamreeve b5a7336
Use correct page ordinal and module type in AADs
adamreeve 968209f
Tidy up ordinal types
adamreeve a2507c5
Lint
rok 3a5e8bb
Fix regular deserialization path
rok ffd4a7e
cleaning
rok 22b2abb
Update data checks in test
adamreeve 9f58792
start non-uniform decryption
rok e6e056a
Add missing doc comments
adamreeve cca1155
Make encryption an optional feature
adamreeve 952892d
Handle when a file is encrypted but encryption is disabled or no decr…
adamreeve 8aa8ba4
Allow for plaintext footer
rok 490e153
work
rok 6763ee9
Fix method name
adamreeve af6c589
work
rok 9cf130d
Minor
rok 6014acd
work
rok 2f09a88
work
rok 9104ab5
work
rok 3b3b75a
Fix reading to end of file
adamreeve 40d3c21
Refactor tests
adamreeve 3f7e841
Fix non-uniform encryption configuration
adamreeve bf4df8a
Don't use footer key for non-encrypted columns
adamreeve 135eef2
Rebase and cleanup
rok 4617870
Cleanup
rok 397d37b
Cleanup
rok 3d3bfd8
Cleanup
rok e53306e
Cleanup
rok 95888bc
Cleanup
rok ed1bb3c
Cleanup
rok ee2cbed
lint
rok fbd23cb
Remove encryption setup
rok e050751
Fix building with ring on wasm
rok 62ac361
file_decryptor into a seperate module
rok 8ab9a43
lint
rok febbe83
FileDecryptionProperties should have at least one key
rok e5a788e
Move cyphertext reading into decryptor
rok 423411d
More tidy up of footer key handling
adamreeve 187e7de
Get column decryptors as RingGcmBlockDecryptor
adamreeve 65cebbe
Use Arc<dyn BlockDecryptor>
adamreeve 53e554e
Fix file metadata tests
adamreeve 55e55ce
Handle reading plaintext footer files without decryption properties
adamreeve 98cc63e
Split up encryption modules further
adamreeve 0ff9404
Error instead of panic for AES-GCM-CTR
adamreeve c6d4dca
load_async
rok dc19abc
new_with_options
rok 329a613
Add tests
rok 00aa47a
get_metadata
rok fb6cdbc
Add CryptoContext to async_reader
rok aa44408
Add row_group_ordinal to InMemoryRowGroup
rok 497abb3
Adjust docstrings
rok 16e9efe
Apply suggestions from code review
rok 95a3097
Review feedback
rok 1e73b25
move file_decryption_properties into ArrowReaderOptions
rok e3e3163
make create_page_aad method of CryptoContext
rok 105c3e9
Review feedback
rok a8a204e
Infer ModuleType in create_page_aad
rok fb3b6b0
add create_page_header_aad
rok 11c4e7a
Review feedback
rok 815e35d
Update parquet/src/arrow/async_reader/store.rs
rok d3df0ab
Review feedback
rok File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we please also document this new flag here
https://github.com/apache/arrow-rs/tree/main/parquet#feature-flags
Maybe we should update the feature support matrix as well