Code

Parquet Modular decryption support#6637

Code

Open

rok wants to merge 70 commits intoapache:mainapache/arrow-rs:mainfrom rok:decryption-basics-forkrok/arrow-rs:decryption-basics-forkCopy head branch name to clipboard

+1,596-91

Commits

Commits on Feb 17, 2025

• first commit

  

  ggershinsky

  authored and

  rok

  committed
  9f240bd

  Copy full SHA for 9f240bd

• Use ParquetMetaDataReader

  rok

  committed
  3c1ca4f

  Copy full SHA for 3c1ca4f

• Fix CI

  rok

  committed
  a1bf0ea

  Copy full SHA for a1bf0ea

• test

  rok

  committed
  d752073

  Copy full SHA for d752073

• save progress

  rok

  committed
  8e2e118

  Copy full SHA for 8e2e118

• work

  rok

  committed
  be10eb3

  Copy full SHA for be10eb3

• Review feedback

  rok

  committed
  46910b2

  Copy full SHA for 46910b2

• page decompression issue

  rok

  committed
  e8e5df2

  Copy full SHA for e8e5df2

• add update_aad

  rok

  committed
  8501cf9

  Copy full SHA for 8501cf9

• Change encrypt and decrypt to return Results

  

  adamreeve

  authored and

  rok

  committed
  1a8fd94

  Copy full SHA for 1a8fd94

• Use correct page ordinal and module type in AADs

  

  adamreeve

  authored and

  rok

  committed
  b5a7336

  Copy full SHA for b5a7336

• Tidy up ordinal types

  

  adamreeve

  authored and

  rok

  committed
  968209f

  Copy full SHA for 968209f

• Lint

  rok

  committed
  a2507c5

  Copy full SHA for a2507c5

• Fix regular deserialization path

  rok

  committed
  3a5e8bb

  Copy full SHA for 3a5e8bb

• cleaning

  rok

  committed
  ffd4a7e

  Copy full SHA for ffd4a7e

• Update data checks in test

  

  adamreeve

  authored and

  rok

  committed
  22b2abb

  Copy full SHA for 22b2abb

• start non-uniform decryption

  rok

  committed
  9f58792

  Copy full SHA for 9f58792

• Add missing doc comments

  

  adamreeve

  authored and

  rok

  committed
  e6e056a

  Copy full SHA for e6e056a

• Make encryption an optional feature

  

  adamreeve

  authored and

  rok

  committed
  cca1155

  Copy full SHA for cca1155

• Handle when a file is encrypted but encryption is disabled or no decryption properties are provided

  

  adamreeve

  authored and

  rok

  committed
  952892d

  Copy full SHA for 952892d

• Allow for plaintext footer

  rok

  committed
  8aa8ba4

  Copy full SHA for 8aa8ba4

• work

  rok

  committed
  490e153

  Copy full SHA for 490e153

• Fix method name

  

  adamreeve

  authored and

  rok

  committed
  6763ee9

  Copy full SHA for 6763ee9

• work

  rok

  committed
  af6c589

  Copy full SHA for af6c589

• Minor

  rok

  committed
  9cf130d

  Copy full SHA for 9cf130d

• work

  rok

  committed
  6014acd

  Copy full SHA for 6014acd

• work

  rok

  committed
  2f09a88

  Copy full SHA for 2f09a88

• work

  rok

  committed
  9104ab5

  Copy full SHA for 9104ab5

• Fix reading to end of file

  

  adamreeve

  authored and

  rok

  committed
  3b3b75a

  Copy full SHA for 3b3b75a

• Refactor tests

  

  adamreeve

  authored and

  rok

  committed
  40d3c21

  Copy full SHA for 40d3c21

• Fix non-uniform encryption configuration

  

  adamreeve

  authored and

  rok

  committed
  3f7e841

  Copy full SHA for 3f7e841

• Don't use footer key for non-encrypted columns

  

  adamreeve

  authored and

  rok

  committed
  bf4df8a

  Copy full SHA for bf4df8a

• Rebase and cleanup

  rok

  committed
  135eef2

  Copy full SHA for 135eef2

• Cleanup

  rok

  committed
  4617870

  Copy full SHA for 4617870

• Cleanup

  rok

  committed
  397d37b

  Copy full SHA for 397d37b

• Cleanup

  rok

  committed
  3d3bfd8

  Copy full SHA for 3d3bfd8

• Cleanup

  rok

  committed
  e53306e

  Copy full SHA for e53306e

• Cleanup

  rok

  committed
  95888bc

  Copy full SHA for 95888bc

• Cleanup

  rok

  committed
  ed1bb3c

  Copy full SHA for ed1bb3c

• lint

  rok

  committed
  ee2cbed

  Copy full SHA for ee2cbed

• Remove encryption setup

  rok

  committed
  fbd23cb

  Copy full SHA for fbd23cb

• Fix building with ring on wasm

  rok

  committed
  e050751

  Copy full SHA for e050751

• file_decryptor into a seperate module

  rok

  committed
  62ac361

  Copy full SHA for 62ac361

• lint

  rok

  committed
  8ab9a43

  Copy full SHA for 8ab9a43

• FileDecryptionProperties should have at least one key

  rok

  committed
  febbe83

  Copy full SHA for febbe83

• Move cyphertext reading into decryptor

  rok

  committed
  e5a788e

  Copy full SHA for e5a788e

• More tidy up of footer key handling

  

  adamreeve

  authored and

  rok

  committed
  423411d

  Copy full SHA for 423411d

• Get column decryptors as RingGcmBlockDecryptor

  

  adamreeve

  authored and

  rok

  committed
  187e7de

  Copy full SHA for 187e7de

• Use Arc<dyn BlockDecryptor>

  

  adamreeve

  authored and

  rok

  committed
  65cebbe

  Copy full SHA for 65cebbe

• Fix file metadata tests

  

  adamreeve

  authored and

  rok

  committed
  53e554e

  Copy full SHA for 53e554e

• Handle reading plaintext footer files without decryption properties

  

  adamreeve

  authored and

  rok

  committed
  55e55ce

  Copy full SHA for 55e55ce

• Split up encryption modules further

  

  adamreeve

  authored and

  rok

  committed
  98cc63e

  Copy full SHA for 98cc63e

• Error instead of panic for AES-GCM-CTR

  

  adamreeve

  authored and

  rok

  committed
  0ff9404

  Copy full SHA for 0ff9404

• load_async

  rok

  committed
  c6d4dca

  Copy full SHA for c6d4dca

• new_with_options

  rok

  committed
  dc19abc

  Copy full SHA for dc19abc

• Add tests

  rok

  committed
  329a613

  Copy full SHA for 329a613

• get_metadata

  rok

  committed
  00aa47a

  Copy full SHA for 00aa47a

• Add CryptoContext to async_reader

  rok

  committed
  fb6cdbc

  Copy full SHA for fb6cdbc

• Add row_group_ordinal to InMemoryRowGroup

  rok

  committed
  aa44408

  Copy full SHA for aa44408

• Adjust docstrings

  rok

  committed
  497abb3

  Copy full SHA for 497abb3

• Apply suggestions from code review

  Show description for 16e9efe
  

  rok

  and

  adamreeve

  committed
  16e9efe

  Copy full SHA for 16e9efe

• Review feedback

  rok

  committed
  95a3097

  Copy full SHA for 95a3097

• move file_decryption_properties into ArrowReaderOptions

  rok

  committed
  1e73b25

  Copy full SHA for 1e73b25

• make create_page_aad method of CryptoContext

  rok

  committed
  e3e3163

  Copy full SHA for e3e3163

• Review feedback

  rok

  committed
  105c3e9

  Copy full SHA for 105c3e9

• Infer ModuleType in create_page_aad

  rok

  committed
  a8a204e

  Copy full SHA for a8a204e

• add create_page_header_aad

  rok

  committed
  fb3b6b0

  Copy full SHA for fb3b6b0

• Review feedback

  rok

  committed
  11c4e7a

  Copy full SHA for 11c4e7a

• Update parquet/src/arrow/async_reader/store.rs

  Show description for 815e35d
  

  rok

  and

  etseidl

  committed
  815e35d

  Copy full SHA for 815e35d

• Review feedback

  rok

  committed
  d3df0ab

  Copy full SHA for d3df0ab