Architecture: Data Lake
Batch
- Storage Transfer Service
- BigQuery Data Transfer Service
- Transfer Appliance
Streaming
- Pub/Sub
Storage decision tree
Data cleansing and normalize
- Cloud Dataprep Data Harvest
- Dataplex ETL
- Dataflow and Cloud Data Fusion for data absorption
Warehouse
- Dataproc and BigQuery
- DataLab -> Vertex AI Workbench
- Looker & Looker Studio
- Data Catalog in Dataplex
- predicate-based search experience for metadata associated with a data entry