WIP

Author: habedi

README.md

@@ -31,17 +31,14 @@ frequently.
 Gaggle tries to help simplify this process by hiding the complexity and letting you work with datasets directly inside
 an analytical database like DuckDB that can handle fast queries.
 
-> [!NOTE]
-> Gaggle is similar of [Hugging Face extension]() for DuckDB.
-> Although, Kaggle datasets are not directly accessible from a remote file system, like Hugging Face datasets.
-
 ### Features
 
-- Has a simple API (just a handful of SQL functions)
-- Allows you search, download, update, and delete Kaggle datasets directly from DuckDB
-- Supports datasets made of CSV, JSON, and Parquet files
+- Has a simple API to interact with Kaggle datasets from DuckDB
+- Allows you to search, download, and read datasets from Kaggle
+- Supports datasets that contain CSV, Parquet, JSON, and XLSX files
 - Configurable and has built-in caching support
-- Thread-safe and memory-efficient
+- Thread-safe, fast, and has a low memory footprint
+- Supports dataset versioning and update checks
 
 See the [ROADMAP.md](ROADMAP.md) for planned features and the [docs](docs) folder for detailed documentation.
 
@@ -94,35 +91,34 @@ make release
 #### Trying Gaggle
 
 ```sql
--- Load the Gaggle extension
-load 'build/release/extension/gaggle/gaggle.duckdb_extension';
+-- Load the Gaggle extension (only needed if you built from source)
+--load 'build/release/extension/gaggle/gaggle.duckdb_extension';
 
--- Set your Kaggle credentials (or use `~/.kaggle/kaggle.json`)
+-- Manually, set your Kaggle credentials (or use `~/.kaggle/kaggle.json`)
 select gaggle_set_credentials('your-username', 'your-api-key');
 
 -- Get extension version
 select gaggle_version();
 
--- Prime cache by downloading the dataset locally (optional, but improves first-time performance)
-select gaggle_download('habedi/flickr-8k-dataset-clean');
-
 -- List files in the downloaded dataset
+-- (Note that if the datasets is not downloaded yet, it will be downloaded and cached first)
 select *
 from gaggle_ls('habedi/flickr-8k-dataset-clean') limit 5;
 
--- Read a Parquet file from local cache using a prepared statement (no subquery in function args)
+-- Read a Parquet file from local cache using a prepared statement
+-- (Note that DuckDB doesn't support subquery in function arguments, so we use a prepared statement)
 prepare rp as select * from read_parquet(?) limit 10;
 execute rp(gaggle_file_paths('habedi/flickr-8k-dataset-clean', 'flickr8k.parquet'));
 
--- Use the replacement scan to read directly via kaggle: URL
+-- Alternatively, we can use a replacement scan to read directly via `kaggle:` prefix
 select count(*)
 from 'kaggle:habedi/flickr-8k-dataset-clean/flickr8k.parquet';
 
 -- Or glob Parquet files in a dataset directory
 select count(*)
 from 'kaggle:habedi/flickr-8k-dataset-clean/*.parquet';
 
--- Optionally, check cache info
+-- Optionally, we check cache info
 select gaggle_cache_info();
 
 -- Enforce cache size limit manually (automatic with soft limit by default)

ROADMAP.md

@@ -39,7 +39,7 @@ It outlines features to be implemented and their current status.
     * [x] CSV and TSV file reading.
     * [x] Parquet file reading.
     * [x] JSON file reading.
-    * [ ] Excel and XLSX file reading.
+    * [ ] Excel (XLSX) file reading.
 * **Querying Datasets**
     * [x] Replacement scan for `kaggle:` URLs.
     * [ ] Virtual table support for lazy loading.
@@ -49,7 +49,7 @@ It outlines features to be implemented and their current status.
 * **Concurrency Control**
     * [x] Thread-safe credential storage.
     * [x] Thread-safe cache access.
-    * [x] Concurrent dataset downloads (with per-dataset serialization to prevent race conditions).
+    * [x] Concurrent dataset downloads.
 * **Network Optimization**
     * [x] Configurable HTTP timeouts.
     * [x] Retry logic with backoff for failed requests.