Add more Sqllogictest-style tests

Author: habedi

README.md

@@ -12,34 +12,34 @@
 [![Docs](https://img.shields.io/badge/docs-read-blue?style=flat&labelColor=282c34&logo=read-the-docs)](https://github.com/CogitatorTech/gaggle/tree/main/docs)
 [![License](https://img.shields.io/badge/license-MIT%2FApache--2.0-007ec6?style=flat&labelColor=282c34&logo=open-source-initiative)](https://github.com/CogitatorTech/gaggle)
 
-Kaggle Datasets for DuckDB
+Access and Query Kaggle Datasets from DuckDB
 
 </div>
 
 ---
 
 Gaggle is a DuckDB extension that allows you to work with Kaggle datasets directly in SQL queries, as if
 they were DuckDB tables.
-It is written in Rust and uses the Kaggle API to search, download, and manage datasets.
+It is written in Rust and uses the Kaggle API to search, download, and manage Kaggle datasets.
 
 Kaggle hosts a large collection of very useful datasets for data science and machine learning.
 Accessing these datasets typically involves manually downloading a dataset (as a ZIP file),
 extracting it, loading the files in the dataset into your data science environment, and managing storage and dataset
 updates, etc.
-This workflow can be become complex, especially when working with multiple datasets or when datasets are updated
+This workflow can quickly become complex, especially when working with multiple datasets or when datasets are updated
 frequently.
 Gaggle tries to help simplify this process by hiding the complexity and letting you work with datasets directly inside
 an analytical database like DuckDB that can handle fast queries.
 In essence, Gaggle makes DuckDB into a SQL-enabled frontend for Kaggle datasets.
 
 ### Features
 
-- Has a simple API to interact with Kaggle datasets from DuckDB
+- Provides a simple API to interact with Kaggle datasets from DuckDB
 - Allows you to search, download, and read datasets from Kaggle
-- Supports datasets that contain CSV, Parquet, JSON, and XLSX files (XLSX requires DuckDB's Excel reader to be available in your DuckDB build)
-- Configurable and has built-in caching support
+- Supports datasets that contain CSV, Parquet, JSON, and XLSX files
+- Configurable and has built-in caching of downloaded datasets
 - Thread-safe, fast, and has a low memory footprint
-- Supports dataset versioning and update checks
+- Supports dataset updates and versioning
 
 See the [ROADMAP.md](ROADMAP.md) for planned features and the [docs](docs) folder for detailed documentation.
 

ROADMAP.md

@@ -39,7 +39,7 @@ It outlines features to be implemented and their current status.
     * [x] CSV and TSV file reading.
     * [x] Parquet file reading.
     * [x] JSON file reading.
-    * [ ] Excel (XLSX) file reading. (Available when DuckDB is built with the Excel reader; replacement scan routes `.xlsx` to `read_excel`.)
+    * [x] Excel (XLSX) file reading.
 * **Querying Datasets**
     * [x] Replacement scan for `kaggle:` URLs.
     * [ ] Virtual table support for lazy loading.
@@ -66,8 +66,8 @@ It outlines features to be implemented and their current status.
     * [x] Detailed error codes for programmatic error handling.
 * **Resilience**
     * [x] Automatic retry on network failures.
-    * [ ] Graceful degradation when Kaggle API is unavailable.
     * [x] Local-only mode for cached datasets (via `GAGGLE_OFFLINE`).
+    * [ ] Graceful degradation when Kaggle API is unavailable.
 
 ### 6. Documentation and Distribution
 

docs/CONFIGURATION.md

@@ -10,7 +10,7 @@ Gaggle supports configuration via environment variables to customize its behavio
 
 - **Description**: Directory path for caching downloaded Kaggle datasets
 - **Type**: String (path)
-- **Default**: `$XDG_CACHE_HOME/gaggle_cache` (typically `~/.cache/gaggle_cache`)
+- **Default**: `$XDG_CACHE_HOME/gaggle` (typically `~/.cache/gaggle`)
 - **Example**:
   ```bash
   export GAGGLE_CACHE_DIR="/var/cache/gaggle"
@@ -144,9 +144,10 @@ These settings control the wait behavior when a download is already in progress.
   - **Type**: Boolean (`1`, `true`, `yes`, `on` to enable)
   - **Default**: `false`
   - **Effects**:
-    - gaggle_download(...) fails if the dataset isn’t cached.
-    - Version checks use cached `.downloaded` metadata when available; otherwise return "unknown".
-    - Search and metadata calls will still attempt network; consider avoiding them in offline mode.
+    - `gaggle_download(...)` fails if the dataset isn’t cached.
+    - `gaggle_version_info` reports `latest_version` as "unknown" if no cache metadata exists.
+    - `gaggle_is_current` and other version checks use cached `.downloaded` metadata when available.
+    - `gaggle_search` and `gaggle_info` also fail fast in offline mode (no network attempts).
   - **Example**:
     ```bash
     export GAGGLE_OFFLINE=1
@@ -187,7 +188,7 @@ export GAGGLE_HTTP_TIMEOUT=120              # 2 minutes
 export GAGGLE_HTTP_RETRY_ATTEMPTS=5         # Retry up to 5 times
 export GAGGLE_HTTP_RETRY_DELAY_MS=2000      # 2 second initial delay
 export GAGGLE_HTTP_RETRY_MAX_DELAY_MS=30000 # Cap backoff at 30s
-export GAGGLE_LOG_LEVEL=WARN                # Production logging (planned)
+export GAGGLE_LOG_LEVEL=WARN                # Production logging
 
 ## Set Kaggle credentials
 export KAGGLE_USERNAME="your-username"
@@ -202,7 +203,7 @@ export KAGGLE_KEY="your-api-key"
 ```bash
 ## Development setup with verbose logging
 export GAGGLE_CACHE_DIR="./dev-cache"
-export GAGGLE_LOG_LEVEL=DEBUG               ## Detailed debug logs (planned)
+export GAGGLE_LOG_LEVEL=DEBUG               ## Detailed debug logs
 export GAGGLE_HTTP_TIMEOUT=10               ## Shorter timeout for dev
 export GAGGLE_HTTP_RETRY_ATTEMPTS=1         ## Fail fast in development
 export GAGGLE_HTTP_RETRY_DELAY_MS=250       ## Quick retry
@@ -230,10 +231,11 @@ export GAGGLE_HTTP_RETRY_MAX_DELAY_MS=60000 ## Cap at 60s
 export GAGGLE_OFFLINE=1
 
 # Attempt to download a dataset (will fail if not cached)
-gaggle download username/dataset-name
+SELECT gaggle_download('username/dataset-name');
 
-# Querying metadata or searching will still attempt network access
-gaggle info username/dataset-name
+# Querying metadata or searching will fail fast in offline mode
+SELECT gaggle_info('username/dataset-name');
+SELECT gaggle_search('keyword', 1, 10);
 ```
 
 ### Configuration Verification
@@ -253,16 +255,19 @@ SELECT gaggle_search('housing', 1, 10);
 
 -- Get dataset metadata
 SELECT gaggle_info('username/dataset-name');
+
+-- Retrieve last error string (or NULL if none)
+SELECT gaggle_last_error();
 ```
 
 ### Retry Policy Details
 
 Gaggle implements retries with exponential backoff for HTTP requests. The number of attempts, initial delay, and
 maximum delay can be tuned with the environment variables above.
 
-### Logging Levels (planned)
+### Logging Levels
 
-Detailed logging control via `GAGGLE_LOG_LEVEL` is planned but not yet implemented.
+Detailed logging control via `GAGGLE_LOG_LEVEL` is implemented.
 
 ### Notes
 

docs/README.md

@@ -16,7 +16,7 @@ The table below includes the information about all SQL functions exposed by Gagg
 | 10 | `gaggle_update_dataset(dataset_path VARCHAR)`                   | `VARCHAR`        | Forces update to latest version (ignores cache). Returns local path to freshly downloaded dataset.                        |
 | 11 | `gaggle_version_info(dataset_path VARCHAR)`                     | `VARCHAR (JSON)` | Returns version info: `cached_version`, `latest_version`, `is_current`, `is_cached`.                                      |
 | 12 | `gaggle_json_each(json VARCHAR)`                                | `VARCHAR`        | Expands a JSON object/array into newline-delimited JSON rows with fields: `key`, `value`, `type`, `path`.                 |
-| 13 | `gaggle_file_paths(dataset_path VARCHAR, filename VARCHAR)`     | `VARCHAR`        | Resolves a specific file's local path inside a downloaded dataset.                                                        |
+| 13 | `gaggle_file_path(dataset_path VARCHAR, filename VARCHAR)`      | `VARCHAR`        | Resolves a specific file's local path inside a downloaded dataset.                                                        |
 
 > [!NOTE]
 > Dataset paths must be in the form `owner/dataset` where `owner` is the username and `dataset` is the dataset name on
@@ -220,6 +220,6 @@ Gaggle is made up of two main components:
     - C-compatible FFI surface
 
 2. **C++ DuckDB Bindings (`gaggle/bindings/`)** that:
-    - Defines the custom SQL functions (for example: `gaggle_ls`, `gaggle_file_paths`, `gaggle_search`)
+    - Defines the custom SQL functions (for example: `gaggle_ls`, `gaggle_file_path`, `gaggle_search`)
     - Integrates with DuckDB’s extension system and replacement scans (`'kaggle:...'`)
     - Marshals values between DuckDB vectors and the Rust FFI

docs/examples/README.md

@@ -23,3 +23,7 @@ Each file is self‑contained and can be executed in the DuckDB shell (or via `d
 ```bash
 make examples
 ```
+
+> [!NOTE] 
+> Some operations (like search and download) need network access unless `GAGGLE_OFFLINE=1`.
+> When offline, these will fail fast if data is not cached locally (not downloaded already).

docs/examples/e1_core_functionality.sql

@@ -26,22 +26,22 @@ limit 5;
 
 -- Section 4: download a dataset
 select '## Download a dataset';
-select gaggle_download('owid/covid-latest-data') as download_path;
+select gaggle_download('uciml/iris') as download_path;
 
 -- Section 5: list files (JSON)
 select '## list files (json)';
 select to_json(
          list(struct_pack(name := name, size := size, path := path))
        ) as files_json
-from gaggle_ls('owid/covid-latest-data');
+from gaggle_ls('uciml/iris');
 
 -- Section 5b: list files (table)
 select '## list files (table)';
-select * from gaggle_ls('owid/covid-latest-data') limit 5;
+select * from gaggle_ls('uciml/iris') limit 5;
 
 -- Section 6: get dataset metadata
 select '## get dataset metadata';
-select gaggle_info('owid/covid-latest-data') as dataset_metadata;
+select gaggle_info('uciml/iris') as dataset_metadata;
 
 -- Section 7: get cache information
 select '## Get cache information';

docs/examples/e2_advanced_features.sql

@@ -6,11 +6,11 @@ load 'build/release/extension/gaggle/gaggle.duckdb_extension';
 select gaggle_set_credentials('your-username', 'your-api-key') as credentials_set;
 
 -- Get path to specific file
-select gaggle_file_paths('habedi/flickr-8k-dataset-clean', 'flickr8k.parquet') as file_path;
+select gaggle_file_path('habedi/flickr-8k-dataset-clean', 'flickr8k.parquet') as file_path;
 
 -- Use the file path with DuckDB's read_parquet via prepared statement (no subqueries in args)
 prepare rp as select * from read_parquet(?) limit 10;
-execute rp(gaggle_file_paths('habedi/flickr-8k-dataset-clean', 'flickr8k.parquet'));
+execute rp(gaggle_file_path('habedi/flickr-8k-dataset-clean', 'flickr8k.parquet'));
 
 -- Section 2: list and process multiple files
 select '## list and process dataset files (json and table)';
@@ -35,7 +35,7 @@ select gaggle_cache_info() as cache_status;
 
 -- Section 4: purge cache if needed
 select '## Purge cache (optional)';
--- select gaggle_purge_cache() as cache_purged;
+-- select gaggle_clear_cache() as cache_cleared;
 
 -- Section 5: Dataset versioning
 select '## Check dataset versions';

docs/examples/e4_json_utils.sql

@@ -0,0 +1,24 @@
+.echo on
+
+-- Example 4: JSON helper utilities
+-- Shows how to expand JSON using gaggle_json_each
+
+select '## Load extension';
+load 'build/release/extension/gaggle/gaggle.duckdb_extension';
+
+select '## Expand JSON values into newline-delimited JSON rows';
+select gaggle_json_each('{"a":1,"b":[true,{"c":"x"}],"d":null}') as rows;
+
+-- Combine with DuckDB JSON functions
+with x as (
+  select gaggle_json_each('{"a":1,"b":[true,{"c":"x"}],"d":null}') as row
+)
+select
+  json_type(row)   as value_type,
+  json_extract(row, '$') as raw,
+  json_extract_string(row, '$.key') as key,
+  json_extract(row, '$.value') as value
+from x;
+
+.echo off
+

docs/examples/e5_cache_ops.sql

@@ -0,0 +1,24 @@
+.echo on
+
+-- Example 5: Cache operations & housekeeping
+-- Demonstrates gaggle_version, gaggle_cache_info, gaggle_clear_cache, gaggle_enforce_cache_limit
+
+select '## Load extension';
+load 'build/release/extension/gaggle/gaggle.duckdb_extension';
+
+select '## Extension version';
+select gaggle_version() as version;
+
+select '## Cache info (path, size, limit)';
+select gaggle_cache_info() as cache_info_json;
+
+select '## Clear cache (optional)';
+-- Uncomment to clear the local dataset cache
+-- select gaggle_clear_cache() as cache_cleared;
+
+select '## Enforce cache size limit (LRU eviction)';
+-- This triggers cleanup based on the configured limit; safe to run repeatedly
+select gaggle_enforce_cache_limit() as enforced;
+
+.echo off
+

gaggle/bindings/gaggle_extension.cpp

@@ -436,6 +436,22 @@ static void GetFilePath(DataChunk &args, ExpressionState &state,
   gaggle_free(file_path_c);
 }
 
+/**
+ * @brief Implements the `gaggle_last_error()` SQL function.
+ * Returns the last error message string or NULL if no error is set.
+ */
+static void GetLastError(DataChunk &args, ExpressionState &state, Vector &result) {
+  result.SetVectorType(VectorType::CONSTANT_VECTOR);
+  const char *err = gaggle_last_error();
+  if (!err) {
+    ConstantVector::SetNull(result, true);
+    return;
+  }
+  ConstantVector::GetData<string_t>(result)[0] =
+      StringVector::AddString(result, err);
+  ConstantVector::SetNull(result, false);
+}
+
 /**
  * @brief Table function to read a Kaggle dataset file as a table
  */
@@ -660,6 +676,9 @@ static void GaggleLsFunction(ClientContext &context, TableFunctionInput &data_p,
  * @brief Registers all the Gaggle functions with DuckDB.
  */
 static void LoadInternal(ExtensionLoader &loader) {
+   // Initialize Rust logging once per process
+  gaggle_init_logging();
+
    // Scalar functions (public)
   loader.RegisterFunction(ScalarFunction(
        "gaggle_set_credentials", {LogicalType::VARCHAR, LogicalType::VARCHAR},
@@ -691,6 +710,8 @@ static void LoadInternal(ExtensionLoader &loader) {
       "gaggle_json_each", {LogicalType::VARCHAR}, LogicalType::VARCHAR, JsonEach));
   loader.RegisterFunction(ScalarFunction(
       "gaggle_file_path", {LogicalType::VARCHAR, LogicalType::VARCHAR}, LogicalType::VARCHAR, GetFilePath));
+  loader.RegisterFunction(ScalarFunction(
+      "gaggle_last_error", {}, LogicalType::VARCHAR, GetLastError));
 
    // Table function: gaggle_ls(dataset_path) -> name,size,path
   TableFunction ls_fun("gaggle_ls", {LogicalType::VARCHAR}, GaggleLsFunction,
@@ -710,11 +731,8 @@ DUCKDB_CPP_EXTENSION_ENTRY(gaggle, loader) { LoadInternal(loader); }
   std::string GaggleExtension::Name() { return "gaggle"; }
   std::string GaggleExtension::Version() const {
     // Return a static-safe version string to avoid calling into FFI at load time
-    return std::string("v0.1.0");
+    return std::string("0.1.0");
   }
 
 } // namespace duckdb
 
-extern "C" {
-DUCKDB_CPP_EXTENSION_ENTRY(gaggle, loader) { duckdb::LoadInternal(loader); }
-}