Skip to content

Commit 1b024c7

Browse files
authored
Merge pull request #240 from cdelfosse/readme-spark-update
Update README.md to use the latest Spark, Delta, and Qbeast format versions
2 parents 3efc2fc + 6237db6 commit 1b024c7

File tree

1 file changed

+5
-5
lines changed

1 file changed

+5
-5
lines changed

README.md

+5-5
Original file line numberDiff line numberDiff line change
@@ -62,24 +62,24 @@ You can run the qbeast-spark application locally on your computer, or using a Do
6262
You can find it in the [Packages section](https://github.com/orgs/Qbeast-io/packages?repo_name=qbeast-spark).
6363

6464
### Pre: Install **Spark**
65-
Download **Spark 3.1.1 with Hadoop 3.2**, unzip it, and create the `SPARK_HOME` environment variable:<br />
65+
Download **Spark 3.4.1 with Hadoop 3.3.4**, unzip it, and create the `SPARK_HOME` environment variable:<br />
6666

6767
>:information_source: **Note**: You can use Hadoop 2.7 if desired, but you could have some troubles with different cloud providers' storage, read more about it [here](docs/CloudStorages.md).
6868
6969
```bash
70-
wget https://archive.apache.org/dist/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz
70+
wget https://archive.apache.org/dist/spark/spark-3.4.1/spark-3.4.1-bin-hadoop3.tgz
7171

72-
tar -xzvf spark-3.3.0-bin-hadoop3.tgz
72+
tar -xzvf spark-3.4.1-bin-hadoop3.tgz
7373

74-
export SPARK_HOME=$PWD/spark-3.3.0-bin-hadoop3
74+
export SPARK_HOME=$PWD/spark-3.4.1-bin-hadoop3
7575
```
7676
### 1. Launch a spark-shell
7777

7878
**Inside the project folder**, launch a **spark shell** with the required dependencies:
7979

8080
```bash
8181
$SPARK_HOME/bin/spark-shell \
82-
--packages io.qbeast:qbeast-spark_2.12:0.4.0,io.delta:delta-core_2.12:2.1.0 \
82+
--packages io.qbeast:qbeast-spark_2.12:0.5.0,io.delta:delta-core_2.12:2.4.0 \
8383
--conf spark.sql.extensions=io.qbeast.spark.internal.QbeastSparkSessionExtension \
8484
--conf spark.sql.catalog.spark_catalog=io.qbeast.spark.internal.sources.catalog.QbeastCatalog
8585
```

0 commit comments

Comments
 (0)