Skip to content

Commit ddcd654

Browse files
ganesh-skumarganesh-skumar
authored
Add FlareDB to Beam runner docs (apache#40435)
* add flaredb runner docs
1 parent 69bfcac commit ddcd654

6 files changed

Lines changed: 371 additions & 1 deletion

File tree

‎website/www/site/content/en/documentation/runners/flaredb.md‎

Lines changed: 210 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,210 @@
1+
---
2+
type: runners
3+
title: "FlareDB Runner"
4+
aliases: /learn/runners/flaredb/
5+
---
6+
<!--
7+
Licensed under the Apache License, Version 2.0 (the "License");
8+
you may not use this file except in compliance with the License.
9+
You may obtain a copy of the License at
10+
11+
http://www.apache.org/licenses/LICENSE-2.0
12+
13+
Unless required by applicable law or agreed to in writing, software
14+
distributed under the License is distributed on an "AS IS" BASIS,
15+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16+
See the License for the specific language governing permissions and
17+
limitations under the License.
18+
-->
19+
20+
# Overview
21+
22+
[FlareDB](https://github.com/flare-db/flare-db) is an Apache Beam native streaming database for running Beam pipelines. It's built in Rust, a modern systems programming language, and it uses a streams-tables architecture inspired by the ideas described in the *Streaming Systems* book (Chp 6).
23+
24+
<br/>
25+
26+
The idea is that streams are data in motion, produced by computations (transforms), and a table is the same data at rest, during a windowing or grouping operation. FlareDB persists the PCollections as durable streams/tables on an append-only Apache Paimon table and runs the computations/transforms over the table using Apache DataFusion, a high-performance query engine.
27+
28+
<br/>
29+
30+
As a result, FlareDB is lightweight, takes fewer resources to run, and makes the computational results queryable without the need for an external database.
31+
32+
The [Beam Capability Matrix](/documentation/runners/capability-matrix/) documents the supported capabilities of the FlareDB Runner.
33+
34+
> **Note:** FlareDB is an independent and open-source runner for Apache Beam. It is not part of, or maintained by the Apache Beam project. The source code is available on [GitHub](https://github.com/flare-db/flare-db).
35+
36+
# How to use FlareDB Runner
37+
38+
Install FlareDB CLI to spawn up and manage FlareDB instance and run Beam pipelines.
39+
40+
## 1. Install the FlareDB CLI
41+
42+
If you are on **Linux or macOS**, please run the following command to install the CLI:
43+
44+
```bash
45+
curl --proto '=https' --tlsv1.2 -LsSf https://github.com/flare-db/flare-db/releases/download/flare-cli-v0.3.2/flare-cli-installer.sh | sh
46+
```
47+
48+
If you are on **Windows** use WSL.
49+
50+
## 2. Initialize FlareDB
51+
52+
After installing the CLI, run:
53+
54+
```bash
55+
flare init
56+
```
57+
58+
This command performs the initial setup by creating the required local directories and downloading the FlareDB binary and Apache Beam worker JAR.
59+
60+
The initialization only needs to be **completed once**. After that, you can use the `flare up` and `flare down` commands to manage the instance.
61+
62+
63+
## 3. Start a FlareDB Instance
64+
65+
Start a local FlareDB instance with:
66+
67+
```bash
68+
flare up
69+
```
70+
71+
Once the instance is running, FlareDB is ready to accept pipeline jobs.
72+
73+
74+
## 4. Configure Your Beam Pipeline
75+
76+
To run an Apache Beam pipeline on FlareDB, add the FlareDB Runner SDK as a dependency to your Beam project. The runner SDK submits the pipeline to the FlareDB instance as a Job.
77+
78+
{{< language-switcher java py >}}
79+
80+
{{< paragraph class="language-java" >}}
81+
Add the FlareDB Runner SDK to your `pom.xml`:
82+
{{< /paragraph >}}
83+
84+
{{< highlight java >}}
85+
<dependency>
86+
<groupId>com.flare-db</groupId>
87+
<artifactId>flaredb-runner</artifactId>
88+
<version>0.3.2</version>
89+
</dependency>
90+
{{< /highlight >}}
91+
92+
{{< paragraph class="language-java" >}}
93+
Set `FlareRunner` as the runner and configure the FlareDB instance and application JAR in your pipeline options:
94+
{{< /paragraph >}}
95+
96+
{{< highlight java >}}
97+
WordCountPipelineOptions options =
98+
PipelineOptionsFactory.fromArgs(args).as(WordCountPipelineOptions.class);
99+
100+
options.setRunner(FlareRunner.class);
101+
options.setJobEndpoint("127.0.0.1:8099");
102+
options.setUberJar("build/libs/wordcount-0.2.0-all.jar");
103+
104+
Pipeline pipeline = Pipeline.create(options);
105+
{{< /highlight >}}
106+
107+
{{< paragraph class="language-java" >}}
108+
Alternatively, you can pass these as CLI arguments while running the pipeline:
109+
{{< /paragraph >}}
110+
111+
{{< highlight java >}}
112+
./gradlew :wordcount:run --args="\
113+
--runner=FlareRunner \
114+
--jobEndpoint=127.0.0.1:8099 \
115+
--uberJar=build/libs/wordcount-0.2.0-all.jar"
116+
{{< /highlight >}}
117+
118+
{{< paragraph class="language-java" >}}
119+
Check out the full Java [WordCount example](https://github.com/flare-db/flare-db/tree/main/example/wordcount/src/main/java/com/flaredb/example) pipeline.
120+
{{< /paragraph >}}
121+
122+
{{< paragraph class="language-py" >}}
123+
Create and activate a virtual environment, then install the `flaredb-runner` package. It includes the `apache-beam` dependency.
124+
{{< /paragraph >}}
125+
126+
{{< highlight py >}}
127+
python3 -m venv .venv
128+
source .venv/bin/activate
129+
pip install flaredb-runner
130+
{{< /highlight >}}
131+
132+
{{< paragraph class="language-py" >}}
133+
Set `FlareRunner` as the runner:
134+
{{< /paragraph >}}
135+
136+
{{< highlight py >}}
137+
import apache_beam as beam
138+
from apache_beam.options.pipeline_options import PipelineOptions
139+
140+
from flaredb_runner.flare_runner import FlareRunner
141+
142+
pipeline_options = PipelineOptions(
143+
job_endpoint="127.0.0.1:8099",
144+
)
145+
146+
with beam.Pipeline(runner=FlareRunner(), options=pipeline_options) as p:
147+
{{< /highlight >}}
148+
149+
{{< paragraph class="language-py" >}}
150+
Run the pipeline:
151+
{{< /paragraph >}}
152+
153+
{{< highlight py >}}
154+
python3 wordcount.py
155+
{{< /highlight >}}
156+
157+
{{< paragraph class="language-py" >}}
158+
Check out the full Python [wordcount example](https://github.com/flare-db/flare-db/blob/main/example/python/wordcount.py) pipeline.
159+
{{< /paragraph >}}
160+
161+
162+
## 5. Stop FlareDB instance
163+
164+
After executing pipelines, run this command to stop FlareDB instance
165+
166+
```bash
167+
flare down
168+
```
169+
170+
## Pipeline options
171+
172+
The FlareDB Runner is configured through the following pipeline options:
173+
174+
<table class="table table-bordered">
175+
<tr>
176+
<th>Option</th>
177+
<th>Usage</th>
178+
<th>Description</th>
179+
</tr>
180+
<tr>
181+
<td>Runner</td>
182+
<td><code>setRunner(FlareRunner.class)</code></td>
183+
<td>Pipeline runner.</td>
184+
</tr>
185+
<tr>
186+
<td>Job endpoint</td>
187+
<td><code>setJobEndpoint("host:port")</code></td>
188+
<td>URL of the FlareDB job service. Defaults to <code>127.0.0.1:8099</code>.</td>
189+
</tr>
190+
<tr>
191+
<td>Uber JAR</td>
192+
<td><code>setUberJar("/path/to/app.jar")</code></td>
193+
<td>Path to the fat JAR staged to workers.</td>
194+
</tr>
195+
<tr>
196+
<td>Job name</td>
197+
<td><code>setJobName("my-job")</code></td>
198+
<td>Name of the submitted job.</td>
199+
</tr>
200+
</table>
201+
202+
## Next steps
203+
204+
- Browse the [FlareDB documentation](https://docs.flare-db.com/).
205+
- See the [Beam Capability Matrix](/documentation/runners/capability-matrix/) to learn about features supported by FlareDB.
206+
- Try more [examples](https://github.com/flare-db/flare-db/tree/main/example) in the FlareDB repository.
207+
- Report bugs or request features in the [FlareDB issue tracker](https://github.com/flare-db/flare-db/issues).
208+
- Contributions are welcome. See the [Contributing Guide](https://github.com/flare-db/flare-db/blob/main/CONTRIBUTING.md) to get started.
209+
210+
FlareDB is released under the Apache License 2.0.

0 commit comments

Comments
 (0)