Skip to content

Commit 88f8d64

Browse files
clehnetsickingsfeilmeiermichaelgrill
authored
Docs: add documentation for Backend Metadata and Timedata (OpenEMS#2314)
Adds some basic backend documentation about Metadata and Timedata. Gives some more starter information on the usage of the influxdb and the aggregated influxdb bundle. --------- Co-authored-by: Thomas Sicking <[email protected]> Co-authored-by: Stefan Feilmeier <[email protected]> Co-authored-by: Michael Grill <[email protected]>
1 parent 7e2a6c1 commit 88f8d64

File tree

13 files changed

+333
-42
lines changed

13 files changed

+333
-42
lines changed

‎build.gradle

+29-23
Original file line numberDiff line numberDiff line change
@@ -247,42 +247,48 @@ task copyBundleReadmes() {
247247
def targetBridge = basePath + "edge/bridge.adoc.d"
248248
def targetDeviceService = basePath + "edge/device_service.adoc.d"
249249
def targetTimedata = basePath + "edge/timedata.adoc.d"
250-
250+
def targetBackendService = basePath + "backend/service.adoc.d"
251+
251252
// initialize target files and directories
252-
[targetController, targetScheduler, targetNature, targetBridge, targetDeviceService, targetTimedata].each { target ->
253-
delete fileTree(dir: target, include: '**/*.adoc')
253+
[targetController, targetScheduler, targetNature, targetBridge, targetDeviceService, targetTimedata, targetBackendService].each { target ->
254+
delete fileTree(dir: target, include: '**/*.adoc')
254255
new File(target + "/_include.adoc").write('')
255256
}
256-
257+
257258
subprojects.each { proj ->
258259
// in each subproject (= bundle)...
259260
proj.file(".").listFiles().each { sourceFile ->
260261
// find the 'readme.adoc' file
261262
if(sourceFile.getName().equalsIgnoreCase("readme.adoc")) {
262263
def bundle = sourceFile.getParentFile().getName()
263264
def target = null
264-
// evaluate the OpenEMS Component ('Backend' or 'Edge')
265+
265266
if(bundle.startsWith("io.openems.edge.")) {
266-
// evaluate the bundle type (e.g. 'Controller')
267-
def edgeBundle = bundle.substring("io.openems.edge.".length())
268-
if(edgeBundle.endsWith(".api")) {
269-
target = targetNature
270-
} else if(edgeBundle.startsWith("controller.")) {
271-
target = targetController
272-
} else if(edgeBundle.startsWith("scheduler.")) {
273-
target = targetScheduler
274-
} else if(edgeBundle.startsWith("bridge.")) {
275-
target = targetBridge
276-
} else if(edgeBundle.startsWith("timedata.")) {
277-
target = targetTimedata
278-
} else {
279-
target = targetDeviceService
280-
}
267+
def edgeBundle = bundle.substring("io.openems.edge.".length())
268+
if(edgeBundle.endsWith(".api")) {
269+
target = targetNature
270+
} else if(edgeBundle.startsWith("controller.")) {
271+
target = targetController
272+
} else if(edgeBundle.startsWith("scheduler.")) {
273+
target = targetScheduler
274+
} else if(edgeBundle.startsWith("bridge.")) {
275+
target = targetBridge
276+
} else if(edgeBundle.startsWith("timedata.")) {
277+
target = targetTimedata
278+
} else {
279+
target = targetDeviceService
280+
}
281281

282282
} else if(bundle.startsWith("io.openems.backend.")) {
283-
// ignore
284-
return
285-
283+
def backendBundle = bundle.substring("io.openems.backend.".length())
284+
if(backendBundle.startsWith("timedata.")) {
285+
target = targetBackendService
286+
} else if(backendBundle.startsWith("metadata.")) {
287+
target = targetBackendService
288+
} else {
289+
return // ignore
290+
}
291+
286292
} else if(bundle.startsWith("io.openems.wrapper")) {
287293
// ignore
288294
return

‎doc/modules/ROOT/nav.adoc

+3
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,9 @@
2121
* OpenEMS Backend
2222
** xref:backend/architecture.adoc[Architecture]
2323
** xref:backend/backend-to-backend.adoc[Backend-to-Backend]
24+
** xref:backend/metadata.adoc[Metadata]
25+
** xref:backend/timedata.adoc[Timedata]
26+
** xref:backend/service.adoc[Service]
2427
** xref:backend/build.adoc[Build OpenEMS Backend]
2528
** xref:backend/deploy.adoc[Deploy OpenEMS Backend]
2629
* xref:component-communication/index.adoc[Component Communication]

‎doc/modules/ROOT/pages/backend/metadata.adoc

+37
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
= Metadata
2+
:sectnums:
3+
:sectnumlevels: 4
4+
:toc:
5+
:toclevels: 4
6+
:experimental:
7+
:keywords: AsciiDoc
8+
:source-highlighter: highlight.js
9+
:icons: font
10+
:imagesdir: ../../assets/images
11+
12+
The https://github.com/OpenEMS/openems/blob/develop/io.openems.backend.common/src/io/openems/backend/common/metadata/Metadata.java[Metadata] interface defines a service, that coordinates OpenEMS Backend communication with one or more OpenEMS Edges and with the OpenEMS UI.
13+
It is responsible for identification and authorization of OpenEMS edges and of Users using the OpenEMS UI.
14+
Beside that the Metadata bundle is responsible for the state of https://github.com/OpenEMS/openems/blob/develop/io.openems.backend.common/src/io/openems/backend/common/metadata/User.java[users] and https://github.com/OpenEMS/openems/blob/develop/io.openems.backend.common/src/io/openems/backend/common/metadata/Edge.java[edges] and related processes.
15+
16+
== Edges
17+
18+
The Metadata service provides access to the https://github.com/OpenEMS/openems/blob/develop/io.openems.backend.common/src/io/openems/backend/common/metadata/Edge.java[Edge] object - the digital twin representation of the OpenEMS Edge instance.
19+
[NOTE]
20+
====
21+
Identification of OpenEMS Edges is done with a unique API key.
22+
Please take care, that the API key includes a lot of randomness, because it is implicitly used to authorize the Edge.
23+
====
24+
25+
== Users
26+
27+
The https://github.com/OpenEMS/openems/blob/develop/io.openems.backend.common/src/io/openems/backend/common/metadata/User.java[User] object includes meta information about a user.
28+
Furthermore it is used for identification/authorization of the user.
29+
The metadata service can be used to change various attributes which are associated with a user (e.g. the users default language).
30+
31+
== Processes
32+
33+
The metadata service provides methods to
34+
35+
* add users to an Edge,
36+
* set and change alerting/notification settings,
37+
* set up new Edge devices.

‎doc/modules/ROOT/pages/backend/service.adoc

+18
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
= Backend Services
2+
:sectnums:
3+
:sectnumlevels: 4
4+
:toc:
5+
:toclevels: 4
6+
:experimental:
7+
:keywords: AsciiDoc
8+
:source-highlighter: highlight.js
9+
:icons: font
10+
:imagesdir: ../../assets/images
11+
12+
The following bundles are available in OpenEMS Backend.
13+
[NOTE]
14+
====
15+
This list is not complete.
16+
====
17+
18+
include::service.adoc.d/_include.adoc[leveloffset=+0]

‎doc/modules/ROOT/pages/backend/service.adoc.d/.gitignore

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
*.adoc

‎doc/modules/ROOT/pages/backend/timedata.adoc

+45
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
= Timedata
2+
:sectnums:
3+
:sectnumlevels: 4
4+
:toc:
5+
:toclevels: 4
6+
:experimental:
7+
:keywords: AsciiDoc
8+
:source-highlighter: highlight.js
9+
:icons: font
10+
:imagesdir: ../../assets/images
11+
12+
Live and historical data of an OpenEMS Edge are handled by a https://github.com/OpenEMS/openems/blob/develop/io.openems.backend.common/src/io/openems/backend/common/timedata/Timedata.java[Timedata] service.
13+
It describes basically methods to write and read Edge data to/from a database. There are different kind of timedata providers within OpenEMS (see xref:service.adoc[Service] for concrete implementations).
14+
15+
Within OpenEMS Backend the only component which uses the different timedata services directly is the https://github.com/OpenEMS/openems/blob/develop/io.openems.backend.core/src/io/openems/backend/core/timedatamanager/TimedataManagerImpl.java[TimedataManager].
16+
It passes Edge relevant data to **all** Timedata service and it reads the data from the **first** Timedata providers which can deliver it.
17+
18+
[NOTE]
19+
====
20+
Following paragraphs describe the new data handlng since OpenEMS version 2023.8.
21+
====
22+
23+
OpenEMS Edges will send different types of data to the OpenEMS Backend:
24+
25+
* `TimestampedDataNotification`
26+
** channel values which have changed
27+
** every 5 minutes a full snapshot of all channel values is sent, including channels which haven't changed over this time period
28+
29+
* `AggregatedDataNotification`
30+
** data is sent in a format that is optimized for fast querying from the database, to allow very fast responses in OpenEMS UI
31+
** aggregated (average or maximum) channel values are sent every 5 minutes
32+
** Backend services handle values differently according to their aggregation type, e.g. 5-minute-average is always stored; maximum is only stored if feasible for fast energy queries (e.g. at the end of a day, depending on a time-zone)
33+
34+
* `ResendDataNotification`
35+
** historic data that is resent after connection problems between Edge and Backend
36+
** this data is always aggregated in the form of the Edge.Timedata service (e.g. RRD4j)
37+
38+
Splitting the data enables OpenEMS to implement different timedata providers,
39+
which handle data differently. This gives more flexibility when scaling OpenEMS.
40+
Also due to performance reasons the computation of `AggregatedDataNotification` is done on the Edge side.
41+
This helps keeping CPU load on the database server low.
42+
43+
To get a better understanding of the new Edge data concept, have a look at the
44+
xref:service.adoc.d/io.openems.backend.timedata.aggregatedinflux.adoc[Aggregated Influx] bundle.
45+

‎io.openems.backend.metadata.file/readme.adoc

+29
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
= Metadata File
2+
3+
OpenEMS Backend implementation for Metadata.
4+
5+
Allows you to configure multiple edges by a single JSON file.
6+
Using this bundle enables you to easily set up an OpenEMS
7+
Backend for testing purposes.
8+
9+
10+
== Example configuration file
11+
12+
```
13+
{
14+
edges: {
15+
edge0: {
16+
comment: "Edge #0",
17+
apikey: "edge0",
18+
setuppassword: "abcdefgh"
19+
},
20+
edge1: {
21+
comment: "Edge #1",
22+
apikey: "edge1",
23+
setuppassword: "1234567"
24+
}
25+
}
26+
}
27+
```
28+
29+
https://github.com/OpenEMS/openems/tree/develop/io.openems.backend.metadata.file[Source Code icon:github[]]

‎io.openems.backend.metadata.file/readme.md

-18
This file was deleted.

‎io.openems.backend.metadata.odoo/readme.adoc

+17
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
= Metadata Odoo
2+
3+
Metadata OpenEMS Backend implementation for https://github.com/odoo/odoo[Odoo].
4+
5+
Enables you to manage multiple edges with one backend.
6+
7+
8+
[NOTE]
9+
====
10+
* This bundle works with Odoo 15. For a seamless integration of Odoo
11+
and OpenEMS the open-source https://github.com/OpenEMS/odoo-openems[Odoo-OpenEMS] module is needed.
12+
====
13+
14+
15+
16+
17+
https://github.com/OpenEMS/openems/tree/develop/io.openems.backend.metadata.odoo[Source Code icon:github[]]

‎io.openems.backend.timedata.aggregatedinflux/readme.adoc

+122
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
= Timedata Aggregated InfluxDB
2+
3+
OpenEMS Backend implementation for aggregated InfluxDB access.
4+
5+
[NOTE]
6+
====
7+
Bundle is needed for highly optimized environments with
8+
thousands of connected edges only. If you do not need that functionality you can skip this chapter.
9+
====
10+
11+
A large number of OpenEMS edges connected to a single backend leads to a performance bottleneck.
12+
The bottleneck is mainly due to the access to the database. This `Aggregated InfluxDB` bundle in
13+
combination with the xref:../io.openems.backend.timedata.influx.adoc[Timedata Influx]
14+
bundle can mitigate this bottleneck. This happens by writing all data as usual via xref:../io.openems.backend.timedata.influx.adoc[Timedata Influx] into
15+
an Influx database (`DB 1`). Furthermore, all "aggregated data" (see xref:timedata.adoc[edge datatypes]) are written via the `Aggregated InfluxDB` bundle to
16+
another database (`DB 2`) located on another server.
17+
Thus `DB 1` holds all data in high resolution, while `DB 2` only holds very few data (5-minute averages for power values
18+
or only daily values for energy). Furthermore, a retention policy of e.g. 90 days is applied to `DB 2`.
19+
Thus `DB 1` provides relatively slow access to a huge amount of data
20+
and `DB 2` provides fast and responsive access to a relatively small database.
21+
22+
Within OpenEMS backend access to both databases is managed by the
23+
https://github.com/OpenEMS/openems/blob/develop/io.openems.backend.core/src/io/openems/backend/core/timedatamanager/TimedataManagerImpl.java[TimedataManager].
24+
The TimedataManager writes edge relevant data to **all** Timedata providers,
25+
whereas it reads data only from the **first** Timedata provider,
26+
which can deliver.
27+
Assuming correct configuration, TimedataManager first returns historical data
28+
from `DB 2` (fast and responsive).
29+
And only in a very few cases it gets the data from `DB 1`
30+
(probably slower).
31+
32+
33+
34+
35+
[CAUTION]
36+
====
37+
The `Timedata.AggregatedInfluxDB` includes a class `AllowedChannels.java`.
38+
This implementation includes a *hardcoded* list of channels:
39+
40+
* `ALLOWED_AVERAGE_CHANNELS` are used to calculate the 5min average values (e.g. average power values).
41+
* `ALLOWED_CUMULATED_CHANNELS` are used to calculate daily values (e.g. energy values).
42+
43+
This list must be adopted to a concrete usecase. It strongly depends on
44+
45+
* your strategy to select component-IDs.
46+
* the components you are using within OpenEMS.
47+
48+
If you detect some widgets within your OpenEMS-UI which hava empty values,
49+
it may have to do with an incorrect hardcoded list.
50+
51+
====
52+
53+
== Configuration Setup
54+
55+
If you expect your backend to handle thousands of connected edges,
56+
the following configuration may provide a good start setup:
57+
58+
*Create database and set retention policy:*
59+
60+
Before starting the OpenEMS backend and after intially setting up the influx servers,
61+
you need to create the databases `influx0` and `aggregated0`
62+
and some retention policies:
63+
64+
[source,shell]
65+
----
66+
67+
##### Influx Server 1 #####
68+
curl -i -XPOST http://127.0.0.1:8082/query --data-urlencode "q=CREATE DATABASE influx0"
69+
70+
##### Influx Server 2 #####
71+
curl -i -XPOST http://127.0.0.1:8081/query --data-urlencode "q=CREATE DATABASE aggregated0"
72+
73+
curl -i -XPOST http://127.0.0.1:8081/query --data-urlencode "q=CREATE RETENTION POLICY rp_max ON aggregated0 DURATION 90d REPLICATION 1"
74+
curl -i -XPOST http://127.0.0.1:8081/query --data-urlencode "q=CREATE RETENTION POLICY rp_avg ON aggregated0 DURATION 90d REPLICATION 1"
75+
76+
----
77+
78+
*OpenEMS backend configuration:*
79+
80+
* bundle `Timedata.AggregatedInfluxDB`
81+
** ComponentID: `timedata0`
82+
** QueryLanguage: `INFLUX_QL`
83+
** URL: http://127.0.0.1:8081
84+
** Apikey: `influxuser:influxpassword`
85+
** Bucket: `aggregated0`
86+
** Retention policy for avg values: `rp_avg`
87+
** Retention policy for max values: `rp_max`
88+
** Measurement Avg: `avg`
89+
** Measurement for max values: `Europe/Berlin=max`
90+
91+
* bundle `Timedata.InfluxDB`
92+
** ComponentID: `timedata1`
93+
** QueryLanguage: `INFLUX_QL`
94+
** URL: http://127.0.0.1:8082
95+
** Org: `-`
96+
** Apikey: `influxuser:influxpassword`
97+
** Bucket: `influx0/autogen`
98+
** Measurement: `data`
99+
100+
* bundle `Core Timedata-Manager`
101+
** Timedata-IDs: `[timedata0, timedata1]`
102+
103+
104+
105+
106+
[NOTE]
107+
====
108+
* Note the different Bucket namings, one with retention policy `autogen` and one without.
109+
* Make sure that your influx databases are located on two different servers
110+
or that you can easily move one database to another server later.
111+
* Be sure that the sequence within the `Core Timedata-Manager` is
112+
configured correctly - the component-ID of the `Timedata.AggregatedInfluxDB`
113+
comes first.
114+
* Influx Database must have at least Version 1.8.10.
115+
* Take care of your edge to backend connections on the edge side. Be sure to understand and select the right `PersistencePriority` for the different datatypes.
116+
* Remember the hardcoded list in _AllowedChannels.java_
117+
* Note that you can shift the load of your databases by choosing different retention policies.
118+
* Be sure you know exactly how Influx handles *reading* and *writing* data when using different retention policies on one database.
119+
====
120+
121+
122+
https://github.com/OpenEMS/openems/tree/develop/io.openems.backend.timedata.influx.aggregatedinflux[Source Code icon:github[]]

‎io.openems.backend.timedata.dummy/readme.adoc

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
= Timedata Dummy
2+
3+
OpenEMS Backend dummy implementation for a timeseries database.
4+
5+
Using this bundle enables you to easily set up an OpenEMS Backend for testing purposes.
6+
7+
8+
https://github.com/OpenEMS/openems/tree/develop/io.openems.backend.timedata.influx[Source Code icon:github[]]

0 commit comments

Comments
 (0)