rsyslog: Index page and starter tutorial #305
Conversation
WalkthroughAdds rsyslog documentation: inserts Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches🧪 Generate unit tests
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: CHILL Plan: Pro 📒 Files selected for processing (4)
✅ Files skipped from review due to trivial changes (1)
🚧 Files skipped from review as they are similar to previous changes (3)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (6)
docs/integrate/rsyslog/tutorial.md (4)
31-40: Fix MD010: replace hard tabs with spaces in SQL block.Tabs trigger markdownlint; also remove the stray trailing tab on Line 39.
Apply this diff:
CREATE TABLE doc.systemevents ( - message TEXT - ,INDEX message_ft USING FULLTEXT(message) - ,facility INTEGER - ,fromhost TEXT - ,priority INTEGER - ,DeviceReportedTime TIMESTAMP - ,ReceivedAt TIMESTAMP - ,InfoUnitID INTEGER - ,SysLogTag TEXT - ); + message TEXT + ,INDEX message_ft USING FULLTEXT(message) + ,facility INTEGER + ,fromhost TEXT + ,priority INTEGER + ,DeviceReportedTime TIMESTAMP + ,ReceivedAt TIMESTAMP + ,InfoUnitID INTEGER + ,SysLogTag TEXT + );
47-47: Avoid weak, hard-coded password in docs.Use a neutral placeholder and nudge readers toward strong secrets.
Apply this diff:
-CREATE USER rsyslog WITH (PASSWORD='pwd123'); +-- Use a strong secret; e.g., from a secret manager or env var +CREATE USER rsyslog WITH (PASSWORD='CHANGEME_STRONG');
71-74: Don’t commit live credentials; simplify DSN; restrict file perms.
- Replace the sample password.
- Consider dropping “/doc” from the URI to avoid potential DB-vs-schema confusion; CrateDB defaults to schema doc.
- Lock down the rsyslog config file.
Apply this diff:
echo 'module(load="ompgsql")' | sudo tee /etc/rsyslog.d/pgsql.conf -echo '*.* action(type="ompgsql" conninfo="postgresql://rsyslog:pwd123@localhost/doc")' | sudo tee -a /etc/rsyslog.d/pgsql.conf +echo '*.* action(type="ompgsql" conninfo="postgresql://rsyslog:CHANGEME_STRONG@localhost")' | sudo tee -a /etc/rsyslog.d/pgsql.conf +sudo chmod 640 /etc/rsyslog.d/pgsql.conf sudo systemctl restart rsyslogPlease confirm that your target CrateDB version doesn’t require an explicit database name in the URI (if it does, we can switch to a DSN with
options=-csearch_path=doc).
33-33: Optional: specify a language analyzer for better full‑text relevance.If logs are primarily English, set an analyzer on the FT index.
Apply this diff:
- ,INDEX message_ft USING FULLTEXT(message) + ,INDEX message_ft USING FULLTEXT(message) WITH (analyzer = 'english')docs/integrate/rsyslog/index.md (2)
4-6: Prefer vendoring the logo instead of hotlinking.External hotlinks can break or slow builds; consider adding the logo to the repo’s static assets and referencing it locally.
28-34: Anchor naming consistency with Guide conventions.Using retrieved learnings: prior integrations use a “-learn” suffix (e.g., azure-functions-learn). Here we use “rsyslog-tutorial”. If consistency matters, consider renaming the anchor/page to
rsyslog-learnand updating links.I can provide a scoped rename diff touching this page, the toctree, and the tutorial anchor if you want to align it.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
docs/integrate/index.md(1 hunks)docs/integrate/rsyslog/index.md(1 hunks)docs/integrate/rsyslog/tutorial.md(1 hunks)
🧰 Additional context used
🧠 Learnings (3)
📓 Common learnings
Learnt from: amotl
PR: crate/cratedb-guide#238
File: docs/integrate/azure-functions/learn.rst:1-1
Timestamp: 2025-08-09T16:19:43.966Z
Learning: In the CrateDB Guide documentation, main integration anchors (e.g., `azure-functions`) are intentionally placed in the `index.md` files of their respective integration folders, while detailed tutorials use the `-learn` suffix (e.g., `azure-functions-learn`) in their `learn.rst` or `learn.md` files. This is a deliberate architectural pattern for the documentation restructuring.
📚 Learning: 2025-08-09T16:19:43.966Z
Learnt from: amotl
PR: crate/cratedb-guide#238
File: docs/integrate/azure-functions/learn.rst:1-1
Timestamp: 2025-08-09T16:19:43.966Z
Learning: In the CrateDB Guide documentation, main integration anchors (e.g., `azure-functions`) are intentionally placed in the `index.md` files of their respective integration folders, while detailed tutorials use the `-learn` suffix (e.g., `azure-functions-learn`) in their `learn.rst` or `learn.md` files. This is a deliberate architectural pattern for the documentation restructuring.
Applied to files:
docs/integrate/rsyslog/tutorial.md
📚 Learning: 2025-08-15T22:25:48.433Z
Learnt from: amotl
PR: crate/cratedb-guide#226
File: docs/integrate/queryzen/index.md:51-51
Timestamp: 2025-08-15T22:25:48.433Z
Learning: CrateDB provides a built-in `sys.summits` table that is suitable for demonstration purposes of any kind. This is a system table that can be used in examples and tutorials.
Applied to files:
docs/integrate/rsyslog/tutorial.md
🪛 markdownlint-cli2 (0.17.2)
docs/integrate/rsyslog/tutorial.md
31-31: Hard tabs
Column: 1
(MD010, no-hard-tabs)
32-32: Hard tabs
Column: 1
(MD010, no-hard-tabs)
33-33: Hard tabs
Column: 1
(MD010, no-hard-tabs)
34-34: Hard tabs
Column: 1
(MD010, no-hard-tabs)
35-35: Hard tabs
Column: 1
(MD010, no-hard-tabs)
36-36: Hard tabs
Column: 1
(MD010, no-hard-tabs)
37-37: Hard tabs
Column: 1
(MD010, no-hard-tabs)
38-38: Hard tabs
Column: 1
(MD010, no-hard-tabs)
39-39: Hard tabs
Column: 1
(MD010, no-hard-tabs)
39-39: Hard tabs
Column: 17
(MD010, no-hard-tabs)
40-40: Hard tabs
Column: 1
(MD010, no-hard-tabs)
🔇 Additional comments (2)
docs/integrate/rsyslog/tutorial.md (1)
58-66: Version/package sanity check.“v8.2202 or higher” and
rsyslog-pgsqlnaming can vary by distro. Add a brief note to verify versions on the target OS.If useful, I can add a small “Troubleshooting: package names by distro” note.
docs/integrate/index.md (1)
65-65: LGTM: toctree entry added in correct alphabetical position.
amotl
added
the
cross linking
amotl
force-pushed
the
rsyslog
branch
2 times, most recently
from
a50a95b to
0fdb113
Compare
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (12)
docs/ingest/telemetry/index.md (2)
5-5: Tighten the title phrasingConsider “Metrics, telemetry, and logs” for brevity and consistency with product copy.
-# Metrics, telemetry, and logging data +# Metrics, telemetry, and logs
62-67: Use canonical brand styling and active phrasing
- rsyslog styles its name in lowercase.
- Add an action-oriented one‑liner.
-::::{grid-item-card} Rsyslog +::::{grid-item-card} rsyslog :link: rsyslog :link-type: ref -Rsyslog is a rocket-fast system for log processing. +Send logs with rsyslog, a rocket‑fast system for log processing. ::::docs/integrate/rsyslog/tutorial.md (10)
2-16: Refactor intro to active, impersonal voiceRemove ego voice and rhetorical questions; lead with value and concrete claims.
-# Storing server logs on CrateDB for fast search and aggregations +# Store server logs on CrateDB for fast search and aggregations @@ -## Introduction +## Introduction @@ -Did you know that CrateDB can be a great store for your server logs? +CrateDB stores server logs efficiently and makes them easy to query. @@ -If you have been using log aggregation tools or even some of the most advanced commercial SIEM systems, you have probably experienced the same frustrations I have: +Common pain points with traditional log stacks and SIEMs include: @@ -* timeouts when searching logs over long periods of time -* a complex and proprietary query syntax -* difficulties integrating queries on logs data into application monitoring dashboards +* timeouts when searching across long time ranges +* proprietary, complex query syntaxes +* awkward integrations with application monitoring dashboards @@ -Storing server logs on CrateDB solves these problems, it allows to query the logs with standard SQL and from any tool supporting the PostgreSQL protocol; its unique indexing also makes full-text queries and aggregations super fast. -Let me show you an example. +CrateDB addresses these issues: query logs with standard SQL from any PostgreSQL‑compatible tool, and use full‑text search and aggregations backed by efficient indexes. +The sections below walk through a minimal setup.
21-25: Make setup directive active and pin the container image for reproducibilityUse imperative voice and pin an image tag to avoid surprises when “latest” updates.
-First, we will need an instance of CrateDB, it may be best to have a dedicated cluster for this purpose, to separate the monitoring system from the systems being monitored, but for the purpose of this demo we can just have a single node cluster on a docker container: +First, start CrateDB. For production, use a dedicated cluster. For this demo, run a single‑node container: @@ -sudo docker run -d --name cratedb --publish 4200:4200 --publish 5432:5432 --env CRATE_HEAP_SIZE=1g crate -Cdiscovery.type=single-node +sudo docker run -d --name cratedb \ + -p 4200:4200 -p 5432:5432 \ + -e CRATE_HEAP_SIZE=1g \ + crate:5.6.0 -Cdiscovery.type=single-nodePlease confirm the current recommended tag (e.g., latest 5.x) for this guide before merging.
27-43: Prefer consistent, active phrasing and CLI alternative placementTighten wording and present the headless tip in active voice.
-Next, we need a table to store the logs, let's connect to `http://localhost:4200/#!/console` and run: +Next, create a table for logs. Open `http://localhost:4200/#!/console` and run: @@ -Tip: if you are on a headless system you can also run queries with {ref}`command-line tools <connect-cli>`. +Tip: On headless systems, run queries with the {ref}`command-line tools <connect-cli>`.
46-55: Avoid hardcoded weak secrets; parameterize the passwordUse an environment variable and avoid committing credentials into config.
--- sql --- Use a strong secret; e.g. from a secret manager or env var. -CREATE USER rsyslog WITH (PASSWORD='pwd123'); +-- Use a strong secret; e.g. sourced from an environment variable. +-- Example: +-- export CRATEDB_RSYSLOG_PASSWORD='...' +CREATE USER rsyslog WITH (PASSWORD=${CRATEDB_RSYSLOG_PASSWORD});Also consider granting DQL if rsyslog needs to read (usually not):
GRANT DML ON TABLE doc.systemevents TO rsyslog;is fine for write‑only.
59-67: Harden and streamline package installation commandsMake the steps non‑interactive and idempotent for copy‑paste reliability.
-sudo add-apt-repository ppa:adiscon/v8-stable -sudo apt-get update -sudo apt-get install rsyslog -sudo debconf-set-selections <<< 'rsyslog-pgsql rsyslog-pgsql/dbconfig-install string false' -sudo apt-get install rsyslog-pgsql +sudo add-apt-repository -y ppa:adiscon/v8-stable +sudo apt-get update -y +sudo debconf-set-selections <<< 'rsyslog-pgsql rsyslog-pgsql/dbconfig-install string false' +sudo apt-get install -y rsyslog rsyslog-pgsqlConfirm the target distro(s) you want to support here; we can add alternatives (e.g., RHEL) if needed.
71-76: Parameterize connection info and make schema resolution explicitExplicitly set dbname and search_path; avoid embedding secrets in config.
-echo 'module(load="ompgsql")' | sudo tee /etc/rsyslog.d/pgsql.conf -echo '*.* action(type="ompgsql" conninfo="postgresql://rsyslog:pwd123@localhost/doc")' | sudo tee -a /etc/rsyslog.d/pgsql.conf +echo 'module(load="ompgsql")' | sudo tee /etc/rsyslog.d/pgsql.conf +echo '*.* action(type="ompgsql" conninfo="postgresql://rsyslog:${CRATEDB_RSYSLOG_PASSWORD}@localhost:5432/crate?options=-c%20search_path%3Ddoc")' | sudo tee -a /etc/rsyslog.d/pgsql.conf sudo chmod 640 /etc/rsyslog.d/pgsql.conf sudo systemctl restart rsyslogNote: CrateDB defaults to the
docschema; the explicitsearch_pathmakes this unambiguous in case defaults change. If you prefer to rely on defaults, keep the original DSN but still parameterize the password.
82-91: Active voice and small clarity tweaksKeep the example concise and action‑oriented.
-Now let's imagine that we want to run a container with [MediaWiki](https://www.mediawiki.org/wiki/MediaWiki) to host an intranet and we want all logs to go to CrateDB, we can just deploy this with: +To generate logs, run a [MediaWiki](https://www.mediawiki.org/wiki/MediaWiki) container and forward its logs to rsyslog: @@ -If we now point a web browser to port 80 at `http://localhost/`, you will see a new MediaWiki page. -Let's play around a bit to generate log entries, just click on "set up the wiki" and then once on Continue. -This will have generated entries in the `doc.systemevents` table with `syslogtag` matching the container id of the container running the site. +Open `http://localhost/` to see the MediaWiki setup page. +Click “set up the wiki”, then “Continue” to generate log entries. +CrateDB now stores new rows in `doc.systemevents`, with `syslogtag` matching the container ID.
95-103: Query section: tighten wording and add active phrasingSmall copy edits for flow; SQL is good.
-We can now use the {ref}`crate-reference:predicates_match` to find the error messages we are interested in: +Use {ref}`crate-reference:predicates_match` to find specific error messages:
113-121: Encourage active summary and consistent SQL styleMinor copy tweak; SQL itself is fine.
-Let's now see which log sources created the most entries: +Show the top log sources by event count:
135-136: Close with active, inclusive voiceSwitch from “I” to “we” and reduce hedging.
-I hope you found this interesting. Please do not hesitate to let us know your thoughts in the [CrateDB Community](https://community.cratedb.com/). +We hope this was useful. Share feedback and questions in the [CrateDB Community](https://community.cratedb.com/).
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
docs/ingest/telemetry/index.md(2 hunks)docs/integrate/index.md(1 hunks)docs/integrate/rsyslog/index.md(1 hunks)docs/integrate/rsyslog/tutorial.md(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
- docs/integrate/index.md
- docs/integrate/rsyslog/index.md
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-08-15T22:25:48.433Z
Learnt from: amotl
PR: crate/cratedb-guide#226
File: docs/integrate/queryzen/index.md:51-51
Timestamp: 2025-08-15T22:25:48.433Z
Learning: CrateDB provides a built-in `sys.summits` table that is suitable for demonstration purposes of any kind. This is a system table that can be used in examples and tutorials.
Applied to files:
docs/integrate/rsyslog/tutorial.md
🔇 Additional comments (1)
docs/integrate/rsyslog/tutorial.md (1)
30-41: Table schema looks compatible; consider neutral analyzer for multilingual logsIf logs are multilingual, prefer the “standard” analyzer; keep “english” if your logs are predominantly English.
- INDEX message_ft USING FULLTEXT(message) WITH (analyzer = 'english'), + INDEX message_ft USING FULLTEXT(message) WITH (analyzer = 'standard'),If you intend English only, keep as-is.
docs/integrate/rsyslog/index.md
Outdated
|
|
||
| [Rsyslog] is a rocket-fast system for log processing. | ||
|
|
||
| It offers high-performance, advanced security features, and a modular design. |
There was a problem hiding this comment.
| It offers high-performance, advanced security features, and a modular design. | |
| It offers high performance, advanced security features, and a modular design. |
There was a problem hiding this comment.
Thanks. Fixed with 165e771.
docs/integrate/rsyslog/tutorial.md
Outdated
| sudo docker run -d --name cratedb \ | ||
| -p 4200:4200 -p 5432:5432 \ | ||
| -e CRATE_HEAP_SIZE=1g \ | ||
| crate:5.6.0 -Cdiscovery.type=single-node |
There was a problem hiding this comment.
Should we use a more recent version?
There was a problem hiding this comment.
Thanks. Adjusted with 165e771 to use latest instead.
docs/integrate/rsyslog/tutorial.md
Outdated
|
|
||
| ```bash | ||
| sudo add-apt-repository -y ppa:adiscon/v8-stable | ||
| sudo apt-get update -y |
There was a problem hiding this comment.
| sudo apt-get update -y | |
| sudo apt update -y |
There was a problem hiding this comment.
apt-get has been replaced apt on recent Debian and Ubuntu versions
There was a problem hiding this comment.
Thanks. Fixed with 165e771.
docs/integrate/rsyslog/tutorial.md
Outdated
| sudo add-apt-repository -y ppa:adiscon/v8-stable | ||
| sudo apt-get update -y | ||
| sudo debconf-set-selections <<< 'rsyslog-pgsql rsyslog-pgsql/dbconfig-install string false' | ||
| sudo apt-get install -y rsyslog rsyslog-pgsql |
There was a problem hiding this comment.
| sudo apt-get install -y rsyslog rsyslog-pgsql | |
| sudo apt install -y rsyslog rsyslog-pgsql |
There was a problem hiding this comment.
Thanks. Fixed with 165e771.
| We will use [rsyslog](https://github.com/rsyslog/rsyslog) to send the logs to CrateDB, for this setup we need `rsyslog` v8.2202 or higher and the `ompgsql` module: | ||
|
|
||
| ```bash | ||
| sudo add-apt-repository -y ppa:adiscon/v8-stable |
There was a problem hiding this comment.
I think add-apt-repository has also been phased out on modern Debian/Ubuntu? Shall we exercise and verify this explicitly? Did you?
There was a problem hiding this comment.
I've just verified the command add-apt-repository does not exist on any of the Docker-based default installations of Ubuntu, probing the three most recent LTS releases 20, 22, and 24.
docker run --rm -it ubuntu:20.04 bashroot@33d8b5a2784f:/# add-apt-repository
bash: add-apt-repository: command not found
It always needed the installation of the venerable software-properties-common package. While many people may have it already, it creates significant installation overhead for those who don't.
# apt install -y software-properties-common
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
[...]
0 upgraded, 123 newly installed, 0 to remove and 1 not upgraded.
Need to get 43.1 MB of archives.
After this operation, 157 MB of additional disk space will be used.
[...]It needs to install zillions of dependency packages and also prompts asking to configure tzdata.
Configuring tzdata
------------------
Please select the geographic area in which you live. Subsequent configuration questions will narrow this down by presenting a list of cities, representing the
time zones in which they are located.
1. Africa 2. America 3. Antarctica 4. Arctic 5. Asia 6. Atlantic 7. Australia 8. Europe 9. Indian 10. Pacific 11. Etc 12. Legacy
Geographic area:
There was a problem hiding this comment.
I am now adding this command to install the software-properties-common package to the procedure. It's the quickest path to success, even if it adds drag by the amount of dependencies it pulls in.
docker run --rm -it ubuntu:24.04 bashsudo apt update --yes
sudo DEBIAN_FRONTEND=noninteractive apt install --yes software-properties-common
docs/integrate/rsyslog/tutorial.md
Outdated
|
|
||
| ```bash | ||
| sudo add-apt-repository -y ppa:adiscon/v8-stable | ||
| sudo apt-get update -y |
There was a problem hiding this comment.
Thanks. Fixed with 165e771.
docs/integrate/rsyslog/tutorial.md
Outdated
| sudo add-apt-repository -y ppa:adiscon/v8-stable | ||
| sudo apt-get update -y | ||
| sudo debconf-set-selections <<< 'rsyslog-pgsql rsyslog-pgsql/dbconfig-install string false' | ||
| sudo apt-get install -y rsyslog rsyslog-pgsql |
There was a problem hiding this comment.
Thanks. Fixed with 165e771.
| ```bash | ||
| sudo docker run -d --name cratedb \ | ||
| -p 4200:4200 -p 5432:5432 \ | ||
| -e CRATE_HEAP_SIZE=1g \ | ||
| crate:latest -Cdiscovery.type=single-node | ||
| ``` |
There was a problem hiding this comment.
@kneth: What do you think about omitting the sudo all around, so the tutorial commands can easily be used more universally, e.g. on macOS, without much ado?
There was a problem hiding this comment.
Correcting myself: rsyslog is not meant to be used on macOS, so nevermind: Versatile users can easily omit the sudo prefix on their own.
... after rsyslog joined the gang.
About
Continue adding tutorials from the community forum.
Preview
References