Skip to content

Initial Icinga Notifications Source #998

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 18 commits into
base: main
Choose a base branch
from
Draft

Initial Icinga Notifications Source #998

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
1731c67
Initial Icinga Notifications Source
oxzi Jul 29, 2025
1fc987a
Use the newly introduced notifications event utils from `igl`
yhabteab Aug 6, 2025
96075cc
Don't limit queries referncing to `{host,service}_id` & Env ID params
yhabteab Aug 6, 2025
0738c96
Reevaluate rules immediately after refetching them
yhabteab Aug 6, 2025
69607bd
Retrieve host and service names from Redis
yhabteab Aug 6, 2025
f8df7ac
Drop superfluous `rulesMutex`
yhabteab Aug 7, 2025
8ecbb9c
go.mod(WIP): require not yet merged branch of IGL
yhabteab Aug 8, 2025
ba9ff93
Notifications: Address Code Review
oxzi Sep 5, 2025
bc8fc55
notifications: IGL Changes For Rules
oxzi Sep 9, 2025
1d2a90c
history: Retry failing callback submissions
oxzi Sep 25, 2025
99a86ff
notifications: Send relative Icinga Web 2 URLs
oxzi Sep 29, 2025
e02a705
notifications: Don't abort for faulty object rules
oxzi Sep 30, 2025
96bfca2
notifications: Reflect RulesInfo IGL update
oxzi Oct 2, 2025
a80cb47
Document Notification Source
oxzi Oct 2, 2025
61a164c
Configurable callback sync telemetry stat name
oxzi Oct 21, 2025
0ab4f90
Minor Tweaks for Icinga Notifications Integration
oxzi Oct 21, 2025
214bfe2
notifications: Fetch customvars from Redis
oxzi Oct 22, 2025
f1e3982
notifications: Evaluate Icinga DB Web Rule Filter
oxzi Oct 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 38 additions & 5 deletions cmd/icingadb/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package main
import (
"context"
"fmt"
"github.com/icinga/icinga-go-library/database"
"github.com/icinga/icinga-go-library/logging"
"github.com/icinga/icinga-go-library/redis"
"github.com/icinga/icinga-go-library/utils"
Expand All @@ -15,6 +16,7 @@ import (
v1 "github.com/icinga/icingadb/pkg/icingadb/v1"
"github.com/icinga/icingadb/pkg/icingaredis"
"github.com/icinga/icingadb/pkg/icingaredis/telemetry"
"github.com/icinga/icingadb/pkg/notifications"
"github.com/okzk/sdnotify"
"github.com/pkg/errors"
"go.uber.org/zap"
Expand Down Expand Up @@ -168,13 +170,44 @@ func run() int {
sig := make(chan os.Signal, 1)
signal.Notify(sig, os.Interrupt, syscall.SIGTERM, syscall.SIGHUP)

go func() {
logger.Info("Starting history sync")
{
var (
callbackName string
callbackKeyStructPtr map[string]any
callbackFn func(database.Entity) bool
)

if err := hs.Sync(ctx); err != nil && !utils.IsContextCanceled(err) {
logger.Fatalf("%+v", err)
if cfg := cmd.Config.NotificationsSource; cfg.ApiBaseUrl != "" {
logger.Info("Starting Icinga Notifications source")

notificationsSource, err := notifications.NewNotificationsClient(
ctx,
db,
rc,
logs.GetChildLogger("notifications-source"),
cfg)
if err != nil {
logger.Fatalw("Can't create Icinga Notifications client from config", zap.Error(err))
}

callbackName = "notifications_sync"
callbackKeyStructPtr = notifications.SyncKeyStructPtrs
callbackFn = notificationsSource.Submit
}
}()

go func() {
logger.Info("Starting history sync")

if err := hs.Sync(
ctx,
callbackName,
callbackKeyStructPtr,
callbackFn,
); err != nil && !utils.IsContextCanceled(err) {
logger.Fatalf("%+v", err)
}
}()
}

// Main loop
for {
Expand Down
12 changes: 12 additions & 0 deletions config.example.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,3 +139,15 @@ redis:
# flapping:
# notification:
# state:

# Icinga DB can act as an event source for Icinga Notifications. If the following block is not empty, Icinga DB will
# submit events to the Icinga Notifications API.
# notifications-source:
# # URL to the API root.
# api-base-url: http://localhost:5680
#
# Username to authenticate against the Icinga Notifications API.
# username: icingadb
#
# # Password for the defined user.
# password: insecureinsecure
Comment on lines +143 to +153
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inconsistent formatting with the rest of the file (and even your added part isn't consistent with itself, comment for username: doesn't have a second # where the other two option comments have).

4 changes: 4 additions & 0 deletions doc/01-About.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@ Icinga DB Web also connects to the Icinga 2 API with its Command Transport to ac

These are the components of Icinga DB embedded into an Icinga setup with Icinga 2 and Icinga Web 2.

Since the Icinga DB daemon always receives the latest information from Redis®, it is an ideal candidate to distribute information further.
In addition to inserting data into a relational database, Icinga DB can also forward events to [Icinga Notifications](https://icinga.com/docs/icinga-notifications/),
as described in the [configuration section](03-Configuration.md#notifications-source-configuration).

## Installation

To install Icinga DB see [Installation](02-Installation.md).
Expand Down
16 changes: 15 additions & 1 deletion doc/03-Configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ ICINGADB_LOGGING_OPTIONS=database:error,high-availability:debug
| runtime-updates | Runtime updates of config objects after the initial config synchronization. |
| telemetry | Reporting of Icinga DB status to Icinga 2 via Redis® (for monitoring purposes). |

## Retention
## Retention Configuration

By default, no historical data is deleted, which means that the longer the data is retained,
the more disk space is required to store it. History retention is an optional feature that allows to
Expand Down Expand Up @@ -174,6 +174,20 @@ ICINGADB_RETENTION_OPTIONS=comment:356
| count | **Optional.** Number of old historical data a single query can delete in a `"DELETE FROM ... LIMIT count"` manner. Defaults to `5000`. |
| options | **Optional.** Map of history category to number of days to retain its data. Available categories are `acknowledgement`, `comment`, `downtime`, `flapping`, `notification` and `state`. |

## Notifications Source Configuration

Icinga DB can act as an event source for [Icinga Notifications](https://icinga.com/docs/icinga-notifications/).
If configured, Icinga DB will submit events to the Icinga Notifications API.

For YAML configuration, the options are part of the `notifications-source` dictionary.
For environment variables, each option is prefixed with `ICINGADB_NOTIFICATIONS_SOURCE_`.

| Option | Description |
|--------------|-----------------------------------------------------------------------------------|
| api-base-url | **Optional.** Icinga Notifications API base URL, such as `http://localhost:5680`. |
| username | **Optional.** Icinga Notifications API user for this source. |
| password | **Optional.** Icinga Notifications API user password. |

## Appendix

### Duration String
Expand Down
4 changes: 2 additions & 2 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ require (
github.com/goccy/go-yaml v1.13.0
github.com/google/go-cmp v0.7.0
github.com/google/uuid v1.6.0
github.com/icinga/icinga-go-library v0.7.2
github.com/icinga/icinga-go-library v0.7.3-0.20251022120618-6600889adc38
github.com/jessevdk/go-flags v1.6.1
github.com/jmoiron/sqlx v1.4.0
github.com/mattn/go-sqlite3 v1.14.32
Expand All @@ -34,7 +34,7 @@ require (
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-runewidth v0.0.12 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/redis/go-redis/v9 v9.10.0 // indirect
github.com/redis/go-redis/v9 v9.14.1 // indirect
github.com/rivo/uniseg v0.2.0 // indirect
github.com/ssgreg/journald v1.0.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
Expand Down
8 changes: 4 additions & 4 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,8 @@ github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/icinga/icinga-go-library v0.7.2 h1:6ilUeE9F9OqxxJXNR9URWDf6zOqsdhjjR9w1MUXY9Kg=
github.com/icinga/icinga-go-library v0.7.2/go.mod h1:HZTiYD+N+9FZIVpPdUEJWJnc6sLvrIRO03jvkdkmUEU=
github.com/icinga/icinga-go-library v0.7.3-0.20251022120618-6600889adc38 h1:5RNrPZCwvqm2/06i9dUCJtcBV+tR8WgUKtHne2sOaA8=
github.com/icinga/icinga-go-library v0.7.3-0.20251022120618-6600889adc38/go.mod h1:L80M/ufoqFJJjZcdnfsTp6eFl06vm3JuvSWlGcDf708=
github.com/jessevdk/go-flags v1.6.1 h1:Cvu5U8UGrLay1rZfv/zP7iLpSHGUZ/Ou68T0iX1bBK4=
github.com/jessevdk/go-flags v1.6.1/go.mod h1:Mk8T1hIAWpOiJiHa9rJASDK2UGWji0EuPGBnNLMooyc=
github.com/jmoiron/sqlx v1.4.0 h1:1PLqN7S1UYp5t4SrVVnt4nUVNemrDAtxlulVe+Qgm3o=
Expand All @@ -63,8 +63,8 @@ github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/redis/go-redis/v9 v9.10.0 h1:FxwK3eV8p/CQa0Ch276C7u2d0eNC9kCmAYQ7mCXCzVs=
github.com/redis/go-redis/v9 v9.10.0/go.mod h1:huWgSWd8mW6+m0VPhJjSSQ+d6Nh1VICQ6Q5lHuCH/Iw=
github.com/redis/go-redis/v9 v9.14.1 h1:nDCrEiJmfOWhD76xlaw+HXT0c9hfNWeXgl0vIRYSDvQ=
github.com/redis/go-redis/v9 v9.14.1/go.mod h1:huWgSWd8mW6+m0VPhJjSSQ+d6Nh1VICQ6Q5lHuCH/Iw=
github.com/rivo/uniseg v0.1.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/rivo/uniseg v0.2.0 h1:S1pD9weZBuJdFmowNwbpi7BJ8TNftyUImj/0WQi72jY=
github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
Expand Down
10 changes: 6 additions & 4 deletions internal/config/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"github.com/creasty/defaults"
"github.com/icinga/icinga-go-library/database"
"github.com/icinga/icinga-go-library/logging"
"github.com/icinga/icinga-go-library/notifications/source"
"github.com/icinga/icinga-go-library/redis"
"github.com/icinga/icingadb/pkg/icingadb/history"
"github.com/pkg/errors"
Expand All @@ -15,10 +16,11 @@ const DefaultConfigPath = "/etc/icingadb/config.yml"

// Config defines Icinga DB config.
type Config struct {
Database database.Config `yaml:"database" envPrefix:"DATABASE_"`
Redis redis.Config `yaml:"redis" envPrefix:"REDIS_"`
Logging logging.Config `yaml:"logging" envPrefix:"LOGGING_"`
Retention RetentionConfig `yaml:"retention" envPrefix:"RETENTION_"`
Database database.Config `yaml:"database" envPrefix:"DATABASE_"`
Redis redis.Config `yaml:"redis" envPrefix:"REDIS_"`
Logging logging.Config `yaml:"logging" envPrefix:"LOGGING_"`
Retention RetentionConfig `yaml:"retention" envPrefix:"RETENTION_"`
NotificationsSource source.Config `yaml:"notifications-source" envPrefix:"NOTIFICATIONS_SOURCE_"`
}

func (c *Config) SetDefaults() {
Expand Down
2 changes: 1 addition & 1 deletion pkg/icingadb/history/retention.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,7 @@ func (r *Retention) Start(ctx context.Context) error {

deleted, err := stmt.CleanupOlderThan(
ctx, r.db, e.Id, r.count, olderThan,
database.OnSuccessIncrement[struct{}](&telemetry.Stats.HistoryCleanup),
database.OnSuccessIncrement[struct{}](telemetry.Stats.Get(telemetry.StatHistoryCleanup)),
)
if err != nil {
select {
Expand Down
Loading
Loading