Alerts visualization / Alerts dashboard

[commanderFx]

I setup multiple Monitors with Triggers which raises alerts like expected.

But I got no reference to relevant data which triggered the alert (e.g. protocol, ip address, command).
Can this be made visible within Malcolm?

I tried to setup an alert visualization as described in the link below, but don't have the option to create an alerting monitor.
Are the necessary plugins installed and the right authorizations assigned for this?

https://docs.opensearch.org/latest/observing-your-data/alerting/dashboards-alerting/

[mmguero]

The plugins you need are included in Malcolm, yet. The documentation on opensearch alert actions may help you, as could monitors. Probably what you're looking for is in ctx.results, you can do some searching around that.

[mmguero]

these resources might also help:

• OpenSearch: including document fields in alert messages
• elastic forums (not opensearch, specifically, but similar tooling)
• triggers, see stuff about ctx
• document level monitors
• opensearch forum
• opensearch forum

[mmguero]

Malcolm loopback hook is something that could maybe fill this need. We discovered a bug (see #856) so I'll update this after I look at that.

[commanderFx]

We're on version 26.01.0 but The Malcolm API Loopback Monitor is created multiple times automatically after removing them.
Where is the Loopback Monitor created initially and how to fix the multiple entries?

I don't see any data coming in on event.provider: malcolm

I suspect this is related to the first issue

[mmguero]

EDIT: As I was preparing my reply, it appears there is something with the latest version of dashboards that is causing problems with the webhook as an alert destination. I'm looking into it and am tracking it at #916.

EDIT Number 2: Upon further investigation, it does work correctly as-is IF you format the alert body in the trigger as JSON, the same way the Malcolm API Loopback Trigger example that is created is formatted:

With that configuration, the debug output of the API container shows:

api-1  | POST /mapi/alert arguments: {"alert": {"monitor": {"name": "Malcolm API Loopback Monitor"}, "trigger": {"name": "Malcolm API Loopback Trigger", "severity": 4}, "period": {"start": "2026-03-10T14:31:01.576Z", "end": "2026-03-10T14:32:01.576Z"}, "results": [{"_shards": {"total": 2, "failed": 0, "successful": 2, "skipped": 0}, "hits": {"hits": [], "total": {"value": 2145, "relation": "eq"}, "max_score": null}, "took": 2, "timed_out": false}], "body": "", "alert": "", "error": ""}}
api-1  | {"alert": {"monitor": {"name": "Malcolm API Loopback Monitor"}, "trigger": {"name": "Malcolm API Loopback Trigger", "severity": 4}, "period": {"start": "2026-03-10T14:31:01.576Z", "end": "2026-03-10T14:32:01.576Z"}, "results": [{"_shards": {"total": 2, "failed": 0, "successful": 2, "skipped": 0}, "hits": {"hits": [], "total": {"value": 2145, "relation": "eq"}, "max_score": null}, "took": 2, "timed_out": false}], "body": "", "alert": "", "error": ""}}
api-1  | {"firstPacket": "2026-03-10T14:31:01.576Z", "lastPacket": "2026-03-10T14:32:01.576Z", "ecs": {"version": "1.6.0"}, "event": {"kind": "alert", "start": "2026-03-10T14:31:01.576Z", "end": "2026-03-10T14:32:01.576Z", "ingested": "2026-03-10T14:32:01.599062Z", "provider": "malcolm", "dataset": "alerting", "module": "alerting", "url": "/dashboards/app/alerting#/dashboard", "id": "EX8efK4sCWW6kmKzgDwc", "reason": "Malcolm API Loopback Trigger", "risk_score": 40, "risk_score_norm": 40, "severity": 40, "severity_tags": "Alert", "hits": 2145}, "rule": {"name": "Malcolm API Loopback Monitor"}}
api-1  | {"_index": "arkime_sessions3-260310", "_id": "260310-EX8efK4sCWW6kmKzgDwc", "_version": 1, "result": "created", "_shards": {"total": 1, "successful": 1, "failed": 0}, "_seq_no": 10000, "_primary_term": 2}

And I can see the event in opensearch:

I'm going to leave that bug open, though, to see if I can handle the non-JSON formatted events more gracefully as well.

In the meantime, perhaps the documentation from opensearch on Alerting and Visualizations can help you find another alternative.

My original reply:

The loopback monitor is created initially here in the startup code of the dashboards-helper container. The fix for the duplicate creation bug was committed here as mmguero-dev/Malcolm@2151d9b.

On the first time starting up, you'll see a message like this (I'm running ./scripts/logs -s dashboards-helper):

dashboards-helper-1  | 2026-03-10T13:57:11Z  /usr/local/bin/shared-object-creation.sh: Creating opensearch alerting objects...
dashboards-helper-1  | 2026-03-10T13:57:12Z  /usr/local/bin/shared-object-creation.sh: opensearch alerting objects creation complete!

Then, upon restarting Malcolm, as it's starting back up:

dashboards-helper-1  | 2026-03-10T14:01:53Z  /usr/local/bin/shared-object-creation.sh: Creating opensearch alerting objects...
dashboards-helper-1  | 2026-03-10T14:01:54Z  /usr/local/bin/shared-object-creation.sh: Alerting monitor "Malcolm API Loopback Monitor" already exists, skipping
dashboards-helper-1  | 2026-03-10T14:01:54Z  /usr/local/bin/shared-object-creation.sh: opensearch alerting objects creation complete!

So as far as what's happening for you, I don't know. I guess I'd recommend removing ALL of these monitors, then shutting down malcolm and starting it back up.

Once you do have it up and running and enabled, you'll see it once in monitors, and it also shows up in notification channels:

So you can then edit the monitor or create another monitor that uses that output destination channel with whatever trigger you want:

You can even do a Send test message in the action configuration. If you want to watch the action actually happening in the logs, set MALCOLM_API_DEBUG=true in ./config/upload-common.env and then ./scripts/restart -s api and you can see it as it scrolls by.

[mmguero]

I should note, the Malcolm API Alert Loopback Monitor is an example, not necessarily what you would want for your alert as-is. It shows how you can send a JSON-formatted event with the fields you want into the database. You'd have to edit the query, trigger, etc. how you want to.

[mmguero]

In the next release I've changed it so that the triggered action will handle even non-JSON messages. For now, with v26.02.0 or earlier, just make sure you format the contents of the resulting message as JSON. A good example is here (this is what the example loopback webhook trigger action uses):

{
  "alert": {
    "monitor": {
      "name": "{{ctx.monitor.name}}"
    },
    "trigger": {
      "name": "{{ctx.trigger.name}}",
      "severity": {{ctx.trigger.severity}}
    },
    "period": {
      "start": "{{ctx.periodStart}}",
      "end": "{{ctx.periodEnd}}"
    },
    "results": {{#toJson}}ctx.results{{/toJson}},
    "body": "{{#toJson}}ctx.payload{{/toJson}}",
    "alert": "{{ctx.alert.id}}",
    "error": "{{ctx.error}}"
  }
}

[commanderFx]

We got the json working. Happy to test the rest as well when upgraded. Thanks!