| SPDX-FileCopyrightText | SPDX-License-Identifier | title | author | footer | description | keywords | color | class | style | ||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
© 2024 Menacit AB <[email protected]> |
CC-BY-SA-4.0 |
Logging course: AI/ML |
Joel Rangsmo <[email protected]> |
© Course authors (CC BY-SA 4.0) |
Introduction to AI/MLs role in log analysis |
|
#ffffff |
|
section.center {
text-align: center;
}
table strong {
color: #d63030;
}
table em {
color: #2ce172;
}
|
Our centralized logging solution can act as a data source for machine learning.
But can it help us improve searching and analysis?
Let's look at common use-cases and how they're implemented in OpenSearch!
- Anomaly detection
- Semantic queries
- Conversational searching
Human brains are trained to identify things out of the ordinary.
We can do the same for computers.
Enables us to sift through enormous amounts of logs and act before a nuance becomes a catastrophe.
- Unusually high API latency
- Web server spawning shell process
- User from finance department logging in to database in the middle of the night
Computationally expensive and quite opaque process.
Shit in, shit out.
Use as guidance for development of static detection.
Traditionally, we've relied on lexical/keyword-based searching.
Natural Language Processing helps us fetch more relevant results.
Requires better understanding of the data we've stored/indexed.
Takes NLP one stage further by performing the same process for search results.
Made popular by Large Language Models like ChatGPT.
Context/previous dialog should be considered to improve experience.
With the terms defined, let's look at how OpenSearch can help!
Most functionality provided by the "ML Commons" plugin.
Ability to run (pre-trained) models on searches and indexed documents.
Provides plumbing for using remote models/providers.
Supports node tagging to optimize things like I/O performance and GPU access.
Provided as a high-level feature accessible through OpenSearch Dashboards.
Relies on the unsupervised Random Cut Forest algorithm to compute anomaly grades/confidence scores.
Let's take it for a spin!
If you can't represent it as number or aggregation, the built-in anomaly detection won't help you.
Still needs quite a bit of guidance, in many cases that effort could be better spent on statically configured thresholds/outliers.
Provides pre-trained sentence transformation models.
Processing implemented through OpenSearch ingest and search pipelines (not to be confused with Logstash pipelines).
Can be combined with traditional keyword-based approaches to create "hybrid queries".
If you wanna play around, check out the semantic search tutorial.
Currently provided as experimental feature.
Integrates with ChatGPT, Amazon Bedrock and Cohere ($$$ but self-hostable) APIs.
Would be neat to see support for a free LLMs like LLama 2 to aid querying of private data.
Just an appetizer, I'm far from an expert!
The features are right there, especially anomaly detection - take them for a spin if you're interested.
