From 918b1264063088a3093cb13fb9c1e2b4e2200c75 Mon Sep 17 00:00:00 2001 From: Akshay Awate Date: Tue, 24 Sep 2024 10:24:58 +0530 Subject: [PATCH 1/4] Create 2024-09-24-prometheus_alerts_with_webex_integration.md Signed-off-by: Akshay Awate --- .../blog/2024-09-24-prometheus_alerts_with_webex_integration.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 content/blog/2024-09-24-prometheus_alerts_with_webex_integration.md diff --git a/content/blog/2024-09-24-prometheus_alerts_with_webex_integration.md b/content/blog/2024-09-24-prometheus_alerts_with_webex_integration.md new file mode 100644 index 000000000..8b1378917 --- /dev/null +++ b/content/blog/2024-09-24-prometheus_alerts_with_webex_integration.md @@ -0,0 +1 @@ + From ab549001dd2d016c3f920ef72dc750086d81841b Mon Sep 17 00:00:00 2001 From: Akshay Awate Date: Tue, 24 Sep 2024 10:39:50 +0530 Subject: [PATCH 2/4] Update 2024-09-24-prometheus_alerts_with_webex_integration.md Signed-off-by: Akshay Awate --- ...rometheus_alerts_with_webex_integration.md | 140 ++++++++++++++++++ 1 file changed, 140 insertions(+) diff --git a/content/blog/2024-09-24-prometheus_alerts_with_webex_integration.md b/content/blog/2024-09-24-prometheus_alerts_with_webex_integration.md index 8b1378917..f662f0e2f 100644 --- a/content/blog/2024-09-24-prometheus_alerts_with_webex_integration.md +++ b/content/blog/2024-09-24-prometheus_alerts_with_webex_integration.md @@ -1 +1,141 @@ +--- +title: Setting Up Prometheus Alerts with Webex Integration +created_at: 2024-09-24 +kind: article +author_name: Akshay Awate +--- +In this guide, we will walk you through integrating Prometheus with Webex to send alerts directly to your Webex spaces. This is especially useful for teams using Webex for collaboration, as it allows you to receive real-time alerts on the platform. + +## Overview +We’ll cover the following: + +1. Installation and prerequisites. +2. Creating a Webex Bot. +3. Retrieving the room_id using the Webex API. +4. Configuring Alertmanager to send alerts to Webex. +5. Securing your Bot Token with Kubernetes secrets. + +1. Installation and Prerequisites + To get started, ensure that you have Prometheus and Alertmanager installed. If not, you can refer to the Prometheus documentation for + installation instructions. + + ## Tools required: + 1. Prometheus and Alertmanager. + 2. Webex account with developer access. + 3. Working Kubernetes cluster (for deploying Alertmanager). + +2. Creating a Webex Bot + To integrate Webex with Prometheus, you'll first need to create a bot in Webex Teams that will handle receiving and sending messages on + your behalf. + + ## Steps to Create a Webex Bot: + 1. Log in to the Webex Developer Portal: Head to the Webex Developer Portal and log in. + 2. Navigate to My Apps: Once logged in, click on "My Apps" from the top navigation bar. + 3. Create a New App: Select "Create a New App" and choose "Create a Bot. + + Fill in the Details: + + **Bot Name:** Choose a name that reflects the bot's purpose (e.g., PrometheusAlertBot). + + **Bot Username:** A unique identifier for your bot. + + **Generate Access Token:** After the bot is created, you’ll be provided with an access_token. Copy this token and save it securely. This + will be used later to authenticate API requests. + +3. Invite the Bot to a Webex Channel +Now that you have a bot, you need to add it to a Webex space where it can post alerts. + +Steps: +Create a Webex Space: In Webex Teams, create a space (channel) for your alerts. + +Invite the Bot: Go to the "Add People" section of your space and invite your bot by using its bot username. + +Once added, your bot will be able to receive and send messages within the space. + +4. Retrieving the room_id Using the Webex API + To configure Prometheus to send alerts to Webex, you'll need the room_id of the space where you invited your bot. + + Steps to Retrieve room_id: + Open your terminal and run the following command, replacing YOUR_ACCESS_TOKEN with the token you generated earlier: + + ``` + curl -s -X GET -H 'Authorization: Bearer YOUR_ACCESS_TOKEN' https://webexapis.com/v1/rooms | jq . + ``` + The response will contain a list of rooms the bot is part of. Look for the room where you invited the bot and copy the room_id. + + This room_id will be used to configure Alertmanager to send alerts to the right Webex space. + +5. Configuring Alertmanager to Send Alerts to Webex + Now that you have the room_id and access_token, it's time to configure Alertmanager to forward alerts to Webex. + + Step 1: Modify alertmanager.yaml + Update your Alertmanager configuration to include a new receiver for Webex: + ``` + receivers: + - name: webexroom + webex_configs: + - room_id: "add_room_id_here" + message: "CPU usage is high" + http_config: + authorization: + type: Bearer + credentials: "add_bot_token_here" + ``` + In this configuration: + + Replace add_room_id_here with the actual room_id you retrieved earlier. + Replace add_bot_token_here with the access token generated for your bot. + Step 2: Secure Your Bot Token Using Kubernetes Secrets + Since the bot token is sensitive information, it’s a best practice to store it securely using Kubernetes secrets. Follow these steps to create a Kubernetes secret for the bot token: + + Create a Secret: + + Save your bot token in base64 encoding (you can encode it using ```echo -n "your_token" | base64```, and create the following secret + configuration file: + + ``` + apiVersion: v1 + kind: Secret + metadata: + name: apc-webex-receiver + type: Opaque + data: + Bearer: your_encoded_token + ``` + Add the Secret to AlertmanagerSpec: + + In your Alertmanager Helm chart or Kubernetes configuration, add the secret name to the alertmanagerSpec: + ``` + alertmanagerSpec: + secrets: + - apc-webex-receiver + ``` + Modify Alertmanager Config to Use Secret: + + Update the credentials_file to point to the secret in your Kubernetes cluster: + ``` + receivers: + - name: webexroom + webex_configs: + - room_id: "your_room_id" + message: "CPU usage is high" + http_config: + authorization: + type: Bearer + credentials_file: /etc/alertmanager/secrets/apc-webex-receiver/Bearer + ``` + This ensures your token is securely managed and not hardcoded into configuration files. + +6. Using Templating for Dynamic Alerts + If you want to loop through multiple alerts and send them as a formatted message, you can use the templating feature in Alertmanager’s message field. + + For example: + ``` + message: >- + {{ range .Alerts }} + *Alert:* {{ .Labels.alertname }} + *Cluster:* {{ .Labels.cluster }} + {{ end }} + ``` +This will format the alerts in a user-friendly way, providing more context in your Webex notifications. From b29644f5f5e790dfb100094a0b5c38b876e4356d Mon Sep 17 00:00:00 2001 From: Akshay Awate Date: Tue, 24 Sep 2024 17:36:59 +0530 Subject: [PATCH 3/4] Update 2024-09-24-prometheus_alerts_with_webex_integration.md Signed-off-by: Akshay Awate --- ...rometheus_alerts_with_webex_integration.md | 24 +++++++++---------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/content/blog/2024-09-24-prometheus_alerts_with_webex_integration.md b/content/blog/2024-09-24-prometheus_alerts_with_webex_integration.md index f662f0e2f..3c7b26551 100644 --- a/content/blog/2024-09-24-prometheus_alerts_with_webex_integration.md +++ b/content/blog/2024-09-24-prometheus_alerts_with_webex_integration.md @@ -16,7 +16,7 @@ We’ll cover the following: 4. Configuring Alertmanager to send alerts to Webex. 5. Securing your Bot Token with Kubernetes secrets. -1. Installation and Prerequisites +**1. Installation and Prerequisites:** To get started, ensure that you have Prometheus and Alertmanager installed. If not, you can refer to the Prometheus documentation for installation instructions. @@ -25,7 +25,7 @@ We’ll cover the following: 2. Webex account with developer access. 3. Working Kubernetes cluster (for deploying Alertmanager). -2. Creating a Webex Bot +**2. Creating a Webex Bot:** To integrate Webex with Prometheus, you'll first need to create a bot in Webex Teams that will handle receiving and sending messages on your behalf. @@ -43,17 +43,15 @@ We’ll cover the following: **Generate Access Token:** After the bot is created, you’ll be provided with an access_token. Copy this token and save it securely. This will be used later to authenticate API requests. -3. Invite the Bot to a Webex Channel -Now that you have a bot, you need to add it to a Webex space where it can post alerts. - -Steps: -Create a Webex Space: In Webex Teams, create a space (channel) for your alerts. - -Invite the Bot: Go to the "Add People" section of your space and invite your bot by using its bot username. - -Once added, your bot will be able to receive and send messages within the space. +**3. Invite the Bot to a Webex Channel** + Now that you have a bot, you need to add it to a Webex space where it can post alerts. + + Steps: + 1. Create a Webex Space: In Webex Teams, create a space (channel) for your alerts. + 2. Invite the Bot: Go to the "Add People" section of your space and invite your bot by using its bot username. + 3. Once added, your bot will be able to receive and send messages within the space. -4. Retrieving the room_id Using the Webex API +**4. Retrieving the room_id Using the Webex API** To configure Prometheus to send alerts to Webex, you'll need the room_id of the space where you invited your bot. Steps to Retrieve room_id: @@ -66,7 +64,7 @@ Once added, your bot will be able to receive and send messages within the space. This room_id will be used to configure Alertmanager to send alerts to the right Webex space. -5. Configuring Alertmanager to Send Alerts to Webex +**5. Configuring Alertmanager to Send Alerts to Webex** Now that you have the room_id and access_token, it's time to configure Alertmanager to forward alerts to Webex. Step 1: Modify alertmanager.yaml From 20dfa6b1047e87454bc5a5d21fac17deb1980d63 Mon Sep 17 00:00:00 2001 From: Akshay Awate Date: Tue, 24 Sep 2024 18:09:12 +0530 Subject: [PATCH 4/4] Update 2024-09-24-prometheus_alerts_with_webex_integration.md Signed-off-by: Akshay Awate --- ...rometheus_alerts_with_webex_integration.md | 113 +++++++++--------- 1 file changed, 55 insertions(+), 58 deletions(-) diff --git a/content/blog/2024-09-24-prometheus_alerts_with_webex_integration.md b/content/blog/2024-09-24-prometheus_alerts_with_webex_integration.md index 3c7b26551..b2f83c43b 100644 --- a/content/blog/2024-09-24-prometheus_alerts_with_webex_integration.md +++ b/content/blog/2024-09-24-prometheus_alerts_with_webex_integration.md @@ -8,66 +8,59 @@ author_name: Akshay Awate In this guide, we will walk you through integrating Prometheus with Webex to send alerts directly to your Webex spaces. This is especially useful for teams using Webex for collaboration, as it allows you to receive real-time alerts on the platform. ## Overview -We’ll cover the following: - -1. Installation and prerequisites. -2. Creating a Webex Bot. -3. Retrieving the room_id using the Webex API. -4. Configuring Alertmanager to send alerts to Webex. -5. Securing your Bot Token with Kubernetes secrets. - -**1. Installation and Prerequisites:** - To get started, ensure that you have Prometheus and Alertmanager installed. If not, you can refer to the Prometheus documentation for - installation instructions. + We’ll cover the following: + + 1. Prerequisites. + 2. Creating a Webex Bot. + 3. Retrieving the room_id using the Webex API. + 4. Configuring Alertmanager to send alerts to Webex. + 5. Securing your Bot Token with Kubernetes secrets. - ## Tools required: +**1. Prerequisites:** + To get started, ensure that you have Prometheus and Alertmanager installed. If not, you can refer to the Prometheus documentation for installation instructions. + + **Tools required:** 1. Prometheus and Alertmanager. 2. Webex account with developer access. 3. Working Kubernetes cluster (for deploying Alertmanager). **2. Creating a Webex Bot:** + To integrate Webex with Prometheus, you'll first need to create a bot in Webex Teams that will handle receiving and sending messages on your behalf. - ## Steps to Create a Webex Bot: - 1. Log in to the Webex Developer Portal: Head to the Webex Developer Portal and log in. - 2. Navigate to My Apps: Once logged in, click on "My Apps" from the top navigation bar. - 3. Create a New App: Select "Create a New App" and choose "Create a Bot. - + **Steps:** + 1. Log in to the [Webex Developer Portal](https://developer.webex.com/): Head to the Webex Developer Portal and log in. + 2. Navigate to My Apps: Once logged in, click on "My Apps" from the top navigation bar. + 3. Create a New App: Select "Create a New App" and choose Create a Bot. + Fill in the Details: - **Bot Name:** Choose a name that reflects the bot's purpose (e.g., PrometheusAlertBot). - - **Bot Username:** A unique identifier for your bot. - - **Generate Access Token:** After the bot is created, you’ll be provided with an access_token. Copy this token and save it securely. This - will be used later to authenticate API requests. + 1. Bot Name: Choose a name that reflects the bot's purpose (e.g., PrometheusAlertBot). + 2. Bot Username: A unique identifier for your bot. + 3. Generate Access Token: After the bot is created, you’ll be provided with an access_token. Copy this token and save it securely. This will be used later to authenticate API requests. -**3. Invite the Bot to a Webex Channel** +**3. Invite the Bot to a Webex Channel:** Now that you have a bot, you need to add it to a Webex space where it can post alerts. - Steps: + **Steps:** 1. Create a Webex Space: In Webex Teams, create a space (channel) for your alerts. 2. Invite the Bot: Go to the "Add People" section of your space and invite your bot by using its bot username. 3. Once added, your bot will be able to receive and send messages within the space. -**4. Retrieving the room_id Using the Webex API** - To configure Prometheus to send alerts to Webex, you'll need the room_id of the space where you invited your bot. - - Steps to Retrieve room_id: - Open your terminal and run the following command, replacing YOUR_ACCESS_TOKEN with the token you generated earlier: - +**4. Retrieving the room_id Using the Webex API:** + To configure Prometheus to send alerts to Webex, you'll need the ```room_id``` of the space where you invited your bot. Steps to Retrieve room_id: + Open your terminal and run the following command, replacing ```YOUR_ACCESS_TOKEN``` with the token you generated earlier: ``` curl -s -X GET -H 'Authorization: Bearer YOUR_ACCESS_TOKEN' https://webexapis.com/v1/rooms | jq . ``` - The response will contain a list of rooms the bot is part of. Look for the room where you invited the bot and copy the room_id. - - This room_id will be used to configure Alertmanager to send alerts to the right Webex space. + The response will contain a list of rooms the bot is part of. Look for the room where you invited the bot and copy the ```room_id```. + This ```room_id``` will be used to configure Alertmanager to send alerts to the right Webex space. **5. Configuring Alertmanager to Send Alerts to Webex** Now that you have the room_id and access_token, it's time to configure Alertmanager to forward alerts to Webex. - Step 1: Modify alertmanager.yaml + **Step 1:** Modify ```alertmanager.yaml``` Update your Alertmanager configuration to include a new receiver for Webex: ``` receivers: @@ -84,14 +77,13 @@ We’ll cover the following: Replace add_room_id_here with the actual room_id you retrieved earlier. Replace add_bot_token_here with the access token generated for your bot. - Step 2: Secure Your Bot Token Using Kubernetes Secrets - Since the bot token is sensitive information, it’s a best practice to store it securely using Kubernetes secrets. Follow these steps to create a Kubernetes secret for the bot token: - - Create a Secret: + **Step 2:** Secure Your Bot Token Using Kubernetes Secrets + Since the bot token is sensitive information, it’s a best practice to store it securely using Kubernetes secrets, create k8s Secret. Save your bot token in base64 encoding (you can encode it using ```echo -n "your_token" | base64```, and create the following secret configuration file: - + + ```secret.yaml``` ``` apiVersion: v1 kind: Secret @@ -101,17 +93,20 @@ We’ll cover the following: data: Bearer: your_encoded_token ``` - Add the Secret to AlertmanagerSpec: - - In your Alertmanager Helm chart or Kubernetes configuration, add the secret name to the alertmanagerSpec: + ```kubectl -n monitoring apply -f secret.yaml``` + + Add the secret to ```AlertmanagerSpec```, In your Alertmanager Helm chart or Kubernetes configuration, add the secret name to the + +```alertmanagerSpec:``` + ``` alertmanagerSpec: secrets: - apc-webex-receiver ``` - Modify Alertmanager Config to Use Secret: +Modify Alertmanager Config to Use Secret: +Update the credentials_file to point to the secret in your Kubernetes cluster: - Update the credentials_file to point to the secret in your Kubernetes cluster: ``` receivers: - name: webexroom @@ -123,17 +118,19 @@ We’ll cover the following: type: Bearer credentials_file: /etc/alertmanager/secrets/apc-webex-receiver/Bearer ``` - This ensures your token is securely managed and not hardcoded into configuration files. +This ensures your token is securely managed and not hardcoded into configuration files. -6. Using Templating for Dynamic Alerts - If you want to loop through multiple alerts and send them as a formatted message, you can use the templating feature in Alertmanager’s message field. - - For example: - ``` - message: >- - {{ range .Alerts }} - *Alert:* {{ .Labels.alertname }} - *Cluster:* {{ .Labels.cluster }} - {{ end }} - ``` -This will format the alerts in a user-friendly way, providing more context in your Webex notifications. +**6. Using Templating for Dynamic Alerts:** + If you want to loop through multiple alerts and send them as a formatted message, you can use the templating feature in Alertmanager’s message field. + + For example: + + ``` + message: >- + {{ range .Alerts }} + *Alert:* {{ .Labels.alertname }} + *Cluster:* {{ .Labels.cluster }} + {{ end }} + ``` + + This will format the alerts in a user-friendly way, providing more context in your Webex notifications.