Added Bucket-config as a configmap in a Block-sync init container.

prombench/docs/eks.md

@@ -107,8 +107,72 @@ Run Prombench tests in [Elastic Kubernetes Service (EKS)](https://aws.amazon.com
     ```bash
     make node_create
     ```
-
-3. **Deploy the Kubernetes Objects**:
+3. **Setting Up Benchmarking Data**    
+
+ When setting up a benchmarking environment, it’s often useful to have pre-generated data available. This data can help speed up testing and make benchmarks more realistic by simulating actual workloads.
+
+In this setup, you have two choices:
+
+Here’s how each option works:
+- **Option 1: Download data from object storage**
+
+   To download data from object storage, create a Kubernetes secret with exact named `bucket-secret` and file name `object-config.yml`  with the necessary credentials as per your object storage. This secret enables access to the stored data.
+> Note: Make sure this secret applied before `3b_prometheus-test_deployment.yaml`
+
+- **Option 2: Skip downloading data**
+
+If you don’t Want to Download data create an empty secret like this -
+
+```yaml
+# Empty Secret to Skip Downloading Data
+apiVersion: v1
+kind: Secret
+metadata:
+  name: bucket-secret
+  namespace: prombench-{{ .PR_NUMBER }} 
+type: Opaque
+stringData:
+  object-config.yml: 
+```  
+
+Regardless of the option chosen, data stored in Prometheus will only be retained based on the configured retention settings (```--storage.tsdb.retention.size```). 
+
+> **⚠️ Warning:** The benchmark will change its basis when the retention size limit is reached and older downloaded blocks are deleted. Ensure that you have sufficient retention settings configured to avoid data loss that could affect benchmarking results. 
+
+4. **Downloading Directory configuration**
+
+PromBench can download data from a specific directory in object storage based on a configuration file. This configuration file specifies the directory name along with the minimum and maximum timestamps.
+> **Note:** Make sure the file is applied before the ```3b_prometheus-test_deployment.yaml``` . 
+
+ - **Option 1: To Download Data from a Specific Directory**
+
+ Create a ConfigMap with the following structure:
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: blocksync-config
+  namespace: prombench-{{ .PR_NUMBER }}
+data:
+  bucket-config.yml: |
+    path: your-directory-name
+    minTime: block-starting-time
+    maxTime: block-ending-time
+```
+Replace the values of ```directory```,```minTime```, and ```maxTime``` with your desired configuration. Here ```minTime``` , ```maxTime``` are the starting and ending time of TSDB block. 
+- **Option 2: To Skip Data Download**
+
+If you do not want to download data, create an empty ConfigMap with the same name:
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: blocksync-config
+  namespace: prombench-{{ .PR_NUMBER }}
+data:
+```
+
+5. **Deploy the Kubernetes Objects**:
 
     ```bash
     make resource_apply

prombench/docs/gke.md

@@ -96,8 +96,71 @@ Run Prombench tests in [Google Kubernetes Engine (GKE)](https://cloud.google.com
     ```bash
     make node_create
     ```
-
-3. **Deploy the Kubernetes Objects**:
+3. **Setting Up Benchmarking Data**    
+ When setting up a benchmarking environment, it’s often useful to have pre-generated data available.This data can help speed up testing and make benchmarks more realistic by simulating actual workloads.
+
+In this setup, you have two choices:
+
+Here’s how each option works:
+- **Option 1: Download data from object storage**
+
+   To download data from object storage, create a Kubernetes secret with exact named `bucket-secret` and file name `object-config.yml`  with the necessary credentials as per your object storage. This secret enables access to the stored data.
+> Note: Make sure this secret applied before `3b_prometheus-test_deployment.yaml`
+
+- **Option 2: Skip downloading data**
+
+If you don’t Want to Download data create an empty secret like this -
+
+```yaml
+# Empty Secret to Skip Downloading Data
+apiVersion: v1
+kind: Secret
+metadata:
+  name: bucket-secret
+  namespace: prombench-{{ .PR_NUMBER }} 
+type: Opaque
+stringData:
+  object-config.yml: 
+```  
+
+Regardless of the option chosen, data stored in Prometheus will only be retained based on the configured retention settings (```--storage.tsdb.retention.size```). 
+
+> **⚠️ Warning:** The benchmark will change its basis when the retention size limit is reached and older downloaded blocks are deleted. Ensure that you have sufficient retention settings configured to avoid data loss that could affect benchmarking results. 
+
+4. **Downloading Directory configuration**
+
+PromBench can download data from a specific directory in object storage based on a configuration file. This configuration file specifies the directory name along with the minimum and maximum timestamps.
+> **Note:** Make sure the file is applied before the ```3b_prometheus-test_deployment.yaml``` .
+
+ - **Option 1: To Download Data from a Specific Directory** 
+
+ Create a ConfigMap with the following structure:
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: blocksync-config
+  namespace: prombench-{{ .PR_NUMBER }}
+data:
+  bucket-config.yml: |
+    path: your-directory-name
+    minTime: block-starting-time
+    maxTime: block-ending-time
+```
+Replace the values of ```directory```,```minTime```, and ```maxTime``` with your desired configuration. Here ```minTime``` , ```maxTime``` are the starting and ending time of TSDB block. 
+- **Option 2: To Skip Data Download**
+
+If you do not want to download data, create an empty ConfigMap with the same name:
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: blocksync-config
+  namespace: prombench-{{ .PR_NUMBER }}
+data:
+```
+
+5. **Deploy the Kubernetes Objects**:
 
     ```bash
     make resource_apply

prombench/docs/kind.md

@@ -114,7 +114,71 @@ If used with the GitHub integration:
    export PR_NUMBER=<PR to benchmark against the selected $RELEASE>
    ```
 
-2. Deploy the Kubernetes objects:
+2. Setting Up Benchmarking Data    
+ When setting up a benchmarking environment, it’s often useful to have pre-generated data available.This data can help speed up testing and make benchmarks more realistic by simulating actual workloads.
+
+In this setup, you have two choices:
+
+Here’s how each option works:
+- **Option 1: Download data from object storage**
+
+   To download data from object storage, create a Kubernetes secret with exact named `bucket-secret` and file name `object-config.yml`  with the necessary credentials as per your object storage. This secret enables access to the stored data.
+> Note: Make sure this secret applied before `3b_prometheus-test_deployment.yaml`
+
+- **Option 2: Skip downloading data**
+
+If you don’t Want to Download data create an empty secret like this -
+
+```yaml
+# Empty Secret to Skip Downloading Data
+apiVersion: v1
+kind: Secret
+metadata:
+  name: bucket-secret
+  namespace: prombench-{{ .PR_NUMBER }} 
+type: Opaque
+stringData:
+  object-config.yml: 
+```  
+
+Regardless of the option chosen, data stored in Prometheus will only be retained based on the configured retention settings (```--storage.tsdb.retention.size```). 
+
+> **⚠️ Warning:** The benchmark will change its basis when the retention size limit is reached and older downloaded blocks are deleted. Ensure that you have sufficient retention settings configured to avoid data loss that could affect benchmarking results. 
+
+3. Downloading Directory configuration
+
+PromBench can download data from a specific directory in object storage based on a configuration file. This configuration file specifies the directory name along with the minimum and maximum timestamps.
+> **Note:** Make sure the file is applied before the ```3b_prometheus-test_deployment.yaml``` . 
+
+ - **Option 1: To Download Data from a Specific Directory**
+
+ Create a ConfigMap with the following structure:
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: blocksync-config
+  namespace: prombench-{{ .PR_NUMBER }}
+data:
+  bucket-config.yml: |
+    path: your-directory-name
+    minTime: block-starting-time
+    maxTime: block-ending-time
+```
+Replace the values of ```directory```,```minTime```, and ```maxTime``` with your desired configuration. Here ```minTime``` , ```maxTime``` are the starting and ending time of TSDB block. 
+- **Option 2: To Skip Data Download**
+
+If you do not want to download data, create an empty ConfigMap with the same name:
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: blocksync-config
+  namespace: prombench-{{ .PR_NUMBER }}
+data:
+```
+
+4. Deploy the Kubernetes objects:
    > **_Note:_** If you encounter a `too many files open` error caused by promtail, increase the default value of `/proc/sys/fs/inotify/max_user_instances` from 128 to 512:
    > ```bash
    > sudo sysctl fs.inotify.max_user_instances=512

prombench/manifests/prombench/benchmark/3b_prometheus-test-pr_deployment.yaml

@@ -48,6 +48,16 @@ spec:
         volumeMounts:
         - name: prometheus-executable
           mountPath: /prometheus-builder
+      - name: data-downloader
+        image: docker.io/prominfra/block-sync:master
+        imagePullPolicy: Always                
+        volumeMounts:
+        - name: instance-ssd
+          mountPath: /data
+        - name: bucket-config
+          mountPath: /config
+        - name: key
+          mountPath: /key
       containers:
       - name: prometheus
         image: quay.io/prometheus/busybox:latest
@@ -67,7 +77,8 @@ spec:
           "--web.external-url=http://{{ .DOMAIN_NAME }}/{{ .PR_NUMBER }}/prometheus-pr",
           "--storage.tsdb.path=/prometheus",
           "--config.file=/etc/prometheus/prometheus.yml",
-          "--log.level=debug"
+          "--log.level=debug",
+          "--storage.tsdb.retention.size=5GB"
         ]
         resources:
           requests:
@@ -93,6 +104,12 @@ spec:
           path: /mnt/disks/ssd0
       - name: prometheus-executable
         emptyDir: {}
+      - name: bucket-config   # Define the Secret volume
+        secret:
+          secretName: bucket-secret
+      - name: key
+        configMap:
+          name: blocksync-config                 
       terminationGracePeriodSeconds: 300
       nodeSelector:
         node-name: prometheus-{{ .PR_NUMBER }}

prombench/manifests/prombench/benchmark/3b_prometheus-test-release_deployment.yaml

@@ -32,6 +32,17 @@ spec:
                 - prometheus
       securityContext:
         runAsUser: 0
+      initContainers:
+      - name: data-downloader
+        image: docker.io/prominfra/block-sync:master
+        imagePullPolicy: Always                  
+        volumeMounts:
+        - name: instance-ssd
+          mountPath: /data
+        - name: bucket-config
+          mountPath: /config
+        - name: key
+          mountPath: /key            
       containers:
       - name: prometheus
         image: quay.io/prometheus/prometheus:{{ .RELEASE }}
@@ -41,7 +52,8 @@ spec:
           "--web.external-url=http://{{ .DOMAIN_NAME }}/{{ .PR_NUMBER }}/prometheus-release",
           "--storage.tsdb.path=/prometheus",
           "--config.file=/etc/prometheus/prometheus.yml",
-          "--log.level=debug"
+          "--log.level=debug",
+           "--storage.tsdb.retention.size=5GB"
         ]
         resources:
           requests:
@@ -63,6 +75,12 @@ spec:
         hostPath:
           # /mnt is where GKE keeps it's SSD.
           path: /mnt/disks/ssd0
+      - name: bucket-config   # Define the Secret volume
+        secret:
+          secretName: bucket-secret
+      - name: key
+        configMap:
+          name: blocksync-config             
       terminationGracePeriodSeconds: 300
       nodeSelector:
         node-name: prometheus-{{ .PR_NUMBER }}