FlakyTestLab

FlakyTestLab was developed with the aim of supporting the detection of flaky tests through reruns, as the tool provides an infrastructure in which multiple environments can be configured, allowing multiple tests to be executed in isolation. In addition, the tool also provides an observability layer that captures and stores logs and test reports (such as XML, TXT files, etc.) and offers tools, including visual ones, enabling the tester to monitor the reruns. FlakyTestLab was implemented using Kubernetes as its foundation and includes many components, as presented below.

Kubernetes was chosen as the base because the tool’s goal is to provide an environment with various configurations, which can be used to run tests in general and increase the occurrence of flaky tests.

FlakyTestLab aims to create an environment where tests in general can be executed and provides disturbance mechanisms. It restricts resources using the mechanisms available in Kubernetes. Additionally, it makes use of resources that would otherwise be wasted to run more tests in parallel. To make this possible, a resource management feature of Kubernetes was used, which involves configuring resources. This allows you to define values for CPU and RAM, ensuring that pods cannot use more resources than what has been allocated to them. This, in turn, limits the applications’ access to the host’s resources, even if they are still available. In Listing 1, an example of this configuration can be seen. Specifically for this project, the Guaranteed QoS ((Quality of Service)[kubernetes_qos]) approach was used, which ensures that resources are guaranteed both in terms of what is requested and the set limit. In other words, a pod will not use less than what was requested, nor will it exceed the defined limit. In this example, to define the Guaranteed QoS, both the request and limit need to be set to the same values

Listing 1

resources:
  requests:
    memory: "512Mi"
    cpu: "500m"
  limits:
    memory: "512Mi"
    cpu: "500m"

Running FlakyTestLab

In its current form, the tool requires the user to have a Linux operating system (tested on Ubuntu 20.04 and 22.04) with available CPU and RAM resources, superuser access and Internet connectivity. With this, it is possible to clone the project repository and run the commands make setup and make cluster_up, where all installation and configuration scripts will be executed, making the tool available for use.

Directory Structure

FlakyTestLab/.
├── .docs/.
├── k8s/.
├── project-outputs/.
├── project-configs/.
│   └── .3 ripme.yaml/.
├── projects/.
├── utils/.
├── Makefile/.
├── README.md/.
├── docker-compose.yaml/.

After installing and configuring the tool, the user should be inside its directory, where the structure will be as shown above. Here, the configuration file for the demo project (ripme.yaml) can be found. In other words, to run a test using the tool, the user needs to execute the two commands for setup and cluster initialization, then add the file with the test parameters following the Directory Structure and the model shown in Listing 2.

Listing 2

project:
  local_path: projects/ripme
  url: https://github.com/RipMeApp/ripme
  branch: main
  commit: 19ea20d
  output_target_path:
    - pod_target: "/target/surefire-reports"
  env:
    ENV_SPACE: development
env_config:
  - cpu: 2000m
    memory: 4000Mi
    command: mvn test --fail-never --batch-mode
    docker_image: maven:3.8.6-openjdk-8-slim
    runs: 100
    parallelism: 1
  - cpu: 2000m
    memory: 4000Mi
    command: mvn test --fail-never --batch-mode
    docker_image: maven:3.8.6-openjdk-8-slim
    runs: 100
    parallelism: 2
  - cpu: 2000m
    memory: 4000Mi
    command: mvn test --fail-never --batch-mode
    docker_image: maven:3.8.6-openjdk-8-slim
    runs: 100
    parallelism: 4
  - cpu: 2000m
    memory: 4000Mi
    command: mvn test --fail-never --batch-mode
    docker_image: maven:3.8.6-openjdk-8-slim
    runs: 100
    parallelism: 8
  - cpu: 2000m
    memory: 4000Mi
    command: mvn test --fail-never --batch-mode
    docker_image: maven:3.8.6-openjdk-8-slim
    runs: 100
    parallelism: 16

As an input for FlakyTestLab, Listing 2 shows the configuration of a project to beexecuted. With these parameters, the tool itself handles cloning the defined project, providing the source code ready for execution. These parameters make the tool flexible, as it is possible, for example, to run a Python or JavaScript project simply by changing the Docker image and the test execution command.

Parameter	Description
local_path	Where the project will be cloned (always projects/<project_name>)
url	The project’s URL on GitHub (or similar)
branch	The target branch
commit	The target commit
output_target_path	List of directories containing test output files (which are then mapped outside of the Kubernetes pods)
env	Environment variables, required for certain projects
env_config	A list defining how many distinct environments the user wants (five in this example)
cpu	Amount of CPU to be allocated to the POD
memory	Amount of RAM to be allocated to the POD
command	The command used to run the tests
docker_image	The Docker image to be used for test execution
runs	How many runs to be executed
parallelism	How many pods to be run in parallel

After defining the project, the tests should be executed and the generated logs and files analyzed. To run the test, the command make up project=ripme is used (make down project=ripme to uninstall the project). This command performs all the necessary steps to execute the project, from cloning the repository to running it in the cluster. It is important to note that for parallel executions or a large volume of runs, the host must be sized to meet these requirements. For example, it will not be possible to run parallel=4 for a project that requires 4GB of RAM on a host with only 8GB of RAM. In such a situation, as many pods as the host can accommodate will be executed, with the remaining ones in a waiting state.

During the execution of the project, it is possible to monitor its status by running commands such as make pods to see the list of running pods and their respective states, and make jobs, where you can track a summary of the state of each job (each job represents a configuration from the env_config list presented in Listing 2). In the example in question, five configurations were executed with four completing and only the configuration with parallel=1 (ftlab-ripme-c2000m-m4000mi-p1 or just P1) still running.

ftlab-ripme-c2000m-m4000mi-p1    Running    66/100        16h        16h
ftlab-ripme-c2000m-m4000mi-p16   Complete   100/100       4h31m      16h
ftlab-ripme-c2000m-m4000mi-p2    Complete   100/100       14h        16h
ftlab-ripme-c2000m-m4000mi-p4    Complete   100/100       9h         16h
ftlab-ripme-c2000m-m4000mi-p8    Complete   100/100       6h31m      16h

Analysis

After the execution is completed, FlakyTestLab directory structure will have more directories, where the test logs and their respective generated files will be stored.

FlakyTestLab/.
├── .docs/.
├── k8s/.
├── project-outputs/.
│   └── ripme/.
│       ├── logs/.
│       |   ├── ftlab-ripme-c2000m-m4000mi-p1/.
│       |   |   ├── ftlab-ripme-c2000m-m4000mi-p1-0-gcp5m/.
│       |   |   |     └── ftlab-ripme-c2000m-m4000mi-p1-0-gcp5m.log.
│       |   |   ├── ftlab-ripme-c2000m-m4000mi-p1-1-hced7/.
│       ├── outputs/.
│           └── ftlab-ripme-c2000m-m4000mi-p1/.
│               └── ftlab-ripme-c2000m-m4000mi-p1-0-gcp5m/.
│                   ├── *.xml.
│                   ├── *.txt.
├── project-configs/.
│   └── ripme.yaml/.
├── projects/.
├── utils/.
├── Makefile/.
├── README.md/.
├── docker-compose.yaml/.

The structure begins with the project name, which allows for the execution of multiple projects. Following that, there are the “logs” and “outputs” directories, containing the logs and generated files, respectively, followed by ftlab-ripme-c2000m-m4000mi-p1, which represents the configuration data, i.e., the Ripme project with the definition of two CPUs (c2000m), four gigabytes of RAM (m4000mi), and running only one pod at a time in parallel (p1). This allows for the individual analysis of test data for each configuration generated by FlakyTestLab. After the tests are completed, FlakyTestLab provides the test data to be analyzed according to the user’s needs. The tool does not provide any type of analysis because, due to its goal of being as generic and broad as possible, defining analysis methods could make the tool more selective.