Skip to content

Latest commit

 

History

History
253 lines (175 loc) · 9.34 KB

k6-performance-tests.rst

File metadata and controls

253 lines (175 loc) · 9.34 KB

OroCommerce Performance Load Testing with k6

k6 is an open-source load-testing tool designed to test the performance of APIs, microservices, and websites. It allows developers and testers to simulate traffic to a system and analyze its performance under various conditions. The key features of k6 include:

  • Script-based tests using JavaScript
  • Support for HTTP/1.1, HTTP/2, and WebSocket protocols
  • Metrics collection and real-time reporting
  • Cloud execution and distributed testing capabilities
  • Integration with CI/CD pipelines

This documentation outlines the steps for performing performance and load tests on the OroCommerce application using the k6 load testing tool.

Repository Structure

The repository is structured as follows:

performance/
├── scripts/                  # Load test scenarios for k6
│   ├── warmingUpTheApp.js
│   ├── storefrontTests.js
│   └── checkoutTest.js
└── summary.html              # Summary test result output (generated by k6-reporter)

Setup Instructions

Installing k6

Debian/Ubuntu

To install k6 on Debian/Ubuntu:

  1. Add the k6 GPG key and repository:

    sudo gpg -k
sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69
echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list
sudo apt-get update

  2. Install k6:

    sudo apt-get install k6

macOS

To install k6 using Homebrew, run:

brew install k6

Windows

To install k6 using the Windows Package Manager, run:

winget install k6

Docker

To pull the latest k6 Docker image, run:

docker pull grafana/k6:latest

Running Tests

Test Parameters

Before running the tests, specify the following parameters in the test commands:

  • BASE_URL: Base URL of the OroCommerce application.
  • USERNAME: Storefront username.
  • PASSWORD: Storefront user password.
  • VU: Number of virtual users.
  • DURATION: Duration of the test (e.g., 60s, 10m).
  • THRESHOLD_95: Pass/fail criteria for your test metrics (95th percentile).
  • SL_ID: Shopping list ID (optional).
  • SHIPPING_METHOD: Shipping method (e.g., "fixed_product_5", "flat_rate_6").
  • PAYMENT_METHOD: Payment method (e.g., "payment_term_1").

Test Commands

Run the following commands to execute the tests:

Debian/Ubuntu

  1. Warm up the application:

    k6 run -e BASE_URL="https://example.com" -e USERNAME="[email protected]" -e PASSWORD="[email protected]" -e VU=1 -e DURATION=60s -e THRESHOLD_95=3000 scripts/warmingUpTheApp.js

  2. Run storefront tests:

    k6 run -e BASE_URL="https://example.com" -e USERNAME="[email protected]" -e PASSWORD="[email protected]" -e VU=1 -e DURATION=600s -e THRESHOLD_95=850 scripts/storefrontTests.js

  3. Run checkout tests:

    k6 run -e BASE_URL="https://example.com" -e USERNAME="[email protected]" -e PASSWORD="[email protected]" -e SL_ID=2 -e SHIPPING_METHOD="fixed_product_5" -e PAYMENT_METHOD="payment_term_1" -e VU=1 -e DURATION=600s -e THRESHOLD_95=850 scripts/checkoutTest.js

Docker

  1. Warm up the application:

    docker run --rm --network host -u "$(id -u):$(id -g)" -v ${PWD}:/home/k6/performance -w /home/k6/performance grafana/k6:latest run -e BASE_URL="https://example.com" -e USERNAME="[email protected]" -e PASSWORD="[email protected]" -e VU=1 -e DURATION=60s -e THRESHOLD_95=3000 scripts/warmingUpTheApp.js

  2. Run storefront tests:

    docker run --rm --network host -u "$(id -u):$(id -g)" -v ${PWD}:/home/k6/performance -w /home/k6/performance grafana/k6:latest run -e BASE_URL="https://example.com" -e USERNAME="[email protected]" -e PASSWORD="[email protected]" -e VU=1 -e DURATION=600s -e THRESHOLD_95=850 scripts/storefrontTests.js

  3. Run checkout tests:

    docker run --rm --network host -u "$(id -u):$(id -g)" -v ${PWD}:/home/k6/performance -w /home/k6/performance grafana/k6:latest run -e BASE_URL="https://example.com" -e USERNAME="[email protected]" -e PASSWORD="[email protected]" -e SL_ID=2 -e SHIPPING_METHOD="fixed_product_5" -e PAYMENT_METHOD="payment_term_1" -e VU=1 -e DURATION=600s -e THRESHOLD_95=850 scripts/checkoutTest.js

Interpreting Results

After running the tests, a summary of the results will be available in performance/summary.html. This summary is generated by k6-reporter and provides detailed metrics on the performance and load handling of the OroCommerce application.

Default Metrics

Metric Description
http_req_duration Measures the total time for the request. Includes sending, waiting, receiving, connecting, and TLS handshaking.
http_req_waiting Measures the waiting time of the request. It is the time spent waiting for the server response.
http_req_connecting Measures the time spent establishing a TCP connection to the remote host.
http_req_tls_handshaking Measures the time spent on the TLS handshake. Includes all blocking phases of the TLS handshake.
http_req_sending Measures the time spent sending data to the remote host.
http_req_receiving Measures the time spent receiving a response from the remote host.
http_req_blocked Measures the time spent blocked before initiating the request. Includes time blocked on DNS resolution, TCP connecting, and TLS handshaking.
iteration_duration Measures the time it takes for one iteration of the script to execute.
group_duration Measures the time it takes for a group to complete. A group is a logical grouping of actions in a script.

Custom Metrics

Metric Description
authentication_post_request Measures the performance of the POST request for authentication.
check_failure_rate Measures the rate of failed checks in the test.
create_sl_request_post_request Measures the performance of the POST request for creating a shopping list request.
create_sl_widget Measures the performance of creating a shopping list widget.
load_about_page_cms_guest_user Measures the performance of loading the About page for a guest user.
load_home_page_guest_user Measures the performance of loading the Home page for a guest user.
load_home_page_logged_in_user Measures the performance of loading the Home page for a logged-in user.
load_login_page Measures the performance of loading the Login page.
load_product_detail_page_guest_user Measures the performance of loading the Product Detail page for a guest user.
load_product_detail_page_logged_in_user Measures the performance of loading the Product Detail page for a logged-in user.
load_product_listing_page_guest_user Measures the performance of loading the Product Listing page for a guest user.
load_product_listing_page_logged_in_user Measures the performance of loading the Product Listing page for a logged-in user.
load_product_search_page_guest_user Measures the performance of loading the Product Search page for a guest user.
load_product_search_page_logged_in_user Measures the performance of loading the Product Search page for a logged-in user.

Key Performance Indicators (KPIs)

Average Response Time

Average response time is calculated as the average duration for a request to be completed. Use the http_req_duration metric in the report to measure this.

Peak Response Time

Peak response time is the maximum time it takes for a request to be completed. Use the http_req_duration metric in the report to find the maximum value.

Error Rate

Error rate is the percentage of requests that results in an error. Use the check_failure_rate metric in the report to measure this.

Throughput

Throughput is the number of requests that your application can handle per unit of time. It is usually measured in requests per second. In the provided HTML file, this information is given as Concurrent Requests Per Second (CRPS) in the title of each tab.

Concurrent Users

Concurrent users is the number of users that are simultaneously connected to your application. In the provided HTML file, this information is given in the title of each tab (e.g., 1 VU is 1 Virtual User, 25 VU is 25 Virtual Users, etc.).

Additional Resources