Add support for Fortio output format (envoyproxy#168)

Add support for Fortio output format

Testing: Added a unit test the compare the output to a golden file

Docs: Added sample data/reports, added docs to README.md

Signed-off-by: Teju Nareddy <[email protected]>

Author: nareddyt

README.md

@@ -9,7 +9,7 @@ Nighthawk currently offers:
 - A load testing client which supports HTTP/1.1 and HTTP/2 over HTTP and HTTPS.
 (HTTPS certificates are not yet validated).
 - A simple [test server](source/server/README.md) which is capable of generating dynamic response sizes, as well as inject delays.
-
+- A binary to transform nighthawk output to well-known formats, allowing integration with other systems and dashboards.
 
 ## Prerequisites
 
@@ -55,12 +55,13 @@ bazel-bin/nighthawk_client  [--trace <uri format>]
 |CONNECT|OPTIONS|TRACE>] [--address-family
 <auto|v4|v6>] [--burst-size <uint32_t>]
 [--prefetch-connections] [--output-format
-<json|human|yaml|dotted>] [-v <trace|debug
-|info|warn|error|critical>] [--concurrency
-<string>] [--h2] [--timeout <uint32_t>]
-[--duration <uint32_t>] [--connections
-<uint32_t>] [--rps <uint32_t>] [--]
-[--version] [-h] <uri format>
+<json|human|yaml|dotted|fortio>] [-v <trace
+|debug|info|warn|error|critical>]
+[--concurrency <string>] [--h2] [--timeout
+<uint32_t>] [--duration <uint32_t>]
+[--connections <uint32_t>] [--rps
+<uint32_t>] [--] [--version] [-h] <uri
+format>
 
 
 Where:
@@ -111,9 +112,9 @@ Release requests in bursts of the specified size (default: 0).
 --prefetch-connections
 Prefetch connections before benchmarking (HTTP/1 only).
 
---output-format <json|human|yaml|dotted>
-Output format. Possible values: {"json", "human", "yaml", "dotted"}.
-The default output format is 'human'.
+--output-format <json|human|yaml|dotted|fortio>
+Output format. Possible values: {"json", "human", "yaml", "dotted",
+"fortio"}. The default output format is 'human'.
 
 -v <trace|debug|info|warn|error|critical>,  --verbosity <trace|debug
 |info|warn|error|critical>
@@ -208,14 +209,15 @@ Nighthawk comes with a tool to transform its json output to its other supported
 USAGE:
 
 bazel-bin/nighthawk_output_transform  --output-format <json|human|yaml
-|dotted> [--] [--version] [-h]
+|dotted|fortio> [--] [--version]
+[-h]
 
 
 Where:
 
---output-format <json|human|yaml|dotted>
+--output-format <json|human|yaml|dotted|fortio>
 (required)  Output format. Possible values: {"json", "human", "yaml",
-"dotted"}.
+"dotted", "fortio"}.
 
 --,  --ignore_rest
 Ignores the rest of the labeled arguments following this flag.
@@ -305,6 +307,21 @@ client.upstream_rq_total                9994        1998.80
 [21:28:18.522403][27849][I] [source/client/client.cc:279] Done.
 ```
 
+## Visualizing the output of a benchmark
+
+Nighthawk supports transforming the output into other well-known formats, such as:
+
+- `dotted`: Provides integration with Prometheus
+- `fortio`: Provides integration with [Fortio's report-only UI](https://github.com/fortio/fortio#report-only-ui)
+
+The following is an example of a nighthawk benchmark visualized via the Fortio UI.
+
+```bash
+fortio report --data-dir ./samples/fortio_data
+```
+
+![Fortio Large Report](./samples/fortio_reports/large.png)
+
 ## Accuracy and repeatability considerations when using the Nighthawk client
 
 - Processes not related to the benchmarking task at hand may add significant noise. Consider stopping any

api/client/options.proto

@@ -47,6 +47,7 @@ message OutputFormat {
     HUMAN = 2;
     YAML = 3;
     DOTTED = 4;
+    FORTIO = 5;
   }
   OutputFormatOptions value = 1;
 }

api/client/transform/BUILD

@@ -0,0 +1,14 @@
+licenses(["notice"])  # Apache 2
+
+load("@envoy//bazel:envoy_build_system.bzl", "envoy_package")
+load("@envoy_api//bazel:api_build_system.bzl", "api_cc_py_proto_library")
+
+envoy_package()
+
+api_cc_py_proto_library(
+    name = "transform",
+    srcs = [
+        "fortio.proto",
+    ],
+    visibility = ["//visibility:public"],
+)

api/client/transform/fortio.proto

@@ -0,0 +1,57 @@
+syntax = "proto3";
+
+package nighthawk.client;
+
+import "google/protobuf/duration.proto";
+import "google/protobuf/timestamp.proto";
+import "google/protobuf/wrappers.proto";
+import "validate/validate.proto";
+
+// This proto represents that output format that Fortio expects when converted to JSON.
+// Nighthawk can fill in this proto, and then unmarshal to the Fortio-compatible JSON.
+// Therefore, this proto may not follow conventions. aip.dev/not-precedent
+message FortioResult {
+  // Disable validate because it doesn't work well with fields that are not all lowercase.
+  // The field cases must match the final output JSON that fortio-ui expects.
+  option (validate.disabled) = true;
+
+  string Labels = 1;
+
+  google.protobuf.Timestamp StartTime = 2;
+
+  uint32 RequestedQPS = 3;
+
+  google.protobuf.Duration RequestedDuration = 4;
+
+  double ActualQPS = 5;
+
+  double ActualDuration = 6;
+
+  uint32 NumThreads = 7;
+
+  DurationHistogram DurationHistogram = 8;
+
+  map<string, uint64> RetCodes = 11;
+
+  string URL = 12;
+}
+
+message DurationHistogram {
+  option (validate.disabled) = true;
+
+  uint64 Count = 1;
+
+  repeated DataEntry Data = 2;
+}
+
+message DataEntry {
+  option (validate.disabled) = true;
+
+  double Start = 1;
+
+  double End = 2;
+
+  double Percent = 3;
+
+  uint64 Count = 4;
+}