Doc

Author: Lars T Hansen

doc/HOWTO-KAFKA.md

@@ -217,19 +217,46 @@ If the nodes running Sonar are behind an http proxy then the Kafka broker cannot
 directly, but must be contacted through a Kafka proxy that accepts the data via an http REST call
 and forwards them to the broker.  The proxy in `util/kafka-proxy` can be used for this and Sonar
 knows how to talk to it.  In this case, configure the `[kafka]` section of Sonar's config with a
-`rest-endpoint` instead of a `broker-address` and leave out the `ca-file`.  Run the proxy behind a
-web server, eg, for nginx I use this:
+`rest-endpoint` instead of a `broker-address`.  The 8090 port is the default for the proxy.
+
+On the backend, there are two possible configurations, one for a standard TLS certificate on a host
+with a well-known name, one one for use with self-signed certificates for other setups.
+
+Note in the following that there are two .ini files, one for Sonar on the nodes and one for the
+Kafka proxy on the backend.  Both of those have `[kafka]` and `[http]` sections.  Do not confuse
+them.
+
+### Well-known host names, standard cert
+
+A host such as `naic-monitor.uio.no` has a fully qualified, globally visible name, and a cert from a
+normal CA.  In this case, Sonar on the cluster nodes is configured *without* a `kafka.ca-file`
+property but *with* an HTTPS `kafka.rest-endpoint`.  On the backend, a standard web server takes
+care of TLS and forwards traffic over plain HTTP to the kafka proxy that runs on the backend but is
+not externally visible.  For example, for nginx I use this:
 
 ```
 	location /kprox {
 		proxy_pass http://localhost:8090;
 	}
 ```
 
-and set up the `rest-endpoint` to be `https://my-kafka-host.uio.no/kprox`.  The 8090 port is the
-default for the proxy.  Then the ini file *for the proxy* is usually pretty simple, these values are
-exactly those that were used in the Sonar config file when it was speaking directly to the Kafka
-broker:
+For Sonar on the cluster nodes, I set up the corresponding `rest-endpoint` to be
+`https://my-kafka-host.uio.no/kprox`.
+
+### Private or no host name, self-signed cert
+
+For test setups or hosts that have no names or at least not globally or organization-wide visible
+names, a self-signed certificate can be used for HTTPS.  In this case we have two files, the server
+certificate (call it sonar-ca.crt) and the secret key (sonar-ca-key.pem).  The certificate is not
+secret and is distributed to all the cluter nodes.  For Sonar, `kafka.ca-file` is set to point to
+that file, along with a `kafka.rest-endpoint` using HTTPS (as above).  On the proxy, the
+`http.ca-file` points to the server certificate and `http.key-file` points to the secret key.  This
+is sufficient for Sonar and the proxy to be communicating over HTTPS.
+
+### Proxy-to-Kafka communication
+
+Then the ini file for the proxy is usually pretty simple, these values are exactly those that were
+used in the Sonar config file when it was speaking directly to the Kafka broker:
 
 ```
 [kafka]

util/kafka-proxy/kprox.go

@@ -1,5 +1,5 @@
 // SPDX-License-Identifier: MIT
-
+//
 // Copyright (c) 2023-2026 Norwegian Ai Cloud
 
 // Kprox is a very simple Kafka REST proxy, written for Sonar but probably generally useful.
@@ -25,20 +25,25 @@
 //
 // # Config file
 //
-// The config file is on .ini format with http, kafka, and debug sections:
+// The config file is on .ini format with http, kafka, and debug sections, all settings have
+// defaults.
+//
+// The http section configures the connection between the remote Sonar and the proxy:
 //
 //	[http]
 //	endpoint = ...        # default /
 //	listen-port = ...     # default 8090
-//  ca-file = ...         # default ""
-//  key-file = ...        # default ""
+//	ca-file = ...         # default ""
+//	key-file = ...        # default ""
 //
 // The proxy listens on for incoming traffic the interface ":{http.listen-port}{http.endpoint}",
 // by default ":8090/".
 //
 // If http.ca-file has a value then http.key-file must also have a value (and vice versa), and
 // the proxy will listen for https traffic only, using that information.
 //
+// The kafka section configures the connection between the proxy and the Kafka broker:
+//
 //	[kafka]
 //	broker-address = ...  # default localhost:9099
 //	ca-file = ...         # default none
@@ -53,16 +58,16 @@
 //
 // kafka.timeout is how long to hold messages without broker contact before discarding them.
 //
-// If -D is present, the [debug] section is honored:
+// The [debug] section is honored if -D is present on the command line:
 //
 //	[debug]
 //	dump = filename
 //	user = username
 //	password = password
 //
 // If there is a debug.dump, all validated incoming data are appended to that file.  If there are
-// debug.user and/or debug.password then the sasl-user / sasl-password fields must be set in the
-// control object and must match the user / password or the message is rejected, not dumped.
+// debug.user and/or debug.password properties then the sasl-user / sasl-password fields must be set
+// in the control object and must match the user / password or the message is rejected, not dumped.
 //
 // # Protocol
 //