Display topic throughput stats

[Haarolean]

topic list: add production/consumption column (one column 2 rows)
like this:

topic overview pane: display production, consumption

[acass91]

Hi @Haarolean is this issue available to be worked on for Hacktoberfest?

[Haarolean]

Hey, yes, it is since it’s marked with the label “hacktoberfest”.On 8 Oct 2022, at 08:48, acass91 ***@***.***> wrote: Hi @Haarolean is this issue available to be worked on for Hacktoberfest? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[acass91]

I can give this one a go.

[Haarolean]

Sure, go ahead :)

[acass91]

Edit - I was missing the necessary package to run the Docker daemon. It builds provided that I skip the tests.

I encountered a build issue with openjdk-18 on Linux.
I tried running ./mvnw clean install -Dmaven.test.skip=true -Pprod, but it looks like it is still trying to run some tests.

[ERROR] Tests run: 244, Failures: 0, Errors: 86, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for kafka-ui 0.0.1-SNAPSHOT:
[INFO] 
[INFO] kafka-ui ........................................... SUCCESS [  0.101 s]
[INFO] kafka-ui-contract .................................. SUCCESS [  7.669 s]
[INFO] kafka-ui-serde-api ................................. SUCCESS [  0.174 s]
[INFO] kafka-ui-api ....................................... FAILURE [ 24.161 s]
[INFO] kafka-ui-e2e-checks ................................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  32.248 s
[INFO] Finished at: 2022-10-08T21:06:07-06:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.22.2:test (default-test) on project kafka-ui-api: There are test failures.
[ERROR] 
[ERROR] Please refer to /home/adam/kafka-ui/kafka-ui-api/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :kafka-ui-api

I see this warning in the dumpstream file.
OpenJDK 64-Bit Server VM warning: Ignoring option --illegal-access=permit; support was removed in 17.0

[acass91]

I am seeing some errors when trying to access the React app on port 3000.

[Haarolean]

@acass91 no idea. Have you set the dev_proxy variable mentioned in kafka-ui-react-app/README.md?

[acass91]

I have not. I followed the Docker method which said to make sure the variable is not set. Access works on port 8080, but it requires a docker-compose -f ./documentation/compose/kafka-ui.yaml down, build, and docker-compose -f ./documentation/compose/kafka-ui.yaml up -d before I can see any changes take effect.

Also, I was looking at using getValue<Topic['bytesInPerSec']>() and getValue<Topic['bytesOutPerSec']>(), but they are always undefined. Is there a quick way to check which metrics are available?

[Haarolean]

@acass91 port 8080 is for the app being run in a docker container, there are both frontend and backend, you just have to use that container as a proxy for the backend part.
Please follow this guide.

Have you tried running a producer/consumer for this matter?

[acass91]

Okay, it loads after creating a .env.local file containing VITE_DEV_PROXY.

Make sure that none of the .env* files contain DEV_PROXY variable
What does this line mean in the guide if I'm actually supposed to define VITE_DEV_PROXY?
Edit - I think I know what you mean now. You do not want a proxy in the container which is running everything. Sorry that wasn't clear to me before.

I have one console producer/consumer pair running. I can see the number of messages increase as I enter text.

[Haarolean]

Okay, it loads after creating a .env.local file containing VITE_DEV_PROXY.

Make sure that none of the .env* files contain DEV_PROXY variable What does this line mean in the guide if I'm actually supposed to define VITE_DEV_PROXY? Edit - I think I know what you mean now. You do not want a proxy in the container which is running everything. Sorry that wasn't clear to me before.

I have one console producer/consumer pair running. I can see the number of messages increase as I enter text.

Probly it means that you don't add this property to any other env file besides a local one.

Which API request's payload are you trying to fetch this info from?

[acass91]

I was using getValue incorrectly. Now I have some formatting to fix.

Edit - How's this?

[Haarolean]

@acass91 pretty good, but let's get rid of zeroes in case of minor units. Also, please format the sizes like we do everywhere in the app (human readable, KBs, MBs, etc)

[acass91]

I'm getting some pre-commit errors that don't appear to be related to my changes.

> [email protected] pre-commit
> pnpm tsc && lint-staged


> [email protected] tsc /home/adam/kafka-ui/kafka-ui-react-app
> tsc --pretty --noEmit

src/components/Brokers/utils/__test__/fixtures.ts:16:7 - error TS2322: Type '{ OneMinuteRate: number; FifteenMinuteRate: number; Count: number; FiveMinuteRate: number; MeanRate: number; }' is not assignable to type 'number'.

16       value: {
         ~~~~~

  src/generated-sources/models/Metric.ts:39:5
    39     value?: number;
           ~~~~~
    The expected type comes from property 'value' which is declared here on type 'Metric'

src/components/Brokers/utils/__test__/fixtures.ts:32:7 - error TS2322: Type '{ Mean: number; StdDev: number; '75thPercentile': number; '98thPercentile': number; Min: number; '95thPercentile': number; '99thPercentile': number; Max: number; '999thPercentile': number; Count: number; '50thPercentile': number; }' is not assignable to type 'number'.

32       value: {
         ~~~~~

  src/generated-sources/models/Metric.ts:39:5
    39     value?: number;
           ~~~~~
    The expected type comes from property 'value' which is declared here on type 'Metric'

src/components/Brokers/utils/__test__/fixtures.ts:54:7 - error TS2322: Type '{ OneMinuteRate: number; FifteenMinuteRate: number; Count: number; FiveMinuteRate: number; MeanRate: number; }' is not assignable to type 'number'.

54       value: {
         ~~~~~

  src/generated-sources/models/Metric.ts:39:5
    39     value?: number;
           ~~~~~
    The expected type comes from property 'value' which is declared here on type 'Metric'

src/components/Brokers/utils/__test__/fixtures.ts:71:7 - error TS2322: Type '{ OneMinuteRate: number; FifteenMinuteRate: number; Count: number; FiveMinuteRate: number; MeanRate: number; }' is not assignable to type 'number'.

71       value: {
         ~~~~~

  src/generated-sources/models/Metric.ts:39:5
    39     value?: number;
           ~~~~~
    The expected type comes from property 'value' which is declared here on type 'Metric'

src/components/Brokers/utils/__test__/fixtures.ts:88:7 - error TS2322: Type '{ OneMinuteRate: number; FifteenMinuteRate: number; Count: number; FiveMinuteRate: number; MeanRate: number; }' is not assignable to type 'number'.

88       value: {
         ~~~~~

  src/generated-sources/models/Metric.ts:39:5
    39     value?: number;
           ~~~~~
    The expected type comes from property 'value' which is declared here on type 'Metric'

src/components/Brokers/utils/__test__/fixtures.ts:105:7 - error TS2322: Type '{ OneMinuteRate: number; FifteenMinuteRate: number; Count: number; FiveMinuteRate: number; MeanRate: number; }' is not assignable to type 'number'.

105       value: {
          ~~~~~

  src/generated-sources/models/Metric.ts:39:5
    39     value?: number;
           ~~~~~
    The expected type comes from property 'value' which is declared here on type 'Metric'

src/components/Brokers/utils/__test__/fixtures.ts:122:7 - error TS2322: Type '{ Value: number; }' is not assignable to type 'number'.

122       value: {
          ~~~~~

  src/generated-sources/models/Metric.ts:39:5
    39     value?: number;
           ~~~~~
    The expected type comes from property 'value' which is declared here on type 'Metric'

src/components/Brokers/utils/__test__/fixtures.ts:134:7 - error TS2322: Type '{ Value: number; }' is not assignable to type 'number'.

134       value: {
          ~~~~~

  src/generated-sources/models/Metric.ts:39:5
    39     value?: number;
           ~~~~~
    The expected type comes from property 'value' which is declared here on type 'Metric'

src/components/Brokers/utils/__test__/fixtures.ts:146:7 - error TS2322: Type '{ OneMinuteRate: number; FifteenMinuteRate: number; Count: number; FiveMinuteRate: number; MeanRate: number; }' is not assignable to type 'number'.

146       value: {
          ~~~~~

  src/generated-sources/models/Metric.ts:39:5
    39     value?: number;
           ~~~~~
    The expected type comes from property 'value' which is declared here on type 'Metric'

src/components/Brokers/utils/__test__/fixtures.ts:163:7 - error TS2322: Type '{ OneMinuteRate: number; FifteenMinuteRate: number; Count: number; FiveMinuteRate: number; MeanRate: number; }' is not assignable to type 'number'.

163       value: {
          ~~~~~

  src/generated-sources/models/Metric.ts:39:5
    39     value?: number;
           ~~~~~
    The expected type comes from property 'value' which is declared here on type 'Metric'


Found 10 errors in the same file, starting at: src/components/Brokers/utils/__test__/fixtures.ts:16

 ELIFECYCLE  Command failed with exit code 2.
husky - pre-commit hook exited with code 1 (error)

[Haarolean]

I'm getting some pre-commit errors that don't appear to be related to my changes.

no idea, it works on master, sooo :)

@provectus/kafka-frontend any ideas?

[acass91]

Hmm, I tried downgrading my Java to 13, doing a clean build, and re-running pnpm gen:sources, but I'm not having any luck.
Here's the diff with my changes.

--- a/kafka-ui-react-app/src/components/Topics/List/TopicTable.tsx
+++ b/kafka-ui-react-app/src/components/Topics/List/TopicTable.tsx
@@ -13,6 +13,27 @@ import { TopicTitleCell } from './TopicTitleCell';
 import ActionsCell from './ActionsCell';
 import BatchActionsbar from './BatchActionsBar';
 
+import BytesFormatted from 'components/common/BytesFormatted/BytesFormatted';
+
+function formatThroughput(row: Topic) {
+  const production = row.bytesInPerSec;
+  const consumption = row.bytesOutPerSec;
+  if (production === undefined && consumption === undefined) {
+    return (<tr>N/A</tr>);
+  } else if (production === undefined) {
+    return (<tr>out: <BytesFormatted value={consumption}/> </tr>);
+  } else if (consumption === undefined)  {
+    return (<tr>in: <BytesFormatted value={production}/></tr>);
+  } else {
+    return (
+      <div>
+        <tr>in: <BytesFormatted value={production}/></tr>
+        <tr>out: <BytesFormatted value={consumption}/></tr>
+      </div>
+    );
+  }
+}
+
 const TopicTable: React.FC = () => {
   const { clusterName } = useAppParams<{ clusterName: ClusterName }>();
   const [searchParams] = useSearchParams();
@@ -85,6 +106,14 @@ const TopicTable: React.FC = () => {
         accessorKey: 'segmentSize',
         cell: SizeCell,
       },
+      {
+        header: 'Throughput',
+        accessorFn: row => formatThroughput(row),
+        enableSorting: false,
+        cell: ({ getValue }) => {
+          return getValue();
+        },
+      },
       {
         id: 'actions',
         header: '',
diff --git a/kafka-ui-react-app/src/components/Topics/Topic/Overview/Overview.tsx b/kafka-ui-react-app/src/components/Topics/Topic/Overview/Overview.tsx
index 2bd1c4a7..bddf6d28 100644
--- a/kafka-ui-react-app/src/components/Topics/Topic/Overview/Overview.tsx
+++ b/kafka-ui-react-app/src/components/Topics/Topic/Overview/Overview.tsx
@@ -88,6 +88,12 @@ const Overview: React.FC = () => {
           <Metrics.Indicator label="Message Count">
             {messageCount}
           </Metrics.Indicator>
+          <Metrics.Indicator label="Production">
+            <BytesFormatted value={data?.bytesInPerSec}/>
+          </Metrics.Indicator>
+          <Metrics.Indicator label="Consumption">
+            <BytesFormatted value={data?.bytesOutPerSec}/>
+          </Metrics.Indicator>
         </Metrics.Section>
       </Metrics.Wrapper>
       <Table isFullwidth> 

[Haarolean]

@acass91 try to commit it with --no-verify, let's see if it fails on CI as well