KAFKA-18404: Remove partitionMaxBytes usage from DelayedShareFetch

[adixitconfluent]

About

This PR aims to remove partitionMaxBytes from share fetch requests(SFR). In the current flow, we cannot fetch more than 1 MB for each topic partition in the share fetch request by default although the overall SFR max bytes is 50MB. This PR has removed this limit by using strategies from a strategy class PartitionMaxBytesStrategy.
PartitionMaxBytesStrategy contains an UNIFORM strategy that divides overall SFR max bytes equally among the acquired topic partitions. For example - If SFR max bytes = 50MB and there are 5 acquired topic partitions, then total data that can be fetched for each topic partition from replica manager can be upto 10MB each.

Testing

Added unit tests for PartitionMaxBytesStrategy and tested with already present unit tests on broker and integration tests for share consumers

[adixitconfluent]

Test failures are unrelated to my changes.

[apoorvmittal10]

Thanks for updating the PR. Can you please help me understand how we will extend this code in future with minimal changes?

[apoorvmittal10]

How is this class or way extensible in future to add more startegies? Shouldn't you need the interface with enum which shall provide you the right implementing strategy class?

[adixitconfluent]

@apoorvmittal10, you can add a strategy to enum StrategyType and add the corresponding code to partitionMaxBytes function for handling the new strategy type. If you do not like this way I can create an interface

public interface PartitionMaxBytesDivisionStrategy {
       public LinkedHashMap<TopicIdPartition, Integer> partitionMaxBytes(int requestMaxBytes, Set<TopicIdPartition> partitions, int acquiredPartitionsSize);
}

Then use subclasses to implement this function. Wdyt?

[apoorvmittal10]

I don't think it's good as you would require to chnage couple of additional lines in DelayedShareFetch. Rather we should write over a single class/interface which should provide the implementation based on StrategyType.

[apoorvmittal10]

What do you think about below implementation and usage?

this.partitionMaxBytesStrategy = PartitionMaxBytesStrategy.type(StrategyType.UNIFORM);

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.kafka.server.share.fetch;

import org.apache.kafka.common.TopicIdPartition;

import java.util.LinkedHashMap;
import java.util.Locale;
import java.util.Set;

public interface PartitionMaxBytesStrategy {

    enum StrategyType {
        UNIFORM;

        @Override
        public String toString() {
            return super.toString().toLowerCase(Locale.ROOT);
        }
    }

    LinkedHashMap<TopicIdPartition, Integer> maxBytes(int requestMaxBytes, Set<TopicIdPartition> partitions, int acquiredPartitionsSize);

    static PartitionMaxBytesStrategy type(StrategyType type) {
        return switch (type) {
            case UNIFORM -> PartitionMaxBytesStrategy::uniformPartitionMaxBytes;
        };
    }

    /**
     * Returns the partition max bytes for a given partition based on the strategy type.
     *
     * @param requestMaxBytes - The total max bytes available for the share fetch request
     * @param partitions - The topic partitions in the order for which we compute the partition max bytes.
     * @param acquiredPartitionsSize - The total partitions that have been acquired.
     * @return the partition max bytes for the topic partitions
     */
    static LinkedHashMap<TopicIdPartition, Integer> uniformPartitionMaxBytes(int requestMaxBytes, Set<TopicIdPartition> partitions, int acquiredPartitionsSize) {
        if (partitions == null || partitions.isEmpty()) {
            throw new IllegalArgumentException("Partitions to generate max bytes is null or empty");
        }
        if (requestMaxBytes <= 0) {
            throw new IllegalArgumentException("Requested max bytes must be greater than 0");
        }
        if (acquiredPartitionsSize <= 0) {
            throw new IllegalArgumentException("Acquired partitions size must be greater than 0");
        }

        LinkedHashMap<TopicIdPartition, Integer> partitionMaxBytes = new LinkedHashMap<>();
            partitions.forEach(partition -> partitionMaxBytes.put(
                partition, requestMaxBytes / acquiredPartitionsSize));
        return partitionMaxBytes;
    }
}

[apoorvmittal10]

Thanks for the changes, some comments and questions.

[apoorvmittal10]

Overall LGTM, some comments to address.

[apoorvmittal10]

Thanks for the changes. I have a comment around tests, I think we should add tests for the functionality in DelayedShareFetch.

[apoorvmittal10]

LGTM, thanks for the PR and fixes.

[AndrewJSchofield]

lgtm

[chia7712]

I apologize for the delayed review. Could you please use Set.of() to address the following warnings?

> Task :share:compileTestJava
Note: /home/jenkins/kafka/share/src/test/java/org/apache/kafka/server/share/fetch/PartitionMaxBytesStrategyTest.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

[apoorvmittal10]

@chia7712 I missed it in review, thanks for pointing out. @adixitconfluent is busy so I just raised this minor PR: #18541

[ijuma]

I cherry-picked this to 4.0 as it made another cherry-pick simpler (#18524).

[apoorvmittal10]

I cherry-picked this to 4.0 as it made another cherry-pick simpler (#18524).

@AndrewJSchofield Do you think it's sensible to include the config for limiting bytes in single share fetch as the partition max bytes is now dynamic, in your doc? I am just thinking in terms of clients if using default value in 4.0 then might see bigger chunk of data flowing to client.