KAFKA-14648: Do not fail clients if bootstrap servers is not immediately resolvable by frankvicky · Pull Request #21080 · apache/kafka

frankvicky · 2025-12-04T18:20:43Z

This PR aims to deliver
KIP-909.

Reviewers: Lianet Magrans 98415067+lianetm@users.noreply.github.com

github-actions · 2025-12-12T03:35:26Z

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

github-actions · 2025-12-17T03:34:11Z

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

github-actions · 2025-12-19T03:35:31Z

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

github-actions · 2026-01-04T03:39:20Z

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

chia7712 · 2026-03-18T21:09:12Z

clients/src/main/java/org/apache/kafka/clients/admin/internals/AdminMetadataManager.java

+
+        @Override
+        public void bootstrap(List<InetSocketAddress> addresses) {
+            // AdminClient handles bootstrap during construction, so this method is not used


KIP-909 states "Consumer, Producer, and Admin Clients: The bootstrap code will be changed.", so this comment is not correct

chia7712 · 2026-03-18T21:21:32Z

clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java

                config.originalsWithPrefix(CommonClientConfigs.METRICS_CONTEXT_PREFIX));
            metrics = new Metrics(metricConfig, reporters, time, metricsContext);
+
+            // Use the appropriate bootstrap configuration determined by AdminBootstrapAddresses


It seems AdminBootstrapAddresses.fromConfig(config); will resolve the dns already, which is conflict with KIP-909

Nikita-Shupletsov · 2026-03-19T06:16:28Z

looks like the test failures may be related to the change

chia7712 · 2026-03-19T15:15:16Z

clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java

+                    unresolvedAddresses.add(InetSocketAddress.createUnresolved(host, port));
+                }
+            }
+            metadataManager.update(Cluster.bootstrap(unresolvedAddresses), time.milliseconds());


The cluster created by Cluster.bootstrap has isBootstrapConfigured=true, so bootstrapCluster will be updated immediately. This makes isBootstrapped() incorrectly return true

Maybe we should leverage MetadataUpdater#bootstrap to set the resolved bootstrap servers once the DNS resolution succeeds in NetworkClient?

chia7712 · 2026-03-19T15:23:10Z

clients/src/main/java/org/apache/kafka/clients/NetworkClient.java

+            long remainingBootstrapTimeMs = bootstrapTimer.remainingMs();
+            long sleepTimeMs = Math.min(Math.min(remainingPollTimeMs, remainingBootstrapTimeMs), bootstrapConfiguration.retryBackoffMs);
+
+            if (sleepTimeMs > 0) {


What happens if users try to interrupt the poll?

chia7712

@frankvicky Just wondering, did we discuss the blocking DNS resolution in poll before? I'm thinking we could use a separate BootstrapResolver class with an independent thread to handle this. We can pass it to Metadata or AdminMetadataManager so we don't have to change NetworkClient too much. NetworkClient#ensureBootstrapped can just poll the metadataUpdater's state with blocking.

chia7712 · 2026-03-21T19:49:52Z

clients/src/main/java/org/apache/kafka/clients/NetworkClient.java

+
+        while (true) {
+            // Check if thread has been interrupted
+            if (Thread.interrupted()) {


Should we propagate the exception directly?

if (Thread.interrupted()) { throw new InterruptException(new InterruptedException()); }

chia7712 · 2026-03-21T19:52:05Z

clients/src/main/java/org/apache/kafka/common/Cluster.java

+     * @param isBootstrapConfigured whether the cluster is bootstrapped
+     * @return a new Cluster instance with the specified bootstrap state
+     */
+    public static Cluster withBootstrapFlag(String clusterId,


This change seems to be unnecessary. Also, it is a public class so we should not make this change here

I understand your concern about adding a new public API. However, the Cluster.withBootstrapFlag() method is necessary for the KIP-909 implementation. Here's why:

The isBootstrapConfigured flag must be preserved across metadata updates:

After DNS resolution succeeds, we create a cluster with isBootstrapConfigured=true

When a partial metadata update occurs (via MetadataSnapshot.mergeWith()), this flag needs to be preserved

This allows AdminMetadataManager.update() to correctly distinguish between bootstrap cluster and real cluster metadata

Package structure constraints:

Cluster is in the org.apache.kafka.common package

MetadataSnapshot is in the org.apache.kafka.clients package

We cannot use package-private methods to access each other

The only options are: public static method or restructuring the package layout

Issues with alternative approaches:

Not preserving the flag: Would break AdminMetadataManager's ability to correctly identify bootstrap clusters, affecting re-bootstrap logic

Modifying package structure: Too large a change with high risk

Other designs: All require similar public API additions or more complex refactoring

Alternative consideration:
If adding this public method is a blocker, the alternative would be to:

Move both Cluster and MetadataSnapshot to the same package, or

Add a ClusterBuilder class in the clients package that has access to create clusters with specific bootstrap flags

we had on offline discussion about this one. Another option to consider is to completely remove the isBootstrapConfigured flag, and rely on checking if metadata has nodes instead (having nodes, either bootstrap nodes or from real metadata, means that we can skip DNS resolution, which is what the isBootstrapConfigured was used for).

The changes are in place now, with that, no need for changes on this Cluster class. Thoughts @chia7712 ?

chia7712 · 2026-03-21T19:54:55Z

clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java

                config.getLong(AdminClientConfig.RETRY_BACKOFF_MS_CONFIG),
                config.getLong(AdminClientConfig.METADATA_MAX_AGE_CONFIG),
                adminAddresses.usingBootstrapControllers());
-            metadataManager.update(Cluster.bootstrap(adminAddresses.addresses()), time.milliseconds());


it seems adminAddresses.addresses() is not used any more. Maybe we could just remove AdminBootstrapAddresses and use a helper method instead?

chia7712 · 2026-03-21T21:19:24Z

did we discuss the blocking DNS resolution in poll before? I'm thinking we could use a separate BootstrapResolver class with an independent thread to handle this. We can pass it to Metadata or AdminMetadataManager so we don't have to change NetworkClient too much. NetworkClient#ensureBootstrapped can just poll the metadataUpdater's state with blocking.

the while loop in ensureBootstrapped is problematic because it introduces an unexpected blocking logic into the event loop. The origin design likely didn't intend to have a loop here. To keep it non-blocking, it should just return immediately if the dns resolution is not ready for now

lianetm · 2026-03-27T14:54:15Z

About the blocking behaviour, agree with removing the while loop within the network client poll, and let the retry happen on the next poll.
But even with that, we still have a blocking call to parseAddresses(single one, not in a while, but still blocking because it triggers InetAddress.getAllByName). I wonder if we could consider running this async? triggered only once when needed from the network poll (so the overhead of the extra thread is only to bootstrap), and with that we don't block on network poll and just conitnue to check if that asyn op finished in time, something like that. Thoughts?

-- update
Adding follow-up thoughts on this (tradeoff between the impact of the blocking we have vs the overhead of any asyn op). With the current shape the blocking is really is a single blocking call to parseAddresses within poll (blocks on single DNS resolution attempt, not for the full bootstrap timer), and the important bit is that the blocking would only happen initially when bootstrapping for the first time (at a point where the network client is really not functional yet). Is my understanding correct? If so then we're probably better off not introducing any overhead for running this async (and just need to handle interrupts?)

lianetm · 2026-03-27T19:36:17Z

clients/src/main/java/org/apache/kafka/clients/NetworkClient.java

+
+        // DNS resolution failed
+        if (bootstrapTimer.isExpired()) {
+            throw new BootstrapResolutionException("Failed to resolve bootstrap servers after " +


from the KIP I get that we want this to propagate to the consumer.poll right?

If so, then we need to catch this exception and send an error event to the app thread, otherwise this error will be swallowed in the background (this

kafka/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkThread.java

Line 165 in 73cb15e

log.error("Unexpected error caught in consumer network thread", e);

)

One option I see is to catch this at the NetworkClientDelegate level, where we have the backgroundEventHandler, and propagate the error there via event (just like we do to propagate metadata errors)

Btw, we really need to have a test at the KafkaConsumerTest level covering this behaviour. E.g, create consumer with invalid host, poll continuously, poll should eventually throw the bootstrap exception (for both consumers). I expect that wouldn't work now for the async.

lianetm · 2026-04-07T15:40:43Z

clients/src/test/java/org/apache/kafka/clients/consumer/KafkaConsumerTest.java

    }

+    @Test
+    public void testAsyncConsumerBootstrapResolutionExceptionPropagatedToPoll() throws InterruptedException {


this behaviour applies equally to both consumers. Should we better parameterize the test to run it for group.protocol=classic too?

KAFKA-14648: Add boostrap timeout config

e5be409

github-actions bot added core Kafka Broker producer consumer tools connect clients triage PRs from the community labels Dec 4, 2025

frankvicky added 2 commits December 5, 2025 16:41

Merge branch 'trunk' into KAFKA-14648

62f852a

fix the error test

e1f8466

github-actions bot added the needs-attention label Dec 12, 2025

Seperate config and state

4a06620

github-actions bot removed the needs-attention label Dec 16, 2025

frankvicky added 2 commits December 16, 2025 10:47

add disable instance to BootstrapConfiguration

b6fe307

Fix tests

2a146be

github-actions bot added the needs-attention label Dec 17, 2025

Merge branch 'trunk' into KAFKA-14648

9922711

github-actions bot removed the needs-attention label Dec 18, 2025

github-actions bot added the needs-attention label Dec 19, 2025

frankvicky added 2 commits January 2, 2026 16:01

Merge branch 'trunk' into KAFKA-14648

668f24e

Fix the failed tests

9f1afdb

github-actions bot removed the needs-attention label Jan 3, 2026

github-actions bot added the needs-attention label Jan 4, 2026

frankvicky added 3 commits January 5, 2026 09:49

Fix failed tests

f7adeda

Merge branch 'trunk' into KAFKA-14648

cf31b6e

Fix failed tests

3168e32

chia7712 reviewed Mar 18, 2026

View reviewed changes

frankvicky added 2 commits March 18, 2026 22:16

address comments

2550e57

Merge branch 'trunk' into KAFKA-14648

4ebbd77

frankvicky added 2 commits March 19, 2026 07:19

fix failed tests

4566ef1

Merge branch 'trunk' into KAFKA-14648

0c15459

chia7712 reviewed Mar 19, 2026

View reviewed changes

frankvicky added 3 commits March 19, 2026 16:06

address comments

e1b31b2

Merge branch 'trunk' into KAFKA-14648

b368e71

address comments

bfdd24c

chia7712 reviewed Mar 21, 2026

View reviewed changes

frankvicky added 5 commits March 25, 2026 12:36

Merge branch 'trunk' into KAFKA-14648

8645123

address comments

e3159c8

address comments

4e745ab

address comments

0621ede

Merge branch 'trunk' into KAFKA-14648

9ceba73

frankvicky added 3 commits March 27, 2026 15:37

add interrupt mechanism

ff5492e

Merge branch 'trunk' into KAFKA-14648

3d9c1b2

address comments

c0ce72a

lianetm reviewed Mar 27, 2026

View reviewed changes

frankvicky added 4 commits March 30, 2026 14:14

Merge branch 'trunk' into KAFKA-14648

faa8054

Merge branch 'trunk' into KAFKA-14648

02ca329

Merge branch 'trunk' into KAFKA-14648

11e5b3f

address comments

752d79c

lianetm reviewed Apr 7, 2026

View reviewed changes

frankvicky added 2 commits April 8, 2026 14:25

Merge branch 'trunk' into KAFKA-14648

46eb423

address comments

f78ddc3

Conversation

frankvicky commented Dec 4, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 12, 2025

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

github-actions bot commented Dec 19, 2025

Uh oh!

github-actions bot commented Jan 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Nikita-Shupletsov commented Mar 19, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chia7712 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chia7712 commented Mar 21, 2026

Uh oh!

lianetm commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

frankvicky commented Dec 4, 2025 •

edited by github-actions bot

Loading

lianetm commented Mar 27, 2026 •

edited

Loading