[Bug] Bookie (v4.1.2) Restarting Suddenly: ZKRegistrationClient Invalidate Cache / NetworkTopology Node Removal

[amjadali-klarity]

Search before reporting

• I searched in the issues and found nothing similar.

Read release policy

• I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

User environment

Pulsar Version: 4.1.2 (Image: docker.io/apachepulsar/pulsar-all:4.1.2)
Platform: Kubernetes (Azure/AKS based on log headers)
Component: BookKeeper / ZooKeeper

Issue Description

Our pulsar-bookie pods are experiencing sudden restarts. The logs indicate that the ZKRegistrationClient is invalidating the cache for the specific bookie address, followed by NetworkTopologyImpl removing the node from the /default-rack.

Error messages

INFO  org.apache.bookkeeper.discover.ZKRegistrationClient - Invalidate cache for pulsar-bookie-1.pulsar-bookie.pulsar.svc.cluster.local:3181
INFO  org.apache.bookkeeper.net.NetworkTopologyImpl - Removing a node: /default-rack/pulsar-bookie-1.pulsar-bookie.pulsar.svc.cluster.local:3181

Reproducing the issue

ZooKeeper logs show standard ruok commands but no explicit session expiration immediately preceding the drop.
The Bookie seems to be under normal load (Compaction usage buckets are mostly at 100%).
Config: diskUsageWarnThreshold = 0.9, isForceGCAllowWhenNoSpace = true.

Additional information

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[lhotari]

Our pulsar-bookie pods are experiencing sudden restarts.

Claude answer made with some prompting:

Diagnosing Pulsar-Bookie Pod Restarts

1. Check pod details and events

kubectl describe pod  -n 

Shows restart count, last termination reason (e.g. OOMKilled, exit code), and events (probe failures, back-offs) scoped to the pod.

2. Get logs from the crashed container

kubectl logs  -n  --previous

The --previous flag retrieves logs from the prior (crashed) instance.

3. Filter events for a specific pod

kubectl get events -n  --sort-by='.lastTimestamp' \
  --field-selector involvedObject.name=

Example:

kubectl get events -n pulsar --sort-by='.lastTimestamp' \
  --field-selector involvedObject.name=pulsar-bookie-0

Useful for spotting node-level evictions or scheduling issues.

4. Common causes

Cause	Indicator	Check
OOMKilled	Exit code 137	See OOMKilled details below
Liveness probe failure	Unhealthy / Killing events	Review probe timeoutSeconds and failureThreshold
Disk pressure	Pod evicted or crash loops	kubectl describe node and PV usage

OOMKilled details

Bookies consume memory well beyond the JVM heap. Direct memory (Netty direct buffers, RocksDB block cache and memtables) and JVM overhead (metaspace, thread stacks, GC) all contribute to total RSS.

-Xmx should be set to roughly 60–70% of the pod's memory limit, leaving the remaining 30–40% for direct memory allocations and JVM overhead.

If memory limits are set lower than memory requests, the container can be killed as soon as usage exceeds the limit. Ensure resources.limits.memory is equal to or greater than resources.requests.memory.

Node-level overprovisioning can also cause OOMKilled even when the pod itself is within its limits. When the node's total committed memory exceeds physical RAM, the Linux kernel's OOM killer may terminate a bookie process. Check node-level metrics:

kubectl describe node 
kubectl top node 

Look at Allocated resources in the describe output to see if the node is overcommitted.

[lhotari]

Reducing Netty direct memory usage

If restarts are related to OOMKilled, reducing Netty's direct memory allocation can help. Setting -Dio.netty.allocator.maxOrder controls the chunk size Netty pre-allocates for direct buffers.

Add it via PULSAR_EXTRA_OPTS in your values.yaml if using pulsar-helm-chart:

bookkeeper:
  configData:
    PULSAR_EXTRA_OPTS: >-
      -Dpulsar.allocator.exit_on_oom=true
      -Dio.netty.recycler.maxCapacityPerThread=4096
      -Dio.netty.allocator.maxOrder=10

• io.netty.allocator.maxOrder=10 — sets the PooledByteBufAllocator chunk size to 8 MB. The default order is 9 (4 MB chunks). Increasing to 10 allows larger contiguous allocations, which can reduce fragmentation for bookie workloads with heavy direct memory usage.
• pulsar.allocator.exit_on_oom=true — forces the process to exit on allocation failure instead of entering an unstable state.
• io.netty.recycler.maxCapacityPerThread=4096 — caps the per-thread object recycler pool size.

(related to #25021 / #25274, change is only made to 4.2.x)

[lhotari]

Also check Netflix ZGC originated tuning recommentations.
Some settings are Linux kernel settings at the node level, however IIRC cloud managed k8s nodes already provide sufficient configuration, but it's worth checking.

check Linux kernel settings with

cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/shmem_enabled
cat /sys/kernel/mm/transparent_hugepage/defrag
cat /sys/kernel/mm/transparent_hugepage/khugepaged/defrag

[amjadali-klarity]

Ok noted. Let me diagnose the pod and see the results