CH-259: Update kafka deployment by alxbrd · Pull Request #843 · MetaCell/cloud-harness

alxbrd · 2026-04-02T12:54:27Z

Closes CH-259

Implemented solution

Update kafka
Update kafka-ui
Configure Kafka in single-node KRaft
Remoe pzoo and zookeeper

How to test this PR

Deploy config with events and check sync works.
Verify Kafka-ui and topic creation / messages.

Sanity checks:

The pull request is explicitly linked to the relevant issue(s)
The issue is well described: clearly states the problem and the general proposed solution(s)
In this PR it is explicitly stated how to test the current change
The labels in the issue set the scope and the type of issue (bug, feature, etc.)
The relevant components are indicated in the issue (if any)
All the automated test checks are passing
All the linked issues are included in one Sprint
All the linked issues are in the Review state
All the linked issues are assigned

Breaking changes (select one):

The present changes do not change the preexisting api in any way
This PR and the issue are tagged as a breaking-change and the migration procedure is well described above

Possible deployment updates issues (select one):

There is no reason why deployments based on CloudHarness may break after the current update
This PR and the issue are tagged as alert:deployment

Test coverage (select one):

Tests for the relevant cases are included in this pr
The changes included in this pr are out of the current test coverage scope

Documentation (select one):

The documentation has been updated to match the current changes
The changes included in this PR are out of the current documentation scope

Nice to have (if relevant):

Screenshots of the changes
Explanatory video/animated gif

Update Kafka-UI to the latest version.

Copilot

Pull request overview

Updates the CloudHarness “events” deployment to run Kafka as a single-node KRaft StatefulSet (removing ZooKeeper and related resources) and refreshes the bundled Kafka UI Helm chart/templates to newer versions and Kubernetes API expectations.

Changes:

Switch events Kafka deployment to KRaft mode using docker.io/apache/kafka:4.0.2, removing ZooKeeper/pzoo manifests and broker configmaps/scripts.
Update documentation for Kafka UI access and local troubleshooting/reset steps for KRaft PVC state.
Update Kafka UI Helm chart (version bump + template updates for namespace, ingress pathType, LoadBalancer IP, etc.).

Reviewed changes

Copilot reviewed 28 out of 28 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
docs/events.md	Update docs for Kafka UI, values-based config overrides, and KRaft PVC reset/local port-forwarding.
applications/events/deploy/values.yaml	Define new Kafka image/ports/KRaft config and remove ZooKeeper-related values.
applications/events/deploy/templates/zoo-config.yml	Remove ZooKeeper ConfigMap template.
applications/events/deploy/templates/services.yml	Update services for Kafka client/controller ports; remove ZooKeeper services.
applications/events/deploy/templates/roles.yml	Remove RBAC previously used for pod labeling/annotation in init scripts.
applications/events/deploy/templates/deployments.yml	Replace prior Kafka+ZooKeeper StatefulSets with a single KRaft Kafka StatefulSet.
applications/events/deploy/templates/broker-config.yml	Remove broker ConfigMap that provided legacy server/log4j configs.
applications/events/deploy/templates-compose/deployments.yaml	Simplify compose deployment to a single Kafka service configured for KRaft.
applications/events/deploy/resources/zookeeper/zookeeper.properties	Remove ZooKeeper configuration.
applications/events/deploy/resources/zookeeper/log4j.properties	Remove ZooKeeper logging configuration.
applications/events/deploy/resources/zookeeper/init.sh	Remove ZooKeeper init script.
applications/events/deploy/resources/broker/server.properties	Remove legacy broker server.properties.
applications/events/deploy/resources/broker/log4j.properties	Remove legacy broker log4j configuration.
applications/events/deploy/resources/broker/init.sh	Remove legacy broker init script (kubectl labeling, listener patching, etc.).
applications/events/deploy/charts/values.yaml	Add ingress `pathType` default and document optional LoadBalancer static IP.
applications/events/deploy/charts/templates/serviceaccount.yaml	Explicitly set namespace on ServiceAccount.
applications/events/deploy/charts/templates/service.yaml	Explicitly set namespace; optionally set `loadBalancerIP`.
applications/events/deploy/charts/templates/secret.yaml	Make Secret creation conditional; explicitly set namespace.
applications/events/deploy/charts/templates/networkpolicy-ingress.yaml	Explicitly set namespace.
applications/events/deploy/charts/templates/networkpolicy-egress.yaml	Explicitly set namespace.
applications/events/deploy/charts/templates/ingress.yaml	Add namespace; make `pathType` configurable; enhance backend port selection.
applications/events/deploy/charts/templates/hpa.yaml	Explicitly set namespace.
applications/events/deploy/charts/templates/deployment.yaml	Explicitly set namespace; adjust Spring config env var handling.
applications/events/deploy/charts/templates/configmap.yaml	Explicitly set namespace; minor template formatting.
applications/events/deploy/charts/templates/configmap_fromValues.yaml	Explicitly set namespace; minor template formatting.
applications/events/deploy/charts/templates/_helpers.tpl	Allow global image registry override (`.Values.global.imageRegistry`).
applications/events/deploy/charts/README.md	Replace inline README with link to upstream helm chart docs.
applications/events/deploy/charts/Chart.yaml	Bump Kafka UI chart version/appVersion.

applications/events/deploy/charts/templates/deployment.yaml

applications/events/deploy/templates-compose/deployments.yaml

applications/events/deploy/charts/templates/ingress.yaml

Copilot

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 2 comments.

Copilot · 2026-04-02T13:50:23Z

applications/events/deploy/templates/services.yml

 kind: Service
 metadata:
  name: broker
 spec:
  clusterIP: None
-  ports:
-  - port: {{ .Values.apps.events.kafka.port }}
-  selector:
-    app: kafka
---
-apiVersion: v1
-kind: Service
-metadata:
-  name: pzoo
-spec:
-  clusterIP: None
-  ports:
-  - name: peer
-    port: 2888
-  - name: leader-election
-    port: 3888
-  selector:
-    app: zookeeper
-    storage: persistent
---
-apiVersion: v1
-kind: Service
-metadata:
-  name: zookeeper
-spec:
  ports:
  - name: client
-    port: 2181
+    port: {{ .Values.apps.events.kafka.port }}
+    targetPort: client
+  - name: controller
+    port: {{ .Values.apps.events.kafka.controllerPort }}
+    targetPort: controller


The broker Service is headless (clusterIP: None) and is used for stable StatefulSet DNS names (e.g., kafka-0.broker.<ns>.svc.cluster.local) referenced by KAFKA_ADVERTISED_LISTENERS/KAFKA_CONTROLLER_QUORUM_VOTERS. Without publishNotReadyAddresses: true, Kubernetes DNS may not publish pod A records until the pod becomes Ready, which can deadlock startup when the broker needs to resolve those names during initialization. Consider setting publishNotReadyAddresses: true on the headless broker Service.

Copilot · 2026-04-02T13:50:24Z

docs/events.md

+## Reset local KRaft state
+
+Kafka metadata and log state are stored on the `kafka` PVC. If a local Minikube deployment gets stuck because of stale KRaft state, delete the broker pod and its PVC before redeploying:
+
+```bash
+kubectl delete pod -n test kafka-0
+kubectl delete pvc -n test data-kafka-0
+```


This section mixes namespaces (test here vs mnp in the port-forward example below) and refers to a kafka PVC, but the PVC created by the StatefulSet is named data-kafka-0 (from the data volumeClaimTemplate). Consider using a single placeholder namespace consistently and referencing the actual PVC/claim name to avoid confusing reset instructions.

Alex Burdusel added 2 commits March 28, 2026 12:54

Update Kafka to 4.0.2 and configured KRaft mode.

094e5ac

Update Kafka-UI to the latest version.

Increase Kafka memory limit to avoid OOM restarts.

8acc7f0

alxbrd requested review from Copilot and filippomc April 2, 2026 12:54

Copilot started reviewing on behalf of alxbrd April 2, 2026 12:55 View session

Copilot AI reviewed Apr 2, 2026

View reviewed changes

Remove redundant template file from events config

04d660c

alxbrd requested a review from Copilot April 2, 2026 13:43

Copilot started reviewing on behalf of alxbrd April 2, 2026 13:44 View session

Copilot AI reviewed Apr 2, 2026

View reviewed changes

filippomc approved these changes Apr 2, 2026

View reviewed changes

filippomc merged commit ae229a2 into develop Apr 2, 2026
12 of 15 checks passed

filippomc deleted the feature/CH-259 branch April 2, 2026 17:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CH-259: Update kafka deployment#843

CH-259: Update kafka deployment#843
filippomc merged 3 commits intodevelopfrom
feature/CH-259

alxbrd commented Apr 2, 2026 •

edited by atlassian bot

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 2, 2026

Uh oh!

Copilot AI Apr 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

alxbrd commented Apr 2, 2026 • edited by atlassian bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Implemented solution

How to test this PR

Sanity checks:

Breaking changes (select one):

Possible deployment updates issues (select one):

Test coverage (select one):

Documentation (select one):

Nice to have (if relevant):

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alxbrd commented Apr 2, 2026 •

edited by atlassian bot

Loading