Add playbooks for SKMO by vakwetu · Pull Request #3684 · openstack-k8s-operators/ci-framework

vakwetu · 2026-02-13T21:59:54Z

Add multi-namespace SKMO scenario and playbooks
Add support for Shared Keystone Multi-region OpenStack (SKMO)
deployments with cross-region Barbican keystone listener:

Playbooks:

prepare-leaf.yaml: Pre-stage hook that creates a TransportURL CR
in the central region for the leaf's barbican-keystone-listener,
copies the generated secret to the leaf namespace, extracts
rootca-internal CA cert from central and adds it to the leaf's
custom-ca-certs bundle, and waits for central Keystone and
openstackclient readiness with retry logic
configure-leaf-listener.yaml: Post-stage hook that patches the
leaf OpenStackControlPlane with the cross-region transport_url
for the barbican-keystone-listener
trust-leaf-ca.yaml: Post-stage hook that extracts the leaf
region's rootca-public and rootca-internal CA certs and adds
them to the central region's custom-ca-certs bundle
ensure-central-ca-bundle.yaml: Ensures the central CA bundle
secret exists before the leaf control plane deployment

Scenario:

va-multi-skmo.yml reproducer scenario configuration
multi-namespace-skmo architecture scenario symlink

openshift-ci · 2026-02-13T22:00:00Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign bshewale for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

softwarefactory-project-zuul · 2026-02-17T18:28:22Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/b7e9a45c2dc540e683237f9d199a301c

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 01m 57s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 18m 44s
❌ cifmw-crc-podified-edpm-baremetal RETRY_LIMIT in 27m 33s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 1h 49m 08s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 04s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 26s

softwarefactory-project-zuul · 2026-02-18T21:09:28Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/9a0c3d521a444791a9cb4da2cf7cedca

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 15m 25s
❌ podified-multinode-edpm-deployment-crc FAILURE in 56m 51s
❌ cifmw-crc-podified-edpm-baremetal FAILURE in 1h 01m 47s
❌ cifmw-crc-podified-edpm-baremetal-minor-update FAILURE in 58m 54s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 59s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 32s

softwarefactory-project-zuul · 2026-02-24T05:02:49Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/17da5110aa71420dbf164b2d3651f1bf

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 27m 13s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 23m 03s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 30m 14s
❌ cifmw-crc-podified-edpm-baremetal-minor-update FAILURE in 2h 12m 02s
✔️ cifmw-pod-zuul-files SUCCESS in 5m 00s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 13s
✔️ cifmw-pod-k8s-snippets-source SUCCESS in 4m 37s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 42s
✔️ cifmw-architecture-validate-hci SUCCESS in 4m 19s
✔️ cifmw-molecule-ci_gen_kustomize_values SUCCESS in 5m 56s

softwarefactory-project-zuul · 2026-02-25T07:56:00Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/2382ae10dbf24bc185e36fba76f1ffc5

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 35m 34s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 21m 39s
❌ cifmw-crc-podified-edpm-baremetal RETRY_LIMIT in 14m 38s
❌ cifmw-crc-podified-edpm-baremetal-minor-update RETRY_LIMIT in 14m 22s
✔️ cifmw-pod-zuul-files SUCCESS in 6m 58s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 10m 22s
✔️ cifmw-pod-k8s-snippets-source SUCCESS in 5m 25s
✔️ cifmw-pod-pre-commit SUCCESS in 11m 33s
✔️ cifmw-architecture-validate-hci SUCCESS in 4m 52s
✔️ cifmw-molecule-ci_gen_kustomize_values SUCCESS in 5m 11s

Configure the multi-namespace SKMO scenario with: - SKMO-specific control-plane kustomization for the central region with barbican-keystone-listener pool_name and RabbitMQ memory reduction (4Gi -> 2Gi) for compact clusters - Barbican keystone listener pool_name configuration for both central (regionOne) and leaf (regionTwo) regions - Automation stages updated to reference SKMO-specific paths - Post-stage hook for populating cross-region transport URL - Wait conditions adjusted for parallel deployment - Correct keystone endpoint values for leaf region - CA trust configuration between central and leaf regions - Symlink net-env for SKMO to reuse multi-namespace networking Depends-On: openstack-k8s-operators/ci-framework#3684 Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Ade Lee <alee@redhat.com>

…ne stage hook_controlplane_config.yml now directly patches the OSCP (Step 6), which requires the CR to exist. Moving the hook from pre_stage_run to post_stage_run for stage 5 (control-plane) ensures the OSCP has been created by the kustomize deploy before the hook tries to patch it. Bootstrap Keycloak remains in pre_stage_run so that the ingress-operator-ca.crt is available when the hook runs post-deploy. Depends-On: openstack-k8s-operators/ci-framework#3684 Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Ade Lee <alee@redhat.com> Made-with: Cursor

fultonj

Based on what I've seen from reviews of others on the ci-framwork team,
I think hooks/playbooks/skmo is a better place for the playbooks than playbooks/skmo.

fultonj · 2026-03-23T19:02:50Z

+      args:
+        executable: /bin/bash
+
+    - name: Create or append leaf CA bundle secret


The ansible shell piping into python caught my attention. I think it can be avoided. I showed lines 134-169 to Claude Code and it suggested something like the following as an alternative.

- name: Get existing CA bundle if present kubernetes.core.k8s_info: api_version: v1 kind: Secret namespace: "{{ leaf_namespace }}" name: "{{ ca_bundle_secret_name }}" register: _existing_bundle - name: Get central rootca certs kubernetes.core.k8s_info: api_version: v1 kind: Secret namespace: "{{ central_namespace }}" name: "{{ item }}" register: _central_certs loop: - "{{ central_rootca_secret }}" - "{{ central_rootca_internal_secret }}" - name: Create or update leaf CA bundle secret kubernetes.core.k8s: state: present definition: apiVersion: v1 kind: Secret metadata: name: "{{ ca_bundle_secret_name }}" namespace: "{{ leaf_namespace }}" data: "{{ (_existing_bundle.resources[0].data | default({})) | combine({ 'skmo-central-rootca.crt': _central_certs.results[0].resources[0].data['tls.crt'], 'skmo-central-rootca-internal.crt': _central_certs.results[1].resources[0].data['tls.crt'] }) }}"

fixed in latest

fultonj · 2026-03-23T19:05:55Z

+      args:
+        executable: /bin/bash
+
+    - name: Copy transport URL secret to leaf namespace


Could you try making this work using the following (suggested by Claude code)?

- name: Get transport URL secret from central namespace kubernetes.core.k8s_info: api_version: v1 kind: Secret namespace: "{{ central_namespace }}" name: "{{ leaf_transport_url_name_secret }}" register: _transport_secret - name: Copy transport URL secret to leaf namespace kubernetes.core.k8s: state: present definition: apiVersion: v1 kind: Secret metadata: name: "{{ leaf_transport_url_secret_copy }}" namespace: "{{ leaf_namespace }}" type: "{{ _transport_secret.resources[0].type }}" data: "{{ _transport_secret.resources[0].data }}"

fixed in latest

fultonj · 2026-03-23T19:08:12Z

+      ansible.builtin.set_fact:
+        central_ca_bundle_secret_name: "{{ skmo_values.data.centralCaBundleSecretName }}"
+
+    - name: Create or append central CA bundle secret


Could you try replacing this task with this instead (again from Claude)?

- name: Get existing central CA bundle if present kubernetes.core.k8s_info: api_version: v1 kind: Secret namespace: "{{ central_namespace }}" name: "{{ central_ca_bundle_secret_name }}" register: _existing_bundle - name: Get leaf rootca certs kubernetes.core.k8s_info: api_version: v1 kind: Secret namespace: "{{ leaf_namespace }}" name: "{{ item }}" register: _leaf_certs loop: - "{{ leaf_rootca_secret }}" - "{{ leaf_rootca_internal_secret }}" - name: Create or update central CA bundle secret kubernetes.core.k8s: state: present definition: apiVersion: v1 kind: Secret metadata: name: "{{ central_ca_bundle_secret_name }}" namespace: "{{ central_namespace }}" data: "{{ (_existing_bundle.resources[0].data | default({})) | combine({ 'skmo-leaf-rootca.crt': _leaf_certs.results[0].resources[0].data['tls.crt'], 'skmo-leaf-rootca-internal.crt': _leaf_certs.results[1].resources[0].data['tls.crt'] }) }}"

fixed in latest

…ooks Add support for Shared Keystone Multi-region OpenStack (SKMO) deployments with cross-region Barbican keystone listener: Playbooks (in hooks/playbooks/skmo/): - prepare-leaf.yaml: Pre-stage hook that creates a TransportURL CR in the central region for the leaf's barbican-keystone-listener, copies the generated secret to the leaf namespace, extracts rootca-internal CA cert from central and adds it to the leaf's custom-ca-certs bundle, and waits for central Keystone and openstackclient readiness with retry logic - configure-leaf-listener.yaml: Post-stage hook that patches the leaf OpenStackControlPlane with the cross-region transport_url for the barbican-keystone-listener - trust-leaf-ca.yaml: Post-stage hook that extracts the leaf region's rootca-public and rootca-internal CA certs and adds them to the central region's custom-ca-certs bundle - ensure-central-ca-bundle.yaml: Ensures the central CA bundle secret exists before the leaf control plane deployment Scenario: - va-multi-skmo.yml reproducer scenario configuration - multi-namespace-skmo architecture scenario symlink Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Ade Lee <alee@redhat.com> Made-with: Cursor

Configure the multi-namespace SKMO scenario with: - SKMO-specific control-plane kustomization for the central region with barbican-keystone-listener pool_name and RabbitMQ memory reduction (4Gi -> 2Gi) for compact clusters - Barbican keystone listener pool_name configuration for both central (regionOne) and leaf (regionTwo) regions - Automation stages updated to reference SKMO-specific paths - Post-stage hooks for cross-region CA trust, transport URL setup, and barbican-keystone-listener configuration - Wait conditions adjusted for parallel deployment - Correct keystone endpoint values for leaf region - CA trust configuration between central and leaf regions - Symlink net-env for SKMO to reuse multi-namespace networking - README documenting the SKMO scenario and deployment stages Depends-On: openstack-k8s-operators/ci-framework#3684 Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Ade Lee <alee@redhat.com>

vakwetu · 2026-03-23T21:22:32Z

addressed comments - added WIP label until testing is done.

…ne stage hook_controlplane_config.yml now directly patches the OSCP (Step 6), which requires the CR to exist. Moving the hook from pre_stage_run to post_stage_run for stage 5 (control-plane) ensures the OSCP has been created by the kustomize deploy before the hook tries to patch it. Bootstrap Keycloak remains in pre_stage_run so that the ingress-operator-ca.crt is available when the hook runs post-deploy. Depends-On: openstack-k8s-operators/ci-framework#3684 Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Ade Lee <alee@redhat.com> Made-with: Cursor

Configure the multi-namespace SKMO scenario with: - SKMO-specific control-plane kustomization for the central region with barbican-keystone-listener pool_name and RabbitMQ memory reduction (4Gi -> 2Gi) for compact clusters - Barbican keystone listener pool_name configuration for both central (regionOne) and leaf (regionTwo) regions - Automation stages updated to reference SKMO-specific paths - Post-stage hooks for cross-region CA trust, transport URL setup, and barbican-keystone-listener configuration - Wait conditions adjusted for parallel deployment - Correct keystone endpoint values for leaf region - CA trust configuration between central and leaf regions - Symlink net-env for SKMO to reuse multi-namespace networking - README documenting the SKMO scenario and deployment stages Depends-On: openstack-k8s-operators/ci-framework#3684 Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Ade Lee <alee@redhat.com>

…ne stage hook_controlplane_config.yml now directly patches the OSCP (Step 6), which requires the CR to exist. Moving the hook from pre_stage_run to post_stage_run for stage 5 (control-plane) ensures the OSCP has been created by the kustomize deploy before the hook tries to patch it. Bootstrap Keycloak remains in pre_stage_run so that the ingress-operator-ca.crt is available when the hook runs post-deploy. Depends-On: openstack-k8s-operators/ci-framework#3684 Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Ade Lee <alee@redhat.com> Made-with: Cursor

vakwetu · 2026-03-27T22:53:33Z

superseded by #3766

Configure the multi-namespace SKMO scenario with: - SKMO-specific control-plane kustomization for the central region with barbican-keystone-listener pool_name and RabbitMQ memory reduction (4Gi -> 2Gi) for compact clusters - Barbican keystone listener pool_name configuration for both central (regionOne) and leaf (regionTwo) regions - Automation stages updated to reference SKMO-specific paths - Post-stage hooks for cross-region CA trust, transport URL setup, and barbican-keystone-listener configuration - Wait conditions adjusted for parallel deployment - Correct keystone endpoint values for leaf region - CA trust configuration between central and leaf regions - Symlink net-env for SKMO to reuse multi-namespace networking - README documenting the SKMO scenario and deployment stages Depends-On: openstack-k8s-operators/ci-framework#3684 Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Ade Lee <alee@redhat.com>

…ne stage hook_controlplane_config.yml now directly patches the OSCP (Step 6), which requires the CR to exist. Moving the hook from pre_stage_run to post_stage_run for stage 5 (control-plane) ensures the OSCP has been created by the kustomize deploy before the hook tries to patch it. Bootstrap Keycloak remains in pre_stage_run so that the ingress-operator-ca.crt is available when the hook runs post-deploy. Depends-On: openstack-k8s-operators/ci-framework#3684 Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Ade Lee <alee@redhat.com> Made-with: Cursor

vakwetu added the do-not-merge/work-in-progress label Feb 13, 2026

vakwetu force-pushed the add_skmo branch 2 times, most recently from 5c879e4 to 4300f16 Compare February 14, 2026 00:16

github-actions Bot added the Ready For Review label Feb 14, 2026

vakwetu force-pushed the add_skmo branch from 4300f16 to 59a1f5e Compare February 16, 2026 22:28

github-actions Bot added Ready For Review and removed Ready For Review labels Feb 16, 2026

vakwetu force-pushed the add_skmo branch from 59a1f5e to ed495c5 Compare February 17, 2026 16:25

github-actions Bot removed the Ready For Review label Feb 17, 2026

vakwetu force-pushed the add_skmo branch from ed495c5 to 91e8651 Compare February 17, 2026 19:20

github-actions Bot added the Ready For Review label Feb 17, 2026

vakwetu force-pushed the add_skmo branch from 91e8651 to 66a26d1 Compare February 17, 2026 22:04

github-actions Bot added Ready For Review and removed Ready For Review labels Feb 17, 2026

vakwetu force-pushed the add_skmo branch from 1652dd1 to b7eb184 Compare February 18, 2026 19:52

github-actions Bot removed the Ready For Review label Feb 18, 2026

vakwetu force-pushed the add_skmo branch from b7eb184 to d470f18 Compare February 19, 2026 02:19

github-actions Bot added the Ready For Review label Feb 19, 2026

vakwetu force-pushed the add_skmo branch from d470f18 to defd4ab Compare February 24, 2026 02:34

github-actions Bot removed the Ready For Review label Feb 24, 2026

vakwetu force-pushed the add_skmo branch from defd4ab to 88d0f06 Compare February 25, 2026 06:19

vakwetu force-pushed the add_skmo branch from 88d0f06 to 05597b1 Compare March 3, 2026 17:23

github-actions Bot added Ready For Review and removed Ready For Review labels Mar 3, 2026

vakwetu requested review from abays and fultonj March 12, 2026 14:20

vakwetu force-pushed the add_skmo branch from 7da97ce to 5e5ae2f Compare March 12, 2026 14:53

github-actions Bot added the Ready For Review label Mar 12, 2026

vakwetu force-pushed the add_skmo branch from 5e5ae2f to d6a83a0 Compare March 20, 2026 18:00

github-actions Bot removed the Ready For Review label Mar 20, 2026

vakwetu force-pushed the add_skmo branch from d6a83a0 to 856e5cb Compare March 20, 2026 18:50

vakwetu force-pushed the add_skmo branch from 856e5cb to e43cd65 Compare March 23, 2026 16:04

github-actions Bot added the Ready For Review label Mar 23, 2026

fultonj reviewed Mar 23, 2026

View reviewed changes

vakwetu force-pushed the add_skmo branch from e43cd65 to 999ffaf Compare March 23, 2026 21:20

github-actions Bot removed Ready For Review labels Mar 23, 2026

vakwetu added the do-not-merge/work-in-progress label Mar 23, 2026

vakwetu closed this Mar 27, 2026

Conversation

vakwetu commented Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci Bot commented Feb 13, 2026

Uh oh!

softwarefactory-project-zuul Bot commented Feb 17, 2026

Uh oh!

softwarefactory-project-zuul Bot commented Feb 18, 2026

Uh oh!

softwarefactory-project-zuul Bot commented Feb 24, 2026

Uh oh!

softwarefactory-project-zuul Bot commented Feb 25, 2026

Uh oh!

fultonj left a comment

Choose a reason for hiding this comment

Uh oh!

fultonj Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

vakwetu Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

fultonj Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

vakwetu Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

fultonj Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

vakwetu Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

vakwetu commented Mar 23, 2026

Uh oh!

vakwetu commented Mar 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vakwetu commented Feb 13, 2026 •

edited

Loading