diff --git a/.wordlist.txt b/.wordlist.txt index 46ca445a7..38f0fcd13 100644 --- a/.wordlist.txt +++ b/.wordlist.txt @@ -516,6 +516,7 @@ kafkatopic kafkatopics kam kamelet +kaoto kasten kastendr kata @@ -548,6 +549,8 @@ kubelet kubeletconfig kubernetes kubevirt +kserve +kuadrant kust kustomization kustomize @@ -587,6 +590,7 @@ machineconfigs machineset macos macosx +mailpit mailto maistra makefile @@ -911,6 +915,7 @@ sigstore siteadmin skipdryrunonmissingresource skopeo +skupper sla slas sme @@ -1113,6 +1118,7 @@ yml yourcluster ypuzljq zefrgjryxxxxxxxxx +ztunnel zh zj zja diff --git a/content/patterns/hybrid-mesh-platform/_index.md b/content/patterns/hybrid-mesh-platform/_index.md new file mode 100644 index 000000000..9d1a90409 --- /dev/null +++ b/content/patterns/hybrid-mesh-platform/_index.md @@ -0,0 +1,106 @@ +--- +title: Hybrid Mesh Platform +date: 2026-05-20 +tier: sandbox +summary: Multi-cluster GitOps platform using a hub-spoke topology with ACM, OpenShift Service Mesh, ACS, and Industrial Edge workloads on OpenShift 4.20. +rh_products: + - Red Hat OpenShift Container Platform + - Red Hat Advanced Cluster Management + - Red Hat OpenShift GitOps + - Red Hat Advanced Cluster Security for Kubernetes + - Red Hat OpenShift Service Mesh + - Red Hat Connectivity Link + - Red Hat OpenShift AI + - Red Hat AMQ Streams + - Red Hat build of Apache Camel + - Red Hat OpenShift Pipelines + - Red Hat Developer Hub + - Red Hat Service Interconnect +industries: + - General + - Industrial +aliases: /hybrid-mesh-platform/ +links: + github: https://github.com/maximilianopizarro/platform-hub-spoke-config + install: getting-started + bugs: https://github.com/maximilianopizarro/platform-hub-spoke-config/issues + feedback: https://docs.google.com/forms/d/e/1FAIpQLScI76b6tD1WyPu2-d_9CCVDr3Fu5jYERthqLKJDUGwqBg7Vcg/viewform +contributor: + name: Maximiliano Pizarro + contact: mailto:maximilianopizarro5@gmail.com + git: https://github.com/maximilianoPizarro +--- + +# Hybrid Mesh Platform + +**Maintainer:** Maximiliano Pizarro, Specialist Solution Architect at Red Hat + +> **Your journey:** This platform deploys in one `helm upgrade`, connects three OpenShift clusters (hub + east + west), and shows IoT sensor data across Grafana and Developer Hub within about 30 minutes. The pages below follow one continuous story — concept, install, operate, scaffold — so you can read straight through or jump to any chapter. + +**Hybrid Mesh Platform** is a multi-cluster GitOps platform using Red Hat products. It implements a hub-spoke topology that centralizes governance with Red Hat Advanced Cluster Management (ACM), delivers [Industrial Edge](/patterns/industrial-edge/) workloads on regional spokes, uses OpenShift Service Mesh in ambient mode for east-west connectivity, layers Connectivity Link (Kuadrant) for API-aware ingress policy, exposes Grafana dashboards for cross-cluster visibility, and integrates Advanced Cluster Security (ACS) for vulnerability and runtime protection. + +**Supported on:** Red Hat OpenShift Container Platform 4.20 (4.14 or newer per cluster). + +Read **concept → mechanics → operations**: start with [Architecture](architecture), install via [Getting Started](getting-started), scaffold workloads via [Scaffolding](scaffolding), then use platform chapters (**Hub Gateway**, **Observability**, **Industrial Edge**) before drilling into the [pattern repository](https://github.com/maximilianopizarro/platform-hub-spoke-config). + +## Overview + +This repository models a **GitOps-first platform** where: + +- **Hub cluster** runs ACM, OpenShift GitOps (Argo CD), observability aggregation, Developer Hub, ACS Central, Mailpit for notifications, and gateway-style HTTP routing with **circuit breaking** for shared services. +- **Spoke clusters** (east/west regions) host **Industrial Edge** patterns: sensor and MQTT-style ingestion, Kafka pipelines, optional ML scoring, and dashboards fed by Prometheus-compatible metrics. +- **Service Mesh 3 ambient** reduces sidecar overhead while retaining ztunnel-based L4 and waypoint-based L7 policy where needed. +- **Hub Gateway** splits traffic into **front** and **API** services per spoke, with per-service **circuit breaking** via `DestinationRule`. +- **Service Interconnect (Skupper)** bridges spoke services and metrics to the hub via a Virtual Application Network (VAN), without VPN or firewall changes. +- **Spoke Gateways** aggregate Industrial Edge services per spoke for simplified cross-cluster exposure. +- **Kiali + OSSM Console** provides service mesh topology visualization on every cluster via the OpenShift Console plugin. +- **Grafana dashboards** roll up cluster and application signals from all clusters. +- **ACS** provides centralized policy, CVE visibility, and SecuredCluster agents on spokes. + +[![Platform hub-spoke overview](/images/hybrid-mesh-platform/arch-overview.png)](/images/hybrid-mesh-platform/arch-overview.png) + +_Hub cluster aggregates observability and Developer Hub; east and west spokes run Industrial Edge workloads connected via Service Interconnect (Skupper). Click the image to open the full diagram._ + +Architecture diagrams illustrate Git, **ACM fleet management**, **ACS Central**, Skupper VAN, Connectivity Link, and Industrial Edge on east/west — use them as the visual companion to the install chapters (see [Architecture](architecture) for ACM and ACS console views). + +## Quick links + +| Topic | Page | +| --- | --- | +| Architecture deep dive | [Architecture](architecture) | +| Install flow | [Getting Started](getting-started) | +| Hub Gateway and Connectivity Link | [Hub Gateway](hub-gateway) | +| Observability | [Observability](observability) | +| Industrial Edge (multi-cluster) | [Industrial Edge](industrial-edge) | +| Scaffolding | [Scaffolding](scaffolding) | +| Branch strategy and customization | [Ideas for customization](ideas-for-customization) | + +## Recommended reading order + +1. [Architecture](architecture) — mental model of hub, spokes, GitOps, Skupper, and observability +2. [Getting Started](getting-started) — bring clusters under GitOps (includes ACM + ApplicationSet detail) +3. [Scaffolding](scaffolding) — deploy Industrial Edge instances on east/west from Developer Hub +4. [Hub Gateway](hub-gateway) — weighted ingress and circuit breaking across spokes +5. [Observability](observability) — Grafana, Kiali, Kafka Console +6. [Industrial Edge](industrial-edge) — factory data pipeline: sensors, Kafka, Camel, ML on multiple spokes + +Screenshots and architecture diagrams in the pattern repository support full-screen review — handy after deploying dashboards and verifying cross-cluster traffic. + +**Next →** [Architecture](architecture) — understand how Git, ACM, and Skupper wire the three clusters together. + +## Red Hat products used + +- Red Hat OpenShift Container Platform +- Red Hat Advanced Cluster Management for Kubernetes +- Red Hat OpenShift GitOps (Argo CD) +- Red Hat Advanced Cluster Security for Kubernetes +- Red Hat OpenShift Service Mesh +- Red Hat Connectivity Link (Kuadrant, Gateway API) +- Red Hat OpenShift AI +- Red Hat AMQ Streams (Apache Kafka) +- Red Hat build of Apache Camel / Camel K +- Red Hat OpenShift Pipelines (Tekton) +- Red Hat Developer Hub (Backstage) +- Red Hat Service Interconnect (Skupper) +- Mailpit (SMTP testing for notifications) +- Observability stack (Prometheus-compatible metrics, Grafana, OpenTelemetry, Kiali) diff --git a/content/patterns/hybrid-mesh-platform/architecture.md b/content/patterns/hybrid-mesh-platform/architecture.md new file mode 100644 index 000000000..1f929e98a --- /dev/null +++ b/content/patterns/hybrid-mesh-platform/architecture.md @@ -0,0 +1,255 @@ +--- +title: Architecture +weight: 20 +aliases: /hybrid-mesh-platform/architecture/ +--- + +# Architecture + +Understand how Git, ACM, Service Interconnect, and Industrial Edge wire the hub and spoke clusters together before you install or scaffold workloads. + +## Hub-spoke theory in multi-cluster Kubernetes + +In multi-cluster Kubernetes, a hub-spoke model designates one administrative cluster (the **hub**) and one or more workload clusters (**spokes**). The hub owns fleet APIs: cluster inventory, policy placement, credentials for spoke registration, and often centralized GitOps controllers that fan out desired state. + +Spokes remain the execution venues for application namespaces, data-plane components (Kafka, MQTT bridges, mesh dataplane), and regional isolation for latency, data residency, or blast-radius control. + +## Why hub-spoke? + +| Benefit | Description | +| --- | --- | +| Centralized management | One control plane for membership, RBAC patterns, and bulk upgrades. | +| Policy enforcement | Kubernetes policies, compliance checks, and security baselines propagate from the hub. | +| Observability | Aggregated metrics, logging, and tracing strategies start at the hub; uniform dashboards span spokes. | +| GitOps consistency | A single Git revision strategy (with branch or overlay variants) drives spoke drift correction. | + +## Platform architecture overview + +The pattern repository uses a **single `main` branch** with cluster-specific directories: + +- Hub chart at `.` (repository root) +- Spoke charts at `east/` and `west/` +- Shared Helm charts under `components/` referenced by hub and spoke Application CRs + +[![Platform architecture overview](/images/hybrid-mesh-platform/arch-overview.png)](/images/hybrid-mesh-platform/arch-overview.png) + +[![Hub-spoke GitOps flow](/images/hybrid-mesh-platform/arch-hub-spoke-flow.png)](/images/hybrid-mesh-platform/arch-hub-spoke-flow.png) + +## Follow the request — one temperature reading end to end + +When a machine sensor on the **east** spoke publishes a temperature sample, the path is: + +1. MQTT (`messaging` broker) → Camel K (`mqtt-to-kafka` integration) +2. Kafka (`dev-cluster` topic) → optional ML scoring (KServe) +3. `line-dashboard` WebSocket consumer + +In parallel: + +- Thanos Querier on east scrapes Istio and Kafka metrics +- Skupper Connector `prometheus-east` tunnels HTTP to the hub +- Hub Grafana datasource `prometheus-east` plots the series +- Hub Gateway routes browser traffic to the east line-dashboard via spoke-gateway and Skupper listener `ie-gateway-east` +- Developer Hub Topology shows the same pods when the catalog entity carries `backstage.io/kubernetes-cluster: east` and spoke API tokens are synced + +[![End-to-end data flow](/images/hybrid-mesh-platform/arch-data-flow.png)](/images/hybrid-mesh-platform/arch-data-flow.png) + +## Components on the hub vs spokes + +| Area | Hub | Spokes | Config path | +| --- | :---: | :---: | --- | +| ACM hub operator & APIs | yes | | `values.yaml` | +| Argo CD / App-of-Apps root | yes | yes | `.` / `east/` / `west/` | +| ApplicationSet (spoke apps) | yes | | `values.yaml` | +| ACS Central | yes | | `values.yaml` | +| ACS Secured Cluster | | yes | `east/` `west/` | +| Developer Hub | yes | | `values.yaml` | +| Hub Gateway (Gateway API) | yes | | `values.yaml` | +| Spoke Gateway (Gateway API) | | yes | `east/` `west/` | +| Industrial Edge workloads | | yes | `east/` `west/` | +| Kafka brokers (regional) | | yes | `east/` `west/` | +| Service Mesh ambient / ztunnel | yes | yes | both | +| Istio CNI (`profile: ambient`) | yes | yes | both | +| Skupper Site (hub listeners) | yes | | `values.yaml` | +| Skupper Site (spoke connectors) | | yes | `east/` `west/` | +| Grafana (multi-cluster dashboards) | yes | | `values.yaml` | +| Grafana (local metrics) | | yes | `east/` `west/` | +| Kiali + OSSM Console plugin | yes | yes | both | +| Connectivity Link (RHCL) | yes | yes | both | +| Kubecost (primary aggregator) | yes | | `values.yaml` | +| Kubecost (agent) | | yes | `east/` `west/` | +| Kafka Console (all clusters) | yes | | `values.yaml` | + +Industrial Edge components exist **only** in spoke charts. The hub chart never includes factory workloads. + +## Advanced Cluster Management (ACM) + +Red Hat Advanced Cluster Management for Kubernetes (ACM) provides fleet-wide visibility and lifecycle for OpenShift clusters. In Hybrid Mesh Platform it anchors hub-spoke registration, policy placement, and integration with OpenShift GitOps via `GitOpsCluster` and related APIs. + +[![ACM fleet management — east and west managed clusters on the hub](/images/hybrid-mesh-platform/ACM.png)](/images/hybrid-mesh-platform/ACM.png) + +### Role in this solution + +- Inventory managed clusters (`hub`, `east`, `west`) and apply governance policies consistently. +- Drive which spokes receive Industrial Edge and platform components through **Placement** rules (`region=east`, `region=west`). +- Coordinate klusterlet agents, `ManagedClusterSet` membership, and secrets required for spoke import. +- Publish **PlacementDecision** objects consumed by the Argo CD ApplicationSet (`clusterDecisionResource` generator). + +### Notable APIs / CRDs + +| Resource | Purpose | +| --- | --- | +| `MultiClusterHub` | Hub installation health | +| `ManagedCluster`, `ManagedClusterSet` | Fleet membership and RBAC grouping | +| `Placement`, `PlacementDecision` | Dynamic cluster selection for GitOps | +| `GitOpsCluster` | Binds placement results to Argo CD cluster secrets | + +Charts: `components/acm-operator`, `components/acm-hub-spoke`. Verify with `oc get managedcluster` and PlacementDecision in `openshift-gitops`. + +## Advanced Cluster Security (ACS) + +Red Hat Advanced Cluster Security for Kubernetes (ACS) centralizes build-time image scanning, deployment-time policy, and runtime detection across the fleet. + +[![ACS Central — hub, east, and west registered](/images/hybrid-mesh-platform/ACS.png)](/images/hybrid-mesh-platform/ACS.png) + +[![ACS Central — policies, vulnerabilities, and runtime visibility](/images/hybrid-mesh-platform/ACS-2.png)](/images/hybrid-mesh-platform/ACS-2.png) + +### Hub-spoke topology + +| Component | Location | Role | +| --- | --- | --- | +| **Central** | Hub | Policy console, vulnerability database, admission coordination | +| **SecuredCluster** | Hub + spokes | Sensor, collector, and admission control per cluster | + +Cluster names in Central: **`hub`**, **`east`**, **`west`**. Init bundles (TLS secrets in namespace `stackrox`) register each SecuredCluster with Central. + +### Service mesh exception + +Namespace `stackrox` is listed in `$noMeshNamespaces` (`components/namespaces`) — **do not** label it `istio.io/dataplane-mode: ambient`. Ambient ztunnel breaks Central ↔ PostgreSQL TLS and Central becomes unreachable. + +### Capabilities used + +- CVE scanning for Industrial Edge and platform images (Quay/internal registry). +- Risk prioritization across namespaces and clusters. +- Optional network and process baselines for regulated factory environments. + +Charts: `components/acs-operator` (hub Central), `components/acs-secured-cluster` (hub + spokes). See [Getting Started](getting-started#advanced-cluster-security-acs) for init bundle generation. + +## GitOps application delivery flow + +1. Hub Argo CD syncs the root Application (operators, ACM, gateway, observability). +2. ACM ApplicationSet reads PlacementDecision and pushes `east/` and `west/` to remote clusters. +3. Each spoke Argo CD reconciles child Applications locally (namespaces → operators → workloads). + +[![GitOps sync sequence](/images/hybrid-mesh-platform/arch-gitops-sync-sequence.png)](/images/hybrid-mesh-platform/arch-gitops-sync-sequence.png) + +Remote deployment model: + +``` +Hub Argo CD → ApplicationSet + → east-spoke-components (source: east/, destination: east cluster) + → east/templates/ generates child Application CRs + → East Argo CD syncs child apps locally + → west-spoke-components (source: west/, destination: west cluster) + → west/templates/ generates child Application CRs + → West Argo CD syncs child apps locally +``` + +## Sync wave ordering + +Components deploy in strict order via Argo CD sync waves. Lower waves complete before higher waves start. + +[![Sync wave ordering](/images/hybrid-mesh-platform/arch-sync-waves.png)](/images/hybrid-mesh-platform/arch-sync-waves.png) + +Sync waves prevent operators from racing workloads — mesh and namespaces land before Industrial Edge and gateways. Typical progression: + +1. Namespaces and RBAC +2. Operator subscriptions (ACM, GitOps, Service Mesh, Skupper, and others) +3. Platform services (Developer Hub, ACS, interconnect sites) +4. Observability stack +5. Industrial Edge and gateway workloads + +## Service Interconnect (Skupper) topology + +Red Hat Service Interconnect creates a **Virtual Application Network (VAN)** that bridges services across clusters without VPN tunnels, direct routes, or firewall changes. Skupper exposes spoke Industrial Edge services and Prometheus metrics to the hub for centralized observability and ingress. + +[![Skupper topology](/images/hybrid-mesh-platform/arch-skupper-topology.png)](/images/hybrid-mesh-platform/arch-skupper-topology.png) + +[![Service Interconnect console topology](/images/hybrid-mesh-platform/service-interconnect-console-topology.png)](/images/hybrid-mesh-platform/service-interconnect-console-topology.png) + +### Skupper Network Observer (console views) + +The Skupper Network Observer visualizes the Virtual Application Network — sites, listeners, connectors, and process-level traffic across hub, east, and west: + +[![Skupper Network Observer — sites and links](/images/hybrid-mesh-platform/service-interconnect-console.png)](/images/hybrid-mesh-platform/service-interconnect-console.png) + +[![Skupper console — components, listeners, and connectors](/images/hybrid-mesh-platform/service-interconnect-console-topology-process.png)](/images/hybrid-mesh-platform/service-interconnect-console-topology-process.png) + +[![Skupper console — process-level topology](/images/hybrid-mesh-platform/service-interconnect-console-process.png)](/images/hybrid-mesh-platform/service-interconnect-console-process.png) + +[![Skupper console — built-in metrics (TCP bytes, latency, connections)](/images/hybrid-mesh-platform/service-interconnect-console-metrics.png)](/images/hybrid-mesh-platform/service-interconnect-console-metrics.png) + +### Hub listeners (namespace `service-interconnect`) + +| Listener | Port | Purpose | +| --- | --- | --- | +| `ie-gateway-east` | 8080 | Spoke-gateway traffic from east | +| `ie-gateway-west` | 8080 | Spoke-gateway traffic from west | +| `prometheus-east` | 9091 | Prometheus metrics from east | +| `prometheus-west` | 9091 | Prometheus metrics from west | +| `kafka-east-tst` | 9092 | Kafka bootstrap (dev-cluster) from east | +| `kafka-west-tst` | 9092 | Kafka bootstrap (dev-cluster) from west | + +### Spoke connectors + +| Connector | Exposes | +| --- | --- | +| `ie-gateway-` | Local spoke-gateway | +| `prometheus-` | Nginx auth proxy → Thanos Querier | +| `kafka--tst` | `dev-cluster-kafka-bootstrap` | +| `kafka--stormshift` | `factory-cluster-kafka-bootstrap` | + +**Link establishment:** Hub `AccessGrant` generates URL, code, and CA. Spoke `AccessToken` redeems the grant over HTTPS, then establishes the inter-router link. Use the Skupper grant-server CA from `skupper-grant-server-ca` — not the OpenShift Ingress CA — or spokes fail with `x509: certificate signed by unknown authority`. + +```bash +oc get secret skupper-grant-server-ca -n openshift-operators \ + -o jsonpath='{.data.ca\.crt}' | base64 -d +``` + +Charts: `components/service-interconnect` (hub), `components/spoke-interconnect` (spokes). + +## Spoke gateway aggregation + +Each spoke runs a Gateway API gateway (`components/spoke-gateway`) that fronts all Industrial Edge services. Skupper exposes **one** gateway per spoke instead of every microservice individually. + +[![Spoke gateway aggregation diagram](/images/hybrid-mesh-platform/arch-spoke-gateway.png)](/images/hybrid-mesh-platform/arch-spoke-gateway.png) + +[![Spoke Gateway API — Connectivity Link view](/images/hybrid-mesh-platform/connectivity-link-spoke-gateway.png)](/images/hybrid-mesh-platform/connectivity-link-spoke-gateway.png) + +Hub-side Gateway API and HTTPRoute policy topology: see [Hub Gateway — Connectivity Link topology](hub-gateway#connectivity-link-topology). + +## Multi-cluster observability pipeline + +Spoke Thanos Querier is reached through nginx auth-proxy Connectors. Hub Listeners `prometheus-east` and `prometheus-west` become Grafana HTTP datasources (no bearer token from hub). + +[![Observability pipeline](/images/hybrid-mesh-platform/arch-observability-pipeline.png)](/images/hybrid-mesh-platform/arch-observability-pipeline.png) + +## Data flow (sensors to dashboard) + +On each spoke, telemetry flows from sensors through MQTT, Camel, and Kafka to dashboards. Strimzi MirrorMaker replicates factory topics toward the hub-centralized data lake (MinIO/S3 archiver on hub and spokes). + +## Comparison with Red Hat Validated Patterns + +The [Multicloud GitOps](/patterns/multicloud-gitops) validated pattern demonstrates fleet GitOps with OpenShift GitOps and ACM: declarative root Application, cluster grouping, and progressive rollout. + +**Hybrid Mesh Platform** extends that foundation with: + +- [Industrial Edge](/patterns/industrial-edge/) on **east** and **west** spokes (not duplicated here — see maintained pattern for OT/factory depth) +- OpenShift Service Mesh 3 **ambient** mode (ztunnel L4, waypoints L7) +- Connectivity Link for Gateway API ingress policy +- Optional OpenShift AI (KServe) on spokes +- ACS Central + SecuredCluster agents +- Service Interconnect for cross-cluster service and metrics exposure + +The result is tuned for factory-style telemetry and east-west observability, not only infrastructure provisioning. + +**Next →** [Getting Started](getting-started) to translate these diagrams into a live deployment, or [Hub Gateway](hub-gateway) for cross-cluster HTTP routing detail. diff --git a/content/patterns/hybrid-mesh-platform/getting-started.md b/content/patterns/hybrid-mesh-platform/getting-started.md new file mode 100644 index 000000000..63e2c52c6 --- /dev/null +++ b/content/patterns/hybrid-mesh-platform/getting-started.md @@ -0,0 +1,324 @@ +--- +title: Getting Started +weight: 10 +aliases: /hybrid-mesh-platform/getting-started/ +--- + +# Getting Started + +Follow these steps to bootstrap the Hybrid Mesh Platform hub-spoke GitOps environment from the [platform-hub-spoke-config](https://github.com/maximilianopizarro/platform-hub-spoke-config) repository. + +## You'll have when finished + +After a successful hub deploy and spoke registration, expect: + +| Component | Verification | +| --- | --- | +| ACM | `east` and `west` in `ManagedCluster` **Available** | +| Argo CD | Root Application **Synced**; ApplicationSet pushing `east/` and `west/` | +| Industrial Edge | Sensors, MQTT, Kafka, `line-dashboard` on each spoke | +| Skupper | Hub `sitesInNetwork: 3`; listeners **Ready** in `service-interconnect` | +| Grafana | Hub dashboards with `prometheus-east` / `prometheus-west` datasources | +| Developer Hub | Industrial Edge catalog + software templates under **Create** | +| Gitea | Route `gitea-gitea.`; orgs `ws-platformadmin`, `app-of-apps` | +| Quay | Route `quay-registry.` (optional image catalog) | + +Then continue with [Scaffolding](scaffolding) to deploy a new edge instance on east or west. + +## Platform operators (reference) + +The hub chart deploys **ACM**, **OpenShift GitOps**, **ACS**, Service Mesh, Skupper, and related operators before application workloads. + +[![OpenShift GitOps — Argo CD Applications on the hub](/images/hybrid-mesh-platform/product-argocd-openshift-gitops.png)](/images/hybrid-mesh-platform/product-argocd-openshift-gitops.png) + +### Advanced Cluster Management (ACM) + +ACM must show **`east`** and **`west`** as **Available** managed clusters before the ApplicationSet can push spoke charts. + +[![ACM fleet management — east and west registered on the hub](/images/hybrid-mesh-platform/ACM.png)](/images/hybrid-mesh-platform/ACM.png) + +Verify: + +```bash +oc get managedcluster +oc get multiclusterhub -n open-cluster-management +``` + +Spoke names must match repository folders (`east`, `west`). Placement labels drive ApplicationSet targeting — see [Step 4](#step-4-import-managed-clusters-in-acm) and [Deploy with ACM and GitOps](#deploy-with-acm-and-gitops). + +### Advanced Cluster Security (ACS) + +ACS Central runs on the hub; **SecuredCluster** agents install on hub and both spokes. All three clusters appear in the Central UI when init bundles are applied. + +[![ACS Central — hub, east, and west clusters](/images/hybrid-mesh-platform/ACS.png)](/images/hybrid-mesh-platform/ACS.png) + +[![ACS Central — policies and vulnerability views](/images/hybrid-mesh-platform/ACS-2.png)](/images/hybrid-mesh-platform/ACS-2.png) + +#### Generating SecuredCluster init bundles + +Generate one init bundle per cluster from Central (do not commit secrets to Git): + +```bash +roxctl -e central.stackrox:443 --password "$ROX_ADMIN_PASSWORD" --insecure-skip-tls-verify \ + central init-bundles generate --output-secrets - | oc apply -n stackrox -f - +``` + +Use cluster names **`hub`**, **`east`**, and **`west`**. Namespace **`stackrox`** must stay **off** Service Mesh ambient — see [Architecture — ACS](architecture#advanced-cluster-security-acs). + +## Prerequisites + +- Red Hat OpenShift Container Platform **4.20** (reference version; 4.14+ supported per cluster) +- **Three clusters:** one hub, one east-region spoke, one west-region spoke (ACM labels drive placement) +- **Helm 3** on your workstation or CI runner (`helm version`) +- **Git** client and hosting account (GitHub in examples) +- **`oc` CLI** logged into the hub as cluster-admin for ACM import (recommended) +- Network access to GitHub (or your fork) and container registries from all clusters + +### Network requirements (connected environments) + +1. Access to public container registries (or mirrored equivalents) +2. Access to your Git repository (fork of `platform-hub-spoke-config`) + +## Repository layout + +``` +. → hub cluster (path: .) +east/ → east spoke cluster (path: east) +west/ → west spoke cluster (path: west) +components/ → shared component charts referenced by all clusters +``` + +Each cluster path is a self-contained Helm chart with its own `Chart.yaml`, `values.yaml`, and `templates/`. + +## Step 1: Fork the repository + +Fork [platform-hub-spoke-config](https://github.com/maximilianopizarro/platform-hub-spoke-config) so you can customize domains, secrets references, and Git URLs without coupling to upstream history. + +Update `gitops.repoUrl` in: + +- `values.yaml` (hub) +- `east/values.yaml` +- `west/values.yaml` + +## Step 2: Configure cluster domains + +Set DNS names for each cluster before install. + +**Hub** (`values.yaml`): + +- `deployer.domain` — hub apps domain (Grafana, Developer Hub, gateway) +- `clusters.hub.domain`, `clusters.east.domain`, `clusters.west.domain` + +**East** (`east/values.yaml`): + +- `deployer.domain` — east spoke apps domain +- `clusters.hub.domain` — hub domain for cross-cluster links + +**West** (`west/values.yaml`): + +- `deployer.domain` — west spoke apps domain +- `clusters.hub.domain` — hub domain for cross-cluster links + +Validate Helm rendering before apply: + +```bash +helm template test-hub . -f values.yaml --set deployer.domain=apps.hub.example.com +helm template test-east east/ -f east/values.yaml +helm template test-west west/ -f west/values.yaml +``` + +## Step 3: Install on the hub + +The hub uses the repository root path (`.`): + +```bash +helm install hybrid-mesh-platform . -f values.yaml --set deployer.domain=apps.hub.example.com +``` + +For production GitOps, create an Argo CD `Application` pointing at your fork on branch `main` (matching `gitops.revision`). Supply values via Helm parameters or a values ConfigMap pattern your organization prefers. + +## Step 4: Import managed clusters in ACM + +From the hub, import east and west using ACM **Import cluster** or automated klusterlet flows. + +Apply labels used by placement rules: + +```bash +# Example — adjust cluster names to match your environment +oc label managedcluster east region=east +oc label managedcluster west region=west +oc label managedcluster east cluster.open-cluster-management.io/clusterset=global +oc label managedcluster west cluster.open-cluster-management.io/clusterset=global +``` + +Ensure spoke credentials are stored per ACM requirements. + +## Step 5: Register spokes as Argo CD cluster secrets + +The ApplicationSet deploys spoke charts remotely. Register each spoke on the hub Argo CD instance: + +```bash +helm upgrade field-content . \ + --set clusters.east.token=sha256~... \ + --set clusters.west.token=sha256~... +``` + +Or create secrets with label `argocd.argoproj.io/secret-type: cluster`. + +Spoke cluster names in ACM and Argo CD must match folder names: **`east`** and **`west`**. + +## Step 6: Verify ApplicationSet generates spoke applications + +Confirm the ACM → GitOps chain: + +1. `Placement` `hub-spoke-placement` selects `region=east` and `region=west` +2. `PlacementDecision` lists those clusters in `openshift-gitops` +3. `GitOpsCluster` `hub-spoke-gitops` registers clusters in Argo CD +4. ApplicationSet creates `east-spoke-components` and `west-spoke-components` +5. Each spoke Argo CD syncs child Applications (namespaces, operators, Industrial Edge, and others) + +**Hub:** + +```bash +oc get applications -n openshift-gitops +# Expect: east-spoke-components, west-spoke-components (Synced / Healthy) +``` + +**Each spoke:** + +```bash +oc get applications -n openshift-gitops +# Expect: east-namespaces, east-operators, east-industrial-edge-tst, etc. +``` + +**PlacementDecision:** + +```bash +oc get placementdecisions.cluster.open-cluster-management.io -n openshift-gitops \ + -l cluster.open-cluster-management.io/placement=hub-spoke-placement -o yaml +``` + +Healthy sync waves progress: **namespaces → operators → platform → observability → Industrial Edge workloads**. + +## Deploy with ACM and GitOps + +### ManagedClusterSet + +Groups clusters for RBAC and placement. Downstream objects reference set membership — not hard-coded cluster names in Git. + +### Placement + +Selects clusters from a ManagedClusterSet using label selectors (for example `region=east`). ACM recomputes decisions as clusters join, leave, or change labels. + +### PlacementDecision + +Publishes the concrete cluster names that satisfied a Placement at a given time. The ApplicationSet `clusterDecisionResource` generator watches these decisions. + +### GitOpsCluster + +Associates Argo CD (`openshift-gitops`) with clusters chosen by placement — bridging ACM fleet selection and Argo CD cluster secrets. + +### ApplicationSet with clusterDecisionResource + +For each cluster in the PlacementDecision, the ApplicationSet creates an Application that: + +- Uses repository path `east/` or `west/` (from `{{name}}`) +- Deploys to remote cluster `{{name}}` via hub-stored cluster credentials + +Adding a spoke with correct labels and a new folder (for example `south/`) automatically generates a new Application when the Placement matches. + +### Troubleshooting: no spoke Applications + +| Check | Command / expectation | +| --- | --- | +| Managed clusters | `oc get managedcluster` — **Available** | +| Cluster names | Must be `east` and `west` (match repo folders) | +| Placement | `hub-spoke-placement` includes both regions | +| PlacementDecision | Decisions list `east`, `west` | +| GitOpsCluster | `hub-spoke-gitops` reconciled; clusters visible in Argo CD UI | +| RBAC | Role `applicationset-placementdecisions` for ApplicationSet controller | +| Spoke charts | `east/` and `west/` valid Helm charts with `Chart.yaml` | + +## Step 7: Kiali multi-cluster (hub) + +Hub Kiali shows mesh topology from east and west **without** Istio multi-cluster trust federation. Each spoke keeps its own control plane; hub Kiali uses remote secrets and links to spoke Kiali UIs. + +### Automated token sync (default) + +With `multiCluster.automateTokens: true` (hub) and `exportTokenForHub: true` (spokes): + +1. Spoke PostSync Job exports `kiali-hub-export` ConfigMap with token +2. Hub PostSync Job reads ACM `ManagedCluster` apiUrl/caBundle and creates `kiali-multi-cluster-secret` +3. CronJobs renew tokens every 6 hours + +### Manual tokens (optional) + +```bash +oc create token kiali-service-account -n openshift-cluster-observability-operator --duration=8760h +helm upgrade field-content . \ + --set multiCluster.automateTokens=false \ + --set clusters.east.kialiToken=sha256~... \ + --set clusters.east.kialiCaData=LS0tLS1... \ + --reuse-values +``` + +With `auth.strategy: openshift`, users **Log in** per remote cluster the first time they open Kiali for that cluster. + +## Step 8: Developer Hub (Keycloak OIDC) + +Developer Hub uses cluster Keycloak (`sso.`) realm `backstage` — not GitHub OAuth. + +```bash +SECRET=$(openssl rand -base64 24) +helm upgrade field-content . \ + --set keycloakOidcClientSecret="$SECRET" \ + --reuse-values +``` + +Or patch after deploy: + +```bash +oc create secret generic developer-hub-oidc-auth \ + --from-literal=OIDC_CLIENT_SECRET="$SECRET" \ + -n developer-hub --dry-run=client -o yaml | oc apply -f - +``` + +Log in at `https://developer-hub.` (for example `platformadmin` / workshop password from your values). + +## Step 9: Continue AI (DevSpaces + Kaoto) + +Do not commit MaaS API keys to Git. Create the DevSpaces secret after deploy: + +```bash +oc create secret generic continue-ai-config -n devspaces \ + --from-literal=CONTINUE_API_KEY='' \ + --from-literal=CONTINUE_API_BASE='https://litellm-prod.apps.maas.redhatworkshops.io/v1' \ + --from-literal=CONTINUE_MODEL='deepseek-r1-distill-qwen-14b' \ + --dry-run=client -o yaml | oc apply -f - +``` + +Industrial Edge catalog loads from an in-cluster ConfigMap. Software templates ship as static assets in the pattern repository under `docs/assets/backstage/software-templates/`. + +### Developer Hub multi-cluster Topology + +Spoke workloads appear in **Topology** / **Kubernetes** only when: + +1. `developer-hub-spoke-tokens` Secret exists (token sync Job completed) +2. Catalog entities include `backstage.io/kubernetes-cluster: east` or `west` + +```bash +oc get secret developer-hub-spoke-tokens -n developer-hub +oc get job -n developer-hub -l job-name=developer-hub-spoke-token-sync-hook +``` + +### Optional: Quay credentials (hub) + +Never commit registry passwords to Git. Generate dockerconfig and pass via Helm at upgrade time if you enable Quay push from pipelines. + +## References + +- [ACM documentation](https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.16/html/about/welcome-to-red-hat-advanced-cluster-management-for-kubernetes) +- [Multicloud GitOps Validated Pattern](/patterns/multicloud-gitops) +- [ApplicationSet Generators](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators/) + +**Next →** [Scaffolding](scaffolding) to deploy Industrial Edge instances from Developer Hub · [Architecture](architecture) for diagrams · [Observability](observability) once metrics are flowing. diff --git a/content/patterns/hybrid-mesh-platform/hub-gateway.md b/content/patterns/hybrid-mesh-platform/hub-gateway.md new file mode 100644 index 000000000..9a841e7a9 --- /dev/null +++ b/content/patterns/hybrid-mesh-platform/hub-gateway.md @@ -0,0 +1,153 @@ +--- +title: Hub Gateway +weight: 30 +aliases: /hybrid-mesh-platform/hub-gateway/ +--- + +# Hub Gateway + +The hub gateway provides centralized HTTP ingress on the hub cluster with behaviors similar to an F5 BIG-IP ADC: VIP-style routing, TLS termination at the edge, and weighted traffic splits across backend services or spoke-derived routes. + +Implementation chart: `components/hub-gateway`. Connectivity Link operator: `components/rhcl-operator`. + +## Connectivity Link topology + +Connectivity Link (Kuadrant) brings multi-cluster ingress and policy using Kubernetes Gateway API — DNS, TLS, rate limiting, and auth patterns layered on `Gateway` and `HTTPRoute` resources. In this platform, Gateway API objects align with hub gateway routing (including weighted splits similar to hardware ADC behavior). Policies may be disabled initially; enable Kuadrant `AuthPolicy`, `RateLimitPolicy`, and DNS/TLS strategies as you harden environments. + +[![Gateway API policy topology — hub HTTPRoute and route rules](/images/hybrid-mesh-platform/connectivity-link-hub.png)](/images/hybrid-mesh-platform/connectivity-link-hub.png) + +_Gateway API policy topology — hub-gateway, HTTPRoute, and route rules in the OpenShift Console._ + +### Hub cluster + +Hub cluster Gateway API resources and HTTPRoute attachment to `hub-gateway-system`: + +[![Hub cluster Gateway API and HTTPRoute](/images/hybrid-mesh-platform/connectivity-link-hub-gateway.png)](/images/hybrid-mesh-platform/connectivity-link-hub-gateway.png) + +### Spoke clusters + +Spoke cluster Gateway API and backend services exposed through the mesh: + +[![Spoke cluster Gateway API and backends](/images/hybrid-mesh-platform/connectivity-link-spoke.png)](/images/hybrid-mesh-platform/connectivity-link-spoke.png) + +Spoke gateway aggregating Industrial Edge services for cross-cluster exposure (single Skupper Connector target per spoke): + +[![Spoke gateway aggregating Industrial Edge services](/images/hybrid-mesh-platform/connectivity-link-spoke-gateway.png)](/images/hybrid-mesh-platform/connectivity-link-spoke-gateway.png) + +[![Spoke gateway architecture — Gateway API aggregation](/images/hybrid-mesh-platform/arch-spoke-gateway.png)](/images/hybrid-mesh-platform/arch-spoke-gateway.png) + +Verify Connectivity Link reconciliation by inspecting `Gateway` status conditions and `HTTPRoute` `spec.parentRefs` — not only Pod labels. Chart path: `components/rhcl-operator`. + +## Gateway API theory + +Kubernetes Gateway API separates concerns: + +| Resource | Role | +| --- | --- | +| `Gateway` | Listens on addresses and ports; attaches TLS and listener policies | +| `HTTPRoute` | Attaches to a `Gateway`; selects backend `Service` resources with matches, filters, and weighted backends | + +Weighted rules enable canary or active-active distributions between service versions or regions when paired with mesh or multi-cluster DNS strategies. + +## Cross-cluster routing architecture + +The hub gateway routes traffic to spoke OpenShift Routes via `ExternalName` services: + +``` +Browser + → Hub OpenShift Router (HTTPS) + → Istio Gateway (port 8080) + → Waypoint Proxy (circuit breaking) + → ExternalName Service (east or west) + → Spoke OpenShift Router (HTTP:80) + → Backend Pod +``` + +### Front / API split + +Traffic uses separate `Service` objects per cluster and traffic type so Kiali and Grafana see distinct destinations: + +| Service | Purpose | +| --- | --- | +| `industrial-edge-east-front` | Static frontend assets for east spoke | +| `industrial-edge-east-api` | Socket.IO / API backend for east spoke | +| `industrial-edge-west-front` | Static frontend assets for west spoke | +| `industrial-edge-west-api` | Socket.IO / API backend for west spoke | + +All four services use `ExternalName` pointing at the spoke Route hostname. Istio still tracks them as separate destination nodes in the traffic graph. + +### HTTPRoute rules + +1. **`/api/*`** → `*-api` services (default **east 100%**, **west 0%** for Socket.IO session affinity) +2. **Catch-all `/*`** → `*-front` services (split by `gateway.weights.east` / `gateway.weights.west`) + +Override API weights with `gateway.apiWeights` when your application supports cross-cluster API load balancing. + +### Key requirements + +| Requirement | Reason | +| --- | --- | +| HTTP port **80** to spokes | Ambient gateways do not apply `DestinationRule` TLS to ExternalName backends; HTTPS causes `CERTIFICATE_VERIFY_FAILED` | +| `insecureEdgeTerminationPolicy: Allow` on spoke Routes | Permits HTTP from hub Envoy to spoke router | +| `ServiceEntry` per external host | Without it, Envoy has no cluster definition → HTTP 500 | +| Per-backend **Host** header rewrite | Spoke router routes by `Host` | +| Session affinity on `/api` | Socket.IO polling and WebSocket upgrade must hit the same backend | + +## Circuit breaking + +Each `ExternalName` service gets a `DestinationRule` with `outlierDetection` and `connectionPool`, enforced by `hub-gateway-system-waypoint` (Istio ambient L7). + +### Default configuration (demo profile) + +| Parameter | Default | Purpose | +| --- | --- | --- | +| `connectionPool.tcp.maxConnections` | 100 | Max concurrent TCP connections | +| `connectionPool.http.h2UpgradePolicy` | `DO_NOT_UPGRADE` | Spokes expect HTTP/1.1 | +| `connectionPool.http.maxRequestsPerConnection` | 10 | Force connection recycling | +| `outlierDetection.consecutive5xxErrors` | 3 | Eject after 3 consecutive 5xx | +| `outlierDetection.interval` | 10s | Health check interval | +| `outlierDetection.baseEjectionTime` | 30s | Minimum ejection duration | +| `outlierDetection.maxEjectionPercent` | 100 | Required when only one endpoint exists | + +`maxEjectionPercent: 100` is required because each ExternalName resolves to a single endpoint; otherwise Istio refuses to eject the last host. + +### Override circuit breaker values + +```yaml +gateway: + circuitBreaking: + enabled: true + outlierDetection: + consecutive5xxErrors: 5 + baseEjectionTime: 60s +``` + +Set `enabled: false` to disable circuit breaking entirely. + +## Relationship to Connectivity Link and Service Mesh + +Connectivity Link layers DNS automation, TLS policies, and advanced controls atop the Gateway API topology shown above. Service Mesh ambient (ztunnel/waypoints) carries east-west traffic between gateway hops and workloads. Start with plain HTTPRoutes for incremental adoption; enable Kuadrant policies when teams require DNS/TLS/rate-limit governance at scale. + +## IoT Dashboard integration + +The Industrial Edge `line-dashboard` (`iot-frontend`) requires an `iot-consumer` sidecar: + +- Bridges MQTT to WebSocket via Socket.IO +- ConfigMap `config.json` with `websocketHost: ""` (empty = same origin) +- Spoke Route `/api` → port 3000 (`iot-consumer`) +- Hub gateway proxies `/api` to the correct spoke backend where Socket.IO terminates + +## Operational notes + +- Align hostnames with `deployer.domain` and corporate DNS wildcard records +- Combine with Service Mesh ambient when east-west encryption between gateway hops and workloads matters +- Monitor gateway Envoy metrics at port 15020 `/stats/prometheus` +- Generate traffic through `hub-gateway-istio` or waypoints before expecting L7 panels in Grafana + +## References + +- [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io/) +- [Connectivity Link documentation](https://docs.redhat.com/en/documentation/red_hat_connectivity_link/) +- [Istio DestinationRule](https://istio.io/latest/docs/reference/config/networking/destination-rule/) + +**Next →** [Observability](observability) for Grafana and Kiali across clusters. diff --git a/content/patterns/hybrid-mesh-platform/ideas-for-customization.md b/content/patterns/hybrid-mesh-platform/ideas-for-customization.md new file mode 100644 index 000000000..12d4f9221 --- /dev/null +++ b/content/patterns/hybrid-mesh-platform/ideas-for-customization.md @@ -0,0 +1,89 @@ +--- +title: Ideas for customization +weight: 70 +aliases: /hybrid-mesh-platform/ideas-for-customization/ +--- + +# Ideas for customization + +Adapt Hybrid Mesh Platform to your fleet size, regions, and operational constraints. The pattern repository follows a **single-branch, multi-directory** model so all clusters stay on `main` while Helm values and folders isolate hub vs spoke configuration. + +## Branch strategy + +All configuration lives on branch **`main`**. Cluster-specific settings use directories — not long-lived environment branches. + +### Repository layout + +``` +. → Hub cluster (root Helm chart) +east/ → East spoke cluster (self-contained Helm chart) +west/ → West spoke cluster (self-contained Helm chart) +components/ → Shared component charts referenced by all clusters +``` + +Each directory (`east/`, `west/`) is an independent Helm chart with its own `Chart.yaml`, `values.yaml`, and `templates/`. The hub uses the repository root (`.`) as its chart. + +### How values and ApplicationSet interact + +| Path / File | Purpose | +| --- | --- | +| `values.yaml` | Hub — operators, observability, gateway, hub-only components | +| `east/values.yaml` | East spoke — subscriptions, domains, spoke Applications | +| `west/values.yaml` | West spoke — subscriptions, domains, spoke Applications | +| `values-lite.yaml` | Minimal hub profile for demos (fewer operators) | +| `components/` | Shared charts referenced by hub and spoke Application CRs | + +The ACM ApplicationSet (`components/acm-hub-spoke`) uses a **`clusterDecisionResource`** generator: + +```yaml +# Conceptual — see pattern repo for full template +source: + path: '{{name}}' # east/ or west/ +destination: + name: '{{name}}' # remote cluster via Argo CD secret +``` + +Each spoke chart generates **child Application CRs** that the spoke's own Argo CD syncs locally — the hub does not apply spoke workloads directly. + +## Adding a new spoke cluster + +1. Provision OpenShift and import the cluster into ACM. +2. Label `ManagedCluster`: `region=`, `cluster.open-cluster-management.io/clusterset=global`. +3. Copy `east/` to a new folder (for example `south/`) and update `clusterName`, `deployer.domain`, `clusters.hub.domain`. +4. Add `clusters.south.domain` (or equivalent) to hub `values.yaml`. +5. Register the spoke as an Argo CD cluster secret on the hub (name must match folder name). +6. Validate: `helm lint south/` and `helm template south/`. +7. Confirm Placement selects the cluster and ApplicationSet creates `south-spoke-components`. + +## Minimal profiles + +For constrained environments (labs, CI, edge PoC), use `values-lite.yaml` on the hub: + +```bash +helm template test . -f values-lite.yaml +``` + +Disables heavy subscriptions while preserving GitOps bootstrap and ApplicationSet paths. + +## Customization ideas + +| Area | Customization | +| --- | --- | +| Regions | Add spokes beyond east/west — new folder + ACM labels + hub gateway weights | +| Gateway weights | `gateway.weights.east` / `west` for canary or active-active front traffic | +| API affinity | `gateway.apiWeights` when Socket.IO can span spokes | +| Circuit breaking | Tune `gateway.circuitBreaking` per environment (see [Hub Gateway](hub-gateway)) | +| Industrial Edge | Scaffold additional factory instances per spoke via [Scaffolding](scaffolding) | +| Observability | Custom Grafana dashboards; additional Prometheus federation | +| Mesh | Waypoints for L7 policy on selected namespaces; keep data-lake off ambient if using MinIO | +| Skupper | Additional listeners/connectors for new spoke services | +| Secrets | Vault or External Secrets for multi-cluster credentials (never commit tokens) | +| OpenShift version | Validate on 4.20+; align operator channels per release | +| Developer Hub | Extend software templates under `docs/assets/backstage/software-templates/` | + +## Related patterns + +- [Multicloud GitOps](/patterns/multicloud-gitops) — fleet GitOps and ACM placement patterns +- [Industrial Edge](/patterns/industrial-edge/) — underlying factory/OT workload pattern (maintained tier) + +**Pattern repository:** [platform-hub-spoke-config](https://github.com/maximilianopizarro/platform-hub-spoke-config) diff --git a/content/patterns/hybrid-mesh-platform/industrial-edge.md b/content/patterns/hybrid-mesh-platform/industrial-edge.md new file mode 100644 index 000000000..d4f32d6f0 --- /dev/null +++ b/content/patterns/hybrid-mesh-platform/industrial-edge.md @@ -0,0 +1,106 @@ +--- +title: Industrial Edge on multiple spokes +weight: 50 +aliases: /hybrid-mesh-platform/industrial-edge/ +--- + +# Industrial Edge on multiple spokes + +The [Industrial Edge](/patterns/industrial-edge/) validated pattern is a **maintained** Red Hat Validated Pattern for discrete manufacturing and operational technology (OT) connectivity on OpenShift: sensors, MQTT, Kafka-centric pipelines, CI/CD for edge software, ML-assisted anomaly detection, and centralized data lakes. + +**Hybrid Mesh Platform** does not replace that pattern. It **extends** it by deploying the Industrial Edge stack independently on **east** and **west** spoke clusters while the hub aggregates observability, gateway access, and GitOps control. + +[![Industrial Edge on spoke](/images/hybrid-mesh-platform/industrial-edge.png)](/images/hybrid-mesh-platform/industrial-edge.png) + +For solution overview, northbound/southbound data flows, demo scenario (MANUela), and maintained-pattern diagrams, see [Industrial Edge](/patterns/industrial-edge/). + +After operators discover Kafka clusters and mesh namespaces receive ambient labels, use this page as the **multi-cluster business narrative** tying workloads together across spokes. + +## What Hybrid Mesh Platform adds + +| Capability | Single-cluster Industrial Edge | Hybrid Mesh Platform | +| --- | --- | --- | +| Deployment scope | One or few factory sites | Hub + east + west spokes from one Git repo | +| GitOps | Pattern-focused repo | ApplicationSet pushes `east/` and `west/` charts remotely | +| Cross-cluster access | Per-site ingress | Hub Gateway + Skupper listeners on hub | +| Observability | Local Grafana/Prometheus | Hub Grafana with Prometheus-East/West datasources | +| Provisioning | Manual / pattern install | Developer Hub templates target `east` or `west` | +| Data lake | Central + factory sites | MirrorMaker + hub `prod-cluster` / MinIO archiver | + +Each spoke runs an **identical** Industrial Edge stack independently — the hub does **not** host factory sensor workloads. + +## Factory pattern + +Factories emit telemetry through MQTT brokers and Camel K integrations. Kubernetes namespaces isolate teams while GitOps keeps spoke clusters aligned with approved Git revisions. + +Each spoke (east/west) runs the same stack independently — the hub aggregates metrics and provides gateway access to line-dashboard and API endpoints. + +## Data stack (per spoke) + +| Stage | Component | Namespace | +| --- | --- | --- | +| Ingress | AMQ Broker (MQTT acceptors), machine sensors | `industrial-edge-tst-all` | +| Integration | Camel K routes (MQTT→Kafka, Kafka→S3) | `industrial-edge-tst-all` | +| Streaming | AMQ Streams / Kafka `dev-cluster` topics | `industrial-edge-tst-all` | +| Replication | Strimzi MirrorMaker (factory→data-lake) | `industrial-edge-stormshift-messaging` | +| Data lake | `prod-cluster` Kafka + Camel S3 archiver | `industrial-edge-data-lake` | +| Processing | OpenShift AI (KServe InferenceService) | `industrial-edge-tst-all` | +| CI/CD | Tekton pipelines (buildah + deploy) | `industrial-edge-ci` | +| Visualization | `line-dashboard` (WebSocket consumer) | `industrial-edge-tst-all` | + +[![OpenShift AI — KServe inference on spoke](/images/hybrid-mesh-platform/openshift-ia.png)](/images/hybrid-mesh-platform/openshift-ia.png) + +End-to-end on a spoke: sensor → MQTT → Camel K → Kafka → optional ML → dashboard; MirrorMaker streams toward centralized storage for model training (see [Architecture](architecture#follow-the-request--one-temperature-reading-end-to-end)). + +## Camel K integrations + +The `mqtt-to-kafka` integration bridges sensor data from the AMQ Broker to Kafka topics. It runs as a Camel K `Integration` CR deployed by the `industrial-edge-tst` chart. + +The **Camel Kaoto** software template in Developer Hub scaffolds additional routes with a DevSpaces-ready project: + +- MQTT→Kafka route (temperature, vibration) +- Kafka→S3/MinIO archiver (data lake) +- Alert→Mailpit route (anomaly notifications) +- Kaoto visual editor + Continue AI code assistant + +**Scaffold:** Developer Hub → Create → Industrial Edge — Camel Routes (Kaoto + Continue AI) → choose **east** or **west**. + +## Spoke Argo CD Applications + +Each spoke deploys these Applications (generated from `east/` or `west/` charts): + +| Application | Chart | Purpose | +| --- | --- | --- | +| `industrial-edge-tst` | `components/industrial-edge-tst` | Sensors, broker, Kafka, Camel, dashboard, ML | +| `industrial-edge-stormshift` | `components/industrial-edge-stormshift` | Factory Kafka + MirrorMaker | +| `industrial-edge-data-lake` | `components/industrial-edge-data-lake` | prod-cluster + S3 archiver | +| `industrial-edge-pipelines` | `components/industrial-edge-pipelines` | Tekton build-and-test pipeline | + +Industrial Edge Applications exist **only** on spokes — never on the hub chart. + +## Service mesh (ambient mode) + +Industrial Edge namespaces run under OSSM 3 ambient mesh with ztunnel L4 metrics. Namespace `industrial-edge-tst-all` carries `istio.io/dataplane-mode: ambient`. + +Ztunnel captures `istio_tcp_connections_opened_total` and related L4 series; L7 `istio_requests_total` flows through waypoint gateways where deployed. + +**Exceptions** (stay off ambient): + +- `spoke-gateway-system` — gateway TLS and ExternalName routing +- `industrial-edge-data-lake` — MinIO TLS and object-store compatibility + +## Cross-cluster exposure + +| Mechanism | Role | +| --- | --- | +| **Spoke Gateway** | Aggregates Industrial Edge services per spoke for one Skupper Connector | +| **Hub Gateway** | Routes browser traffic to east/west front and API backends with circuit breaking | +| **Skupper** | Listeners `ie-gateway-*`, `prometheus-*`, `kafka-*` on hub | + +Deploy new edge instances on east or west via [Scaffolding](scaffolding). Verify traffic in [Observability](observability) and [Hub Gateway](hub-gateway). + +## References + +- [Industrial Edge Validated Pattern](/patterns/industrial-edge/) +- [Multicloud GitOps](/patterns/multicloud-gitops) — fleet GitOps foundation +- [Pattern repository](https://github.com/maximilianopizarro/platform-hub-spoke-config) — charts `industrial-edge-tst`, `industrial-edge-stormshift`, `industrial-edge-pipelines`, data lake charts diff --git a/content/patterns/hybrid-mesh-platform/observability.md b/content/patterns/hybrid-mesh-platform/observability.md new file mode 100644 index 000000000..6d75cf4a0 --- /dev/null +++ b/content/patterns/hybrid-mesh-platform/observability.md @@ -0,0 +1,201 @@ +--- +title: Observability +weight: 40 +aliases: /hybrid-mesh-platform/observability/ +--- + +# Observability + +Observability ties together metrics, logs, traces, and mesh visualization so operators can compare east and west Industrial Edge clusters from the hub. It sits between [Architecture](architecture) (why telemetry crosses Skupper) and [Industrial Edge](industrial-edge) (what applications emit). + +Grafana panels, Kiali graphs, and Kafka Console views help you confirm that factory telemetry and mesh traffic match the architecture diagrams after install. + +[![Grafana multi-cluster dashboards](/images/hybrid-mesh-platform/product-grafana-observability.png)](/images/hybrid-mesh-platform/product-grafana-observability.png) + +[![Kiali service mesh](/images/hybrid-mesh-platform/product-kiali-service-mesh.png)](/images/hybrid-mesh-platform/product-kiali-service-mesh.png) + +[![Kafka Console](/images/hybrid-mesh-platform/product-kafka-console-amq-streams.png)](/images/hybrid-mesh-platform/product-kafka-console-amq-streams.png) + +[![Observability pipeline](/images/hybrid-mesh-platform/arch-observability-pipeline.png)](/images/hybrid-mesh-platform/arch-observability-pipeline.png) + +## Grafana dashboard views + +Multi-cluster fleet dashboards on the hub (east/west traffic, Service Mesh L4/L7, Kafka health): + +[![Grafana — east-west traffic and Service Mesh](/images/hybrid-mesh-platform/product-grafana-observability-2.png)](/images/hybrid-mesh-platform/product-grafana-observability-2.png) + +[![Grafana — multi-cluster Istio metrics (ztunnel L4)](/images/hybrid-mesh-platform/product-grafana-observability-3.png)](/images/hybrid-mesh-platform/product-grafana-observability-3.png) + +[![Grafana — extended fleet KPI panels](/images/hybrid-mesh-platform/product-grafana-observability-4.png)](/images/hybrid-mesh-platform/product-grafana-observability-4.png) + +## Kiali and mesh topology views + +[![Kiali — service mesh traffic graph](/images/hybrid-mesh-platform/product-kiali-service-mesh-2.png)](/images/hybrid-mesh-platform/product-kiali-service-mesh-2.png) + +## Kafka Console views + +[![Kafka Console — multi-cluster clusters and topics](/images/hybrid-mesh-platform/product-kafka-console-amq-streams-2.png)](/images/hybrid-mesh-platform/product-kafka-console-amq-streams-2.png) + +[![Kafka Console — broker and topic detail over Skupper](/images/hybrid-mesh-platform/product-kafka-console-amq-streams-3.png)](/images/hybrid-mesh-platform/product-kafka-console-amq-streams-3.png) + +## Observability architecture + +| Layer | Technology | Role | +| --- | --- | --- | +| Metrics | User Workload Monitoring / Prometheus | RED/USE signals, Kafka lag, mesh L4/L7 stats | +| Dashboards (hub) | Grafana + multi-cluster datasources | Fleet and factory KPI views (`components/grafana-dashboards`) | +| Dashboards (spoke) | Grafana local | Per-cluster ztunnel L4, Kafka, workloads (`components/spoke-dashboards`) | +| Mesh UI | Kiali + OSSM Console plugin | Traffic graphs in OpenShift Console | +| Kafka UI | Streams for Apache Kafka Console | Hub UI for all spoke Kafka clusters (`components/kafka-console`) | +| Cross-cluster metrics | Skupper + Grafana datasource | Prometheus via VAN | +| Tracing | OpenTelemetry Collector | Distributed traces (`components/opentelemetry`) | +| Mesh scraping | `components/istio-monitoring` | istiod, gateways, waypoints, ztunnel, Kafka JMX | + +**Hub:** Grafana aggregates hub Thanos plus HTTP datasources `prometheus-east` / `prometheus-west` (Skupper listeners). Kafka Console registers hub `prod-cluster` and remote bootstrap services over Skupper. + +**Spokes:** Local Grafana and Kiali use local Thanos. Connectors expose Thanos and Kafka bootstrap to the hub. + +## Service Mesh metrics (OSSM 3 ambient + ztunnel) + +Use channel **`stable-3.2`** for the Service Mesh operator. Tech Preview (`candidates` / 3.0.0-tp.2) does **not** deploy ztunnel. + +| Metric | Producer | Notes | +| --- | --- | --- | +| `istio_tcp_connections_opened_total` | ztunnel | Primary spoke/hub L4 signal | +| `istio_tcp_sent_bytes_total` / `received` | ztunnel | Bytes per workload namespace | +| `istio_requests_total` | Waypoints, ingress gateways | L7; hub `hub-gateway-istio` when traffic flows | +| `kafka_server_kafkaserver_brokerstate` | Strimzi JMX | `3` = Running — use in gauge panels | +| `kafka_network_requestmetrics_requestspersec_total` | Strimzi JMX | API activity — bargauge panels | +| `kafka_server_replicamanager_leadercount` / `partitioncount` | Strimzi JMX | piechart / bargauge on fleet views | + +### Prerequisite: ambient profile on Istio CNI + +`IstioCNI` CR must include `profile: ambient`. Namespace-only ambient labels are insufficient — without the profile, ztunnel never becomes Ready and `istio_tcp_*` series are absent. + +Grant UWM `ClusterRoleBinding` to `cluster-monitoring-view` in `istio-system`, `ztunnel`, `hub-gateway-system`, and Industrial Edge namespaces. + +```bash +oc get ds -n ztunnel +oc logs -n istio-cni $(oc get pods -n istio-cni -o name | head -1) | grep AmbientEnabled +# Expect: AmbientEnabled: true +``` + +## Kiali and OSSM Console plugin + +Each cluster runs Kiali with an OSSMConsole CR. The dynamic plugin adds **Service Mesh** to the OpenShift Console. + +### Fixing 401 Unauthorized on the plugin + +The plugin proxies API calls to Kiali, which queries Thanos Querier (`:9091`). Kiali's service account needs cluster monitoring read access: + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: kiali-monitoring-rbac +roleRef: + kind: ClusterRole + name: cluster-monitoring-view +subjects: +- kind: ServiceAccount + name: kiali-service-account + namespace: openshift-cluster-observability-operator +``` + +Kiali CR `external_services.prometheus`: + +```yaml +prometheus: + auth: + type: bearer + use_kiali_token: true + thanos_proxy: + enabled: true + url: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091 +``` + +With ztunnel running, Kiali shows **L4** traffic graphs; **L7** appears for HTTP routed through waypoints. + +Hub Kiali multi-cluster details: [Getting Started — Kiali](getting-started#step-7-kiali-multi-cluster-hub). + +## Grafana + Thanos (dashboards with data) + +**Hub Grafana** uses a ServiceAccount token for local Thanos and HTTP URLs for remote spokes (Skupper auth proxy injects bearer token — hub Grafana does not store spoke tokens). + +**Spoke Grafana** uses local Prometheus/Thanos only. Do not point spoke dashboards at hub Skupper listener names unless intentionally cross-querying. + +| Dashboard | Visualizations | Data sources | +| --- | --- | --- | +| `east-west-traffic` | Gauges (broker state), donut pie (leaders East/West), bargauge (partitions, Kafka API req/s) | Hub + Prometheus-East/West | +| `multi-cluster-istio` | Timeseries + L4 bargauge per cluster | Mixed datasources | +| `local-metrics` | ztunnel readiness gauge, Kafka bargauge/piechart, L4 timeseries | Local Thanos per spoke | + +Notes: + +- Hub gateway / Istio HTTP panels may show **no data** until clients generate traffic through `hub-gateway-istio` or waypoints +- Kafka panels use `kafka_network_requestmetrics_*` and `kafka_server_replicamanager_*` — not legacy `brokertopicmetrics` with `_objectname` filters + +Enable User Workload Monitoring on spokes: + +```yaml +# cluster-monitoring-config ConfigMap +enableUserWorkload: true +``` + +## Multi-cluster metrics via Skupper + +On each spoke, Thanos Querier sits behind an nginx **auth proxy** (injects bearer token). Skupper Connector exposes the proxy; hub Listener `prometheus-east` / `prometheus-west` becomes a Grafana HTTP datasource. + +See [Architecture — Service Interconnect](architecture#service-interconnect-skupper-topology) for listener/connector names. + +## Kafka Console (hub) + +The Streams for Apache Kafka Console on the hub registers: + +- `prod-cluster` (hub, full metrics in `industrial-edge-data-lake`) +- dev/factory clusters on east and west via Skupper bootstrap listeners + +### Metrics configuration + +`metricsSources` type `openshift-monitoring` is broken in Console operator 0.12.x (`Prometheus URL is not configured` in logs). Use **`type: standalone`**: + +- URL: `https://thanos-querier.openshift-monitoring.svc.cluster.local:9091` +- Bearer token via `kubernetes.io/service-account-token` Secret +- TrustStore: `openshift-service-ca.crt` ConfigMap (PEM) +- `ClusterRoleBinding` for `cluster-monitoring-view` + +Each `kafkaCluster` entry **must** include `namespace` — without it, logs report `namespace is required for metrics retrieval`. + +Only hub `prod-cluster` shows full broker metrics by default. Spoke clusters over Skupper show topics and nodes; full metrics require broker DNS fixes below. + +### Broker DNS over Skupper + +Common error: `Timed out waiting for a node assignment` / `listNodes` fails — console reaches bootstrap over Skupper but broker advertised DNS from spokes does not resolve on the hub. + +**Fix:** + +1. **Spokes:** Strimzi `advertisedHost` per broker with cluster suffix (`dev-cluster-broker-0-east`, and similar) +2. **Hub:** headless Services + `EndpointSlice` in `components/kafka-console/templates/broker-dns.yaml` (Helm `lookup` of Skupper ClusterIPs) + +Argo CD excludes `Endpoints` from managed resources — use `EndpointSlice` so broker DNS syncs via GitOps. + +Re-sync `kafka-console` after Skupper listeners are healthy. Confirm `listNodes` returns 200 in the Console UI. + +## Grafana dashboard inventory + +| Dashboard | Scope | Datasources | +| --- | --- | --- | +| `east-west-traffic` | Hub | Hub, Prometheus-East, Prometheus-West | +| `multi-cluster-istio` | Hub | Hub, Prometheus-East, Prometheus-West | +| `local-metrics` | Each spoke | Local Prometheus (UWM/Thanos) | + +## References + +- [OpenShift Observability](https://docs.redhat.com/en/documentation/openshift_container_platform/latest/html/monitoring/) +- [Red Hat Service Interconnect 2.1](https://docs.redhat.com/en/documentation/red_hat_service_interconnect/2.1) +- [OSSM 3.2 ambient mode](https://docs.redhat.com/en/documentation/red_hat_openshift_service_mesh/3.2/html/installing/ossm-istio-ambient-mode) +- [Kiali on OSSM 3.2](https://docs.redhat.com/en/documentation/red_hat_openshift_service_mesh/3.2/html/observability/kiali-operator-provided-by-red-hat) + +Charts: `components/observability`, `components/grafana-dashboards`, `components/spoke-dashboards`, `components/kiali`, `components/kafka-console`, `components/opentelemetry`, `components/istio-monitoring`, `components/service-interconnect`, `components/spoke-interconnect`. + +**Next →** [Industrial Edge](industrial-edge) for the factory data pipeline on multiple spokes. diff --git a/content/patterns/hybrid-mesh-platform/scaffolding.md b/content/patterns/hybrid-mesh-platform/scaffolding.md new file mode 100644 index 000000000..0d41e0ab5 --- /dev/null +++ b/content/patterns/hybrid-mesh-platform/scaffolding.md @@ -0,0 +1,116 @@ +--- +title: Scaffolding +weight: 60 +aliases: /hybrid-mesh-platform/scaffolding/ +--- + +# Scaffolding Industrial Edge on east and west + +Developer Hub **Create** templates deploy new Industrial Edge instances to east or west spokes. This chapter is the operational follow-up to [Architecture](architecture) and [Getting Started](getting-started) — turn a running platform into additional factory instances without hand-editing GitOps repos. + +[![Developer Hub](/images/hybrid-mesh-platform/product-developer-hub.png)](/images/hybrid-mesh-platform/product-developer-hub.png) + +## What you need before scaffolding + +| Requirement | How to verify | +| --- | --- | +| Developer Hub reachable | `https://developer-hub.` loads | +| Signed in as `platformadmin` (or catalog user) | Keycloak OIDC; user listed in `catalog-users.yaml` | +| Gitea org `ws-platformadmin` exists | PostSync Job `gitea-admin-setup` succeeded in namespace `gitea` | +| Spoke tokens synced | `oc get secret developer-hub-spoke-tokens -n developer-hub` | +| Templates catalog loaded | **Create** shows three Industrial Edge templates | + +`platformadmin` is the Gitea root-equivalent user (`gitea_admin` is the server admin account). Workshop users receive orgs `ws-user1`, `ws-user2`, …; `platformadmin` uses **`ws-platformadmin`**. + +## Software templates + +Templates are static files in the pattern repository: + +``` +docs/assets/backstage/software-templates/ +``` + +Published catalog URL (when GitHub Pages is enabled on the pattern repo): + +``` +https://maximilianopizarro.github.io/platform-hub-spoke-config/assets/backstage/software-templates/templates-catalog.yaml +``` + +| Template | Target | Result | +| --- | --- | --- | +| Industrial Edge: IoT Manufacturing (Multi-Cluster) | east or west | IoT namespace, sensors, Kafka, deployment, Tekton pipeline | +| Industrial Edge — Camel Routes (Kaoto + Continue AI) | east or west | Camel routes, DevSpaces devfile, Continue AI config | +| Industrial Edge — Delete Instance | east or west | Removes Argo CD Application + Gitea repo + notification | + +After Argo CD syncs `developer-hub`, open **Catalog → Systems → industrial-edge** and use **Create** for the templates. + +## Step-by-step — deploy on east + +1. Open Developer Hub → **Create**. +2. Choose **Industrial Edge: IoT Manufacturing (Multi-Cluster)**. +3. Set **Instance Name** (for example `edge-factory-1`), **Owner** `platformadmin`, **Target Cluster** `east`. +4. Set **Hub cluster apps domain** (for example `apps.`). +5. Run the template. + +### Behind the scenes + +| Step | Action | +| --- | --- | +| `fetch:template` | Skeleton from template integration (GitHub Pages assets) | +| `publish:github` | Repo `ws-platformadmin/edge-factory-1-east` on Gitea | +| `catalog:register` | New Component in Developer Hub catalog | +| `http:backstage:request` | Argo CD Application `gen-platformadmin-edge-factory-1-east` on cluster **east** | +| Notification | Sent to `user:default/platformadmin` | + +Repeat with **Target Cluster** `west` for a west instance (`edge-factory-1-west` repo naming). + +## Verify Topology and Kubernetes tabs + +1. Open the new catalog entity (or platform `line-dashboard-east` for the reference stack). +2. **Topology** tab — requires `backstage.io/kubernetes-cluster: east` (or `west`) and `backstage.io/kubernetes-id` matching deployment labels. +3. **Kubernetes** tab — pods in `industrial-edge-tst-all` on the selected cluster. + +If Topology is empty: + +```bash +oc get secret developer-hub-spoke-tokens -n developer-hub -o jsonpath='{.data.EAST_SA_TOKEN}' | wc -c +oc get managedserviceaccount -n east +oc logs -n developer-hub -l app.kubernetes.io/name=backstage --tail=20 | grep -i kubernetes +``` + +Confirm `ManagedServiceAccount` and hub token sync Job completed ([Getting Started](getting-started#step-9-continue-ai-devspaces--kaoto)). + +## Gitea organizations + +| Org | Purpose | +| --- | --- | +| `developer-hub` | Platform-owned repos | +| `ws-` | Per-user scaffold repos (for example `ws-platformadmin`) | +| `app-of-apps` | ApplicationSet-managed GitOps repos — delete repo to trigger Argo CD prune | + +The `app-of-apps` org is created by the Gitea PostSync Job. Use it when wiring an ApplicationSet with a Gitea generator: each generated repo maps to one Argo CD Application; removing the repo lets prune clean up spoke resources. + +## Quay vs internal registry + +| Stage | Registry | +| --- | --- | +| Tekton buildah push | `image-registry.openshift-image-registry.svc:5000//:latest` | +| Deployment pull | Same internal image (no pull secret on OpenShift) | +| Catalog display | `quay.io//` annotation for metadata only | + +On-prem Quay (`quay-registry.`) is for catalog metadata and optional mirror; default pipelines do not require Quay credentials unless you add an explicit push step. + +## Delete an instance + +1. **Create** → **Industrial Edge — Delete Instance**. +2. Enter the same **name**, **owner**, and **target cluster** used at create time. +3. Template deletes the Argo CD Application and Gitea repo. +4. Unregister the catalog entity manually if it still appears in the catalog UI. + +## References + +- [Getting Started](getting-started) — Developer Hub OIDC and spoke token sync +- [Industrial Edge](industrial-edge) — workload components per spoke +- [Hub Gateway](hub-gateway) — expose line-dashboard through hub ingress + +**Next →** [Hub Gateway](hub-gateway) for cross-cluster HTTP routing, or [Observability](observability) to confirm metrics after workloads are running. diff --git a/static/images/hybrid-mesh-platform/ACM.png b/static/images/hybrid-mesh-platform/ACM.png new file mode 100644 index 000000000..b205514e9 Binary files /dev/null and b/static/images/hybrid-mesh-platform/ACM.png differ diff --git a/static/images/hybrid-mesh-platform/ACS-2.png b/static/images/hybrid-mesh-platform/ACS-2.png new file mode 100644 index 000000000..794851c51 Binary files /dev/null and b/static/images/hybrid-mesh-platform/ACS-2.png differ diff --git a/static/images/hybrid-mesh-platform/ACS.png b/static/images/hybrid-mesh-platform/ACS.png new file mode 100644 index 000000000..2e347b8e2 Binary files /dev/null and b/static/images/hybrid-mesh-platform/ACS.png differ diff --git a/static/images/hybrid-mesh-platform/arch-data-flow.png b/static/images/hybrid-mesh-platform/arch-data-flow.png new file mode 100644 index 000000000..1626fead5 Binary files /dev/null and b/static/images/hybrid-mesh-platform/arch-data-flow.png differ diff --git a/static/images/hybrid-mesh-platform/arch-gitops-sync-sequence.png b/static/images/hybrid-mesh-platform/arch-gitops-sync-sequence.png new file mode 100644 index 000000000..d99c88889 Binary files /dev/null and b/static/images/hybrid-mesh-platform/arch-gitops-sync-sequence.png differ diff --git a/static/images/hybrid-mesh-platform/arch-hub-spoke-flow.png b/static/images/hybrid-mesh-platform/arch-hub-spoke-flow.png new file mode 100644 index 000000000..e6359696e Binary files /dev/null and b/static/images/hybrid-mesh-platform/arch-hub-spoke-flow.png differ diff --git a/static/images/hybrid-mesh-platform/arch-observability-pipeline.png b/static/images/hybrid-mesh-platform/arch-observability-pipeline.png new file mode 100644 index 000000000..e4b6079b9 Binary files /dev/null and b/static/images/hybrid-mesh-platform/arch-observability-pipeline.png differ diff --git a/static/images/hybrid-mesh-platform/arch-overview.png b/static/images/hybrid-mesh-platform/arch-overview.png new file mode 100644 index 000000000..773239727 Binary files /dev/null and b/static/images/hybrid-mesh-platform/arch-overview.png differ diff --git a/static/images/hybrid-mesh-platform/arch-skupper-topology.png b/static/images/hybrid-mesh-platform/arch-skupper-topology.png new file mode 100644 index 000000000..0cba33c4b Binary files /dev/null and b/static/images/hybrid-mesh-platform/arch-skupper-topology.png differ diff --git a/static/images/hybrid-mesh-platform/arch-spoke-gateway.png b/static/images/hybrid-mesh-platform/arch-spoke-gateway.png new file mode 100644 index 000000000..5a9870f30 Binary files /dev/null and b/static/images/hybrid-mesh-platform/arch-spoke-gateway.png differ diff --git a/static/images/hybrid-mesh-platform/arch-sync-waves.png b/static/images/hybrid-mesh-platform/arch-sync-waves.png new file mode 100644 index 000000000..873b9b391 Binary files /dev/null and b/static/images/hybrid-mesh-platform/arch-sync-waves.png differ diff --git a/static/images/hybrid-mesh-platform/connectivity-link-hub-gateway.png b/static/images/hybrid-mesh-platform/connectivity-link-hub-gateway.png new file mode 100644 index 000000000..c3eaf8157 Binary files /dev/null and b/static/images/hybrid-mesh-platform/connectivity-link-hub-gateway.png differ diff --git a/static/images/hybrid-mesh-platform/connectivity-link-hub.png b/static/images/hybrid-mesh-platform/connectivity-link-hub.png new file mode 100644 index 000000000..3ea74718f Binary files /dev/null and b/static/images/hybrid-mesh-platform/connectivity-link-hub.png differ diff --git a/static/images/hybrid-mesh-platform/connectivity-link-spoke-gateway.png b/static/images/hybrid-mesh-platform/connectivity-link-spoke-gateway.png new file mode 100644 index 000000000..36fa6f8de Binary files /dev/null and b/static/images/hybrid-mesh-platform/connectivity-link-spoke-gateway.png differ diff --git a/static/images/hybrid-mesh-platform/connectivity-link-spoke.png b/static/images/hybrid-mesh-platform/connectivity-link-spoke.png new file mode 100644 index 000000000..df6b48cd3 Binary files /dev/null and b/static/images/hybrid-mesh-platform/connectivity-link-spoke.png differ diff --git a/static/images/hybrid-mesh-platform/industrial-edge.png b/static/images/hybrid-mesh-platform/industrial-edge.png new file mode 100644 index 000000000..84702f10f Binary files /dev/null and b/static/images/hybrid-mesh-platform/industrial-edge.png differ diff --git a/static/images/hybrid-mesh-platform/openshift-ia.png b/static/images/hybrid-mesh-platform/openshift-ia.png new file mode 100644 index 000000000..484fdec6d Binary files /dev/null and b/static/images/hybrid-mesh-platform/openshift-ia.png differ diff --git a/static/images/hybrid-mesh-platform/product-argocd-openshift-gitops.png b/static/images/hybrid-mesh-platform/product-argocd-openshift-gitops.png new file mode 100644 index 000000000..08df9f8fb Binary files /dev/null and b/static/images/hybrid-mesh-platform/product-argocd-openshift-gitops.png differ diff --git a/static/images/hybrid-mesh-platform/product-developer-hub.png b/static/images/hybrid-mesh-platform/product-developer-hub.png new file mode 100644 index 000000000..901d1f82b Binary files /dev/null and b/static/images/hybrid-mesh-platform/product-developer-hub.png differ diff --git a/static/images/hybrid-mesh-platform/product-grafana-observability-2.png b/static/images/hybrid-mesh-platform/product-grafana-observability-2.png new file mode 100644 index 000000000..4b40ba5d1 Binary files /dev/null and b/static/images/hybrid-mesh-platform/product-grafana-observability-2.png differ diff --git a/static/images/hybrid-mesh-platform/product-grafana-observability-3.png b/static/images/hybrid-mesh-platform/product-grafana-observability-3.png new file mode 100644 index 000000000..cb727dccb Binary files /dev/null and b/static/images/hybrid-mesh-platform/product-grafana-observability-3.png differ diff --git a/static/images/hybrid-mesh-platform/product-grafana-observability-4.png b/static/images/hybrid-mesh-platform/product-grafana-observability-4.png new file mode 100644 index 000000000..3b8144ec9 Binary files /dev/null and b/static/images/hybrid-mesh-platform/product-grafana-observability-4.png differ diff --git a/static/images/hybrid-mesh-platform/product-grafana-observability.png b/static/images/hybrid-mesh-platform/product-grafana-observability.png new file mode 100644 index 000000000..a6e848554 Binary files /dev/null and b/static/images/hybrid-mesh-platform/product-grafana-observability.png differ diff --git a/static/images/hybrid-mesh-platform/product-kafka-console-amq-streams-2.png b/static/images/hybrid-mesh-platform/product-kafka-console-amq-streams-2.png new file mode 100644 index 000000000..1f65f00a4 Binary files /dev/null and b/static/images/hybrid-mesh-platform/product-kafka-console-amq-streams-2.png differ diff --git a/static/images/hybrid-mesh-platform/product-kafka-console-amq-streams-3.png b/static/images/hybrid-mesh-platform/product-kafka-console-amq-streams-3.png new file mode 100644 index 000000000..1bb78ab84 Binary files /dev/null and b/static/images/hybrid-mesh-platform/product-kafka-console-amq-streams-3.png differ diff --git a/static/images/hybrid-mesh-platform/product-kafka-console-amq-streams.png b/static/images/hybrid-mesh-platform/product-kafka-console-amq-streams.png new file mode 100644 index 000000000..9b959986a Binary files /dev/null and b/static/images/hybrid-mesh-platform/product-kafka-console-amq-streams.png differ diff --git a/static/images/hybrid-mesh-platform/product-kiali-service-mesh-2.png b/static/images/hybrid-mesh-platform/product-kiali-service-mesh-2.png new file mode 100644 index 000000000..3e83953f1 Binary files /dev/null and b/static/images/hybrid-mesh-platform/product-kiali-service-mesh-2.png differ diff --git a/static/images/hybrid-mesh-platform/product-kiali-service-mesh.png b/static/images/hybrid-mesh-platform/product-kiali-service-mesh.png new file mode 100644 index 000000000..057657422 Binary files /dev/null and b/static/images/hybrid-mesh-platform/product-kiali-service-mesh.png differ diff --git a/static/images/hybrid-mesh-platform/service-interconnect-console-metrics.png b/static/images/hybrid-mesh-platform/service-interconnect-console-metrics.png new file mode 100644 index 000000000..c16a77906 Binary files /dev/null and b/static/images/hybrid-mesh-platform/service-interconnect-console-metrics.png differ diff --git a/static/images/hybrid-mesh-platform/service-interconnect-console-process.png b/static/images/hybrid-mesh-platform/service-interconnect-console-process.png new file mode 100644 index 000000000..c1ef94c13 Binary files /dev/null and b/static/images/hybrid-mesh-platform/service-interconnect-console-process.png differ diff --git a/static/images/hybrid-mesh-platform/service-interconnect-console-topology-process.png b/static/images/hybrid-mesh-platform/service-interconnect-console-topology-process.png new file mode 100644 index 000000000..d463e6a6c Binary files /dev/null and b/static/images/hybrid-mesh-platform/service-interconnect-console-topology-process.png differ diff --git a/static/images/hybrid-mesh-platform/service-interconnect-console-topology.png b/static/images/hybrid-mesh-platform/service-interconnect-console-topology.png new file mode 100644 index 000000000..0edbabf39 Binary files /dev/null and b/static/images/hybrid-mesh-platform/service-interconnect-console-topology.png differ diff --git a/static/images/hybrid-mesh-platform/service-interconnect-console.png b/static/images/hybrid-mesh-platform/service-interconnect-console.png new file mode 100644 index 000000000..a07c8080b Binary files /dev/null and b/static/images/hybrid-mesh-platform/service-interconnect-console.png differ