MOC 2.0 · Platform State

WhatEight roll-ups computed from the cluster records further down the page.

WhyThe platform's state is spread across five repositories, so a reader needs one place that says how big it is. Every number here is derived from the detail below rather than typed in, because a hand-maintained total drifts from the thing it summarises the first time anyone edits one and not the other.

Look forTiles labelled partial. That means a figure is incomplete, not that it is small — a value no source records is excluded from the total rather than counted as zero.

WhatThe connection map: identity sources, AWS, the edge, the hubs, the workload clusters and storage.

WhyTwo constraints shaped this. Hardware was scarce — a third of the GPUs failed before the build started — so rather than spend three machines on a control plane per cluster, every workload cluster's control plane runs as pods on a shared hub. And the identity provider sits outside the data centre, because the thing that lets you log in must not depend on the thing you log in to fix. The diagram exists to make both visible at once.

Look forBoxes marked NO LINKS IN SOURCES — real components whose connections nothing records, drawn unconnected rather than wired up by guesswork. Also any ?, and the absence of control-plane nodes on the workload clusters: that absence is the architecture working.

WhatEach cluster as declared in Git — node pools, components, tenant projects, isolation tiers, VLANs and machines.

WhyUnder hosted control planes a workload cluster owns only worker nodes. Its API server, etcd and scheduler are pods in a clusters-<name> namespace on its hub, which is why a workload cluster's size is a node-pool count and its control plane appears in the hub's namespace list instead. Hubs are ordinary clusters and do have their own nodes, but they are installed by a different route and so are declared in a different repository.

Look forGPU counts marked needs data; the source line under a hub's node count; and whether the node-pool replicas in oac-apps agree with the machine inventory — two independent sources counting the same thing.

WhatProgramme phases from 2026-07-08 to 2026-09-19, against the September 1 deadline.

WhyThe deadline was fixed and the hardware came from decommissioning the previous platform, so building could not begin until the old cloud was switched off. That sequencing is why the shape looks the way it does: roughly six weeks of deciding, then about twelve days of building.

Look forBars still running to the right edge — work that never landed inside the window, which is where the next phase inherits its risk.

WhatRFC status derived from issue state, and the log of decisions already taken with the alternative each one beat.

WhySeveral decisions were explicitly provisional — hosting identity in a commercial AWS account was recorded as a pilot, with compliance to be worked out later. A decision recorded without its rejected alternative cannot be re-evaluated when the constraints change, and the constraints are about to change, so both halves are kept.

Look forItems sitting in Under review with old dates, and decisions whose stated rationale was time-boxed rather than permanent.

WhatIssues opened and closed each week in the tracker.

WhyIssue flow is the only continuous record of where effort actually went; the repositories show what was built but not what it cost to decide. The July spike is requirements work, not building.

Look forWeeks where opened far exceeds closed. Roughly two thirds of the tracker is still open, and that backlog is where the compliance work now lives.

WhatDefects and surprises found during the build, with current status, each citing an issue.

WhyThe expensive failures were consequences of the architecture rather than faults in it. Consolidating control planes onto hubs moved the scheduler out of the workload cluster — and took the metric the invoices are built from with it. Keeping these listed is what stops a decision's cost from being forgotten once the decision looks settled.

Look forAnything still Open, especially serious or critical, and whether a fix created its own follow-on problem.

WhatEvery tenant project across all clusters: who asked for it, what it is for, which groups hold edit on it, and the quota it runs under.

WhyA project is the unit of tenancy — it is a namespace, a quota, a limit range, a network policy and a set of RoleBindings created together by the user-projects chart from one entry in a values file. Because that entry is in Git, who requested a project and why is recorded rather than remembered, which is also what makes the tenancy auditable.

Look forProjects whose quota is not the chart default, and the difference between the production clusters, where projects are tracked in a values file, and development, where they are created more loosely.

WhatRealm settings, federated providers, per-cluster OIDC clients, groups and authentication flows, read from the Terraform.

WhyMOC serves users from many institutions it does not control, so it federates rather than holding passwords. But federation alone would let anyone with a valid university account in, so the first-broker flow links only accounts that already exist — the platform refuses unknown users by construction rather than by policy document. The realm flags are that intent expressed as configuration.

Look forClients marked no cluster — a cluster being prepared before its configuration exists. Values shown as variable, which the repository alone cannot resolve. And the realm flags: registration disabled, email login disabled, duplicate emails disallowed.

WhatEvery machine either documented in open-accelerator-infra’s hardware table or named in its Ansible inventory, with the VLANs its NICs carry, its BMC address, and whether a cluster claims it.

WhyThe cluster views only show machines that belong to a cluster, which is a minority of the fleet. The rest is the capacity question: how much idle hardware exists, how much is broken, and how much is being held for the RHOSO 18 build. A platform assembled from decommissioned hardware is defined as much by what is unusable as by what is running.

Look forMachines flagged flagged (boot failures, unreachable BMCs, disk faults) and those reserved for RHOSO 18. Also the inventory only rows: the hardware table documents the production side — rack R4PAC10 plus the prod A100s and H100s — so all eight development machines appear from the inventory alone, with no NIC or resource-class data. That is a gap in the source, not in the parser. The table records its own unknowns too: three R440s are listed as ?? R440-1 ?? and are skipped.

WhatVLANs and switch ports from ansible-switches, joined to CIDRs from the hardware document and Terraform.

WhyA pod is attached only to the cluster network, but the storage array serves NFS and S3 on a separate, faster VLAN. That single fact forces a second network interface on every node, a proxy for object storage, and wider storage subnets — so the physical VLAN table is where an apparently abstract Kubernetes problem becomes a cabling one.

Look forVLANs carrying real ports but claimed by no declared cluster — including the MOCSEC-* set, which is the NIST environment already wired at the network layer. Also whether a storage VLAN's CIDR matches what Portworx declares; the two are maintained in different repositories.

WhatNode pools each workload cluster declares, and the GPU count that implies.

WhyGPUs are the scarce resource and billing is by reservation rather than use, so declared capacity is the number that generates an invoice. The metric that would show real utilisation is unavailable under hosted control planes, so this section deliberately reports what was reserved and says why the other half is missing.

Look forSegments whose GPU count is unknown, and the note on the missing billing metric — the gap between reserved and used is the open policy question, and it currently cannot be measured.