Taking Over a Running Cluster

GPU Managed Services Takeover

What handing over a running cluster actually involves: the audit, the credential handover, the build standard, and the first month of live operation.

Author
Justin Ritchie
President
Arc Compute
Connect on LinkedIn→

Most GPU clusters that get handed over were not failing. They were working, and still not worth what they cost.

‍

That is what makes this decision harder than it looks. There is rarely a single bad night to point at. The engineer who understood the east-west fabric leaves. Firmware drift, optic replacement, and OEM ticket triage turn into steady unplanned work nobody was hired to do. Utilization sits below what the business modelled and no one can say precisely why, because SNMP polls on a 5 minute interval never captured the fault that caused it. Nothing on the dashboard looks broken, which is the problem. A cluster can run for months while quietly failing to earn what it cost.

‍

Below is what moving it actually involves. The five stages and who is accountable at each, what gets checked on the data center floor that no dashboard will show you, the access you need to collect, the standard a cluster has to meet before a serious operator will take it on, and what should happen in the first month after control transfers. It is worth reading even if you never change providers, because it doubles as a scorecard for whoever runs your cluster today.

‍

‍

‍

‍

‍

‍

What does a GPU managed services takeover actually involve?

‍

A GPU managed services takeover runs through five stages: an audit of the cluster while the outgoing team still operates it, an access and documentation handover, a written gap assessment against a defined build standard, remediation to close those gaps, and a stabilization period once control transfers. The audit always comes before ownership.
‍

The five stages of a GPU cluster takeover

Stage Duration What happens Who is accountable
Audit About 1 week Read-only review of physical plant, switch and compute telemetry, logs, documentation, and ticket history Outgoing provider still operates the cluster. Incoming operator has read access only
Gap assessment 2 to 5 days Written delta between the cluster as found and the standard required for handover Incoming operator writes it. Customer decides what gets funded
Remediation Scope dependent Recabling, network correction, patching, firmware alignment, documentation rebuild Customer's own team, or the incoming operator under a separate scope of work
Handover 2 to 5 days New credentials issued, all prior credentials removed, management plane rebuilt from a clean install Incoming operations team, with customer security sign-off on credential removal
Stabilization 30 days First period of live operation. Recurring faults resolved, failure baselines established, maintenance cadence agreed Incoming operations team, reviewed with the customer at day 30

Figure 1. The five stages of a GPU cluster takeover. The sequence matters more than the timeline, and no serious operator should take responsibility for an environment it has not inspected.

‍

‍

‍

Why running clusters change hands

Nobody calls us about a cluster that is working. The trigger is rarely one dramatic outage either. It is accumulation.

‍

  • The person who understood it left. One departure is often enough. ManpowerGroup’s 2026 Talent Shortage Survey of 39,063 employers across 41 countries found 72% report difficulty filling roles, with artificial intelligence skills ranking as the hardest capability to hire globally for the first time. Replacing that person takes longer than the business will tolerate.
  • The team ran out of bandwidth. Firmware, optics, ticket triage, and vendor coordination are steady work, and not the work most infrastructure teams were hired to do.
  • The incumbent provider stopped earning it. Recurring faults that never close out, tickets that bounce between vendors, no documentation anyone can act on.
  • Utilization stopped justifying the spend. The cluster works and finance still wants to know why the return is not there.

‍

Any of these is a fair reason to hand a cluster to somebody else, and several show up in the infrastructure challenges we hear most weeks months before anyone frames it as a managed services decision.

‍

‍

‍

Why the audit starts on the data center floor

‍

Remote audits are convenient and incomplete. Counters, logs, and telemetry describe electrical behaviour. They do not describe a cable pressed against an exhaust fan.

‍

On a recent takeover, our infrastructure team opened the cage and knew inside a minute that the whole site needed recabling. Where 3 and 4 metre runs were appropriate, 20 metre cables had been ordered for everything, so the excess sat in bundles crammed against server chassis. During recabling the team pulled out fibre whose jacketing had melted and fused together after sustained exposure to GPU exhaust.

‍

Read that site remotely and it presents as intermittent link failures with no obvious cause. Walk the floor and the answer takes 30 seconds.
‍

If we had not looked at it physically, we would have spent weeks asking why links were randomly failing.

Josh Gelata, Lead Infrastructure, Arc Compute


The floor inspection covers cable routing and strain, airflow and exhaust paths, optic seating, power distribution, and whether labelling matches reality. None of it is visible from a dashboard, and all of it produces faults that monitoring will describe inaccurately.

‍

‍

‍

The access list, and the conversation nobody wants to have

‍An audit is only as good as the access behind it. This is the checklist we hand the customer before anything else begins.

‍

  • Out-of-band management on every server
  • Every switch in the environment, compute fabric and management alike
  • Firewalls, where they sit in the customer’s scope rather than the facility’s
  • Management and monitoring infrastructure, including whatever provisioning system pushes images
  • System logs and the full support ticket history, not a summary of it
  • A complete inventory of users and credentials, mapped to the systems they belong to

‍

Ticket history is the most undervalued item there. A cluster with a port that has been flapping for eight months has already documented its own problem. Nobody read it back.

‍

The credential inventory is the hardest, and the difficulty is political rather than technical. The customer is asking their incumbent provider for every login at the precise moment that provider learns it is being replaced. Nothing prevents a departing team from handing over something technically complete and practically useless. On one engagement it took 5 days to map credentials to the 20 or so systems they belonged to, because the first version arrived as an unsorted list with no indication of what opened what.

‍

When documentation does not exist, and frequently it does not, we rebuild state from the devices: configuration pulled from switches, firmware inventoried from servers, topology reconstructed from what is physically cabled. Slower than reading a diagram, and more accurate than the diagram usually was.

‍

‍

‍

What "ready to hand over" actually means

‍

Before anyone assumes operational accountability, the cluster has to meet a defined standard, and the customer has to see exactly where their environment falls short. We put it in writing. It is the least comfortable document in the engagement and the most useful one.

‍

The build standard applied before a takeover

RequirementStandard appliedWhy it is not negotiable
East-west compute fabricFull non-blocking, rail-optimized leaf and spine. No oversubscription between any two GPUsNVIDIA's enterprise reference architectures specify a full non-blocking fat tree for the compute fabric. A distributed job runs at the speed of its slowest path, so one oversubscribed link caps the entire cluster
Fabric separationThree distinct networks: compute east-west, converged north-south and storage, and out-of-band managementNVIDIA reference architectures separate these deliberately. Collapsed fabrics turn a management event into a workload event
In-band management bandwidth25 Gbps to each server and at least 100 Gbps from that switch back to the spine holding image and provisioning serversPushing an image to 128 servers is not a small job. Over a 1 Gbps management path it becomes an overnight task and slows every rebuild for the life of the cluster
North-south and storageA throughput floor matched to the GPU class rather than a line-for-line reference buildUnder-provisioning starves data loading and presents as low GPU utilization rather than as a network fault
Security postureNo known unpatched vulnerabilities anywhere in the environment at handoverInherited exposure becomes the incoming operator's liability from day one

Network topology requirements align with NVIDIA Enterprise Reference Architecture guidance.

‍

The east-west line draws the most pushback, because closing it costs money, and it has the least room for negotiation. Front-end and storage networks get more latitude: they need to clear a throughput floor appropriate to the GPUs in the cluster, not match a reference diagram line for line. Which fabric you use to get there is a separate decision, and we worked through the tradeoffs in InfiniBand vs Ethernet for AI clusters.

‍

‍

‍

When the cluster was not built with the vendors you would have picked

‍

This is the part most takeover discussions skip. We standardise on Arista for the network, along with specific compute and storage vendors, because that combination gives us signal we can act on before something fails. Inherited clusters rarely match, and pretending otherwise helps nobody.

‍

So the handover document carries a plainly worded section: here is what we can commit to on this hardware, here is what would be possible on a different build, and here are the operational promises that get harder or impossible as a result. If the switching platform in place only exposes interval polling, predictive optic replacement is not something anyone can honestly promise. Faster triage and a documented failure baseline are. Those are different commitments, and the difference belongs in the contract rather than in the first incident review.

‍

‍

‍

Handover, and the 30 days that follow

Once the gaps are closed and control transfers, the first moves are sequenced and unglamorous.

‍

  1. New credentials are issued for the incoming operations team.
  2. Every prior credential is removed from switches, firewalls, and out-of-band management.
  3. The management cluster gets a fresh install rather than an inherited configuration, with nodes re-imaged in a rolling sequence.
  4. Rolling firmware and patch alignment follows, scheduled against workload availability.

‍

GPU nodes stay in production through most of that. The management plane gets rebuilt first so that when GPU nodes do need attention, the tooling touching them is known to be clean.

‍

Workloads then set the pace. A 24-node cluster running a job across all 24 nodes has no maintenance window until that job finishes, and the customer decides how long the window gets. Some work cannot be done with a reboot either. NVIDIA’s DGX GB200 NVL72 release notes document conditions where GPU reset operations fail with kernel Xid errors and recovery requires a full power cycle. Knowing which procedures a fleet needs is the difference between a planned window and an unplanned outage.

‍

‍

‍

Why the first 30 days matter, and whether you actually need them

‍

A cluster that has just changed hands is not a cluster anyone understands yet. An audit tells you what the environment looks like at rest. It does not tell you which optic degrades under thermal load on the third day of a training run, or which node drops a link every time an image gets pushed to it. Those patterns only appear once the new team is operating the thing live.

‍

Thirty days is roughly how long a cluster takes to show its real failure behaviour under production workload. By the end of it, three things should exist: every recurring fault from the audit is either fixed or formally accepted, a failure baseline predicts which optics and nodes are most likely to go next, and the customer has a documented maintenance window process rather than an ad hoc one.

‍

Is it necessary? On a well run cluster the month passes quietly and what you get is the baseline, which is worth having on its own. On a neglected cluster it is where the remaining surprises land, and it is far better that they land inside a defined window with an agreed escalation process than in the middle of your quarter. Either way, a provider unwilling to commit to a stabilization period is telling you something about how confident they are.

‍

‍

‍

Why we will not operate a cluster we cannot see into

‍

Most inherited clusters are monitored by polling. A collector queries each device on an interval using Simple Network Management Protocol (SNMP) and records what it finds. If the interval is 5 minutes and a fault lasts 40 seconds, that fault never happened as far as monitoring is concerned.

‍

Here is the version everyone recognises. Someone is on a video call at 3pm. Around 3:04 it stutters badly for 40 seconds, then recovers, and they raise a ticket afterwards. With 5 minute polls there is no record of 3:04, so the ticket gets closed as unreproducible. With streamed state you go to 3:04 directly and find that a large file transfer saturated the uplink. Same fault, two outcomes, and only one of them stops it happening again next week.

‍

Scale that to a GPU cluster and the stakes change, because the failure that hurts is intermittent long before it is total. An optic degrades before it dies. A link flaps before it drops. Arista describes its state streaming architecture as continuous, streaming every device state change as it occurs, against polling models it characterises as operating on 10 to 15 minute granularity.
‍

Polling based monitoring compared with streaming telemetry

CapabilityPolling based monitoringStreaming telemetry
Detection windowOnly what is true at the moment of the pollEvery state change as it occurs
Intermittent faultsFrequently missed between intervalsCaptured with the surrounding context
Optic degradationUsually detected after the link dropsTrending signals available before failure
Root cause on a past eventLimited to whatever the interval capturedHistorical state available for forensic review
Operational postureOpen a ticket after the impactSchedule a window before the impact

Source: Arista Networks network telemetry and state streaming documentation.

‍

The arithmetic explains why this is not a theoretical preference. On a rail-optimized fabric, 8 nodes at 8 GPUs each involve roughly 256 optics once both ends of the server to leaf and leaf to spine links are counted, plus at least 3 switches, all healthy, for a distributed run to proceed. Scale to hundreds of nodes and the question stops being whether an optic fails during a training run.
‍

419

Unexpected interruptions Meta reported across 54 days of pre-training on a cluster of up to 16,384 H100 GPUs, with hardware issues accounting for the majority.

Source: Meta Llama 3 technical report, cited in Lablup operational analysis of large scale training, arXiv 2026


Alibaba, publishing its own production data, attributed 15.8% of all training failures to network issues, split as 9.1% network interface cards and 6.7% optics. Those are not exotic components. They are the parts of a cluster that fail on a schedule, which is exactly why they should be visible before they fail.

‍

‍

‍

What you own, what we own, and when the cluster has to stop

Day to day accountability depends on the workload above the hardware, and the two profiles behave very differently.
‍

Why maintenance planning differs by workload

WorkloadWindow approachWhat a failing optic causesDesign requirement
Inference clusterRolling maintenance, no user impactAffected node drains, a standby node joins the poolSpare node capacity held in rotation
Training clusterFull pause, negotiated in advanceThe running job stops and restarts from checkpointPredictive optic signalling and short, pre-agreed windows

A cluster sized at 128 inference nodes is typically built with 130, keeping two in a rotating maintenance state at all times.

‍

For training there is no equivalent move. Every node in the job has to be up, which means every optic in the path has to be up, and one failed optic ends the run. That scheduling discipline is also what decides whether a cluster ever reaches the utilization it was funded for, which we examined in The 5% Problem.

‍

The ownership line needs writing down as well. In a managed engagement we own everything below the operating system: hardware faults, firmware, patching, coordination with original equipment manufacturers, ticket triage, and remote hands scheduling. In a colocation facility our responsibility stops at the power distribution unit, which we monitor and escalate but the facility fixes. Agreeing that boundary early prevents the argument that otherwise happens mid-incident, when nobody has patience for it.

‍

‍

‍

The question to ask at your next renewal
‍

Whether or not you ever change providers, one question is worth putting to whoever operates your cluster today.
‍

Which optics, links, or nodes are most likely to fail in the next 30 days, and what is the plan when they do?

The question a competent operator answers in a sentence

‍

An operator running streaming telemetry with a real failure baseline answers immediately. An operator running interval polls will tell you everything looks fine, which is a different statement than everything is fine. That answer, more than any service level agreement, tells you what you are paying for.

‍

We take over, remediate, and operate clusters other teams built, alongside the turnkey GPU clusters and private AI cloud environments we design from scratch. It starts with an assessment rather than a contract, and that assessment is useful even if you decide to keep operating the cluster yourself. If that is where you are, our team can walk you through what one covers.

‍

‍

‍

Sources

  1. NVIDIA Enterprise Reference Architectures, HGX AI Factory networking physical topologies
  2. NVIDIA Enterprise Reference Architecture, key building blocks
  3. NVIDIA DGX GB200 NVL72 release notes, known issues
  4. Arista Networks, Network Visibility with CloudVision and advanced network telemetry
  5. Arista Networks, State Streaming solution brief
  6. Lablup, operational analysis of large-scale pre-training, arXiv, citing Meta Llama 3 interruption data
  7. SHIFT: Exploring the Boundary of RDMA Network Fault Tolerance, arXiv, citing Alibaba production failure attribution
  8. ManpowerGroup, 2026 Talent Shortage Survey
About the Author
Justin Ritchie
President
Arc Compute

Justin founded Arc Compute to help organizations access and deploy the GPU infrastructure required for modern AI and high-performance computing workloads. As President, he leads the company’s vision, partnerships, and growth initiatives, working closely with customers to turn infrastructure challenges into scalable, production-ready solutions.

Connect on LinkedIn→
Continue Your Research

Explore Other related resources