The GPU Isn't The Bottleneck

How Trading Firms Cut AI Inference Latency On-Premises

Where AI inference latency comes from in trading systems, and what firms change on-premises: bare metal, colocation, right-sized GPUs, serving stack tuning.

Author
Samuel Zeman
EMEA Account Executive
Arc Compute
Connect on LinkedIn→

A trading desk running inference in the cloud usually has the same two complaints. The bill moves unpredictably, and performance slips exactly when the market gets busy and the signal matters. That combination moved the proprietary trading firm Lynx Trading Technologies out of the cloud and into 2 NVIDIA HGX B200 systems in a Secaucus colocation facility, 4 weeks from quote to installation.

‍

‍

Most of what a move like that buys back has little to do with the accelerator. A server 50 kilometers from the matching engine starts roughly 250 microseconds behind one in the same building, each way, and in an audited benchmark the same NVIDIA GH200 returned one model's signal in 4.70 microseconds and another's in 15.80 with no hardware change.

Most firms trying to cut AI inference latency start by looking at the GPU, and the microseconds are usually going elsewhere: the distance to the venue, the hops between feed and accelerator, the queue a request waits in, and model size.

Which matters depends on the workload, because most desks run two that do not share a latency problem. Real-time machine learning in the signal path, covering order book prediction and market making, is judged in microseconds. Language model work on filings, news and sentiment is judged in hundreds of milliseconds. Infrastructure built for one is often wrong for the other.

‍

Where AI inference latency actually comes from

Four contributors account for most of the budget, and they are not equally compressible.

Distance. Light in optical fiber travels at roughly 5 microseconds per kilometer, according to CommScope's white paper on latency in optical fiber systems, which is why venue proximity settles much of the budget before compute enters the picture. The CME Group matching engine runs out of CyrusOne's Aurora I facility in Aurora, Illinois rather than downtown Chicago, as Databento documents, so where a firm racks its hardware is already a latency decision.

5 microseconds
per kilometer of optical fiber
Propagation delay in optical fiber, at a refractive index of 1.5. Source: CommScope, Latency in Optical Fiber Systems.

Network hops and data movement. Every virtual switch, hypervisor boundary and gateway between the feed handler and the GPU adds transit time and, more importantly, variance to it. The same applies inside the box, where movement from the network interface card to GPU memory across the PCIe bus sits in the hot path, and whether RDMA is available can matter more than headline throughput.

Queueing and batching. Requests left waiting for batch capacity raise latency under load, which is how a system that benchmarks cleanly degrades at the moment volume spikes. On the language model path this is time to first token (TTFT), which ClickHouse's engineering breakdown defines as covering network transit, queueing and prefill. On the real-time path it is the gap between a healthy median and an unacceptable tail.

Model size and memory. A larger model means more weight traffic per step, not just more arithmetic, which is why memory bandwidth rather than peak floating point throughput sets the floor on single-request latency.

Where the inference latency budget goes
ContributorWhat sets itHow compressible
Distance to the venueFiber route length at roughly 5 microseconds per kilometerOnly by moving the server
Network hopsVirtual switches, hypervisor boundaries, gatewaysHigh on dedicated bare metal
Queueing and prefillScheduler behaviour, batch capacity, prompt lengthHigh, if the serving stack is exposed
Decode step timeMemory bandwidth and model weight trafficModerate, via model size and quantisation
Sources: CommScope; ClickHouse engineering guidance on inference latency; vLLM optimisation documentation.

Why does cloud GPU inference add latency for trading?

Shared cloud GPU inference adds latency through 4 mechanisms: physical distance from the matching engine, virtualized network hops between the feed and the accelerator, queueing behind other tenants' requests, and limited control over instance placement. The arithmetic mean usually looks acceptable, while the tail that actually governs a trading decision does not.

This is why the measurement that matters is p50, p99 and p99.9 under realistic market data load, not a single average. A quant team can design around a known 2 millisecond path but has nothing to design around when that path runs at 2 milliseconds most of the time and 40 milliseconds whenever a neighbouring workload saturates a shared link. Shared instances also hide where an accelerator sits relative to a venue, which makes proximity planning awkward.

Cost behaves the same way, since inference for live trading is effectively always on through market hours and a per-hour rate applies whether or not the model is doing useful work. Practitioners discussing self-hosted inference economics on Hacker News kept returning to 2 points: the appeal was predictable cost rather than the lowest cost, and a locally hosted model does not change unless the team changes it.

‍

What GPUs do trading firms use for inference?

Trading firms select GPUs by memory bandwidth and deterministic behaviour rather than peak training throughput. NVIDIA and STAC published audited results in April 2026 showing single-digit microsecond p99 inference latency on a GH200 Grace Hopper Superchip, alongside a separate open-source reference running on an RTX PRO 6000 Blackwell Server Edition, with model size rather than GPU class determining the outcome.

The audited STAC-ML Markets (Inference) Tacana figures, published by NVIDIA engineers Nikolay Markovskiy and Martin Marciniszyn Mehringer, show how quickly latency scales with model size rather than hardware tier.

Audited p99 inference latency by model size, GH200 Grace Hopper Superchip
ModelRelative sizep99 latency, 1 instancep99 latency, 4 instances
LSTM_ABaseline4.70 microseconds4.61 microseconds
LSTM_BAbout 6x LSTM_A7.10 microseconds7.10 microseconds
LSTM_CAbout 200x LSTM_A15.80 microsecondsNot published
FP16 precision, STAC-ML Markets (Inference) Tacana suite, audited by STAC. Source: NVIDIA Developer Blog, 2 April 2026.

Latency also stayed roughly flat as concurrent model instances rose, which NVIDIA attributes to CUDA green contexts partitioning the GPU. Right-sizing the model to the decision it supports is therefore its own latency lever, and usually a cheaper one than upgrading silicon.

‍

Do GPUs or FPGAs win the lowest-latency path?

FPGAs hold the lowest audited inference latency in capital markets, while GPUs cover a wider range of model sizes with easier development. On the STAC-ML Tacana suite, an audited FPGA submission shown at the STAC Summit in May 2026 came in below 2 microseconds p99 on the smallest model, against 4.70 microseconds for the GPU system.

Audited p99 latency on the same STAC-ML Tacana model suite, FPGA and GPU submissions
ModelFPGA submissionGPU submission (GH200)
LSTM_ABelow 2 microseconds4.70 microseconds
LSTM_BBelow 3 microseconds7.10 microseconds
LSTM_CBelow 8 microseconds15.80 microseconds
FPGA figures are stated as upper bounds and include a PCIe round trip of about 1 microsecond. Sources: STAC Summit presentation, May 2026; NVIDIA Developer Blog, 2 April 2026.

These are different tools rather than competing ones. FPGA designs win the deterministic hot loop, but cost compile cycles, specialist engineers and a slower path to production, while GPUs win everything behind it: larger models, frequent retraining, simulation and the language model workload. Hybrid architectures that put the latency-sensitive signal on an FPGA and the heavier work on GPUs are common for that reason, a point we developed in why GPUs are now essential in high frequency trading.

‍

What firms change when inference moves on-premises

Four levers open up once the environment stops being shared.

Dedicated bare metal. Removing the hypervisor removes a source of jitter and hands back control over interrupt handling, memory placement and device assignment, which the techniques NVIDIA used to reach single-digit microseconds quietly assume.

Placement. Colocating inside or next to the venue facility collapses the distance term outright, and for firms running inference in a general-purpose cloud region it tends to be the largest single reduction available.

Right-sized models and GPUs. A smaller distilled model on a mid-tier accelerator will often beat a large model on a flagship card along a latency-bound signal path, which argues for starting capacity planning from the decision latency budget rather than the hardware.

Serving stack tuning. This is the lever teams most often leave untouched, and it differs by path. Real-time models are compiled and served through stacks such as TensorRT and Triton Inference Server, where gains come from kernel selection, quantisation and model residency. Language model serving exposes a different trade-off, which vLLM's documentation states plainly: smaller max_num_batched_tokens values around 2048 improve inter-token latency, while larger values improve TTFT. None of this is adjustable in a managed endpoint.

Which latency levers are available by deployment model
LeverShared cloud instanceDedicated bare metal, colocated
Placement relative to the venueRegion level onlyFacility level
Hypervisor removedNoYes
Scheduler and batch tuningLimited or noneFull
Model version stabilityProvider controlled on managed endpointsTeam controlled
Cost behaviour at low utilisationCharged per hour regardlessFixed, independent of load
Tuning parameters per vLLM optimisation documentation. Cost behaviour reflects standard commercial terms rather than a published benchmark.

Do you need to own the hardware to get low latency?

No, because ownership and latency are separate decisions. Latency is determined by dedicated access to the accelerator, control over the serving stack and physical proximity to the venue, and all 3 can be obtained through colocation with leased equipment, managed dedicated infrastructure or outright purchase. Ownership changes the economics and the balance sheet treatment rather than the microseconds.

The trade-offs run both ways. Buying ties up capital in hardware with a defined refresh cycle and brings facility, spares and staffing obligations smaller teams underestimate, while leasing preserves flexibility and moves that burden elsewhere at a higher cost per unit of compute.

The Lynx deployment took the colocated route, on air-cooled systems built by Aivres. The gains reported in that case study, closer exchange proximity and steadier costs, are qualitative rather than benchmarked.

For a firm of 10 to 200 people in New York, Chicago, Toronto or London, the question is narrower: whether the team can run a colocated cage and a tuned serving stack without hiring 2 infrastructure engineers. That is what managed GPU infrastructure arrangements exist to solve.

‍

How do compliance rules affect where trading AI runs?

Regulation shapes placement more than most infrastructure plans allow for. In the European Union, the Digital Operational Resilience Act (Regulation (EU) 2022/2554) entered into application on 17 January 2025 and, as the European Insurance and Occupational Pensions Authority sets out, establishes an oversight framework for critical Information and Communication Technology (ICT) third-party providers to address concentration risk.

Concentration risk is a regulatory argument for infrastructure diversity, and a firm routing its entire signal path through a single hyperscaler region sits close to the case that provision was drafted to describe. Timing obligations push the same way. Under Commission Delegated Regulation (EU) 2017/574, known as Regulatory Technical Standard 25, firms using a high frequency algorithmic trading technique must keep business clocks within 100 microseconds of UTC and timestamp to 1 microsecond or better, against 1 millisecond for other trading activity. A system that cannot demonstrate deterministic timing inside that envelope has an evidentiary problem, not only a performance one.

Timing and resilience obligations that shape placement
RequirementApplies toObligation
Regulatory Technical Standard 25High frequency algorithmic tradingClocks within 100 microseconds of UTC, 1 microsecond granularity
Regulatory Technical Standard 25All other trading activityClocks within 1 millisecond of UTC, 1 millisecond granularity
Digital Operational Resilience ActEU financial entitiesOversight of critical ICT third-party providers, concentration risk addressed
Sources: Commission Delegated Regulation (EU) 2017/574, Annex Table 2; Regulation (EU) 2022/2554, in application since 17 January 2025 (EIOPA).

Data residency sits alongside both, since where model weights, feature stores and inference logs reside is now a question auditors ask directly, and one we examined in why cloud-only data sovereignty strategies fall short.

‍

Start with the latency budget, not the hardware

The firms that handle this well begin with the same unglamorous exercise. They separate the real-time signal path from the language model workload, write down the end-to-end budget for a single decision on each, and only then ask what hardware and placement the remainder allows.

If your team is mapping that budget, or has traced the bottleneck to a layer you do not control, Arc Compute builds and operates dedicated GPU environments for financial services firms where placement, isolation and serving stack control are designed in rather than worked around.

‍

Sources

About the Author
Samuel Zeman
EMEA Account Executive
Arc Compute

Sam drives customer engagement and growth across the EMEA region, partnering with organizations to deliver GPU infrastructure solutions tailored to their AI and high-performance computing requirements. Based in Slovakia, he works closely with customers throughout the purchasing process, helping turn infrastructure needs into production-ready deployments.

Connect on LinkedIn→
Continue Your Research

Explore Other related resources