A trading desk running inference in the cloud usually has the same two complaints. The bill moves unpredictably, and performance slips exactly when the market gets busy and the signal matters. That combination moved the proprietary trading firm Lynx Trading Technologies out of the cloud and into 2 NVIDIA HGX B200 systems in a Secaucus colocation facility, 4 weeks from quote to installation.

Most of what a move like that buys back has little to do with the accelerator. A server 50 kilometers from the matching engine starts roughly 250 microseconds behind one in the same building, each way, and in an audited benchmark the same NVIDIA GH200 returned one model's signal in 4.70 microseconds and another's in 15.80 with no hardware change.
Most firms trying to cut AI inference latency start by looking at the GPU, and the microseconds are usually going elsewhere: the distance to the venue, the hops between feed and accelerator, the queue a request waits in, and model size.
Which matters depends on the workload, because most desks run two that do not share a latency problem. Real-time machine learning in the signal path, covering order book prediction and market making, is judged in microseconds. Language model work on filings, news and sentiment is judged in hundreds of milliseconds. Infrastructure built for one is often wrong for the other.
Where AI inference latency actually comes from
Four contributors account for most of the budget, and they are not equally compressible.
Distance. Light in optical fiber travels at roughly 5 microseconds per kilometer, according to CommScope's white paper on latency in optical fiber systems, which is why venue proximity settles much of the budget before compute enters the picture. The CME Group matching engine runs out of CyrusOne's Aurora I facility in Aurora, Illinois rather than downtown Chicago, as Databento documents, so where a firm racks its hardware is already a latency decision.
Network hops and data movement. Every virtual switch, hypervisor boundary and gateway between the feed handler and the GPU adds transit time and, more importantly, variance to it. The same applies inside the box, where movement from the network interface card to GPU memory across the PCIe bus sits in the hot path, and whether RDMA is available can matter more than headline throughput.
Queueing and batching. Requests left waiting for batch capacity raise latency under load, which is how a system that benchmarks cleanly degrades at the moment volume spikes. On the language model path this is time to first token (TTFT), which ClickHouse's engineering breakdown defines as covering network transit, queueing and prefill. On the real-time path it is the gap between a healthy median and an unacceptable tail.
Model size and memory. A larger model means more weight traffic per step, not just more arithmetic, which is why memory bandwidth rather than peak floating point throughput sets the floor on single-request latency.
Why does cloud GPU inference add latency for trading?
Shared cloud GPU inference adds latency through 4 mechanisms: physical distance from the matching engine, virtualized network hops between the feed and the accelerator, queueing behind other tenants' requests, and limited control over instance placement. The arithmetic mean usually looks acceptable, while the tail that actually governs a trading decision does not.
This is why the measurement that matters is p50, p99 and p99.9 under realistic market data load, not a single average. A quant team can design around a known 2 millisecond path but has nothing to design around when that path runs at 2 milliseconds most of the time and 40 milliseconds whenever a neighbouring workload saturates a shared link. Shared instances also hide where an accelerator sits relative to a venue, which makes proximity planning awkward.
Cost behaves the same way, since inference for live trading is effectively always on through market hours and a per-hour rate applies whether or not the model is doing useful work. Practitioners discussing self-hosted inference economics on Hacker News kept returning to 2 points: the appeal was predictable cost rather than the lowest cost, and a locally hosted model does not change unless the team changes it.
What GPUs do trading firms use for inference?
Trading firms select GPUs by memory bandwidth and deterministic behaviour rather than peak training throughput. NVIDIA and STAC published audited results in April 2026 showing single-digit microsecond p99 inference latency on a GH200 Grace Hopper Superchip, alongside a separate open-source reference running on an RTX PRO 6000 Blackwell Server Edition, with model size rather than GPU class determining the outcome.
The audited STAC-ML Markets (Inference) Tacana figures, published by NVIDIA engineers Nikolay Markovskiy and Martin Marciniszyn Mehringer, show how quickly latency scales with model size rather than hardware tier.
Latency also stayed roughly flat as concurrent model instances rose, which NVIDIA attributes to CUDA green contexts partitioning the GPU. Right-sizing the model to the decision it supports is therefore its own latency lever, and usually a cheaper one than upgrading silicon.
Do GPUs or FPGAs win the lowest-latency path?
FPGAs hold the lowest audited inference latency in capital markets, while GPUs cover a wider range of model sizes with easier development. On the STAC-ML Tacana suite, an audited FPGA submission shown at the STAC Summit in May 2026 came in below 2 microseconds p99 on the smallest model, against 4.70 microseconds for the GPU system.
These are different tools rather than competing ones. FPGA designs win the deterministic hot loop, but cost compile cycles, specialist engineers and a slower path to production, while GPUs win everything behind it: larger models, frequent retraining, simulation and the language model workload. Hybrid architectures that put the latency-sensitive signal on an FPGA and the heavier work on GPUs are common for that reason, a point we developed in why GPUs are now essential in high frequency trading.
What firms change when inference moves on-premises
Four levers open up once the environment stops being shared.
Dedicated bare metal. Removing the hypervisor removes a source of jitter and hands back control over interrupt handling, memory placement and device assignment, which the techniques NVIDIA used to reach single-digit microseconds quietly assume.
Placement. Colocating inside or next to the venue facility collapses the distance term outright, and for firms running inference in a general-purpose cloud region it tends to be the largest single reduction available.
Right-sized models and GPUs. A smaller distilled model on a mid-tier accelerator will often beat a large model on a flagship card along a latency-bound signal path, which argues for starting capacity planning from the decision latency budget rather than the hardware.
Serving stack tuning. This is the lever teams most often leave untouched, and it differs by path. Real-time models are compiled and served through stacks such as TensorRT and Triton Inference Server, where gains come from kernel selection, quantisation and model residency. Language model serving exposes a different trade-off, which vLLM's documentation states plainly: smaller max_num_batched_tokens values around 2048 improve inter-token latency, while larger values improve TTFT. None of this is adjustable in a managed endpoint.
Do you need to own the hardware to get low latency?
No, because ownership and latency are separate decisions. Latency is determined by dedicated access to the accelerator, control over the serving stack and physical proximity to the venue, and all 3 can be obtained through colocation with leased equipment, managed dedicated infrastructure or outright purchase. Ownership changes the economics and the balance sheet treatment rather than the microseconds.
The trade-offs run both ways. Buying ties up capital in hardware with a defined refresh cycle and brings facility, spares and staffing obligations smaller teams underestimate, while leasing preserves flexibility and moves that burden elsewhere at a higher cost per unit of compute.
The Lynx deployment took the colocated route, on air-cooled systems built by Aivres. The gains reported in that case study, closer exchange proximity and steadier costs, are qualitative rather than benchmarked.
For a firm of 10 to 200 people in New York, Chicago, Toronto or London, the question is narrower: whether the team can run a colocated cage and a tuned serving stack without hiring 2 infrastructure engineers. That is what managed GPU infrastructure arrangements exist to solve.
How do compliance rules affect where trading AI runs?
Regulation shapes placement more than most infrastructure plans allow for. In the European Union, the Digital Operational Resilience Act (Regulation (EU) 2022/2554) entered into application on 17 January 2025 and, as the European Insurance and Occupational Pensions Authority sets out, establishes an oversight framework for critical Information and Communication Technology (ICT) third-party providers to address concentration risk.
Concentration risk is a regulatory argument for infrastructure diversity, and a firm routing its entire signal path through a single hyperscaler region sits close to the case that provision was drafted to describe. Timing obligations push the same way. Under Commission Delegated Regulation (EU) 2017/574, known as Regulatory Technical Standard 25, firms using a high frequency algorithmic trading technique must keep business clocks within 100 microseconds of UTC and timestamp to 1 microsecond or better, against 1 millisecond for other trading activity. A system that cannot demonstrate deterministic timing inside that envelope has an evidentiary problem, not only a performance one.
Data residency sits alongside both, since where model weights, feature stores and inference logs reside is now a question auditors ask directly, and one we examined in why cloud-only data sovereignty strategies fall short.
Start with the latency budget, not the hardware
The firms that handle this well begin with the same unglamorous exercise. They separate the real-time signal path from the language model workload, write down the end-to-end budget for a single decision on each, and only then ask what hardware and placement the remainder allows.
If your team is mapping that budget, or has traced the bottleneck to a layer you do not control, Arc Compute builds and operates dedicated GPU environments for financial services firms where placement, isolation and serving stack control are designed in rather than worked around.
Sources
- NVIDIA Developer Blog, Achieving Single-Digit Microsecond Latency Inference for Capital Markets (2 April 2026).
- NVIDIA, RTX PRO 6000 Blackwell Server Edition datasheet.
- NVIDIA, H200 Tensor Core GPU product page.
- CommScope, Latency in Optical Fiber Systems (white paper).
- Databento, Where is the CME matching engine located?.
- ClickHouse, LLM inference latency: TTFT, tokens per second, and what to measure.
- vLLM documentation, Optimization and Tuning.
- Commission Delegated Regulation (EU) 2017/574 (Regulatory Technical Standard 25), Annex.
- Emissions-EUETS.com, Time-stamping and business clocks synchronisation under MiFID II (Annex Table 2 values).
- European Insurance and Occupational Pensions Authority, Digital Operational Resilience Act (DORA).
- Hacker News, Cost of self hosting Llama-3 8B-Instruct (practitioner discussion).
- STAC Summit presentation, Barriers broken in ML inference latency (May 2026), audited STAC-ML Markets (Inference) FPGA results.
- Arc Compute, Cutting Costs and Latency in 4 Weeks: Inside a Trading Firm's GPU Upgrade.




