What if a server could use a PCIe device physically installed in another machine, while Linux and the device's native driver continued to treat it like a normal local device?
That is the question behind my PCIe-over-Network project.
The goal is not to build another remote GPU API. It is not to replace CUDA, the NVIDIA driver, NVMe, or the Linux PCI stack with a new application interface. The goal is to preserve the familiar PCIe programming model at the host boundary while moving the physical device somewhere else.
TCP/IP is the first transport and validation environment. It gives me a standard, widely available baseline to prove that the PCIe operations path is functionally correct. The next transport direction is RDMA, where the goal is to reduce data-movement overhead and latency without changing what the host sees above.
Why I Started This
Modern AI infrastructure still couples expensive devices tightly to individual servers. A GPU can be idle in one machine while another machine has CPU and memory capacity but no accelerator. An NVMe device can be stranded for the same reason.
This physical coupling turns hardware placement into a workload-placement constraint. We often have to upgrade compute and accelerator capacity together even when demand for them grows at different rates.
I wanted to ask a more basic architectural question: why should the location of a PCIe device be part of the application contract?
If the operating system and driver can continue to see the interface they already understand, then compute and devices can begin to evolve as separate resource pools. GPU, storage, and other PCIe functions become composable infrastructure rather than permanent properties of one server.
Just as importantly, the current experiment is a pure software approach on stock hardware. There is no proprietary PCIe switch in the path, no custom GPU, and no requirement for an application to speak a special remote-device protocol.
The Real Problem Is Not Enumeration
Making a device name appear in lspci is relatively easy. Making that device behave closely enough to a local PCIe function that an existing vendor driver can initialize and use it is the real problem.
A native PCIe driver expects much more than vendor ID and device ID. It expects configuration space, MMIO, interrupts, DMA mappings, ordering, reset, and error behavior to form one coherent contract.
Normally these operations live on a local hardware bus. A network introduces distance, packetization, variable latency, and new failure modes. So the architecture has to bridge two very different worlds without forcing the software above to understand that difference.
The conceptual path is simple:
Compute / guest VM
|
Linux PCI stack + native driver
|
PCIe virtualization / transport layer
|
TCP/IP today
RDMA next
|
Remote device service
|
Physical PCIe device
GPU / NVMe / other functions
The important invariant is the interface at the top. The transport underneath can evolve.
Evidence, Part One: The Operating System Sees the Devices
In the clean alpha validation on July 27, 2026, lspci in the compute-side VM showed the remote mixed-device topology. The following is the exact four-device excerpt from the validation log:
42:01.0 Display controller: Red Hat, Inc. Virtio 1.0 GPU (rev 01) Subsystem: Red Hat, Inc. Virtio 1.0 GPU Flags: bus master, fast devsel, latency 0, IRQ 11, NUMA node 1, IOMMU group 9 Memory at fa008000 (32-bit, non-prefetchable) [size=4K] Memory at e0202000000 (64-bit, prefetchable) [size=16K] Capabilities: [98] MSI-X: Enable- Count=3 Masked- Capabilities: [84] Vendor Specific Information: VirtIO: <unknown> Capabilities: [70] Vendor Specific Information: VirtIO: Notify Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg Capabilities: [50] Vendor Specific Information: VirtIO: ISR Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg Kernel driver in use: virtio-pci 42:02.0 VGA compatible controller: NVIDIA Corporation AD107 [GeForce RTX 4060] (rev a1) (prog-if 00 [VGA controller]) Subsystem: ZOTAC International (MCO) Ltd. AD107 [GeForce RTX 4060] Flags: bus master, fast devsel, latency 0, IRQ 10, NUMA node 1, IOMMU group 10 Memory at f9000000 (32-bit, non-prefetchable) [size=16M] Memory at e0000000000 (64-bit, prefetchable) [size=8G] Memory at e0200000000 (64-bit, prefetchable) [size=32M] I/O ports at c000 [size=128] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Legacy Endpoint, MSI 00 Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100] Virtual Channel Capabilities: [128] Power Budgeting <?> Capabilities: [420] Advanced Error Reporting Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Capabilities: [bb0] Physical Resizable BAR Kernel driver in use: nvidia Kernel modules: nvidiafb, nvidia_drm, nvidia 42:03.0 Non-Volatile memory controller: Red Hat, Inc. QEMU NVM Express Controller (rev 02) (prog-if 02 [NVM Express]) Subsystem: Red Hat, Inc. QEMU NVM Express Controller Flags: bus master, fast devsel, latency 0, IRQ 11, NUMA node 1, IOMMU group 11 Memory at fa000000 (64-bit, non-prefetchable) [size=16K] Capabilities: [40] MSI-X: Enable+ Count=65 Masked- Capabilities: [80] Express Root Complex Integrated Endpoint, MSI 00 Capabilities: [60] Power Management version 3 Kernel driver in use: nvme Kernel modules: nvme 42:04.0 Non-Volatile memory controller: Red Hat, Inc. QEMU NVM Express Controller (rev 02) (prog-if 02 [NVM Express]) Subsystem: Red Hat, Inc. QEMU NVM Express Controller Flags: bus master, fast devsel, latency 0, IRQ 10, NUMA node 1, IOMMU group 12 Memory at fa004000 (64-bit, non-prefetchable) [size=16K] Capabilities: [40] MSI-X: Enable+ Count=65 Masked- Capabilities: [80] Express Root Complex Integrated Endpoint, MSI 00 Capabilities: [60] Power Management version 3 Kernel driver in use: nvme Kernel modules: nvme
This matters because it demonstrates a single host-visible PCIe tree spanning more than one class of device. The experiment is not limited to a hard-coded GPU-only interface.
But lspci is only the first checkpoint. Enumeration proves that Linux can discover the functions and read their PCI identity. It does not, by itself, prove that a complex device has initialized or that its native software stack can operate it.
Evidence, Part Two: The Native NVIDIA Stack Reaches the GPU
The stronger evidence came from running stock nvidia-smi in the compute-side VM. It completed successfully and reported the remote physical GPU. This is the exact command output from the validation log:
Mon Jul 27 10:05:37 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 595.84 Driver Version: 595.84 CUDA Version: N/A | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 4060 Off | 00000000:42:02.0 Off | N/A | | 0% 36C P8 N/A / 115W | 0MiB / 8188MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+
The bus address matches the RTX 4060 enumerated by lspci. NVIDIA's unmodified management tool identified the GPU model, communicated through the stock driver, and reported the device's 8,188 MiB of memory.
That is why nvidia-smi is such an important progress marker. It exercises a far deeper path than PCI enumeration. The result shows that the system carried enough of the native PCIe behavior across TCP/IP for the vendor driver to bring up the physical GPU and answer a real management query.
lspci showed that the remote GPU could look local. nvidia-smi showed that the stock NVIDIA stack could actually talk to it.The clean validation also found no NVIDIA Xid, fallen-bus event, GPU firmware timeout, soft lockup, or kernel panic after the run. Three consecutive nvidia-smi queries completed successfully without resetting the environment between them.
The NVMe and virtio functions demonstrate that the topology is not designed as a GPU-only API. The NVIDIA result is currently the strongest evidence because it crosses the harder boundary from enumeration into real vendor-driver initialization and management.
Why This Matters
For me, the milestone changes the question from can a remote PCIe device appear? to how far can native PCIe behavior be preserved across a network?
If the model continues to mature, the infrastructure implication is larger than remote GPU access. It points toward a software-defined device fabric where compute, accelerators, and storage can be managed as independent pools and composed for a workload when needed.
This is also why preserving the native interface matters. The application should not need to know where the GPU is. Existing tools, drivers, and operating-system abstractions remain the integration point. The network becomes part of the infrastructure path instead of becoming part of the application API.
TCP Is the Beginning, Not the Architecture
There is an important distinction here. Moving from TCP to RDMA is not simply a faster socket replacement.
PCIe was designed around local latency and ordering assumptions. If every control operation becomes a network round trip, transport optimization alone will not solve the entire problem. The deeper engineering work is deciding which semantics must remain synchronous, which state can be maintained closer to the compute side, how DMA mappings persist, how interrupts travel, and how failure and reset are represented across a distributed system.
So the longer-term architecture may become more interesting than simple PCIe tunneling:
That is the direction I want to explore next. The interface above stays familiar. The execution model underneath can change dramatically.
What I Am — and Am Not — Claiming
This is an alpha functional milestone. It demonstrates successful PCI discovery, native driver binding, and a real NVIDIA management operation through the TCP/IP-backed PCIe path.
It does not yet establish production latency, throughput, CUDA workload performance, sustained DMA behavior, scale, long-duration stability, or recovery under every device and network failure.
The next phase is engineering rather than proving the basic idea: sustained workloads, performance measurement, reconnect and recovery behavior, failure injection, and an RDMA transport.
But one line has been crossed. A physical GPU on the other side of a network was no longer just a remote resource exposed through a special API. To Linux and the stock NVIDIA stack, it behaved locally enough to be discovered, initialized, and queried through the interfaces they already know.
That is the idea I find most interesting: change where the hardware lives without changing how software thinks about it.