`simvx.graphics.gpu.multi_device`¶

Explicit multi-adapter (multi-GPU) foundation: workload-split offload.

The gated plumbing for an explicit-multi-adapter renderer (one independent VkDevice per physical GPU) where whole SubViewport / offscreen scene-render-units (SRUs) are offloaded to secondary GPUs and composited on the primary (GPU 0). It is deliberately off by default: a single-device system, and any multi-device system that does not opt in, run the single-device path unchanged (no extra device, no transfer work, no behaviour change). The active multi-device path requires a system exposing more than one Vulkan physical device, and is not exercised by the single-device test suite.

Three coherent, unit-testable pieces live here:

class:

MultiDeviceManager enumerates physical devices and, only when physical_device_count > 1 and the caller opted in, creates an independent logical device per physical GPU (reusing

func:

~simvx.graphics.gpu.device.create_logical_device). When the count is 1 or the opt-in is off it holds exactly the existing single (primary) device and is a transparent passthrough.
func:

assign_srus is the pure device-assignment policy (no Vulkan): given the ordered SRUs and a device count it decides which render on GPU 0 vs which offload to GPU 1+. TAA-safe by construction (each SRU keeps its temporal history on its one assigned device, so there is no cross-device reprojection).
class:

CrossDeviceTransfer selects how a finished offscreen colour image moves from a secondary device to the primary for compositing. The staging-copy path (secondary -> host-visible staging -> primary) is the guaranteed floor and is the only path implemented end-to-end; the dma-buf / VK_KHR_external_memory_fd zero-copy path is an optimisation for device pairs that both expose the extension; it is gated behind a capability and currently raises a clear, actionable error.

Scope and current limits. Rendering an SRU on a second VkDevice requires that device to own its full set of rendering resources: its own pipelines, descriptor pools, transform/material SSBOs, and mesh+texture residency. The

class:: ~simvx.graphics.renderer.forward.Renderer is built around exactly one device, so the multi-device path constructs a second one against a secondary facade rather than sharing it (see
class:: ~simvx.graphics.gpu.secondary_engine.SecondaryRenderContext). Device management, the assignment policy and the transfer interface are covered by the unit tests; per-device renderer construction and the offload-record-and-composite loop need a system exposing two or more Vulkan devices and are not exercised by the single-device test suite (see :meth:MultiDeviceManager.attach_renderer /
attr:: DeviceSlot.renderer).

Module Contents¶

Classes¶

`DeviceSlot`	One physical+logical GPU participating in the multi-device renderer.
`MultiDeviceManager`	Owns the per-physical-GPU logical devices for the offload renderer.
`SRUAssignment`	Which device renders one SRU and whether its result must be transferred.
`TransferMethod`	Selectable cross-device image-transfer strategies (enum-like constants).
`CrossDeviceTransfer`	Moves a finished offscreen colour image from a secondary to the primary.
`OffloadRoute`	The per-SRU offload decision the recording path consults.
`SRUOffloadCoordinator`	Decides + drives where each SubViewport SRU renders across the devices.

Functions¶

`assign_srus`	Decide which device renders each SRU (pure, unit-testable, no Vulkan).
`select_transfer_method`	Pick the transfer strategy for one SRU assignment (pure, no Vulkan).

Data¶

`log`
`__all__`

API¶

simvx.graphics.gpu.multi_device.log¶: ‘getLogger(…)’

simvx.graphics.gpu.multi_device.__all__¶: [‘DeviceSlot’, ‘MultiDeviceManager’, ‘SRUAssignment’, ‘assign_srus’, ‘CrossDeviceTransfer’, ‘Transfe…

class simvx.graphics.gpu.multi_device.DeviceSlot¶

One physical+logical GPU participating in the multi-device renderer.

index 0 is always the primary (compositing) GPU; 1+ are secondaries that render offloaded SRUs. On a single-GPU run there is exactly one slot (index == 0) wrapping the engine’s existing device, and nothing else is created.

renderer is the per-device :class:~simvx.graphics.renderer.forward.Renderer duplicate. It is populated for the primary slot from the engine’s existing renderer and, on a multi-device run, for each secondary by

Meth:: MultiDeviceManager.attach_renderer. It stays None for secondaries until per-device renderer construction runs (see the module docstring): this layer never fabricates a non-functional renderer to look complete.

index: int¶: None

physical_device: Any¶: None

queue_families: simvx.graphics.gpu.device.QueueFamilies¶: None

device: Any¶: None

graphics_queue: Any¶: None

present_queue: Any¶: None

compute_queue: Any¶: None

transfer_queue: Any¶: None

name: str = <Multiline-String>¶

renderer: Any¶: None

property is_primary: bool¶

class simvx.graphics.gpu.multi_device.MultiDeviceManager(*, primary_physical_device: Any, primary_queue_families: simvx.graphics.gpu.device.QueueFamilies, primary_device: Any, primary_graphics_queue: Any, primary_present_queue: Any, physical_devices: list[Any], enabled: bool, capabilities: simvx.graphics.gpu.capabilities.RenderCapabilities | None = None, primary_compute_queue: Any = None, primary_transfer_queue: Any = None, find_queue_families: Any = None)¶

Owns the per-physical-GPU logical devices for the offload renderer.

Construct with the primary device’s already-created handles (the engine’s existing single device, so the primary slot is never re-created) plus the enumerated physical devices and the opt-in flag. When enabled is true and more than one physical device is present, a secondary :class:DeviceSlot is created per additional physical GPU with its own independent VkDevice via

Func:: create_logical_device. Otherwise the manager holds exactly the single primary slot and :attr:multi_gpu is False (the single-device path, unchanged).

The manager does NOT own the primary device’s lifetime (the engine created and destroys it); :meth:destroy only tears down the secondary devices it created itself.

Initialization

property multi_gpu: bool¶: True only when an opted-in multi-device renderer is active (>= 2 slots).

property device_count: int¶: Number of logical devices the manager owns (1 on the single-device path).

property primary: simvx.graphics.gpu.multi_device.DeviceSlot¶

property secondaries: list[simvx.graphics.gpu.multi_device.DeviceSlot]¶

property slots: list[simvx.graphics.gpu.multi_device.DeviceSlot]¶

slot(index: int) → simvx.graphics.gpu.multi_device.DeviceSlot¶

attach_renderer(index: int, renderer: Any) → None¶

Bind a per-device :class:Renderer to slot index.

The primary renderer is the engine’s existing one. Each secondary needs its OWN renderer (its device’s pipelines / descriptor pools / SSBOs / residency). This setter is the seam through which a caller hands the constructed per-device renderer back to the manager so the offload loop can record into it.

register_teardown(hook: Any) → None¶

The offload coordinator registers its :meth:SRUOffloadCoordinator.destroy here so its secondary facades / targets / staging buffers are freed while the secondary VkDevice\ s are still alive (they own those resources). Invoked first by :meth:destroy.

destroy() → None¶

Destroy only the SECONDARY logical devices this manager created.

The primary device is owned by the engine and left untouched. Safe to call on the single-GPU path (no secondaries => no-op). Registered teardown hooks run FIRST so per-secondary GPU resources are freed before the devices.

class simvx.graphics.gpu.multi_device.SRUAssignment¶

Which device renders one SRU and whether its result must be transferred.

device_index 0 means the SRU renders on the primary and is already resident for compositing (needs_transfer is False). A non-zero index means the SRU is offloaded to that secondary and its finished colour image must be moved to the primary before compositing (needs_transfer is True).

sru_id: int¶: None

device_index: int¶: None

property needs_transfer: bool¶

simvx.graphics.gpu.multi_device.assign_srus(srus: list[Any], device_count: int, *, sru_id: Any = None, cost: Any = None) → list[simvx.graphics.gpu.multi_device.SRUAssignment]¶

Decide which device renders each SRU (pure, unit-testable, no Vulkan).

Policy (TAA-safe by construction: an SRU is assigned to exactly one device, so its temporal history never crosses devices):

device_count <= 1: EVERY SRU stays on GPU 0. This is the single-device / unopted path and produces exactly the per-SRU work of a single-device frame (no transfer is ever flagged).
device_count >= 2: the main scene is implicitly GPU 0 (it is not an SRU and is not in this list). Among the SRUs, the cheapest stay on GPU 0 (composited locally) and the heaviest independent SRUs are offloaded to the secondaries round-robin. Concretely: sort SRUs by descending cost and walk them, sending the next heaviest to the least-loaded secondary while that keeps the primary from being the bottleneck; ties and the remainder stay on GPU 0.

The returned list preserves the INPUT order of srus (the P1 producer-before-consumer topological order), so a consumer SRU still follows the producer it samples; only the device choice is decided here.

Args: srus: Ordered SRU plans (SubViewportSRU or any object exposing the cost inputs). Order is preserved in the result. device_count: Number of devices available (MultiDeviceManager.device_count). sru_id: Optional accessor sru -> int for the stable id; defaults to reading sru.sru_id. cost: Optional accessor sru -> int overriding the default instance-count heuristic (handy for tests).

class simvx.graphics.gpu.multi_device.TransferMethod¶

Selectable cross-device image-transfer strategies (enum-like constants).

NONE¶

‘none’

SRU already lives on the primary device: compositing samples it directly.

STAGING_COPY¶

‘staging_copy’

Secondary image -> host-visible staging buffer -> primary image. The guaranteed floor, works on any pair of devices (the colour RenderTarget already carries TRANSFER_SRC|TRANSFER_DST).

DMABUF¶

‘dmabuf’

Zero-copy via VK_KHR_external_memory_fd (dma-buf import/export). Requires the external-memory extensions to be enabled on both devices. Not implemented: selecting it raises, and :func:select_transfer_method never returns it.

simvx.graphics.gpu.multi_device.select_transfer_method(assignment: simvx.graphics.gpu.multi_device.SRUAssignment, capabilities: simvx.graphics.gpu.capabilities.RenderCapabilities | None, *, prefer_dmabuf: bool = True) → str¶

Pick the transfer strategy for one SRU assignment (pure, no Vulkan).

An SRU on the primary needs no transfer -> :data:TransferMethod.NONE.
An offloaded SRU uses :data:TransferMethod.DMABUF when the caller prefers it AND the capability snapshot reports the external-memory-fd path enabled; otherwise the always-available :data:TransferMethod.STAGING_COPY.

The capability gate is :attr:RenderCapabilities.external_memory_fd_enabled: the extension must be enabled at device creation, not merely probed available. A probed-but-not-enabled device cannot export/import fds, so DMABUF must not be selected for it (that would later raise with no working path). On a single-device run the extension is never enabled, so the field is False, the staging-copy floor is always chosen, and the dma-buf raise is never reached.

class simvx.graphics.gpu.multi_device.CrossDeviceTransfer¶

Moves a finished offscreen colour image from a secondary to the primary.

Holds the source (secondary) and destination (primary) :class:DeviceSlot plus the chosen :class:TransferMethod. Cross-device synchronisation is explicit: each device signals a fence/timeline when its half of the copy is done, and the primary’s composite waits on the import being complete. The sync handles are carried on this object so cross-device wiring has a single seam.

Implemented:

Meth:: run_staging_copy records the staging-copy sequence (secondary GPU copy-to-buffer -> host-visible staging -> primary copy-to-image). The CPU-roundtrip floor: correct on any device pair, slower than dma-buf. It needs both devices’ live command pools and a host-visible staging allocation per device, so it runs only on a system exposing two or more Vulkan devices and is not exercised by the single-device test suite.

Not implemented:

Meth:: run_dmabuf raises a clear, actionable error until VK_KHR_external_memory_fd is enabled on both devices and the import/export is wired.

src: simvx.graphics.gpu.multi_device.DeviceSlot¶: None

dst: simvx.graphics.gpu.multi_device.DeviceSlot¶: None

method: str¶: None

src_done: Any¶: None

dst_ready: Any¶: None

width: int¶: 0

height: int¶: 0

bytes_per_pixel: int¶: 8

extra: dict¶: ‘field(…)’

run(*, src_image: Any = None, dst_image: Any = None) → None¶

Execute the selected transfer. Dispatches by :attr:method.

Data:: TransferMethod.NONE is a no-op (single-device / primary-resident SRU). The other two dispatch to their handlers.

run_staging_copy(*, src_image: Any = None, dst_image: Any = None) → None¶

Staging-copy floor: secondary image -> host staging -> primary image.

The CPU-roundtrip floor: correct on any device pair, slower than dma-buf. src_image is the finished SRU colour image on the SECONDARY device (left in SHADER_READ_ONLY_OPTIMAL by the offscreen pass; the RenderTarget carries TRANSFER_SRC_BIT). dst_image is the primary-device image the main scene samples for this SubViewport feed (carries TRANSFER_DST_BIT).

Exact sequence and every wait, recorded here as the contract this method implements. All staging buffers are host-visible|host-coherent, built once per (transfer, size) into :attr:extra by :meth:ensure_staging:

SOURCE (secondary) device, one-shot command buffer on the secondary command pool:

a. barrier src_image: SHADER_READ_ONLY_OPTIMAL -> TRANSFER_SRC_OPTIMAL (src stage FRAGMENT_SHADER, dst stage TRANSFER; src access SHADER_READ, dst access TRANSFER_READ). b. vkCmdCopyImageToBuffer src_image -> src_staging (tightly packed, bufferRowLength=0, bufferImageHeight=0). c. barrier src_image: TRANSFER_SRC_OPTIMAL -> SHADER_READ_ONLY_OPTIMAL so the secondary can render into / sample it again next frame. d. submit on src.graphics_queue signalling :attr:src_done (a fence on the secondary device). WAIT: the host read in step 2 blocks on src_done (vkWaitForFences); the GPU copy must complete before the bytes are mapped. The offscreen SRU render fence is itself waited on before this submit by the caller (_record_offloaded_sru), so the colour image is fully written first.
HOST: map src_staging (secondary device), memmove the height * row_bytes bytes into dst_staging (primary device), unmap both. Both are host-coherent so no explicit flush/invalidate is needed. WAIT: gated on src_done from 1d before the read; the write into dst_staging happens-before the primary GPU read in 3 because step 3 is submitted only after this memmove returns on the same (host) thread.
DESTINATION (primary) device, one-shot command buffer on the primary command pool:

a. barrier dst_image: SHADER_READ_ONLY_OPTIMAL -> TRANSFER_DST_OPTIMAL. b. vkCmdCopyBufferToImage dst_staging -> dst_image. c. barrier dst_image: TRANSFER_DST_OPTIMAL -> SHADER_READ_ONLY_OPTIMAL so the main composite pass samples it. d. submit on dst.graphics_queue signalling :attr:dst_ready (a fence on the primary device). WAIT: the main composite pass that samples dst_image runs after dst_ready; the caller waits on it before recording the frame’s main pass (or, in the same-frame in-cmd model, this transfer is submitted + waited before the main pass is recorded).

The implementation records exactly this. It needs both devices’ command pools + a host-visible staging buffer per device, supplied via

Meth:: ensure_staging. On a single-device run the method is NONE and this is never reached; it executes only on a system with two or more Vulkan devices, which the test suite does not cover.

ensure_staging(src_command_pool: Any, dst_command_pool: Any) → dict¶

Allocate (once) the per-device host-visible staging buffers + cache pools.

Builds one host-visible|host-coherent buffer on each device, sized width * height * bytes_per_pixel (the raw colour image bytes). Cached on

Attr:: extra ['staging'] so a steady-state per-frame transfer reuses them; reallocated only when the size changes (a SubViewport resize). Returns the staging dict. Allocates on both devices, so it is reached only on a multi-device run.

destroy() → None¶: Free the per-device staging buffers this transfer allocated.

abstractmethod run_dmabuf(*, src_image: Any = None, dst_image: Any = None) → None¶

Zero-copy dma-buf transfer (VK_KHR_external_memory_fd). Not implemented.

Export the secondary colour image’s memory as an opaque fd / dma-buf, import it on the primary as an external image, and composite directly with a single cross-device semaphore wait (no host roundtrip). Requires the external-memory-fd extension enabled on BOTH devices at init.

Raises until that path is implemented; :data:TransferMethod.STAGING_COPY is used instead.

class simvx.graphics.gpu.multi_device.OffloadRoute¶

The per-SRU offload decision the recording path consults.

Computed once per frame by :class:SRUOffloadCoordinator.plan from the ordered SRU list. device_index 0 means “render this SRU on the primary” (offloaded is False, transfer is

Data:: TransferMethod.NONE); a non-zero index means the SRU renders on that secondary’s renderer and its colour image is moved to the primary via
Attr:: transfer before the main pass samples it.

sru_id: int¶: None

device_index: int¶: None

transfer: str¶: None

property offloaded: bool¶: True when this SRU renders on a secondary device (not GPU 0).

class simvx.graphics.gpu.multi_device.SRUOffloadCoordinator(manager: simvx.graphics.gpu.multi_device.MultiDeviceManager, capabilities: simvx.graphics.gpu.capabilities.RenderCapabilities | None = None, *, prefer_dmabuf: bool = True, content_scale: tuple[float, float] = (1.0, 1.0), secondary_renderer_factory: Any = None)¶

Decides + drives where each SubViewport SRU renders across the devices.

Glue between the (already-tested) :func:assign_srus policy, the

Func:: select_transfer_method selector, and the :class:MultiDeviceManager device topology. The recording path (SceneAdapter.render_sru_from_plan / render_to_target) consults a coordinator, when one is present, to decide per SRU whether to take the primary-device path or route the SRU to a secondary device and transfer the result back.

Constructed only when the manager is actively multi-GPU (opted in AND >= 2 devices). On the single-device / unopted path the engine builds no coordinator, so the recording path’s coordinator is None branch is taken and the frame is unchanged.

The decision logic (plan / route_for) is pure and unit-tested with no Vulkan. :meth:render_offloaded is the seam where a secondary-assigned SRU is recorded on its device’s renderer and transferred back; that needs a per-device renderer (DeviceSlot.renderer), which is None until one is attached or a factory is supplied, so it raises a clear, capability-gated error rather than silently dropping the SRU.

Initialization

property active: bool¶: True only when the backing manager is an opted-in multi-device renderer.

plan(srus: list[Any], *, sru_id: Any = None, cost: Any = None) → list[simvx.graphics.gpu.multi_device.OffloadRoute]¶

Compute + cache this frame’s per-SRU routes from the ordered SRU list.

Returns one :class:OffloadRoute per SRU in INPUT order (the P1 producer-before-consumer topological order is preserved). When the coordinator is inactive (single-device / unopted) every route stays on GPU 0 with :data:TransferMethod.NONE, so the caller takes the primary path.

sru_id / cost are forwarded to :func:assign_srus so a caller whose ordered items are not SubViewportSRU (e.g. the synchronous path’s live nodes) can supply the id + offload-cost accessors.

route_for(sru_id: int) → simvx.graphics.gpu.multi_device.OffloadRoute | None¶

The cached route for sru_id from the last :meth:plan, or None.

None means “no decision recorded” (the SRU was not in the planned list); callers treat that as “render on the primary”.

transfer_for(route: simvx.graphics.gpu.multi_device.OffloadRoute, *, width: int = 0, height: int = 0) → simvx.graphics.gpu.multi_device.CrossDeviceTransfer¶

Build the :class:CrossDeviceTransfer for an offloaded route.

Pairs the secondary (route.device_index) and primary (0) device slots with the chosen transfer method so cross-device wiring has a single seam.

render_offloaded(route: simvx.graphics.gpu.multi_device.OffloadRoute) → Any¶

Return the per-device renderer for an offloaded SRU, building it if needed.

Resolution order:

An explicitly attached renderer (MultiDeviceManager.attach_renderer) is returned as-is (a caller may pre-build and bind one; also the unit-test seam).
Otherwise, if a :attr:_secondary_renderer_factory was injected, lazily build the secondary :class:SecondaryRenderContext facade, run the factory to construct + setup() a Renderer(facade) on the secondary device, bind it to the slot, and return it.
With neither (a single-device system: no factory is injected because no secondary device exists), raise the clear, capability-gated error. NEVER reached on the single-device / unopted path: that path builds no coordinator at all, so the seam’s coordinator is None branch keeps it unchanged.

render_sru_offloaded(sru: Any, primary_dst_image: Any) → bool¶

Render one SRU on its secondary device and transfer it to the primary image.

End-to-end multi-GPU SubViewport offload for one SRU (the

Class:: ~simvx.graphics.renderer.render_packet.SubViewportSRU plan):

Resolve the SRU’s route (must be offloaded; else returns False so the caller takes the primary path).
Ensure the secondary render context (facade + Renderer + residency).
Mirror the SRU’s mesh geometry (and, when textured-residency is wired, its sampled textures) onto the secondary device via :class:SecondaryResidency.
Size / create the secondary offscreen :class:RenderTarget for this SRU.
Record + submit the SRU’s draws into that target on the secondary device (one submit on the secondary graphics queue; the offscreen RenderTarget leaves the colour image in SHADER_READ_ONLY_OPTIMAL).
Run the :class:CrossDeviceTransfer (staging-copy floor) to move the secondary colour image into primary_dst_image (the SubViewport’s primary-device bindless image the main scene samples).

Returns True when the SRU was handled on a secondary, False when it was not offloaded (caller renders it on the primary). The actual per-device GPU record + submit (step 5) is delegated to the injected secondary renderer; this method owns the orchestration + the cross-device transfer. Runs only on a system with two or more Vulkan devices; never reached on the single-device path, and not covered by the test suite.

destroy() → None¶: Tear down all lazily-built secondary render contexts (facade + target + transfer).

simvx.graphics.gpu.multi_device¶