DiFfRG::MapCompletion Class Reference

Deferred landing of QuadratureIntegrator::map() results in host memory. More...

#include <map_completion.hh>

Classes
struct	PendingCopy

Static Public Member Functions
static void	record (void *dst, const void *src, const size_t bytes)
	Register a device->host result that still has to be copied from staging into dst.
static void	record_work (std::function< void()> job)
	Queue a host-side map() to be run at flush time instead of now.
static void	flush ()
	Fence, land every pending map result, then exchange slices between MPI ranks.
static void	discard (const char *reason)
	Drop every pending copy without performing it, and abandon the MPI plan.
static bool	has_pending (const void *src)
	Whether src already has an unlanded result queued.
static bool	deferral_enabled ()
	Whether the caller has opened a DeferredMaps scope.
static void	set_deferral (const bool enabled)

Static Private Member Functions
static std::vector< PendingCopy > &	pending ()
static std::vector< std::function< void()> > &	deferred_work ()
static bool &	deferral ()

Detailed Description

Deferred landing of QuadratureIntegrator::map() results in host memory.

Why this exists. map() used to finish with

Kokkos::deep_copy(space, dest_view, dest_device_view);

where dest_view wraps caller memory — a dealii::Vector element range, i.e. ordinary pageable host memory. A device-to-pageable cudaMemcpyAsync is not actually asynchronous: the driver has to stage it, so the call blocks until the copy (and therefore the kernels feeding it) has completed. Measured on a YangMills solve: cudaMemcpyAsync totalled 15.933 s against 15.966 s of kernel time — the host sat inside that copy for essentially the entire GPU execution. The consequence is that there is zero host run-ahead: the GPU drains after every one of the ~3400 map() calls, and all host preparation for the next flow is exposed as GPU idle time. On a part where the kernels are slow (an RTX 4070) that costs a few percent; on one where they are ~6x faster (an A100) it is over half the wall clock.

The fix is to copy into a pinned staging buffer instead, which really is asynchronous, and to defer the final pinned->caller memcpy until someone needs the numbers. Integrators register that pending copy here; flush() fences once and lands all of them together.

Deferral is opt-in and scoped, via DeferredMaps. Outside such a scope map() keeps its original contract — the result is in dest when it returns — so every existing caller (tests, Examples, downstream models) is unaffected. That matters because the failure mode of a silent contract change here is not a compile error but wrong numbers.

Inside a DeferredMaps scope the caller promises not to read any dest until the scope ends (or it calls flush_maps() itself). Use it around a run of independent flows.

Not thread safe: the flow evaluation path is single-threaded (the parallelism lives inside the Kokkos kernels), and this registry is only touched from it.

Member Function Documentation

deferral()

static bool & DiFfRG::MapCompletion::deferral ( )

inlinestaticprivate

deferral_enabled()

static bool DiFfRG::MapCompletion::deferral_enabled ( )

inlinestatic

Whether the caller has opened a DeferredMaps scope.

deferred_work()

static std::vector< std::function< void()> > & DiFfRG::MapCompletion::deferred_work ( )

inlinestaticprivate

discard()

static void DiFfRG::MapCompletion::discard ( const char * reason )

inlinestatic

Drop every pending copy without performing it, and abandon the MPI plan.

Only for stack unwinding, where the destinations may already be gone.

flush()

static void DiFfRG::MapCompletion::flush ( )

inlinestatic

Fence, land every pending map result, then exchange slices between MPI ranks.

Always fences, even with nothing pending, so this is a drop-in replacement for the Kokkos::fence() it supersedes at the end of a residual evaluation.

The MPI step is driven by MapScheduler's plan, not by the local pending list. A rank that owns no slice of a given map records no pending copy, so a pending-list-driven decision would have some ranks enter the collective and others skip it – a silent hang. See MapScheduler.

has_pending()

static bool DiFfRG::MapCompletion::has_pending ( const void * src )

inlinestatic

Whether src already has an unlanded result queued.

An integrator reuses one staging buffer, so a second map() from the same integrator before a flush would overwrite the first result. The integrator asks this and flushes first.

pending()

static std::vector< PendingCopy > & DiFfRG::MapCompletion::pending ( )

inlinestaticprivate

record()

static void DiFfRG::MapCompletion::record ( void * dst, const void * src, const size_t bytes )

inlinestatic

Register a device->host result that still has to be copied from staging into dst.

record_work()

static void DiFfRG::MapCompletion::record_work ( std::function< void()> job )

inlinestatic

Queue a host-side map() to be run at flush time instead of now.

Host kernels are synchronous – tbb::parallel_for and the host Kokkos backends all return only once the work is done. Running one inside a DeferredMaps scope therefore blocks the host thread, and with it the launch of every device map issued after it. The device itself keeps working on whatever is already in flight, so the cost is not the host map's own time but the device idling once its queue drains.

Deferring removes the trap. flush() runs this queue before it fences, so the host work happens while the device work already issued is still running, and the caller no longer has to know that interleaving GPU, CPU, GPU is slower than GPU, GPU, CPU.

The job owns copies of everything it needs (std::function requires a copy-constructible target, which every kernel argument is), so it does not depend on the caller's locals still being alive – only on dest and the integrator, which the DeferredMaps contract already requires to outlive the scope.

set_deferral()

static void DiFfRG::MapCompletion::set_deferral ( const bool enabled )

inlinestatic

---

The documentation for this class was generated from the following file:

• /home/runner/work/DiFfRG_current/DiFfRG_current/DiFfRG/include/DiFfRG/physics/integration/map_completion.hh