NoMapsHere Class Reference#

DiFfRG: DiFfRG::NoMapsHere Class Reference

DiFfRG

Discretization Framework for functional Renormalization Group flows

Decides, without any user input, which rank computes which part of each map(). More...

#include <map_scheduler.hh>

Public Member Functions
	NoMapsHere ()

	~NoMapsHere ()

	NoMapsHere (const NoMapsHere &)=delete

NoMapsHere &	operator= (const NoMapsHere &)=delete

Static Public Member Functions
static bool	active ()
	Whether any NoMapsHere scope is currently open.

Detailed Description

Decides, without any user input, which rank computes which part of each map().

What is split

Only the external coordinate grid of a map() – never a quadrature reduction. Every individual integral is therefore still summed entirely on one rank, in the original order, so the result is bitwise identical to a serial run at any rank count. That is not a nicety: the flow kernels feed a stiff DAE (see common/tbb.hh), where a last-bit change in a residual changes the accepted step sequence and hence the whole trajectory. It also makes rebalancing numerically free – moving a partition boundary changes which rank computes a point, never what it computes.

Splitting applies to grids of any dimension. SubCoordinates is a window into the linear index range, matching part() above, so the split needs no per-axis structure; it used to derive a per-axis box, which is the same set of points only for dim == 1, and multi-dimensional grids were therefore assigned whole to a single rank. That mattered: the multi-angle vertex flows are both the most expensive and the ones with multi-dimensional external grids.

How the split is chosen

Per call, with G external grid points and a quadrature volume Q = prod(grid_size):

S = G * Q                                   (kernel evaluations in this map)
r = clamp(S / quantum, 1, min(n_ranks, G))  (how many ranks to spread it over)
owners = the r least-loaded ranks

Inside a batch, quantum is the fill threshold, not a launch-overhead threshold: below roughly 5e4 evaluations these register-heavy kernels do not occupy a modern device anyway, so splitting further buys nothing and costs a launch plus a gather. Deriving it from launch cost (~1e3 evaluations) would oversplit by two orders of magnitude.

The practical consequence is the one that matters for load balance: a cheap flow is assigned whole to the least-loaded rank rather than chopped into slivers, so a block of many small flows spreads across ranks instead of every rank doing a sliver of every flow.

That reasoning is an argument about opportunity cost, and it holds only while there are other maps to take the ranks this one leaves out. Outside a DeferredMaps scope there are none: map() flushes on return, so the map is the whole batch and every rank without a slice sits in the Allgatherv until the owners finish. There the fill threshold gives away ranks for nothing, and the threshold that applies is instead internal::launch_threshold – a slice must still cover the launch that computes it, and that is the only remaining constraint. set_batched() is how the scheduler learns which case it is in. An explicit user override wins over both.

Mixed device and host models

A rank holds two independent resources – its GPU and its share of the node's cores – and both r and the ownership choice are made against the one the calling integrator actually uses. r follows from that space's fill threshold (a device is saturated by its resident threads, a host by workers x a grain), and the slices are charged to a separate per-resource budget, so a rank that has just taken a large TBB flow is still the natural home for the next GPU flow.

That is what makes a model mixing GPU_exec and TBB_exec integrators use both at once rather than merely correctly: within a DeferredMaps scope the device path launches asynchronously and returns, so a host map issued after it runs concurrently with it, and each resource is levelled across ranks on its own. What the budgets deliberately do not do is convert between the two – there is no exchange rate at which a device evaluation equals a host one, and assuming one is exactly the mistake a single shared budget makes.

Two limits remain, and both are outside this class. Within one resource the weight per evaluation is 1, so flows with equal S but different per-evaluation cost still mis-weigh (calibration, plan stage 4). And the overlap only exists inside a deferral scope: a host map issued with deferral off flushes – and therefore synchronises every rank – before the device work behind it can catch up.

Determinism

Every rank runs the same program and issues the same sequence of map() calls, so a schedule that is a pure function of that sequence is identical on all ranks without any communication. The running per-rank load is part of that state and is reset by complete(). A plan checksum is verified inside complete(); a mismatch aborts the job rather than hanging it.

A scope in which map() must not be called.

map() is a collective: every rank must issue the same sequence of them. Distributed FE assembly breaks that guarantee, because each rank visits only its own cells – so a map() from inside a cell worker is issued a different number of times on different ranks, and the run hangs in the next Allgatherv with no indication of which model did it.

The rule "no map() inside FE assembly" already existed as documentation. This makes it enforceable: the assemblers open one of these around every mesh loop, and a map() inside aborts the job with a message naming the problem instead of hanging.

Deliberately a process-wide counter rather than thread_local: assembly runs cell workers on TBB worker threads, and a thread_local flag set on the main thread would be invisible to exactly the threads that need checking.

Constructor & Destructor Documentation

◆ NoMapsHere() [1/2]

DiFfRG::NoMapsHere::NoMapsHere ( )

◆ ~NoMapsHere()

DiFfRG::NoMapsHere::~NoMapsHere ( )

◆ NoMapsHere() [2/2]

DiFfRG::NoMapsHere::NoMapsHere ( const NoMapsHere & )

delete

Member Function Documentation

◆ active()

static bool DiFfRG::NoMapsHere::active ( )

static

Whether any NoMapsHere scope is currently open.

◆ operator=()

NoMapsHere & DiFfRG::NoMapsHere::operator= ( const NoMapsHere & )