Value domains

The element type of a tensor has two parts: a dtype, how values are stored, and a domain, which values are allowed. Both are optional in a type. The domain defaults to Real, and the dtype defaults to the one the domain implies: bool for Mask, i64 for Index, and the module default float, f32, otherwise.

Built-in domains

over C must name one of the type’s axes, and only Logit, Prob and LogProb take it (E0503). Only Index takes a size in brackets. A combination of domain and dtype that makes no sense, such as Mask f32, Index[V] f32, Prob over C i64 or Unit bool, is E0503.

Values are assumed to be finite: NaN isn’t tracked, and an infinite end of a range is open, so -inf from masking leaves Real values Real.

Messages show a range by name when it has one (NonNeg, Pos, Unit, Signed, or a user domain), not at all for Real, and otherwise as an interval: [b C] [0, 2]. A dtype is shown only when it isn’t the one the domain implies: [b 3 h w] u8.

Widening

A value fits where a domain is expected when every value it may hold is allowed there. There is no conversion and no run-time cost.

• A range fits a range that contains it: Unit fits NonNeg, Signed, Real and any user domain that contains $[0, 1]$.
• Any numbers fit Logit over C: plain Real is implicitly a logit, because a linear layer’s output really is one. LogProb over C fits Logit over C too. Prob over C doesn’t.
• A semantic domain is matched by position: Prob over C fits Prob over K when C and K are the same axis position in the two types. Along any other axis it is just its range, so it fits Unit or NonNeg.
• Mask, Binary and Additive fit only themselves, and Index[V] fits only Index[V] with an equal V (E0507).

Domains are checked at calls (shapeless parameters too), at returns, and in the built-ins that need one. A value that doesn’t fit is E0501; its help suggests to T when the to table has a conversion, and as T otherwise.

Range inference

The checker tracks a closed or open interval for every numeric value, through elementwise arithmetic, activations, clamp and comparisons, so most domains are inferred: sigmoid(x) is Unit and sigmoid(x) * 2 - 1 is Signed. The rules:

• + - * / by the corners of the operands’ ranges; neg, abs, relu, clamp, maximum and minimum exactly.
• exp, expm1, log, log1p, sqrt, rsqrt, sigmoid, tanh, softplus and log_sigmoid at the ends of the range, with exact anchors such as exp(0) = 1; erf, gelu and silu by sign and bounds; sin and cos give $[-1, 1]$; floor, ceil and round give whole ends; pow only for exponents 1 and 2 or a non-negative base.
• sum, prod, cumsum and cumprod keep only the sign; max and min keep the range; mean keeps ends of -1, 0 and 1 and widens others slightly; std, var and norm give NonNeg; logsumexp is at least the lower end.
• @, einsum, layer_norm, convolution and anything else not listed give Real.
• A partial function (log, log1p, sqrt, rsqrt, the divisor of /) of values not proven to be inside its domain gives Real.

Ends are rounded outward to f32, so the ranges hold for the values torch computes. Ranges are approximate: x - x on Signed values is $[-2, 2]$, not 0. When the checker can’t prove a range, use as.

A call to an elementwise function gets its range from the function’s body, and a call to an extern fn whose Python function’s math the checker knows gets its range from that math (The prelude).

Semantic domains

softmax(z, over=C) gives Prob over C and log_softmax gives LogProb over C. Both take logits, so softmax of a Prob is E0501 (“softmax applied twice?”), and passing a Prob where a Logit is wanted says “softmax applied before cross_entropy?”.

Logits shifted or scaled by plain numbers stay logits, and so do negated logits, where, masked_fill, and reductions over another axis. Prob and LogProb become their ranges under arithmetic. mean over another axis keeps Prob over C, since a mixture of distributions is one. Removing the axis C (reducing it, indexing it away, grouping it in a pattern) leaves just the range.

Discrete domains

Mask comes from comparisons, !, and, or, any, all and causal. Where a mask is needed (!, and, or, the condition of where, the mask of masked_fill), anything else is E0501. Arithmetic on a mask is E0505; its help suggests to Additive for +, to Binary for the other operators, ! for -m, and and or or for two masks.

Binary times Binary, their maximum and minimum, and abs, relu and rounding of one stay Binary; other arithmetic gives $[0, 1]$. Additive plus Additive stays Additive.

Index[V] connects values to shapes: emb[ids] with emb: [V D] and ids: [b t] Index[V] is checked, so token ids from a different vocabulary are E0507. argmax, argmin, argsort and topk give indices. Plain integer indices are still allowed, unchecked.

Function edges

For log, log1p, rsqrt and the divisor of / and div:

• a range that only touches the edge, such as log of $[0, \cdot]$, or a divisor whose range contains 0 and isn’t all of Real, is the warning W0501;
• a range with a finite lower end that reaches past the edge (log of Signed), a range wholly outside, or a divisor that is exactly 0 is the error E0506.

sqrt only has the error, since sqrt(0) is fine. A range unbounded on that side, as for Real, gives nothing, because nothing is known. log of a sigmoid suggests log_sigmoid of its input, and log of a Prob suggests log_softmax.

User domains

domain Temp = [0.05, 10] declares a range domain. [ and ] include an end, ( and ) exclude it. The ends must be numbers, possibly negative, or inf (E0503), the range can’t be empty, and the name can’t be a built-in domain or a dtype (E0503). Each end covers the number as written and its nearest f32. A user domain works wherever a built-in range does, and is shown by name.