11: mut and gradients

E1101

mut where it isn’t allowed. A parameter that isn’t a model or optimizer state, or a parameter of an extern fn, is declared mut; mut marks an argument whose parameter isn’t mut, an argument of a built-in or of Mlp.init, or the gradient of update; or what follows mut isn’t a name.

dim D = 64

fn double(mut x: [D]) -> [D] = x * 2

See mut parameters.

E1102

Change to something not passed as mut. A function assigns a field of, or passes as mut, a model or optimizer state it may not change: a parameter without mut, a value that may be one, or a Python variable.

dim D = 64

model Mlp:
    param w: [D D] = zeros

fn reset(m: Mlp):
    m.w = m.w * 0

See What counts as a change.

E1103

Argument for a mut parameter without mut. The function changes this parameter, so every call writes mut before its argument, as in reset(mut m). update changes its model and optimizer state, so it’s written update(mut m, mut opt, g).

dim D = 64

model Mlp:
    param w: [D D] = zeros

fn reset(mut m: Mlp):
    m.w = m.w * 0

fn f(mut m: Mlp):
    reset(m)

See mut parameters.

E1104

New value for a mut parameter. A mut parameter is assigned a new model or new optimizer state, or used as a loop variable. The caller would still have what it passed, so change the parameter’s fields, or update the state with update, instead.

dim D = 64

model Mlp:
    param w: [D D] = zeros

fn reset(mut m: Mlp):
    m = Mlp.init(seed=0)

See What counts as a change.

E1105

Function that can’t be differentiated. The first argument of value_and_grad isn’t the name of a fn, or the fn’s first parameter isn’t a model, or it has a mut parameter.

dim D = 64

model Mlp:
    param w: [D D] = zeros

fn loss(x: [b D], m: Mlp) -> [] = sum(x @ m.w)

fn f(m: Mlp, x: [b D]) -> []:
    l, g = value_and_grad(loss, x, m)
    return l

See value_and_grad.

E1106

Gradient of something that isn’t a []. value_and_grad differentiates a function that returns a single float, such as a loss. Reduce a tensor to one number first, with mean or sum.

dim D = 64

model Mlp:
    param w: [D D] = zeros

fn out(m: Mlp, x: [b D]) -> [b D] = x @ m.w

fn f(m: Mlp, x: [b D]) -> [b D]:
    l, g = value_and_grad(out, m, x)
    return l

See value_and_grad.

E1107

Gradient of a function that doesn’t read the model. The differentiated function reads no param of its model, so the gradient would always be zero.

dim D = 64

model Mlp:
    param w: [D D] = zeros

fn loss(m: Mlp, x: [b D]) -> [] = sum(x)

fn f(m: Mlp, x: [b D]) -> []:
    l, g = value_and_grad(loss, m, x)
    return l

See value_and_grad.

E1108

Operation that gradients don’t have. Gradients can be added to and subtracted from gradients of the same model, negated, and multiplied or divided by a [] or a number; norm(g) gives their norm. Any other operation on one is an error.

dim D = 64

model Mlp:
    param w: [D D] = zeros

fn f(g: Grad[Mlp]) -> Grad[Mlp] = g * g

See Gradient types.

W1101

Param that a differentiated function never reads. Its gradient is always zero. Remove the param, or make it a buffer if it isn’t trained.

dim D = 64

model Mlp:
    param w: [D D] = zeros
    param unused: [D] = zeros

fn loss(m: Mlp, x: [b D]) -> [] = sum(x @ m.w)

fn f(m: Mlp, x: [b D]) -> []:
    l, g = value_and_grad(loss, m, x)
    return l

See value_and_grad.