15: Optimizers, gradient transforms and schedules

E1501

Optimizer that isn’t given a model. sgd and adamw make optimizer state for a model, given first, by position. The state’s type comes from the model’s, so the model can’t be a tensor, a Python value or missing.

dim D = 64

fn f(x: [b D]):
    opt = adamw(x, lr=3e-4)

See Optimizer state.

E1502

Optimizer state or gradient of another model. update was given optimizer state or a gradient of another model type than the model it updates, or a parameter or return type that is optimizer state, such as AdamW[Mlp], got state for another model.

dim D = 64

model Mlp:
    param w: [D D] = zeros

model Other:
    param w: [D D] = zeros

fn step(mut m: Mlp, mut opt: AdamW[Other], g: Grad[Mlp]):
    update(mut m, mut opt, g)

See update and Optimizer state.

E1503

Value isn’t the expected optimizer state. Optimizer state was expected, by update or by a parameter or return type such as AdamW[Mlp], and the value is something else: a tensor, a model, or the state of another optimizer, such as SGD[Mlp] for AdamW[Mlp].

dim D = 64

model Mlp:
    param w: [D D] = zeros

fn step(mut m: Mlp, mut opt: SGD[Mlp], g: Grad[Mlp]):
    update(mut m, mut opt, g)

fn f(mut m: Mlp, g: Grad[Mlp]):
    opt = adamw(m, lr=3e-4)
    step(mut m, mut opt, g)

See Optimizer state.

E1504

Learning rate that isn’t a number or a Schedule. An optimizer’s lr=, warmup’s then, or a Schedule parameter or return type got something other than a Python number or a schedule, such as a tensor. A learning rate that changes is a schedule, as in lr=cosine(3e-4, 1000).

dim D = 64

model Mlp:
    param w: [D D] = zeros

fn f(m: Mlp, lr: []):
    opt = adamw(m, lr=lr)

See Schedules.

E1505

update or a gradient transform given the wrong kind of value. update’s first argument isn’t a model or its third isn’t a gradient, or clip_norm wasn’t given a gradient first.

dim D = 64

fn f(x: [b D]):
    y = clip_norm(x, 1.0)

See update and Gradient transforms.