Models and Training

The spotter works, but look at what it took. forward, loss and predict each take six weight tensors, in the right order, and pass them on. Python made the weights, ran the training loop and saved them, and predict_saved loaded them back by position, w[0] to w[5], with one as each. That’s how you’d call ShapeScript from an existing PyTorch codebase, and it works. But ShapeScript can also hold the whole training program, and when it does, it checks the parts PyTorch leaves to discipline:

• State is declared. A model’s tensors are the fields of a model declaration. The spotter’s six weights become one value, m: Spotter, so there’s nothing to keep in order and nothing hides in an attribute.
• Gradients are values. value_and_grad(loss, m, ...) returns the loss and its gradient. There’s no .grad to zero and no no_grad mode to forget.
• Changes are visible. A function that changes a model says so with mut, in its signature and at every call.
• Randomness and modes are arguments. dropout takes a Mode, Train(seed=...) or Eval, so evaluation can’t run with dropout left on.
• The loop is yours. There are no hooks or callbacks: the training loop is a function you write and call, and the standard library’s is one you can read on one screen.

The chapter has three parts, and each changes spotter.ss a little more:

1. Models, mut and Gradients: the spotter’s weights become a model, with gradients and an optimizer.
2. Data, Modes and Logs: a record for one labelled picture, dropout, and what training reports.
3. The Training Loop and Checkpoints: the training loop in ShapeScript, saving and resuming, one loop for several models, and the standard library’s fit.

By the end, Python only provides the pictures. The spotter trains itself, and gets 97% of new pictures right.

Exercises

1. In Chapter 10’s train.py, list everything that Python had to get right that the checker couldn’t see: the order of the six weights, their shapes, and so on. Which of them does a model declaration take over?