13: Data records and datasets

E1301

Data record has no such field. x.name reads a field that the batch’s or example’s data record doesn’t declare. The help suggests a field with a similar name.

dim D = 64

data Digit:
    img: [D]

fn f(batch: [b] Digit) -> [b D] = batch.imgs

See Batches.

E1302

Invalid data field. A data record’s field has ?k in its type, has no shape, is a host scalar, a model, a data or logs record, a dataset, a gradient, optimizer state, a Schedule, a Mode or a LogSink, or is named with a Python keyword or a name that starts with _. Every example of a record must have the same shapes, so that examples can be stacked into batches; for values of different lengths, pad them to a dim and add a Mask field.

data Sentence:
    tokens: [?n]

See Data records.

E1303

Value isn’t the expected data. A parameter or return type is a batch of a data record or a dataset, and the value is a batch or dataset of another record, a batch with another number of batch axes, a dataset where a batch is expected (or the other way round), or a tensor. Records are matched by declaration.

dim D = 64

data Digit:
    img: [D]

data Pair:
    a: [D]

fn mean_img(batch: [b] Digit) -> [D] = mean(batch.img, over=b)

fn f(p: [b] Pair) -> [D] = mean_img(p)

See Batches and Datasets.

E1304

Invalid dataset stage. shuffle or batches is given something that isn’t a dataset, as in x |> batches(64) for a tensor x; shuffle comes after batches, which would shuffle whole batches; batches is used twice; or the batch size is known to be less than 1.

dim D = 64

data Digit:
    img: [D]

fn f(train: Dataset[Digit]) -> int:
    n = 0
    for batch in train |> batches(64) |> shuffle(seed=0):
        n = n + 1
    return n

See Datasets.