Lexical structure

Source text

A source file is UTF-8. A byte order mark is ignored. Spaces, tabs, form feeds and carriage returns between tokens are ignored, except that indentation is significant and may only use spaces.

Comments and doc comments

# starts a comment that runs to the end of the line. ## starts a doc comment.

A doc comment documents the declaration on the line directly below it. Consecutive lines that hold only a ## comment form one doc comment; each line’s ## and one space after it are removed. A doc comment documents a dim, domain, fn, extern fn, import python, model, or a model’s field, and the editor shows it when you hover the declared name. A doc comment that isn’t directly above a declaration, or that follows code on its line, is a warning (W0201).

Math in comments

Any comment, # or ##, can hold LaTeX math: $...$ inline, or $$...$$ displayed. The rules for finding math are Pandoc’s:

An opening $ can’t be followed by whitespace, and a closing $ can’t come after whitespace or be followed by a digit, so costs $3 and $4 is text.
\$ is a literal dollar sign, a run of three or more $ is text, and nothing inside a Markdown code span (`$x$`) is math.
Inline math stays on one line. Display math may continue over consecutive comment lines of the same kind (all # or all ##, each with nothing but the comment on it), but not past a blank comment line, and can’t be empty. A comment after code is a block of its own.

Math accepts the LaTeX subset listed in the README’s Math in comments table. Anything else is shown as written, with a hint: H0601 for a command or environment outside the subset, and H0602 for malformed math. Math never causes an error; the hints come from shapescript check and the editor, not from build or the Python import hook.

Tokens

Keywords: dim, domain, fn, extern, return, import, as, to, and, or, inf, true, false, for, in, if, else.

Contextual words are identifiers with a special meaning in one place: python after import; over after a domain in a type; devices at the start of a top-level line (in a notebook cell, only when a name follows it; Devices); model at the start of a top-level line when a name follows it; export at the start of a top-level line when fn follows it (Export); and param and buffer at the start of a line in a model’s block (Models). Elsewhere they are ordinary names, so model = Mlp.init(seed=0) in a notebook cell binds a value named model. int and float are ordinary names too, which mean host scalars in a type.

Token	Form	Notes
identifier	`[A-Za-z_][A-Za-z0-9_]*`	Not a keyword, and not `_` alone
placeholder	`_`	The pipe placeholder (Pipes)
unknown size	`?` followed by an identifier, as in `?k`	Only in types (Unknown sizes)
integer	`[0-9][0-9_]*`	Underscores are ignored: `50_000`. At most 2^64 − 1 (E0208)
grouped size	`[0-9][0-9_][A-Za-z][A-Za-z0-9_]`, as in `16gh`	A number touching a name: one axis of size 16 × `gh` (Grouped sizes). The name starts at the first letter. A float wins where both match, so `2e3` is a float
float	`1.5`, `1e-6`, `2.5e3`	Digits on both sides of the `.`; `1.` and `.5` aren’t numbers, so `0..n` is `0`, `..`, `n`
string	`"..."` on one line	Escapes `\n`, `\t`, `\"`, `\\`; any other escape is kept as written. A string without its closing quote is E0102
operators	`+ - * / @ ! == != < <= > >= \|> -> => = . .. ...`	`..` separates the ends of a range, in a `for` (Statements) or at the start of an iterator chain; `=>` is a lambda’s (Iterator chains)
brackets	`( ) [ ]`
punctuation	`, :`

inf is infinity; -inf is unary minus applied to it, so x-inf means x - inf. true and false are the two bools. Any other character is E0101.

Lines and indentation

The layout pass turns line breaks and indentation into NEWLINE, INDENT and DEDENT tokens:

Inside ( ) or [ ], line breaks and indentation are ignored, so a bracketed expression can span lines.
Blank lines and lines with only a comment are ignored.
A line whose first token is |> continues the line before it. No NEWLINE comes between them, and its indentation isn’t checked.
Otherwise each line ends with a NEWLINE. A line indented more than the current block opens a block (INDENT); a line indented less closes blocks (one DEDENT each) until its indentation matches an enclosing block’s. If it matches none, that’s E0104.
Indentation is counted in spaces. A tab in indentation is E0103.

An unmatched ) or ] is E0105, and a bracket that is never closed is E0106.