7. INDENT / DEDENT tokenization, spaces only

• Status: Accepted
• Date: 2026-09-02

Context

0001 makes indentation the only block delimiter. The parser is a conventional recursive-descent parser and wants a flat token stream with explicit block boundaries — it should not be counting spaces. Something has to translate leading whitespace into structure.

The classic hazard is mixing tabs and spaces: two lines that look identically indented but are not, so the block structure the reader sees is not the one the compiler sees.

Decision

The lexer maintains an indentation stack and emits synthetic tokens:

• At the start of each logical line, measure the leading run of spaces.
• Greater than the top of the stack ⇒ push it, emit one INDENT.
• Less than the top ⇒ pop until the top matches, emitting one DEDENT per pop. If no stack entry matches the new level exactly ⇒ E0007 inconsistent dedent.
• Equal ⇒ emit nothing.
• NEWLINE is emitted at the end of each non-blank logical line.
• At end of file, emit a final NEWLINE then one DEDENT per remaining level.

Rules:

• Indentation is spaces only. A tab in the leading whitespace of a line is E0006. This removes the tab/space ambiguity entirely rather than trying to define an equivalence.
• The indent width is not fixed. Any consistent increase opens a block; the matching decrease closes it. (The style guide recommends 4 spaces; the language does not require it.)
• Blank lines and comment-only lines carry no indentation and never produce INDENT/DEDENT/NEWLINE.
• While any bracket is open, indentation is ignored and no NEWLINE is emitted (see 0003).

Consequences

• The parser sees INDENT/DEDENT as ordinary tokens and its block rule is INDENT stmt+ DEDENT. No whitespace logic in the parser.
• Tabs-in-indentation being a hard error will surprise users pasting from tab-configured editors. The error message says exactly this and how to fix it.
• Not fixing the indent width means a file could use 2 spaces at one level and 4 at the next. Legal but ugly; the formatter (post-M1) will normalise it.
• Tab inside a line (e.g. between tokens, in a string) is fine; only leading indentation is restricted.

Alternatives considered

• Tabs allowed, tab = 8 (or 4) columns. Every "tab width" choice is wrong for someone; the ambiguity is the problem, not the number.
• Tabs or spaces but not both in one file. Still two ways to indent; tooling and snippets from elsewhere still clash.
• Fixed 4-space indent, enforced. Tempting, but a language rule where a style rule suffices. Left to the formatter.
• Parser-driven layout (Haskell's parse-error(t) rule). Powerful, but notoriously hard to specify and to give good errors for.