1. Lexical structure

This section defines how Korrin source text is divided into tokens. It is normative. The implementation is crates/korrin/src/lexer/.

Governing decisions: 0001, 0003, 0006, 0007.

1.1 Source encoding

A Korrin source unit is a sequence of Unicode scalar values, encoded as UTF-8. Byte offsets into this sequence are used to report positions.

The only significant line terminator is U+000A (line feed, \n). A U+000D (carriage return, \r) is ignored wherever it appears, so \r\n is accepted as a line ending.

1.2 Logical lines and layout tokens

The lexer produces four synthetic tokens that do not correspond to any run of source characters:

A logical line is one or more physical lines that the lexer treats as a single statement-bearing unit. A physical line break is not a logical line break when:

1. any bracket — (, [, or { — opened earlier is still unclosed; or
2. the last token before the break was one after which the line cannot end: a binary operator (+ - * / % == != < <= > >=), =, ,, ., or a word operator (and, or, not).

There is no line-continuation character. A logical line may not be split except by the two rules above.

total = (1 +
         2 +
         3)
print(total)
# => 6

1.3 Blank lines and comments

A comment starts with # and runs to the end of the physical line. There are no block comments.

A physical line that, after removing leading spaces, is empty or begins with # is a blank line. Blank lines produce no tokens — not even NEWLINE — and never affect indentation.

1.4 Indentation

Indentation is the run of space characters (U+0020) at the start of a physical line that begins a logical line. Its width is the count of those spaces.

• A tab (U+0009) anywhere in that leading run is an error (E0006). Korrin indents with spaces only.
• Tabs elsewhere on a line — between tokens, inside a string — are not restricted (though between tokens a tab is simply whitespace).

The lexer maintains a stack of indentation widths, initially [0]. For each logical line, let w be its indentation width and t the top of the stack:

• w > t: push w; emit one INDENT.
• w < t: pop while the top is greater than w, emitting one DEDENT per pop. If the resulting top is not equal to w, that is an inconsistent dedent (E0007).
• w == t: emit nothing.

At EOF, a NEWLINE is emitted if the last logical line had tokens, then one DEDENT for every stack entry above the base level 0.

The indentation width that opens a block is not fixed by the language — any consistent increase works — but the style guide requires 4 spaces.

if true:
    a = 1
    if true:
        b = 2
    c = 3
print(a + c)
# => 4

1.5 Identifiers

An identifier begins with _ or a character in Unicode XID_Start, and continues with _ or characters in XID_Continue (Unicode UAX #31). Identifiers are compared by exact scalar-value sequence; there is no case folding and no normalization.

1.6 Keywords

The following 19 identifiers are reserved and may not be used as names (ADR 0006):

and     break   class   continue  elif
else    false   fn      for       if
in      nil     not     or        pass
return  super   true    while

self is not reserved — it is the conventional name of a method's first parameter (ADR 0005) and is otherwise an ordinary identifier.

1.7 Literals

Integer literals

One or more ASCII digits: 0, 7, 1000. The value must fit in a signed 64-bit integer, otherwise E0005. There are no digit separators, sign characters (a leading - is the unary operator), or radix prefixes in Milestone 1.

Float literals

Digits, then either a fraction, an exponent, or both:

• fraction: . followed by one or more digits — 3.14, 0.5;
• exponent: e or E, an optional + or -, then one or more digits — 1e9, 2.5e-3.

A . is only a decimal point when a digit follows it, so x.field and 3.name tokenize as an access, not a malformed number. A . or identifier character glued to the end of a number (1.2.3, 10abc) is E0004.

print(type(10))
print(type(10.0))
# => int
# => float

String literals

Text between double quotes: "hello". A string literal may not contain a raw newline — an unclosed string at end of line is E0002. Single quotes are not string delimiters.

The recognized escape sequences are exactly:

Any other \x is E0003. Raw strings and triple-quoted strings do not exist.

Interpolation

Every string literal is a template (ADR 0015). A { begins a hole; the text up to the matching } is a Korrin expression, and its value is spliced into the string when the string is evaluated (§4.2).

• {{ and }} are literal { and }. A lone } is also literal.
• A hole is tokenized as ordinary tokens, so a string inside a hole needs no escaping: "{m["k"]}".
• An empty hole "{}" is E0103. A { not closed before the end of the line is E0002.
• A hole may not contain a newline (a string may not span lines).

A string with no holes is lexically a single string token. A string with at least one hole is lexed as STRSTART, then a run of STRTEXT chunks and { expression } holes, then STREND — the hole tokens carrying their real source positions.

1.8 Operators and punctuation

(  )  [  ]  {  }  ,  :  .
+  -  *  /  %
=  ==  !=  <  <=  >  >=

! is not a token on its own; it exists only as part of !=. A lone ! is E0001, which suggests != or not.