Sikau

Build an interpreter · Day 2 · about 35 min

A real lexer

Line and column numbers, strings, comments and helpful errors.

Where did it go wrong?

Positions cost almost nothing to track and pay off at every later stage: the parser reports grammar mistakes with them, and the evaluator can blame the right line when something goes wrong at run time.

"Syntax error" is useless; "Unexpected character '@' at 12:7" sends you straight to the problem. Good error messages start in the lexer: every token remembers the line and column where it began, so later stages can point at the exact spot. Today you'll also add strings, comments and the comparison operators <= and >=.

let greeting = "Namaste!\n"; // a comment
let ok = 3 <= 5;

Tracking line and column

Keep two counters next to the index: line starts at 1 and goes up at every newline (resetting col to 1); col goes up with every other character. Before reading a token, remember where it starts, then add { line, col } to it. When a multi-character token is read, move col forward by its length.

if (ch === "\n") { i++; line++; col = 1; continue; }
if (ch === " " || ch === "\t" || ch === "\r") { i++; col++; continue; }
const l = line, c = col;
// … read a number into source.slice(i, j) …
add("number", source.slice(i, j), l, c);
col += j - i;
i = j;

Columns count characters, starting at 1, which matches how editors show positions.

The rest of this lesson

  • Strings and commentsexplain
  • Where is the @?predict
  • Escapesfill
  • Source to tokensmatch
  • Upgrade the lexerwrite code
  • Badge quizquiz
  • What you learnedrecap

Continue this lesson free — run real code in your browser

Continue with Google

See all 10 lessons of Build an interpreter