Sikau

Build an interpreter · Day 1 · about 30 min

Text into tokens

Every language starts by chopping text into tokens.

Programs are just text

When you run JavaScript, something has to read your text and decide what it means. That something is an interpreter (or a compiler). In this course you'll build one for Chiya, a small language with variables, functions, closures and arrays — and by the end, loops you add yourself.

  • Lexer: text → tokens (let, x, =, 5).
  • Parser: tokens → a syntax tree that shows the structure.
  • Evaluator: walks the tree and computes values.
  • Each stage only talks to the next one, so each can be tested on its own.
let price = 180;
let withTax = fn(p) { p + p * 13 / 100 };
print(withTax(price));

What a token is

A token is the smallest piece of code that means something on its own. The lexer reads characters and groups them: digits into numbers, letters into names, = and == into operators. Spaces and line breaks only separate tokens and are thrown away. Words the language reserves, such as let and fn, become keyword tokens instead of names.

tokenize("let price = 180;")
// [ { type: "let",    value: "let" },
//   { type: "ident",  value: "price" },
//   { type: "=",      value: "=" },
//   { type: "number", value: "180" },
//   { type: ";",      value: ";" },
//   { type: "eof",    value: "" } ]

Operators and punctuation use themselves as their type ("=", ";"), which makes the parser read naturally later: if (peek().type === "(").

The rest of this lesson

  • Reading one character at a timeexplain
  • Count the tokenspredict
  • Lexing x == 10order
  • What type does it get?match
  • Your first lexerwrite code
  • Badge quizquiz
  • What you learnedrecap

Continue this lesson free — run real code in your browser

Continue with Google

See all 10 lessons of Build an interpreter