STAGE 1 · THE SCANNER

First Cut: Text Becomes Tokens

Before a computer can mean anything by your code, it has to stop seeing characters and start seeing words.

What you're wiring up

You are building blu, a tiny language with integers, booleans, let, if/else, and functions. The first unit on the assembly line is the scanner: it walks the source one character at a time and emits tokens — LET, IDENT("x"), PLUS, INT(5). No grammar yet, no meaning. Just honest labeling.

Every token gets stamped with the line and column it came from. That feels like bureaucracy today, and it is the entire raw material for Stage 6's error messages. Positions flow token to AST node to runtime value; if you skip them now you will retrofit them later, painfully.

The scanner never crashes. Something unrecognizable in the input becomes an ILLEGAL token with a position, and the walk continues. A tool that dies on bad input can only ever report one problem.

# let x = 5;   →   blu tokens x.blu
1:1  LET    let
1:5  IDENT  x
1:7  ASSIGN =
1:9  INT    5
1:10 SEMI   ;
2:1  EOF

Assembly steps

[ 01 ]

Define the token vocabulary: a Token struct with type, literal, line and column, plus every type blu needs — operators, delimiters, INT, IDENT, keywords, EOF, ILLEGAL.

hint

Keep keywords out of the switch. Read a whole identifier, then one map[string]TokenType lookup decides let versus IDENT("letter").

[ 02 ]

Build the character walker: a Lexer holding the input, a read position, and the current line and column, with readChar to advance and peekChar to look ahead without consuming.

hint

Update line and column inside readChar and nowhere else. Every drifting-position bug comes from a second place mutating them.

[ 03 ]

Implement NextToken for single-character tokens, then identifiers and keywords, integer literals, and the two-character operators ==, !=, <=, >=.

hint

For == versus =, read the =, then peek. If the next char is also =, consume it. The same shape handles all four.

[ 04 ]

Skip whitespace and // line comments between tokens, and emit ILLEGAL with a position for anything unrecognized instead of panicking.

[ 05 ]

Wire the blu tokens subcommand: lex a file and print one token per line as LINE:COL TYPE LITERAL.

hint

This printer is the harness's only window into your lexer. Match the golden format exactly, including the final EOF line.

Go deeper (after it passes)

Lexing is the universal entry ramp: config parsers, query languages, log formats. Light pairing with DDIA M4 — Wire Formats That Survive Change, which is the same bytes-in, structure-out move over the network instead of over a file.