Now that the lexer is actually lexing, we can start parsing. This is where the Tree in Abstract Syntax Tree really comes in. What the parser is going to do is take a flat sequence of tokens and transform it into a shape that represents the actual structure of the code.

The first part of the language I’ve built is the lexer. It takes the program text as input and produces a vector of tokens. Tokens are the individual units that the parser will work with, rather than it having to work directly with characters. A token could be a bunch of different things. It could be a literal value (like a number or string), or it could be an identifier, or a specific symbol (like a plus sign).

I’ve been learning Rust for a little while now, and I’m a firm believer that the best way to learn a new language is to actually build something with it. Not something like an example project from a book or tutorial for the language, but something non-trivial that you actually find interesting.

In that vein, I’ve been thinking a lot lately about a building a small programming language. It’s something that I find interesting, and Rust is a good fit for it. It’s also something that lets me start out relatively small and simple, but still always have opportunities for growing more complex.

This post, and the posts that follow it are going to be a sort of experiment. Ordinarily, with an ongoing project, I would just post about it on the fediverse. This time, I’m going to try to write blog posts about it. I expect I’ll continue to post in-the-moment stuff on the fediverse, but I also want to write more semi-regular posts about how it’s going overall. They probably won’t be super detailed, and they certainly won’t be tutorials, but they will be more detailed and more explanatory than much of what I post on the fediverse. And, as with what I would otherwise write on the fediverse, these posts aren’t going to be perfect. I’m not an expert at this; there are going to be mistakes, and I’m going to end up backtracking. But I figure that’s slightly more interesting to read about than if I did everything correctly on the first try.