Now that the lexer is actually lexing, we can start parsing. This is where the Tree in Abstract Syntax Tree really comes in. What the parser is going to do is take a flat sequence of tokens and transform it into a shape that represents the actual structure of the code.
The first part of the language I’ve built is the lexer. It takes the program text as input and produces a vector of tokens. Tokens are the individual units that the parser will work with, rather than it having to work directly with characters. A token could be a bunch of different things. It could be a literal value (like a number or string), or it could be an identifier, or a specific symbol (like a plus sign).
I’ve been learning Rust for a little while now, and I’m a firm believer that the best way to learn a new language is to actually build something with it. Not something like an example project from a book or tutorial for the language, but something non-trivial that you actually find interesting.
In that vein, I’ve been thinking a lot lately about a building a small programming language. It’s something that I find interesting, and Rust is a good fit for it. It’s also something that lets me start out relatively small and simple, but still always have opportunities for growing more complex.
This post, and the posts that follow it are going to be a sort of experiment. Ordinarily, with an ongoing project, I would just post about it on the fediverse. This time, I’m going to try to write blog posts about it. I expect I’ll continue to post in-the-moment stuff on the fediverse, but I also want to write more semi-regular posts about how it’s going overall. They probably won’t be super detailed, and they certainly won’t be tutorials, but they will be more detailed and more explanatory than much of what I post on the fediverse. And, as with what I would otherwise write on the fediverse, these posts aren’t going to be perfect. I’m not an expert at this; there are going to be mistakes, and I’m going to end up backtracking. But I figure that’s slightly more interesting to read about than if I did everything correctly on the first try.
A fairly important part of Tusker (my iOS Mastodon app) is displaying images. And a bunch of varieties of images: user avatars, post attachments, custom emojis, user profile headers, as well as a few other types of rarely-shown images that get lumped in with attachments. And displaying lots of images in a performant way means caching. Lots of caching.
In the beginning, there was nothing. Then I started displaying images and almost immediately realized there would need to be some amount of caching. Otherwise, just scrolling down a little bit and then back up would result in images being re-loaded that were present mere seconds ago.
The very first implementation was super simple. It was basically just a dictionary of image URLs to the
Data for the image at that URL. This fulfilled the primary goals of being 1) super easy to build and 2) mostly working for the simplest of use cases. But, the blog post doesn’t end here, so clearly there are some issues remaining.
About a year ago, I was poking around the CloudFlare dashboard for my website, specifically the Security section of the Analytics page. To my surprise, it reported that it had blocked 78 thousand “bad browser” threats in the last 24 hours (almost one a second). Now, I don’t have very much on my website. My blog doesn’t get much traffic, and the only other thing that does is my fediverse instance. And that volume of inbound traffic is nowhere near what I would expect for my small instance, which probably doesn’t federate with more than a couple hundred others. I found the Firewall section of the dashboard, which shows the details of individual blocked requests. To my surprise, almost all of the blocked requests were to a subdomain I previously used as an update server for my Minecraft mods. Forge, a Minecraft mod loader, provides a mechanism by which mods can specify a URL that Forge can use to get a JSON object describing the latest versions of the mod, in order to notify the player if an update is available. A few years ago, I built a small tool to generate JSON files in Forge’s update format using Git repo tags from the GitHub API. This was running on my server, but some time in the couple years since I’ve stopped actively building mods for Minecraft, I shut it down. And in the time since then, CloudFlare has decided that all the traffic to the update server is a threat and should therefore be blocked. CloudFlare keeps a little bit of information about each blocked request going back quite a while, so this provides a surprising amount of information about the usage of my mods.