And while that technique worked perfectly well, there were a number of downsides that I wasn’t entirely satisfied with. First: SwiftSoup alone is a big dependency, at around ten thousand lines of code. Since it’s shipping in my app, I take the view that I’m responsible for making sure it doesn’t break—and that’s a lot of surface area to cover (moreover, since it was ported from a Java library, the code is extremely un-Swift-y). Second: it’s not especially fast. Partly because of its aforementioned nature as a Java port (everything is a class, lots of unnecessary copies), but primarily because it’s just doing more than I need it to (the first rule of optimization is do less work): it was parsing a string into a tree and then I was flattening that tree back into a string.

The last time I needed to do something similar, I used lol-html, Cloudflare’s streaming HTML rewriter. But, while that worked (and is still in use for my RSS reader app), it’s written in Rust, and I didn’t want to introduce significant additional complexity to the build process for Tusker. So, writing a new HTML parser in Swift seemed like a great idea.

## The Architecture

Parsing HTML is no small task, and the spec stands at some 42 thousand words. Fortunately, I get to ignore about half of that. Parsing HTML is divided into two primary stages: tokenizing and tree construction. The tokenization stage takes the character stream as input and produces a stream of tokens (text characters, comments, start/end tags, etc.). The tree construction stages then uses the token stream to build up the actual DOM tree. The main way I’m improving over SwiftSoup in terms of not doing unnecessary work is by skipping tree construction altogether.

This is based on lol-html’s architecture, which usually uses a simulated tree construction stage (because the tokenizer gets feedback from the tree constructor and may have its state altered based on the current position in the tree) and only switches to the slower, real one when necessary. In my case, though, I don’t even simulate tree construction, since everything I want from the output can be generated directly from the token stream.

This does mean that in certain circumstances the output of the tokenizer could be incorrect, but that’s not a problem for my use case. For Tusker, I’m only ever going to use it to parse fragments—not whole documents—and the Mastodon backend sanitizes incoming HTML. What’s more, from what I can tell of lol-html’s simulator, most of the cases where (without the tree construction stage) ambiguity arises or feedback is needed aren’t going to come up in my use case. They involve other tag namespaces (SVG, MathML), or things like <template> tags—both of which I’m comfortable with handling incorrectly.

### Tokenizing

The tokenizer is the single biggest component of this project, weighing in at about 1500 lines of code. Fortunately though, the spec is quite clear. It describes in exacting detail the state machine that a spec-compliant tokenizer needs to mimic. To make things simple, all of my code is an almost one-for-one translation of the spec into Swift. There are a few places where I diverged, mostly in the name of performance, but where I didn’t, the code is quite easy to follow.

### Attributed Strings

Converting the token stream to an NSAttributedString is somewhat more complicated, but not terribly so. The gist of it is that the converter tracks what the style of the string should be at the current point in the token stream. Then, when a text character arrives from the tokenizer, it’s added to a buffer representing characters for which the current styles apply (called a “run”). When an open/close tag token is emitted, the current style is adjusted. After the current styles change, the current run is finished and an attributed string is built for the current text and styles and is appended to the overall string.

## Optimization

### Named Character References

The spec requires HTML tokenizers to have some special handling for named character references (things like &) which aren’t well-formed (e.g., missing the trailing semicolon). Because I was sticking very closely to the spec, this was originally implemented rather inefficiently: when a named character reference began, it would consume further characters one by one while there were any named character references beginning with the accumulated characters. I was just using a regular Swift dictionary to store all of the valid character references, so this was effectively an O(n) operation for each character in the reference.

That is, to put it mildly, ridiculously inefficient. Instead, the new approach just consumes as many alphanumeric characters as possible until it hits a semicolon. Then, if there’s no valid character reference, it backs up character by character until it finds a valid reference. This optimizes for the common case of a properly-formed reference, but (as far as I can tell) preserves the error-handling behavior that the spec requires.

### Static Dispatch

The way I implemented the tokenizer, there’s a separate method that follows the procedure for each state, each of which returns the next token. The tokenizer itself is also an Iterator that yields tokens. The Iterator.next implementation just switches over the current state and dispatches to the correct method.

It’s fairly common for the procedure for tokenizing to involve switching to a different state. Originally, I did this by changing the state and then calling next(). One small optimization I made relatively early was, in instances where the new state is constant, replacing the next() call with a direct call to the appropriate method for the new state—effectively statically dispatching whenever the state changes.

state = .tagName
return next()
// vs.
state = .tagName
return tokenizeTagName()

### UI/NSFont Caching

A number of the tags handled by the attributed string converter alter the current font. Originally, I implemented this by looking at the currently-applied styles and using them to assemble a SymbolicTraits that could be passed to UIFontDescriptor.withSymbolicTraits on the base font descriptor. When profiling, though, I found a non-trivial amount of time was being spent inside withSymbolicTraits to look up the actual font that should be used. So, the converter instead caches the known styles and their corresponding font objects.

### Using Swift.Array

One of the first optimizations I’d implemented was a type called InlineArray3 which was a collection type that stored up to three elements inline. Growing beyond that would cause it to switch to a regular Swift array, the storage for which is out of line. The motivation for this was the fact that the vast majority of the HTML tags that the parser sees will have very few attributes, so you might be able to gain a little bit of performance by storing them inline.

After implementing a simple performance testing harness (discussed later), I went back and tested this, and it turned out that just using Array was a fair bit faster. Just goes to show that you shouldn’t try to optimize things without having concrete data.

### Looping in Hot Recursive Functions

For a number of states in the tokenizer (such as while tokenizing a tag name), the state machines stays in the same state for multiple iterations before emitting a token. Initially, I implemented this just by having the tokenization function recursively call itself. One small but measurable performance improvement I got came from turning hot recursive paths into loops. So, rather than the function calling itself recursively, the entire function body would be wrapped in an infinite loop and the “recursive call” would simply continue on to the next loop iteration.

This was one of the more surprising optimizations I made, since I’d expected this to make no difference. Given tail-call optimization, I’d think both approaches would generate basically identical code. And indeed, looking at the disassembly before and after this change seems to confirm that. There are differences, and though they seem very minor, I guess they’re enough to make a difference (though a very, very small one—only about 5ms over 10k iterations).

### Avoiding Enums with Associated Values

Lots of states of the tokenizer involve building up the current token until it’s ready to emit. This means the parser needs to store the in-progress token. My Token data type is a Swift enum with a handful of different cases. So, the natural way of representing the current token was as an instance variable of an optional Token. Modifying the current token means, then, pattern matching against a Token and then constructing the updated Token and assigning it back.

Unfortunately, this does not permit in-place modification, meaning that all of the enum’s associated values need to be copied. For primitive types, this might not be a problem. But for more complex types, like arrays and strings, this results in a bunch of extra copies. So, rather than having a single currentToken property, there are separate properties representing the data for each possible token—each of which can be modified in-place:

private var currentToken: Token?
// vs.
private var currentStartTag: (String, selfClosing: Bool, attributes: [Attribute])?
private var currentEndTag: String?
// etc.

### Using Unicode.Scalar instead of Character

My tokenizer originally operated on a stream of Swift Characters, which represent extended grapheme clusters. The tokenizer, however, only cares about individual Unicode code points—and indeed is specified to operate on code points. So, I changed the tokenizer to operate on the Unicode.Scalar type. This resulted in a significant speed-up, since the there was no longer any time being spent on the grapheme breaking algorithm.

### Grouping Character Tokens

When emitting the document text, the tokenizer is specified to emit individual characters. As an optimization, though, I introduced an additional token type which emits a whole string of characters in one go. Since long, uninterrupted runs of characters that don’t require further parsing are quite common in actual HTML documents, this eliminates a bunch of the overhead associated with handling a token that comes from switching back and forth between the tokenizer and whatever’s consuming the tokens (that overhead is amortized over the sequence of character tokens).

## Conclusion

To both test the final performance of the new end-to-end HTML to NSAttributedString conversion process as well as guide most of the optimizations I described above, I wrote a small benchmarking harness that converts the HTML from the last ten thousand posts on my timeline to attributed strings using the old and new methods. This is basically the exact use case I have for this project, so I’m confident that the benchmarking results are fairly representative.

All told, the new process is about 2.7× faster when both are built in release mode, and a whopping 8× faster in debug (not terribly important, but nice for me since a debug build is often what’s on my phone).

I’m very happy with how this project turned out. I greatly enjoyed overengineering/optimizing something fairly self-contained, and the end result meets all of the goals/requirements I had for my use case. If you’re curious to see how it works in more detail, check out the source code.

This work is heavily based on this article by Adam Langley, which provides a great deal of information about implementing passkeys. My goal here is to fill in some more of the details, and provide some Elixir-specific information. As with Adam’s article, I’m not going to use any WebAuthn libraries (even though that may be advisable from a security/maintenance perspective) since I think it’s interesting and helpful to understand how things actually work.

Providing an exhaustive, production-ready implementation is a non-goal of this post. I’m going to make some slightly odd decisions for pedagogical reasons, and leave some things incomplete. That said, I’ll try to note when I’m doing so.

To start, I’m using the default Phoenix template app (less Tailwind) in which I’ve also generated the default, controller-based authentication system. Some parts of the password-specific stuff have been stripped out altogether, others will get changed to fit with the passkey authentication setup we’re going to build.

## Database schema

The first thing we’ll need is a backend schema for passkeys. The only data we need to store for a particular passkey is a unique identifier, and the public key itself. We also want users to be able to have multiple passkeys associated with their account (since they may want to login from multiple platforms that don’t cross-sync passkeys), so the users schema will have a one-to-many relationship with the passkeys.

The actual model I’ll call UserCredential, since that’s closer to what the WebAuthn spec calls them. Here’s the schema, it’s pretty simple:

defmodule PhoenixPasskeys.Accounts.UserCredential do
  use Ecto.Schema
  import Ecto.Changeset

  @primary_key {:id, :binary, []}

  schema "users_credentials" do
    # DER-encoded Subject Public Key Info: https://datatracker.ietf.org/doc/html/rfc5280#section-4.1.2.7
    field :public_key_spki, :binary
    belongs_to :user, PhoenixPasskeys.Accounts.User
    timestamps()
  end

  def changeset(credential, attrs) do
    credential
    |> cast(attrs, [:id, :public_key_spki, :user_id])
  end
end

There are a couple things to note here:

1. First, we’re explicitly specifying that the primary key is a binary, since that’s what the WebAuthn API provides as the credential ID.
2. The public_key_spki field contains the data for the public key itself and the algorithm being used. We don’t care about the specific format, though, since we don’t have to parse it ourselves.

To the generated User schema, I also added the has_many side of the relationship. There’s also a migration to create the credentials table. I won’t show it here, since it’s exactly what you’d expect—just make sure the id column is a binary.

## Registration JavaScript

WebAuthn, being a modern web API, is a JavaScript API. This is a distinct disadvantage, if your users expect to be able to login from JavaScript-less browsers. Nonetheless, we proceed with the JS. Here’s the first bit:

document.addEventListener("DOMContentLoaded", () => {
    const registrationForm = document.getElementById("registration-form");
    if (registrationForm) {
        registrationForm.addEventListener("submit", (event) => {
            event.preventDefault();
            registerWebAuthnAccount(registrationForm);
        });
    }
});

If we find the registration <form> element (note that I’ve added an ID (and also removed the password field)), we add a submit listener to it which prevents the form submission and instead calls the registerWebAuthnAccount function we’ll create next.

Before we get there, we’ll also write a brief helper function to check whether passkeys are actually available:

async function supportsPasskeys() {
    if (!window.PublicKeyCredential || !PublicKeyCredential.isConditionalMediationAvailable) {
		return false;
	}
	const [conditional, userVerifiying] = await Promise.all([
		PublicKeyCredential.isConditionalMediationAvailable(),
		PublicKeyCredential.isUserVerifyingPlatformAuthenticatorAvailable(),
	]);
	return conditional && userVerifiying;
}

The conditional mediation check establishes that WebAuthn conditional UI is available. This is what we’ll use for login, and is part of what makes using passkeys a nice, slick experience. Conditional UI will let us start a request for a passkey that doesn’t present anything until the user interacts with the browser’s passkey autofill UI.

The user-verifying platform authenticator check establishes that, well, there is a user-verifying platform authenticator. The platform authenticator part means an authenticator that’s part of the user’s device, not removable, and the user-verifying part means that the authenticator verifies the presence of the user (such as via biometrics).

In the registerWebAuthnAccount function, the first thing we need to do is check that both these conditions are met and passkeys are supported by the browser. If not, we’ll just bail out and registration won’t be possible.

async function registerWebAuthnAccount(form) {
    if (!(await supportsPasskeys())) {
        return;
    }
}

Next, we’ll setup a big ol’ options object that we’ll pass to the WebAuthn API to specify what sort of credential we want. There’s going to be a whole lot of stuff here, so lets take it one piece at a time.

async function registerWebAuthnAccount(form) {
    // ...
    const createOptions = {
        publicKey: {
			rp: {
				id: "localhost",
				name: "Phoenix Passkeys",
			}, 
        },
    };
}

The RP is the Relying Party—that is, us, the party which relies on the credentials for authentication. The ID is the domain of the RP—just localhost for this example, though in reality you’d need to use your actual domain in production. The name is just a user-facing name for the RP.

async function registerWebAuthnAccount(form) {
    // ...
    const createOptions = {
        publicKey: {
            // ...
            user: {
                id: generateUserID(),
                name: form["user[email"].value,
                displayName: "",
            },
        },
    };
}

Next up is some info about the user who the credential is for. The user’s “name” will just be the email that they entered in the form. The displayName value is required, per the spec, but we don’t have any other information so we just leave it blank and let the browser display the name only. The ID is where this gets a little weird, since we’re just generating a random 64-byte value:

function generateUserID() {
	const userID = new Uint8Array(64);
	crypto.getRandomValues(userID);
	return userID;
}

If you were adding a passkey to an existing user account, you could use a server-specified value for the user ID and the browser would replace any existing credentials with the same user ID and RP ID. But, since the user is registering a new account at this point, we assume they don’t want to do that (and, moreover, we don’t have any ID we can use yet). The user ID will also be returned upon a successful login, which would let us look up the user that’s logging in. However, since we’re only going to allow a credential to belong to a single user, we can use the credential’s ID to uniquely determine the user.

Next up, we specify the types of public keys that we’ll accept. We’re going to support Ed25519, ES256, and RS256, since those are recommended by the WebAuthn spec:

async function registerWebAuthnAccount(form) {
    // ...
    const createOptions = {
        publicKey: {
            // ...
            pubKeyCredParams: [
                {type: "public-key", alg: -8}, // Ed25519
                {type: "public-key", alg: -7}, // ES256
                {type: "public-key", alg: -257}, // RS256
            ],
        },
    };
}

During the login process, the server will generate a challenge value that the client will sign and the server will verify was signed with the user’s private key. The spec also requires that we provide a challenge when creating a credential, but we’re not going to use it for anything (since the user is just now creating the credential, we have nothing trusted that we can verify the initial challenge against), so we just provide an empty value:

async function registerWebAuthnAccount(form) {
    // ...
    const createOptions = {
        publicKey: {
            // ...
            challenge: new Uint8Array(),
        },
    };
}

Lastly, we give our requirements for authenticators that we want to use:

async function registerWebAuthnAccount(form) {
    // ...
    const createOptions = {
        publicKey: {
            // ...
            authenticatorSelection: {
                authenticatorAttachment: "platform",
                requireResidentKey: true,
            },
        },
    };
}

We want only the platform authenticator, not anything else, like removable security keys. We also specify that we want a resident key. This usage of “resident” is deprecated terminology that’s enshrined in the API. Really it means we want a discoverable credential, so that the authenticator will surface it during the login process without us having to request the credential by ID. This is important to note, since it’s what prevents needing a separate username entry step.

Now that we (finally) have all the configuration options in place, we can actually proceed with the credential creation. We pass the options to navigator.credentials.create to actually create the WebAuthn credential. If that fails, we’ll just take the easy way out and alert to inform the user (in an actual service, you’d probably want better error handling).

async function registerWebAuthnAccount(form) {
    // ...
    const credential = await navigator.credentials.create(createOptions);
    if (!credential) {
        alert("Could not create credential");
        return
    } 
}

From the credential we get back, we need a few pieces of information. First is the decoded client data, which is an object that contains information about the credential creation request that occurred.

async function registerWebAuthnAccount(form) {
    // ...
    const clientDataJSON = new TextDecoder().decode(credential.response.clientDataJSON);
    const clientData = JSON.parse(clientDataJSON);
}

The clientDataJSON field of the response object contains the JSON-serialized object in an ArrayBuffer, so we decode that to text and then parse the JSON. With the decoded object, we do a consistency check with a couple pieces of data: the type of the request, whether or not it was cross-origin, and the actual origin being used.

async function registerWebAuthnAccount(form) {
    // ...
    if (clientData.type !== "webauthn.create" || (("crossOrigin" in clientData) && clientData.crossOrigin) || clientData.origin !== "http://localhost:4000") {
        alert("Invalid credential");
        return;
    } 
}

If it was not a creation request, the request was cross-origin, or the origin doesn’t match, we bail out. Note that in production, the origin should be checked against the actual production origin, not localhost. And again, in reality you’d want better error handling than just an alert.

Next, we need to get the authenticator data which is encoded in a binary format and pull a few pieces of data out of it. You can see the full format of the authenticator data in the spec, but the parts we’re interested in are the backed-up state and the credential ID, which is part of the attested credential data.

async function registerWebAuthnAccount(form) {
    // ...
    const authenticatorData = new Uint8Array(credential.response.getAuthenticatorData());
    const backedUp = (authenticatorData[32] >> 4) & 1;
    const idLength = (authenticatorData[53] << 8) | authenticatorData[54];
    const id = authenticatorData.slice(55, 55 + idLength);
}

We get the backed-up bit from the flags byte at offset 32. Then we get the length of the credential ID, which is encoded as a big-endian, 16-bit integer at bytes 53 and 54. The ID itself immediately follows the length, thus starting at byte 55.

Before proceeding, we check that the backed-up bit is set, indicating that the credential is backed-up and safe from the user losing the current device. If it’s not, we won’t let the user register with this passkey. I choose to do this since it’s recommended by Adam Langley’s blog post, but whether it’s actually necessary may depend on your specific circumstances.

async function registerWebAuthnAccount(form) {
    // ...
    if (backedUp !== 1) {
        alert("Can't register with non backed-up credential");
        return;
    } 
}

The last piece of data we need out of the credential is the actual public key:

async function registerWebAuthnAccount(form) {
    // ...
    const publicKey = credential.response.getPublicKey();
}

And with all that in place, we can actually initiate the registration. We’ll assemble a form data payload with all of the requisite values:

async function registerWebAuthnAccount(form) {
    // ...
    const body = new FormData();
    body.append("_csrf_token", form._csrf_token.value);
    body.append("email", form["user[email]"].value);
    body.append("credential_id", arrayBufferToBase64(id));
    body.append("public_key_spki", arrayBufferToBase64(publicKey)); 
}

We’ll use the body in a POST request to the registration endpoint. The response we get back will contain a status value to indicate whether the request was successful or not. If it was successful, the backend will have set the session cookie, and we can redirect to the home page and the new user will be logged in. If registration failed, we’ll alert the user.

async function registerWebAuthnAccount(form) {
    // ...
    const resp = await fetch(form.action, {
        method: "POST",
        body,
    });
    const respJSON = await resp.json();
    if (respJSON.status === "ok") {
        window.location = "/";
    } else {
        alert("Registration failed");
    }
}

And with that, we’re done with JavaScript (for now) and can move on to the backend part of registering an account.

## Creating a user

In the UserRegistrationController module that comes with the Phoenix auth template, we’ll change the create function. By default, it registers the user using the parameters from the signup form and then redirects to the homepage. Instead, we’re going to register using the passkey we created on the client and then respond with the JSON that our JavaScript is expecting.

The first thing we need to do is extract the values that were sent by the frontend and decode the base 64-encoded ones.

defmodule PhoenixPasskeysWeb.UserRegistrationController do
  def create(conn, %{
        "email" => email,
        "credential_id" => credential_id,
        "public_key_spki" => public_key_spki
      }) do
    credential_id = Base.decode64!(credential_id)
    public_key_spki = Base.decode64!(public_key_spki)
  end
end

Those get passed to the Accounts.register_user function, which we’ll update shortly to handle. If account creation succeeded, we’ll still send the confirmation email as the existing code did. After that, instead of redirecting, we’ll log the user in by setting the session cookie and then respond with the “ok” status for the frontend. If account creation fails, we’ll just respond with the “error” status so the frontend can alert the user.

defmodule PhoenixPasskeysWeb.UserRegistrationController do
  def create(...) do
    # ...
    case Accounts.register_user(email, credential_id, public_key_spki) do
      {:ok, user} ->
        # Send confirmation email...

        conn
        |> UserAuth.log_in_user_without_redirect(user)
        |> json(%{status: :ok})

      {:error, _changeset} ->
        json(conn, %{status: :error})
    end
  end
end

Let’s update the register_user function. Instead of just creating a changeset for the user and then inserting it, we need to also create the authenticator. To avoid potentially leaving things in a broken state, we wrap both of these in a database transaction.

defmodule PhoenixPasskeys.Accounts do
  def register_user(email, credential_id, public_key_spki) do
    Repo.transaction(fn ->
      user =
        %User{}
        |> User.registration_changeset(%{email: email})
        |> Repo.insert()
        |> case do
          {:ok, user} -> user
          {:error, changeset} -> Repo.rollback(changeset)
        end
    end)
  end
end

First, we create a user with the given email. If the user creation fails, we abort and rollback the transaction. Then, we can create a credential belonging to the new user with the credential ID and public key we received from the client. As with the user, if creating the credential fails, we rollback the transaction.

defmodule PhoenixPasskeys.Accounts do
  def register_user(email, credential_id, public_key_spki) do
    Repo.transaction(fn ->
      # ...
      %UserCredential{}
      |> UserCredential.changeset(%{
        id: credential_id,
        public_key_spki: public_key_spki,
        user_id: user.id
      })
      |> Repo.insert()
      |> case do
        {:ok, _credential} -> nil
        {:error, changeset} -> Repo.rollback(changeset)
      end      
    end)
  end
end

We don’t need to do anything with the newly created credential, so we can just ignore it once it’s been created.

And finally, from the transaction function, we return the user:

defmodule PhoenixPasskeys.Accounts do
  def register_user(email, credential_id, public_key_spki) do
    Repo.transaction(fn ->
      # ...
      user
    end)
  end
end

The last thing we need to to complete the registration flow is to set the new user’s session cookie so that they’re logged in immediately. The UserAuth module that’s generated as part of the Phoenix auth template has a log_in_user function that does exactly this. But it also redirects the connection to another endpoint. We don’t want to do that, since we’re sending a JSON response, so I’ve split the function into two: one that only sets the session, and the existing function that sets the session and then redirects.

defmodule PhoenixPasskeysWeb.UserAuth do
  def log_in_user(conn, user, params \\ %{}) do
    user_return_to = get_session(conn, :user_return_to)
    conn
    |> log_in_user_without_redirect(user, params)
    |> redirect(to: user_return_to || signed_in_path(conn))
  end

  def log_in_user_without_redirect(conn, user, params \\ %{}) do
    token = Accounts.generate_user_session_token(user)
    conn
    |> renew_session()
    |> put_token_in_session(token)
    |> maybe_write_remember_me_cookie(token, params)
  end
end

And with that, everything is in place and you can now create an account with a passkey!

## Login form

Now that the user’s got an account, they need to be able to login with it. That means once again interacting with the WebAuthn API and writing a bunch of JavaScript. But before we get there, we need some slight changes to the backend.

In the HTML for the login page, we’ll add an ID to the <form> element so that we can find it from JS. We’ll also remove the password field, which is obviously no longer necessary. Lastly, but certainly not least, we need to send a challenge to the client.

The challenge is a value that the user’s device will cryptographically sign with their private key. The result will get sent back to the server, and we’ll verify it against the public key we have stored thus authenticating them. We’ll send the challenge just in a hidden form field:

<input type="hidden" id="challenge" value={@webauthn_challenge} />

In the session controller, we’ll need to generate and assign the challenge to the connection.

defmodule PhoenixPasskeysWeb.UserSessionController do
  def new(conn, _params) do
    conn
    |> put_webauthn_challenge()
    |> render(:new, error_message: nil)
  end
end

WebAuthn expects the challenge to be a value up to 64 bytes long, so we’ll use the Erlang crypto module to generate one of that length. The value is encoded as URL-safe base 64 (the same as normal base 64, but with dash and underscore rather than plus and slash) without padding. We encode it this way since that’s the format in which it will later be returned as part of the clientDataJSON, so when we extract that value we can directly compare it to the challenge value we generated.

defmodule PhoenixPasskeysWeb.UserSessionController do
  defp put_webauthn_challenge(conn) do
    challenge =
      :crypto.strong_rand_bytes(64)
      |> Base.url_encode64(padding: false)

    conn
    |> put_session(:webauthn_challenge, challenge)
    |> assign(:webauthn_challenge, challenge)
  end
end

Note that the challenge string is also stored in the session, so that we can later check that the challenge that the client signed matches the challenge we generated. It’s safe to store this in the session, even though it’s sent to the client, because the cookie is encrypted and signed so the client can’t tamper with it.

## Login JavaScript

With the first part of the backend changes taken care of, it’s time for more JavaScript, baby!

We’ll follow a similar outline to the registration setup (and the same caveat applies about error handling).

document.addEventListener("DOMContentLoaded", () => {
    // ...
    const loginForm = document.getElementById("login-form");
	if (loginForm) { 
        loginWebAuthnAccount(loginForm);
    }
});

async function loginWebAuthnAccount(loginForm) {
    if (!(await supportsPasskeys())) {
        return;
    }
}

The first thing we need to do is grab the challenge from the hidden form field, then we can construct the options object for getting the credential (which is, thankfully, much simpler than for creation).

async function loginWebAuthnAccount(loginForm) {
    // ...
    const challenge = loginForm.challenge.value;
    const getOptions = {
        mediation: "conditional",
        publicKey: {
            challenge: base64URLToArrayBuffer(challenge),
            rpId: "localhost",
        },
    };
}

In the options, we specify that we want conditional mediation. As noted before, this means that the browser won’t display any UI, except for autofill, for this credential request until the user accepts the autofill suggestion. In the public key options, we also give the decoded challenge value and specify the our Relying Party ID (again, this would need to be the actual domain in production).

Now, we can actually make the credential request and then, if we get a credential back, encode and send all the values to the backend. We need to send the ID of the credential, so that the backend can find its public key and the corresponding user. We also need the client data JSON, which we send as text decoded from the ArrayBuffer it’s returned as. We also need to send the authenticator data as well as the signature itself.

async function loginWebAuthnAccount(loginForm) {
    // ...
    const credential = await navigator.credentials.get(getOptions);
    if (!credential) {
        alert("Could not get credential");
        return;
    }

    const clientDataJSON = new TextDecoder().decode(credential.response.clientDataJSON);

    const body = new FormData();
    body.append("_csrf_token", loginForm._csrf_token.value);
    body.append("raw_id", arrayBufferToBase64(credential.rawId));
    body.append("client_data_json", clientDataJSON);
    body.append("authenticator_data", arrayBufferToBase64(credential.response.authenticatorData));
    body.append("signature", arrayBufferToBase64(credential.response.signature));
}

We can then send a request to the login endpoint. If the backend request fails (or if we failed to get the credential) we just alert the user (again, you’d probably want something better in reality). If the login attempt was successful, the server will have set the session cookie, and so we can just redirect to the homepage and the user will be logged in.

async function loginWebAuthnAccount(loginForm) {
    // ...
    const resp = await fetch("/users/log_in", {
        method: "POST",
        body,
    });
    const respJSON = await resp.json();
    if (respJSON?.status === "ok") {
        window.location = "/";
    } else {
        alert("Login failed");
    }
}

With that in place, let’s move on to the backend half of the login request.

## Validating a login attempt

As with signup, we’ll modify the existing log in endpoint to actually validate the WebAuthn login attempt.

The first step is extracting all of the information the frontend provides in the params and decoding it:

defmodule PhoenixPasskeysWeb.UserSessionController do
  def create(conn, params) do
    id = params |> Map.get("raw_id") |> Base.decode64!()
    authenticator_data = params |> Map.get("authenticator_data") |> Base.decode64!()
    client_data_json_str = params |> Map.get("client_data_json")
    signature = params |> Map.get("signature") |> Base.decode64!()
  end
end

Next, we’re going to validate all of the information we got from the client. Before we get to that, there are a handful of helper functions we’ll use. First, looking up a credential by its ID:

defmodule PhoenixPasskeys.Accounts do
  def get_credential(id) do
    Repo.get(UserCredential, id)
    |> Repo.preload(:user)
  end
end

Next, a function in the controller that verifies the signature against the provided data:

defmodule PhoenixPasskeysWeb.UserSessionController do
  defp verify_signature(credential, client_data_json_str, authenticator_data, signature) do
    with {:ok, pubkey} <- X509.PublicKey.from_der(credential.public_key_spki),
         client_data_json_hash <- :crypto.hash(:sha256, client_data_json_str),
         signed_message <- authenticator_data <> client_data_json_hash,
         true <- :public_key.verify(signed_message, :sha256, signature, pubkey) do
      true
    else
      _ ->
        false
    end
  end
end

X509 comes from the x509 package, which is the only third-party piece of code we’re using. It’s a fairly thin wrapper around the Erlang public_key and crypto modules, and mostly serves to save me from having to deal with Erlang records in my code. Its from_der helper function is used to parse the public key from the encoded format.

Next, we hash the client data JSON and append that hash to the authenticator data from the client. This value is what should match the signature using the public key we’ve got, so finally we check that. If all these steps succeeded, we return true, and false otherwise.

The last helper function will receive the decoded client data make sure it’s got all of the values that we expect. If the crossOrigin value is present and is not false, the client data is invalid and the login attempt will be rejected.

defmodule PhoenixPasskeysWeb.UserSessionController do
  defp check_client_data_json(%{"crossOrigin" => crossOrigin}) when crossOrigin != false do
    false
  end
end

Otherwise, we check that the data has the expected type and origin, and we extract the challenge value (note that we’re checking the origin again here, and this would need to change in production):

defmodule PhoenixPasskeysWeb.UserSessionController do
  defp check_client_data_json(%{
         "type" => "webauthn.get",
         "challenge" => challenge,
         "origin" => "http://localhost:4000"
       }) do
    {:ok, challenge}
  end 
end

And lastly, if neither of the previous patterns matched, the client data fails validation:

defmodule PhoenixPasskeysWeb.UserSessionController do
  defp check_client_data_json(_), do: false
end

Now, let’s put all those parts together and validate the login attempt.

defmodule PhoenixPasskeysWeb.UserSessionController do
  def create(conn, params) do
    # ...
    with credential when not is_nil(credential) <- Accounts.get_credential(id),
         true <- verify_signature(credential, client_data_json_str, authenticator_data, signature),
         {:ok, client_data_json} <- Jason.decode(client_data_json_str),
         {:ok, challenge} <- check_client_data_json(client_data_json),
         true <- challenge == get_session(conn, :webauthn_challenge),
         true <- :binary.part(authenticator_data, 0, 32) == :crypto.hash(:sha256, "localhost"),
         true <- (:binary.at(authenticator_data, 32) &&& 1) == 1 do
      conn
      |> delete_session(:webauthn_challenge)
      |> UserAuth.log_in_user_without_redirect(credential.user)
      |> json(%{status: :ok})
    else
      _ ->
        json(conn, %{status: :error})
    end
  end
end

Here’s everything that we’re doing:

1. Lookup the credential with the given ID.
2. Use the public key we had stored to verify the signature on the authenticator data and client data JSON.
3. Decode the client data JSON.
4. Check that all the values in the client data are what we expect, and extract the challenge that was signed.
5. Ensure the challenge that the user signed matches what we previously generated.
6. Extract the hash of the origin from the authenticator data and ensure it matches our origin (this would not be localhost in production).
7. Check that the authenticator data has the user presence bit set (indicating that a person was actually present on the client’s end).

If all of those steps succeeded, we remove the old challenge value from the session (since it’s no longer needed), actually log the user in, and then respond with the “ok” status that the JavaScript is expecting. If any step failed, we’ll respond with the “error” status and the frontend will alert the user.

Since there’s a lot going on here, it’s worth being clear about what exactly in this process lets us authenticate and prove that the user is who they claim to be. Since the signature verification step is using the public key that we stored during the registration process, we know that anyone that can produce a valid signature using that public key must be the user (or at any rate, have their private key). The value that they’re signing is, essentially, the challenge: a securely generated random value. The user isn’t directly signing the challenge, but this is still safe, since the challenge value is included in the client data JSON, that hash of which is included in the signed message.

So: the challenge value that was signed by the user must be in the client data, and the challenge value in the client data must be the one we generated. Given that, we know that the user whose key was used to sign the message is the one trying to log in now. That we’re verifying with the stored public key prevents an attacker from using an arbitrary key to sign the login attempt. And that the signed challenge matches the challenge the server generated means an attacker can’t reuse a previous response to login (a replay attack).

At long last, we finally have the ability to log in to our application using a passkey. Only a few minor things to go, so let’s forge ahead.

## Handling login if the user enters an email

Although we’re presenting the conditional UI, there’s nothing preventing the user from typing their email into the field and then clicking “Sign in,” so we should probably handle that to. This can be done fairly simply by reusing our existing code for conditional login.

We’ll change the loginWebAuthnAccount to take an additional parameter, conditional, which will be a boolean indicating whether this login attempt is to setup the conditional UI or triggered by submitting the login form.

If it’s false, we won’t request conditional mediation and instead we’ll look up the credentials corresponding to the email the user entered and ask WebAuthn for one of those:

async function loginWebAuthnAccount(loginForm, conditional) {
    // ...
    let allowCredentials = [];
    if (!conditional) {
		const email = loginForm["user[email]"].value;
		const resp = await fetch(`/users/log_in/credentials?email=${email}`);
		const respJSON = await resp.json();
		allowCredentials = respJSON.map((id) => {
			return {
				type: "public-key",
				id: base64ToArrayBuffer(id),
			};
		});
    }

    const getOptions = {
		mediation: conditional ? "conditional" : "optional",
		publicKey: {
			challenge: base64URLToArrayBuffer(challenge),
			rpId: "localhost",
			allowCredentials,
		}
    };
    // ...
}

The “optional” value for the mediation option means that the authenticator isn’t required to display UI, but will do so if its policies dictate that. The allowCredentials array contains objects describing all of the credentials that we want to accept—specifically, their binary IDs as ArrayBuffers.

We look up the user’s credentials so that the one we actually request from the authenticator matches the account that the user is trying to log in with. To handle this, we’ll also wire up an additional route on the backend that returns the base 64-encoded IDs of all the credentials belonging to the user with a given email.

defmodule PhoenixPasskeysWeb.UserSessionController do
  def credentials(conn, %{"email" => email}) do
    ids =
      Accounts.get_credentials_by_email(email)
      |> Enum.map(fn cred -> Base.encode64(cred.id) end)

    json(conn, ids)
  end
end

The get_credentials_by_email function is quite simple. It just looks up a user by email, preloading any credentials they have and then returning them:

defmodule PhoenixPasskeys.Accounts do
  def get_credentials_by_email(email) do
    Repo.one(from u in User, where: u.email == ^email, preload: :credentials)
    |> case do
      %{credentials: credentials} -> credentials
      nil -> []
    end
  end
end

Back in the JS, we can tweak the setup code to pass true for the conditional parameter in the initial request and also register a submit handler on the login form that will invoke it with false:

document.addEventListener("DOMContentLoaded", () => {
    // ...
	const loginForm = document.getElementById("login-form");
	if (loginForm) {
		loginWebAuthnAccount(loginForm, true);
		loginForm.addEventListener("submit", (event) => {
			event.preventDefault();
			loginWebAuthnAccount(loginForm, false);
		});
	}
});

And so we’ve handled the case where the user ignores the conditional UI and still types in their email to log in.

## Passkey reset

A not-infrequent argument against passkeys is that if your phone falls into a lake, you lose access to all of your passkey-backed accounts. One could argue that this isn’t true because the definition of passkeys means that they’re backed up and thus protected from this sort of event (and indeed, earlier we only permitted registering with backed-up credentials). I think the argument isn’t particularly interesting, however, because you can still have a “Forgot my passkey” option that works just like it does now with passwords.

This is less secure than a passkey implementation that has no “Forgot” option. But it’s no less secure than current password-based systems, and I think the UX/security tradeoff here falls on the UX side—people will, inevitably, lose access to their passkeys while retaining access to their email.

Implementing isn’t too complicated, fortunately, since we can reuse much of the registration code. First, the JavaScript. The only change necessary is attaching the registration function to the reset form as well.

document.addEventListener("DOMContentLoaded", () => {
    // ...
	const resetForm = document.getElementById("reset-passkey-form");
	if (resetForm) {
		resetForm.addEventListener("submit", (event) => {
			event.preventDefault();
			registerWebAuthnAccount(resetForm);
		});
	}
});

In the HTML for the reset form, we we need to include the email in the same form field as the registration form (and also add an ID to the <form> element):

<input type="hidden" name="user[email]" value={@user.email} />

In the function for the reset route (which is changed to POST from PUT, to match the signup route), we take the credential ID and public key and use them to update the user’s credentials:

defmodule PhoenixPasskeysWeb.UserResetPasswordController do
  def update(conn, %{
        "credential_id" => credential_id,
        "public_key_spki" => public_key_spki
      }) do
    credential_id = Base.decode64!(credential_id)
    public_key_spki = Base.decode64!(public_key_spki)
    case Accounts.reset_user_credentials(conn.assigns.user, credential_id, public_key_spki) do
      :ok ->
        conn
        |> put_flash(:info, "Passkey reset successfully.")
        |> UserAuth.log_in_user_without_redirect(conn.assigns.user)
        |> json(%{status: :ok})

      {:error, _} ->
        conn
        |> put_flash(:error, "Error resetting passkey")
        |> json(%{status: :error})
    end
  end
end

After updating, we log the user in if applicable and return JSON with the appropriate status.

The reset_user_credentials function works very similarly to the original reset password function that was part of the template: it deletes all the user’s existing sessions, and then removes their existing credentials and creates a new one:

defmodule PhoenixPasskeys.Accounts do
  def reset_user_credentials(user, credential_id, public_key_spki) do
    Ecto.Multi.new()
    |> Ecto.Multi.delete_all(
      :old_credentials,
      from(a in UserCredential, where: a.user_id == ^user.id)
    )
    |> Ecto.Multi.insert(
      :new_credential,
      UserCredential.changeset(%UserCredential{}, %{
        id: credential_id,
        public_key_spki: public_key_spki,
        user_id: user.id
      })
    )
    |> Ecto.Multi.delete_all(:tokens, UserToken.user_and_contexts_query(user, :all))
    |> Repo.transaction()
    |> case do
      {:ok, _} -> :ok
      {:error, _, changeset, _} -> {:error, changeset}
    end
  end
end

It’s worth noting that this does have slightly weaker security properties than the phx.gen.auth reset password implementation. With that approach, leaking a password reset token does not necessarily result in an account takeover, since whoever obtained the leaked token may not know the target user’s email address. Since the auth template forces the user to re-login after a reset, this prevents someone without the email address from gaining access even if they change the password.

But since logging in with a passkey is functionally a single factor, resetting it means gaining access to the account. So, a leaked reset token gives the bearer control over the account. This is an argument against having a reset option, but whether this is a concern in practice depends on your specific circumstances.

## Conclusion

As noted, this is not a complete implementation. There are a handful of places where I’ve left things unfinished since this isn’t meant to be production-level code. There are also a few places where there are security decisions that need to be made on a more contextual basis, that I’ve tried to note. And, of course, you wouldn’t really want to only permit signing in with passkeys and wholesale drop the passwords column from your database.

Nonetheless, I hope this has been a helpful look at how to implement passkeys in an Elixir/Phoenix application. You can find the complete repo here, and it may also be useful to look at the specific commit where passkey support was added.

To create a custom trait, you define a type conforming to UITraitDefinition which serves as the key for your trait:

struct PureBlackDarkModeTrait: UITraitDefinition {
    static let defaultValue = true
    static let affectsColorAppearance = true
}

The default value is, well, the default value. That is, what will be read when the trait’s not explicitly defined in the trait environment. The trait definition also specifies whether this trait can effect the appearance of colors, meaning whether dynamic UIColors should be reëvaluted when the trait’s value changes.

Then, you can define an extension on UITraitCollection which provides more idiomatic access to the trait (rather than always explicitly looking it up with the type):

extension UITraitCollection {
    var pureBlackDarkMode: Bool {
        self[PureBlackDarkModeTrait.self]
    }
}

One place where UIKit differs slightly from SwiftUI is that trait setters are defined on a separate type: UIMutableTraits. So, one more extension:

extension UIMutableTraits {
    var pureBlackDarkMode: Bool {
        get { self[PureBlackDarkModeTrait.self] }
        set { self[PureBlackDarkModeTrait.self] = newValue }
    }
}

And with that, I can continue using my custom trait just as I was prior to iOS 17, but without any of the pile of hacks. Reading the trait from a UIColor’s dynamicProvider closure works exactly as you expect and updates when appropriate.

It’s a pretty minor thing, objectively speaking, but this is a strong contender for my favorite feature of iOS 17.

## Presenting a UIViewController

The first problem we need to contend with is how to bust out of the containing UIHostingController from within the SwiftUI view tree. In UIKit-land, we’d “escape” the current view controller by presenting another one. SwiftUI has presentation modifiers like sheet and fullScreenCover, which are analogous to the UIKit present(_:animated:) and modalPresentationStyle APIs, but the builtin SwiftUI API doesn’t work for us since the whole point of this is controlling the presentation animation, which SwiftUI doesn’t expose.

So, to get at the presentation animation APIs, we need to present the VC ourselves. Which means that from inside SwiftUI, we need to have access to a view controller whose present method we can call. Rather than trying to walk up the view tree to find the UIHostingController that contains the SwiftUI view tree, we can use UIViewControllerRepresentable to create our own VC and let the framework manage the child VC relationship. We can still call present, because UIKit will handle forwarding the presentation up the hierarchy until it finds one that can actually handle it.

The representable will take a function that creates a view controller—so we can defer its creation until it’s actually used—as well as a Binding<Bool> representing whether the VC is currently presented, following the SwiftUI pattern for presentations.

The actual view controller that the representable, uh, represents will be completely empty and unconfigured: we only need it to exist so that we can call present, we don’t need it to display anything.

struct ViewControllerPresenter: UIViewControllerRepresentable {
    let makeVC: () -> UIViewController
    @Binding var isPresented: Bool
    
    func makeUIViewController(context: Context) -> UIViewController {
        return UIViewController()
    }
    
    func updateUIViewController(_ uiViewController: UIViewController, context: Context) {
    }
}

In the update method, we can read the value of the binding and present or dismiss the VC as necessary.

func updateUIViewController(_ uiViewController: UIViewController, context: Context) {
    if isPresented {
        if uiViewController.presentedViewController == nil {
            let presented = makeVC()
            uiViewController.present(presented, animated: true)
        }
    } else {
        if let presentedViewController = uiViewController.presentedViewController,
           !presentedViewController.isBeingDismissed {
			uiViewController.dismiss(animated: true)
        }
    }
}

A couple things of note:

1. makeVC is called only once, when the VC is first presented. This means that, while it’s okay to access SwiftUI state in when constructing the VC, updates to the state will not be reflected in the VC, so we have to take care to either pass bindings or store mutable state inside of an immutable container (i.e., a class).
2. We make sure to check isBeingDismissed before dismissing the presented VC. Otherwise, if there’s another view update during the dismissal—which is entirely possible—we’ll double call dismiss and potentially dismiss the VC containing the presenter representable.

Just this already works pretty well: you can present and dismiss a view controller using a SwiftUI binding. But the view controller also needs to be able to dismiss itself, without having any knowledge of the binding, and still keep the binding value up-to-date. To accomplish that, we’ll make the coordinator the delegate of the presentation controller, so it can use the will-dismiss method to update the binding state.

func updateUIViewController(_ uiViewController: UIViewController, context: Context) {
    if isPresented {
        if uiViewController.presentedViewController == nil {
            let presented = makeVC()
            presented.presentationController!.delegate = context.coordinator
            uiViewController.present(presented, animated: true)
        }
    } else {
        // ...
    }
}

func makeCoordinator() -> Coordinator {
    return Coordinator(isPresented: $isPresented)
}

class Coordinator: NSObject, UIAdaptivePresentationControllerDelegate {
    @Binding var isPresented: Bool
    
    init(isPresented: Binding<Bool>) {
        self._isPresented = isPresented
    }
    
    func presentationControllerWillDismiss(_ presentationController: UIPresentationController) {
        isPresented = false
    }
}

Lastly, if you try this combined with another presented VC, you’ll notice a rather annoying problem: a view update of the presenter representable when another VC is presented results in that VC getting dismissed. Because the presentedViewController doesn’t just look at the target’s presented VC, but walks up the VC hierarchy until it finds the actual VC responsible for presentation. So if another VC was presented, it will be returned and, since isPresented is false, be dismissed prematurely. To solve this, we can keep track of when the representable itself actually triggered the presentation.

class Coordinator: NSObject, UIAdaptivePresentationControllerDelegate {
    @Binding var isPresented: Bool
    var didPresent = false
    // ...
    func presentationControllerWillDismiss(_ presentationController: UIPresentationController) {
        isPresented = false
        didPresent = false
    }
}

This doesn’t use a SwiftUI state property, since we’re deliberately going to change it during view updates. When the representable presents the VC, it can set this flag to true and later verify that it’s true before dismissing the VC:

func updateUIViewController(_ uiViewController: UIViewController, context: Context) {
    if isPresented {
        if uiViewController.presentedViewController == nil {
            // ...
            context.coordinator.didPresent = true
        }
    } else {
        if context.coordinator.didPresent,
           let presentedViewController = uiViewController.presentedViewController,
        // ...
    }
}

Lastly, we can declare an extension on View called presentViewController which takes the same arguments as the modifier type and applies it to self.

## The Approach

Now that we have the ability to present a custom view controller, let’s think about what that VC actually needs to contain for the matched geometry effect. The transition we want to achieve is split into two parts: the matched geometry and everything else. The frames of the views being matched should animate smoothly from source to destination, and the views themselves should be visible the entire time. The rest of the presented content, however, is not matched and should appear with some other animation. There are multiple options, but I’m going to go with a simple fade-in.

So, when put all together, there will be three layers which are (back to front): the the source view, the non-matched part of the presented view, and finally the matched view(s).

The presented and matched layers will each be their own UIHostingController, containing the respective parts of the SwiftUI view tree that we want to display. They’ll be grouped into a single container view controller, which is the VC that will actually be presented.

## Collecting Matched Geometry Sources

The first step in building the actual effect we’re after is collecting all of the views we want to use as sources as well as their geometries. The views themselves are necessary in addition to the frames because, unlike SwiftUI^[2], we’re displaying the matched views outside of their original position in the view tree.

To send this information up through the view tree, we’ll use a custom preference. The value of the preference will be a dictionary which maps the matched geometry’s ID to a tuple of an AnyView and a CGRect. The view is the type-erased view that’s being matched, and the rect is the frame of the source view. The important part of the preference key is the reducer which, rather than simply overwriting the current value, merges it with the new one. This means that, if there are multiple matched geometry sources in the view tree, reading the preference from higher up in the tree will give us access to all of the sources.

struct MatchedGeometrySourcesKey: PreferenceKey {
	static let defaultValue: [AnyHashable: (AnyView, CGRect)] = [:]
	static func reduce(value: inout Value, nextValue: () -> Value) {
		value.merge(nextValue(), uniquingKeysWith: { _, new in new })
	}
}

The modifier then uses a GeometryReader to get the frame. It resolves the frame in the global coordinate space, which is what we want, since we’ll present a view controller that fills the window. The geometry reader is placed in a background modifier so that it doesn’t affect the layout of the wrapped view.

struct MatchedGeometrySourceModifier<Matched: View>: ViewModifier {
	let id: AnyHashable
    let matched: Matched

	func body(content: Content) -> some View {
		content
			.background(GeometryReader { proxy in
				Color.clear
					.preference(key: MatchedGeometrySourcesKey.self, value: [
                        id: (AnyView(matched), proxy.frame(in: .global))
                    ])
			})
	}
}

Also of note here is why the matched property is necessary, rather than just using the method’s content parameter. When applying the modifier, SwiftUI doesn’t invoke the body method with the actual wrapped view. Rather, it receives an instance of _ViewModifier_Content, which seems to be a placeholder for the real content. And it can’t be used outside of the modifier (trying to do so will result in nothing rendering), so our source modifier needs to store a copy of the actual wrapped view.

We can then add a little extension to View to make the API nice and SwiftUI-y.

extension View {
    func matchedGeometrySource<ID: Hashable>(id: ID) -> some View {
        self.modifier(MatchedGeometrySourceModifier(id: AnyHashable(id), matched: self))
    }
}

Next, let’s read the source data that’s collected by the preference and pass it off to the presented view controller. This too will be a modifier, like the SwiftUI presentation ones.

struct MatchedGeometryPresentationModifier: ViewModifier {
    @Binding var isPresented: Bool
    
    func body(content: Content) -> some View {
        content
        	.backgroundPreferenceValue(MatchedGeometrySourcesKey.self) { sources in
                Color.clear
                    .presentViewController(makeVC(sources: sources), isPresented: $isPresented)
            }
    }
    
    private func makeVC(sources: [AnyHashable: (AnyView, CGRect)]) -> () -> UIViewController {
        return {
            return MatchedGeometryViewController(sources: sources)
		}
    }
}

The backgroundPreferenceValue gives us access to the value of the preference and let’s us use it to build part of the view tree. We can’t use the onPreferenceChange modifier since it requires the preference value conform to Equatable, which isn’t possible for the source since it uses AnyView. So instead, we use the background preference modifier which always gives us access to the current value of the preference, without having to check whether it’s changed. But we don’t actually want to show anything in the background, so we attached the VC presentation modifier to the clear color.

## The Container View Controller

The container VC is where the bulk of the work is happening, so we’ll start with a simple version that just displays all the sources in the right positions—without any animations or other content—to validate our approach.

class MatchedGeometryViewController: UIViewController {
    let sources: [AnyHashable: (AnyView, CGRect)]
    var matchedHost: UIHostingController<MatchedContainerView>!
    
    init(sources: [AnyHashable: (AnyView, CGRect)]) {
        self.sources = sources
        
        super.init(nibName: nil, bundle: nil)
    }
    
    required init?(coder: NSCoder) {
        fatalError("init(coder:) has not been implemented")
    }
}

The container VC will have a hosting controller that’s dedicated to the views that we’re matching between the source and the destination. We’ll set it up in viewDidLoad.

override func viewDidLoad() {
    super.viewDidLoad()
    
    let sources = self.sources.map { (id: $0.key, view: $0.value.0, frame: $0.value.1) }
    let matchedContainer = MatchedContainerView(sources: sources, state: state)
    matchedHost = UIHostingController(rootView: matchedContainer)
    matchedHost.view.autoresizingMask = [.flexibleWidth, .flexibleHeight]
    matchedHost.view.frame = view.bounds
    matchedHost.view.backgroundColor = .clear
    matchedHost.view.layer.zPosition = 100
    addChild(matchedHost)
    view.addSubview(matchedHost.view)
    matchedHost.didMove(toParent: self)
}

A couple notes on this:

1. We turn the sources dictionary into an array (and take the opportunity to flatten out the nested tuples) because we’ll need a RandomAccessCollection to use with ForEach to display all the views.
2. We make the background color clear and raise the z-index so that when we add the content, the matched views appear above it and don’t completely obscure it.

The container view for the matched views will, for now, just display all of the views in a ZStack and fix them at their source frames.

struct MatchedContainerView: View {
    let sources: [(id: AnyHashable, view: AnyView, frame: CGRect)]
    @ObservedObject var state: MatchedGeometryState
    
    var body: some View {
        ZStack {
            ForEach(sources, id: \.id) { (id, view, frame) in
                view
                	.frame(width: frame.width, height: frame.height)
                	.position(x: frame.midX, y: frame.midY)
                    .ignoresSafeArea()
            }
		}
    }
}

Note that we use the middle x/y coordinates of the source frame, since the position modifier is anchored at the center of the view. We also make the matched view ignore the safe area, since the coordinates we’re using for the position are in the global coordinate space, which extends past the safe area.

Testing this out, we can see that the matched views are indeed displayed at the same position (roughly: the container VC is still actually being presented as a sheet, which is slightly offsetting everything relative to the global coordinate space).

VStack {
	Image("pranked")
		.resizable()
		.aspectRatio(contentMode: .fit)
		.matchedGeometrySource(id: "image")
		.frame(width: 100)
	
	Button {
		presented.toggle()
	} label: {
		Text("Present")
	}
}
.matchedGeometryPresentation(isPresented: $presented)

## Non-Matched Content

Next, let’s build the destination side of the setup and actually display the real content we want to present, not just the matched views. We’ll use a modifier like the source one to collect the geometry of the destination views, with much the same geometry reader technique.

struct MatchedGeometryDestinationModifier<Matched: View>: ViewModifier {
    let id: AnyHashable
    let matched: Matched
    
    func body(content: Content) -> some View {
        content
        	.background(GeometryReader { proxy in
                Color.clear
                	.preference(key: MatchedGeometryDestinationFrameKey.self, value: proxy.frame(in: .global))
                    .onPreferenceChange(MatchedGeometryDestinationFrameKey.self) { newValue in
						// TODO
                    }
            })
    }
}

extension View {
    func matchedGeometryDestination<ID: Hashable>(id: ID) -> some View {
        self.modifier(MatchedGeometryDestinationModifier(id: AnyHashable(id), matched: self))
    }
}

The difference here is that the preference is only going to contain the frame, so that we can use the onPreferenceChange modifier to listen for changes and update the container VC’s state. Unlike the source, we’re not going to listen for this preference anywhere higher in the view tree. The reason for this is that we need the value of the preference to update state, not to construct another part of the view tree. And onPreferenceChange is the only modifier we have available for that—and it requires the preference’s value be Equatable which it can’t be if we put the AnyView in it.

The preference key itself is very simple: it just holds an optional rect.

struct MatchedGeometryDestinationFrameKey: PreferenceKey {
    static let defaultValue: CGRect? = nil
    static func reduce(value: inout CGRect?, nextValue: () -> CGRect?) {
        value = nextValue()
    }
}

Before we can fill in the todo comment, we need somewhere to put the destination state. We’re going to make a separate ObservableObject which will contain several pieces of state we need in various places for the animation. But for now we just need somewhere to stick the destinations.

class MatchedGeometryState: ObservableObject {
    @Published var destinations: [AnyHashable: (AnyView, CGRect)] = [:]
}

Back in the destination view modifier, we’ll get the state object from the environment, and update the destinations dictionary in an on-change modifier:

struct MatchedGeometryDestinationModifier<Matched: View>: ViewModifier {
    let id: AnyHashable
    let matched: Matched
    @EnvironmentObject private var state: MatchedGeometryState
    
    func body(content: Content) -> some View {
        content
        	.background(GeometryReader { proxy in
                Color.clear
                	.preference(key: MatchedGeometryDestinationFrameKey.self, value: proxy.frame(in: .global))
                    .onPreferenceChange(MatchedGeometryDestinationFrameKey.self) { newValue in
						if let newValue {
                            state.destinations[id] = (AnyView(matched), newValue)
                        }
                    }
            })
    }
}

Getting the object from the environment is all well and good, but we still need to create it and inject it. For the animation, we’ll want access to some of the state properties in the source modifier, so the object needs to be created somewhere on the source side, rather than being owned by the container view controller. Since the presentation modifier needs to be at a higher level in the view tree than the source modifiers (which means it serves as the “namespace” for the transition, like the Namespace.ID parameter of SwiftUI’s effect), this is a natural place to put it.

struct MatchedGeometryPresentationModifier: ViewModifier {
    @Binding var isPresented: Bool
    @StateObject private var state = MatchedGeometryState()
    
    func body(content: Content) -> some View {
        content
        	.environmentObject(state)
        	// ...
    }
    
    // ...
}

We also pass it into the VC’s constructor in the makeVC function, and then in the constructore it’s stored in a state property on the VC (code omitted for brevity^[3]).

But we’re not done yet. We still need actual content to present, which is where the destination modifier will be used, so we’ll add another property to the presentation modifier and update our View extension function:

extension View {
    func matchedGeometryPresentation<Presented: View>(isPresented: Binding<Bool>, @ViewBuilder presenting: () -> Presented) -> some View {
        self.modifier(MatchedGeometryPresentationModifier(isPresented: isPresented, presented: presenting()))
    }
}
struct MatchedGeometryPresentationModifier<Presented: View>: ViewModifier {
    @Binding var isPresented: Bool
    let presented: Presented
    @StateObject private var state = MatchedGeometryState()
    // ...
}

The presented view also gets passed into the VC, and we’ll make it generic over the view type—to avoid any more type-erasing then we strictly need. We’ll also add another hosting controller which will display the presented content.

class MatchedGeometryViewController<Content: View>: UIViewController {
    let sources: [AnyHashable: (AnyView, CGRect)]
    let content: Content
    let state: MatchedGeometryState
    var contentHost: UIHostingController<ContentContainerView<Content>>!
    var matchedHost: UIHostingController<MatchedContainerView>!
    // ...
}

And we’ll setup the new hosting controller in viewDidLoad much the same as the other one, but this time providing the content:

class MatchedGeometryViewController<Content: View>: UIViewController {
    // ...
    override func viewDidLoad() {
        // ...
        let contentContainer = ContentContainerView(content: content, state: state)
        contentHost = UIHostingController(rootView: contentContainer)
        contentHost.view.autoresizingMask = [.flexibleWidth, .flexibleHeight]
        contentHost.view.frame = view.bounds
        addChild(contentHost)
        view.addSubview(contentHost.view)
        contentHost.didMove(toParent: self)
    }
}

The ContentContainerView is very simple: it just displays the content and provides access to our state object through the environment—finally bringing us full-circle to how the destination modifier will access it.

struct ContentContainerView<Content: View>: View {
    let content: Content
    let state: MatchedGeometryState
    
    var body: some View {
        content
        	.environmentObject(state)
    }
}

Now, if we tweak our test code to add some content to the presentation, we can see that

// ...
.matchedGeometryPresentation(isPresented: $presented) {
    VStack {
        Image("pranked")
        	.resizable()
        	.aspectRatio(contentMode: .fit)
        	.matchedGeometryDestination(id: "image")
        
        Text("Hello!")
	}
}

But it doesn’t look quite right yet. We are displaying all the layers, but we’re always displaying all the layers—which is why we’ve got two copies of the image visible. To fix that, we’ll need to build:

## The Animation

With everything in place, let’s actually put the pieces together and build the animation. First off, we’ll set the modal presentation style and transitioning delegate so we can completely control the presentation animation.

class MatchedGeometryViewController<Content: View>: UIViewController, UIViewControllerTransitioningDelegate {
    init(sources: [AnyHashable: (AnyView, CGRect)], content: Content, state: MatchedGeometryState) {
        // ...
        modalPresentationStyle = .custom
        transitioningDelegate = self
    }
}

We’ll also make the VC conform to the transitioning delegate protocol. Ordinarily, I’d declare this conformance in an extension, but Swift classes with generic types cannot declare @objc protocol conformances in extensions, only on the class itself.

The methods we’ll implement for this protocol are all simple: the actual work is offloaded to other classes. For the presenting animation controller, we’ll just instantiate another class.

func animationController(forPresented presented: UIViewController, presenting: UIViewController, source: UIViewController) -> UIViewControllerAnimatedTransitioning? {
	return MatchedGeometryPresentationAnimationController<Content>()
}

Note that the animation controller class is generic over the same content view type as the container VC. This is so that, in the implementation of the animation, we can cast the UIViewController we’re given to the concrete type of the VC that we know is being presented.

In the animation controller class, the transitionDuration method will just return 1 second. We’re actually going to use a spring animation, so this duration isn’t quite accurate, but we need something to go here, and this is the duration we’ll use when configuring the animator, even if the timing is ultimately driven by the spring.

class MatchedGeometryPresentationAnimationController<Content: View>: NSObject, UIViewControllerAnimatedTransitioning {
    func transitionDuration(using transitionContext: UIViewControllerContextTransitioning?) -> TimeInterval {
        return 1
    }
    func animateTransition(using transitionContext: UIViewControllerContextTransitioning) {
        // TODO
    }
}

All the actual work will happen inside the animateTransition method. The first thing we need to do is pull out our view controller (using the .to key, because it’s the destination of the presentation animation) and add it to the transition container view.

func animateTransition(using transitionContext: UIViewControllerContextTransitioning) {
    let matchedGeomVC = transitionContext.viewController(forKey: .to) as! MatchedGeometryViewController<Content>
    let container = transitionContext.containerView
    
    container.addSubview(matchedGeomVC.view)
}

Next, let’s get the fade-in working for the non-matched content. We start of by setting that layer’s opacity to 0, making it completely transparent. Then we set up a UIViewPropertyAnimator with the same spring configuration we’re going to ultimately use for the matched geometry effect. That animator will set the layer’s opacity to 1, bringing it fully opaque. And when the animation completes, we need to inform the transition context.

func animateTransition(using transitionContext: UIViewControllerContextTransitioning) {
    // ...
    
    matchedGeomVC.contentHost.view.layer.opacity = 0
    let spring = UISpringTimingParameters(mass: 1, stiffness: 150, damping: 15, initialVelocity: .zero)
    let animator = UIViewPropertyAnimator(duration: self.transitionDuration(using: transitionContext), timingParameters: spring)
    animator.addAnimations {
        matchedGeomVC.contentHost.view.layer.opacity = 1
    }
    animator.addCompletion { _ in
        transitionContext.completeTransition(true)
    }
}

We can ignore the position parameter the animation completion handler receives, because we’re never going to stop or cancel the animation.

Testing the animation now, you can see that the non-matched content does indeed fade in. But there’s still an issue, even aside from the fact that we’re not handling the matched geometry effect yet. Specifically, the source and destination views are visible during the animation, which breaks the illusion that a single view is moving seamless from one place to another.

To take care of this, we’ll add another property to the state object. It will store whether the animation is currently taken place.

class MatchedGeometryState: ObservableObject {
    // ...
    @Published var animating: Bool = false
}

The source and destination modifiers can then use this to make their respective wrapped views entirely transparent during the animation.

struct MatchedGeometrySourceModifier<Matched: View>: ViewModifier {
    // ...
    @EnvironmentObject private var state: MatchedGeometryState
    func body(content: Content) -> some View {
        // ...
        	.opacity(state.animating ? 0 : 1)
    }
}
struct MatchedGeometryDestinationModifier<Matched: View>: ViewModifier {
    // ...
    func body(content: Content) -> some View {
        // ...
        	.opacity(state.animating ? 0 : 1)
    }
}

Then the animation controller can set the animating state to true when the animation starts and back to false when it ends.

func animateTransition(using transitionContext: UIViewControllerContextTransitioning) {
    // ...
    matchedGeomVC.state.animating = false
    // ...
    animator.addCompletion { _ in
        transitionContext.completeTransition(true)
        matchedGeomVC.state.animating = false
    }
    animator.startAnimating()
}

Alright, now we can move on to the meat and potatoes of actually matching the views.

The first step is adding another property to the state object. currentFrames will store, well, the current frames of each matched view. Initially, it will contain the source frames, and then setting it to the destination frames will trigger the SwiftUI side of the animation.

class MatchedGeometryState: ObservableObject {
    // ...
    @Published var currentFrames: [AnyHashable: CGRect] = [:]
    @Published var mode: Mode = .presenting
    
    enum Mode {
        case presenting, dismissing
    }
}

Then we can update the matched container view to use the current frame, rather than always using the source one. The other change we’re going to make to the matched views is to blend between the source and destination matched views.

This handles the possibility that there are visual differences between the two, beyond their sizes and positions. We’ll perform this blending by fading in the matched view from the destination over top of the matched view from the source^[4]. By keeping them at the same position, we can ensure it still appears to be a single view that’s changing.

We also need a property for the mode—whether we’re presenting or dismissing—since this fade depends on the direction of the animation. When presenting, the destination view’s opacity should go from 0→1, going from transparent to fully opaque, but when dismissing it should go the other way around.

In the MatchedContainerView, we’ll split creating the actual matched view into a separate function, since there’s a fair bit of logic that goes into it:

struct MatchedContainerView: View {
    // ...
    var body: some View {
        ZStack {
            ForEach(sources, id: \.id) { (id, view, _) in
            	matchedView(id: id, source: view)
            }
        }
    }
}

The matchedView function will acually generate this view

struct MatchedContainerView: View {
	// ...
    func matchedView(id: AnyHashable, source: AnyView) -> some View {
		let frame = state.currentFrames[id]!
        let dest = state.destinations[id]!.0
        let destOpacity: Double
        if case .presenting = state.mode {
            destOpacity = state.animating ? 1 : 0
        } else {
            destOpacity = state.animating ? 0 : 1
        }
        return ZStack {
            source
            dest
            	.opacity(destOpacity)
        }
        .frame(width: frame.width, height: frame.height)
	    .position(x: frame.midX, y: frame.midY)
        .ignoresSafeArea()
        .animation(.interpolatingSpring(mass: 1, stiffness: 150, damping: 15, initialVelocity: 0), value: frame)
    }
}

This function gets the current frame for the view from the state object based on its ID. It also looks up the corresponding destination view. They’re layered together in a ZStack with an opacity modifier applied to the destination, so that it fades in on top of the source.

Note that we only apply the opacity to the destination view. If the opacity animation also applied to the source view (albeit in reverse), the animation would pass through intermediate states where both views are partially transparent, resulting in the background content being visible through the matched view, which can look strange.

The frame and position modifiers also move to the stack, so that they apply to both the source and destination views. We also add an animation modifier to the stack, using a spring animation and making sure to match the configuration to the UISpringTimingParameters we used for the view controller animation.

Next, we need to return to the UIKit animation controller, since we have yet to actually kick off the matched geometry animation.

This was the trickiest part to figure out when I was building this, since we need to wait for the destination SwiftUI view tree to update and layout in order actually know where all the destinations are. If we were to just kick off the animation immediately, there would be no destination views/frames, and the force unwraps above would fail. What’s more, providing a default for the animation and swapping out the real destination frame while the animation is in-flight ends up looking rather janky, since the spring animation has a fair bit of momentum. So, the destination needs to fully laid out before we can even start the SwiftUI side of the animation.

Since SwiftUI doesn’t seem to provide any way of forcing the view tree to update and fully layout immediately, we have to set up some listener that will fire once the destination is ready. Since we already have an ObservableObject containing the destination info in a @Published property, we can subscribe to that with Combine.

func animationTransition(using transitionContext: UIViewControllerContextTransitioning) {
    // ...
    container.addSubview(matchedGeomVC.view)
    
    let cancellable = matchedGeomVC.state.$destinations
    	.filter { destinations in
            matchedGeomVC.sources.allSatisfy { source in
            	destinations.keys.contains(source.key)
            }
		}
    	.first()
    	.sink { _ in
            
        }
    
    matchedGeomVC.view.layer.opacity = 0
    // ...
    animator.addCompletion { _ in
		// ...
		cancellable.cancel()
		matchedGeomVC.matchedHost?.view.isHidden = true
	}
    animator.startAnimation()
}

We use the .filter to wait until the state’s destinations dictionary contains entries for all of the sources. We also use the .first() operator, since we only want the sink closure to fire once—triggering the animation multiple times would mess up the animation and the state tracking we’re doing.

Subscribing to the publisher creates an AnyCancellable which we cancel in the animator’s completion handler—both because, if the destinations never became available, we don’t want to keep listening, and because we need to keep the cancellable alive for the duration of the animation (otherwise it would be deinitialized when the animateTransition method returns, cancelling our subscription).

When the publisher fires and all the state is present, we need to do a few things:

1. Add the matched hosting controller (as noted before, we can no longer do this immediately, since all the requisite state for the MatchedContainerView isn’t available initially).
2. Prepare the initial state of all the properties that are involved in the animation.
3. Set the new values to start the animation.

This takes care of the first two points, and is fairly straightforward:

.sink { _ in
    matchedGeomVC.addMatchedHostingController()

    matchedGeomVC.state.mode = .presenting
    matchedGeomVC.state.currentFrames = matchedGeomVC.sources.mapValues(\.1)
}

Just make sure to factor the code for adding the matched hosting controller out of viewDidLoad and into a separate method.

The third point is slightly trickier. We can’t just immediately set the new values, even in a withAnimation closure. Setting the new values needs to take place in a new transaction, once the view has already been updated with the intiial values. So, we wait one runloop iteration and then set the new values to kick off the animation.

.sink { _ in
	// ...
    DispatchQueue.main.async {
        matchedGeomVC.state.animating = true
        matchedGeomVC.state.currentFrames = matchedGeomVC.state.destinations.mapValues(\.1)
	}
}

And with that, the presentation animation is finally complete! We can present a SwiftUI view fullscreen, and have certain parts of the source view smoothly animate to their positions in the destination view, with everything else fading in in between the source and destination views.

(The hitch towards the end is an artifact of the simulator screen recording, it’s not actually present in the simulator or on-device.)

## Dismiss Animation

The dismiss animation is implemented in a very similar manner, so I won’t go over it in detail.

There’s another animation controller, which is returned from animationController(forDismissed:) which does pretty much the same thing as the presentation animation but in reverse. It sets the state to .dismissing, and the current frames to the destination frames. Then, one runloop iteration later, it sets animating = true and switches the current frames to the source frames.

Unlike the presentation animation, the dismiss one doesn’t need the workaround with the $destinations publisher, since when the dismiss is happening, we know that the presented view must already be all setup and laid-out.

The only slight wrinkle the dismiss animation needs is that, in order to work with the UIViewController presentation abstraction we built, there also needs to be a presentation controller that notifies its delegate when the dismiss transition begins:

class MatchedGeometryViewController<Content: View>: UIViewController, UIViewControllerTransitioningDelegate {
	// ...
    func presentationController(forPresented presented: UIViewController, presenting: UIViewController?, source: UIViewController) -> UIPresentationController? {
        return MatchedGeometryPresentationController(presentedViewController: presented, presenting: presenting)
    }
}

class MatchedGeometryPresentationController: UIPresentationController {
    override func dismissalTransitionWillBegin() {
        super.dismissalTransitionWillBegin()
        delegate?.presentationControllerWillDismiss?(self)
    }
}

## Future Work

There are a few loose ends I have deliberately left as an exercise for you, dear reader.^{(definitely not because I didn’t feel like spending the time to work through them)}

1. Recreating the properties and anchor parameters of matchedGeometryEffect

I chose not to implement these because I don’t have any need for them, but given how we’re displaying and positioning the matched views, it should be pretty clear how you could go about adding similar functionality.

2. Handling device rotation

If the user’s device is rotated while the matched VC is presented, the source frames will become invalid. There are a couple ways you could approach this. One is figuring out why the source frames aren’t updated while the VC is presented. Or, if that doesn’t have an easy fix, detecting when the window size changes while the VC is presented and then using a different animation for the dismissal.

3. Conditional matched views

Because of the way we’ve implemented the $destinations publisher workaround, if there are source views that never end up getting a destination view, the entire matched part of the animation will just never run. This is rather less than ideal, particularly if there are views that you want to match that may not always be present in the destination.

## Conclusion

Overall, I’m very happy with how this implementaiton turned out. I won’t claim it’s straightforward, but I think it’s relatively un-hacky for what it’s doing and has been very reliable in my testing. And, if you’ve got the latest Tusker release, you’re already running this code.

One of the more common feature requests for Mastodon (third only, I’d guess, to quote posts and search) is the ability to migrate accounts with their posts. Without this ability, people get trapped on the first instance they try^[2]. Maybe they’d prefer to be on a different instance, closer to a particular community, or somewhere with different moderation standards. But all their posts are stuck right where they are, and it ends up taking a huge, calamitous event—like an instance shutting down—to overcome that inertia and actually move.

So what’s standing in the way, and why aren’t posts portable already? Well, here’s an (abriged) example of the ActivityPub representation of a Mastodon post:

{
  "@context": ["https://www.w3.org/ns/activitystreams"],
  "id": "https://mastodon.social/users/shadowfacts/statuses/15287",
  "type": "Note",
  "content": "<p>Hello, world!</p>"
}

Notice anything about it? The ID of post tells you where it is. However it doesn’t just identify where the document can be found; it also identifies where it’s hosted. This property is true of all object identifiers in Mastodon (and just about every other serivce that implements ActivityPub).

It’s as if the cannonical URL of this post were https://5.161.136.163/2023/activitypub-portable-identity/. All of a sudden, this post isn’t controlled by me, the person who holds the domain shadowfacts.net, but instead it’s controlled by my server host, who controls that IP address. I couldn’t change hosting providers without losing all of my content and my social graph^[3] (in this example, the readers of my blog). Sound familiar?

It doesn’t have to be like this. My posts don’t belong to the service I’m using to host them. They belong to me. The identity of the posts I published should be tied to me and not to the service I’m using to host them. That is, the IDs need to be resolvable through something (hint: a domain) that I control before ultimately reaching my host.

Nothing about the ActivityPub spec, as I read it, prohibits this. §3.1 says that object identifiers need to be publicly dereferencable. It notes that their authority (i.e., domain and port) has to belong to the originating server—but there’s nothing to suggest that “server” is distinct from a domain, or that the server-to-host mapping can’t be many-to-one.

To make this vision of portable identities to work, there needs to be an additional layer that performs the mapping of identities to hosts.

For people who own domains, this is easy: they just point the DNS records at their host and tell their host that their identity should be served from their domain. Migrating your identity to another host is then a matter of updating DNS records.

But this shouldn’t be accessible only to those who have the resources and technical know-how to administer a domain^[4]. There needs to be some public identity resolver that anyone can use. The simplest way to implement this, I think, is as a service (or services, there’s no reason this has to be a centralizing force) that lets anyone point a subdomain at their host. The identity resolver needs to have a way of mapping any URL back to its host and I think this is easiest (avoiding the need to agree on URL formats) and least resource-intensive method (because you’re not hosting actual content). Having a separate service that’s built on top of DNS lets you abstract away the techinical details from people who don’t care, and would allow a more straightforward signup experience by providing an API to let people create an identity while signing up for an instance, rather than forcing it to be a separate process.

The hard part however, is the social one: we collectively need to agree that the identity resolution layer is infrastructure and not somewhere moderation actions should take place. To use the web analogy again: people publishing objectionable content get kicked off web hosts, but it’s far rarer that their domain name is taken away. The same has to be true of any shared identity provider/resolver, otherwise we end up in the exact same situation we’re in now.

Adding to the social problem is the risk that vanity URLs become popular, and then the desire shifts to migrating between identity resolvers. Solving this problem is possible, although it ends up requiring a larger architectural change to existing ActivityPub implementations. One possible approach, using DIDs, is discussed below. Another approach is splitting the identity resolver from the apparent username. A username that’s written as @someone@example.com could be looked up by using WebFinger on example.com which would then respond with the actual AP ID of the user as https://someone.net which would in turn be hosted by example.com (i.e., DNS records for someone.net point to example.com). This breaks a number of assumptions Mastodon and clients make about the mention format but it is architecturally possible. I do think this is a problem worth solving, but in the interest of supporting account portability in terms of what people usually care about—moderation decisions, servers going down—it isn’t a requirement.

The key advantage of this approach is that resolving a portable identity or object ID into a concrete ActivityPub object is no different than it is now. You just make an HTTP request, and get back an AP object. Where it’s actually hosted, and therefore changing the host, does not matter to you. All your references to the migrated people and posts do not change.

## UX

So under this vision, what does the complete migration process actually look like?

1. You export an archive of your account from your current instance.
2. You import that archive into your new account.
3. You point your identity at the new server (either by changing your DNS records, or by updating your information in the resolver’s directory described before).

In some ways, that’s actually simpler than the current migration process. You don’t have to remember to make your new account point back at your old account before initiating the migration. You don’t have to deal with the distinction between data that’s part of the direct server-to-server migration (your followers) and what’s not (who you follow). You don’t have to retry the migration process because some of your followers’ servers missed the notice and didn’t follow your new account. From everyone else’s point of view, the migration just seamlessly happens.

## Getting There

If you look at Mastodon issue #12432 “Support Post Migration”, you will find an incredibly long thread spanning three and a half years of discussion. The discussion primarily comes down to the fact that Mastodon’s architectural decisions are not conducive to this approach.

As noted before, all of Mastodon’s AP identifiers are scoped to the host, not to the user’s identity. Transferring posts to another server would mean changing those identifiers and breaking every existing reference to a post—both on the old instance and, more complicatedly, every other instance in the network that’s aware of them. The former is possible, if potentially resource-intensive. But the latter is simply not: unless an instance has authorized fetch (off by default) enabled from day 0 and tracks every instance that requests every post (Mastodon does not), there is no way to know who has a copy of a post that’s moved. After an account migration, there will be stale links. As a result, the discussion around adding post migration has centered on moving posts and just accepting that links will be broken. Maybe just having the posts there on your new account is enough?

I am not familiar enough with the Mastodon codebase to say whether moving to the vision I’ve outlined is feasible. If it’s not, I think Mastodon should continue to pursue alternate methods—if only because of the sheer number of users and the absolute clarity that this is a feature people want. But I think it’s apparent that the approach I’ve outlined here would be a more complete and transparent migration process.

There is, to the best of my knowledge, only one single ActivityPub project that supports multiple domains: Takahē. Multiple accounts across different domains being backed by the same host doesn’t get us all the way to portable identity. But the architectural decisions required to support it go a long way towards that vision. I have not taken the time to trawl through the code and work out if it’s actually using the domain to look up AP objects in its database or if it, like Mastodon and others, is still just extracting the database ID from a path component and using that for the lookup. Either way, by virtue of supporting multiple domains already, I think Takahē is much closer to reaching this vision.

Update: There is an AP implementation, Bovine, which stores and identifies AP objects by their AP identifier, which goes a long way towards making this implementation of portable identity possible.

Migrating existing accounts and content to portable identities, though, is an open and much harder question. The identifier for just about every AP object that’s already out there specifies the host. Making those objects portable would mean either making existing domains identity resolvers rather than hosts (not feasible) or getting every instance to update all of its references to the moved objects (basically where we’re at now, and it’s not feasible either). This is the biggest question regarding this approach, and I readily admit that I do not have a good answer to it.

### Technical Miscellanea

What follows isn’t anything cohesive, just some thoughts that occurred while writing this post and thinking about the architecture I’ve described.

#### So what’s the primary key in my database?

Well, it can still be whatever you want. And the ActivityPub identifier URL can still be derived therefrom. The key is just that looking up an object uses the entire URL, treating it as an opaque identifier rather than trying to parse it and pull out pieces.

#### What data actually gets migrated?

Essentially every Activity that you generated on your old host.

Ideally, you’d also want to migrate any relevant activities from other people (think likes/reblogs/replies), in order to preserve the complete context. This can’t be done just by transferring the activities themselves, since that could let importers forge activities from other people. So, what’s actually transferred would need to be a list of IDs of all the relevant activities. The new instance can then dereference them and add them to its databse. This could be a massive collection, and so this part of the import should probably be throttled and done in the background.

Attachments are another wrinkle, given Mastodon’s approach of transcoding everything that’s uploaded. The simplest approach, I think, is that the import process shouldn’t transcode anything unless it exceeds the usual instance-defined limits. In the common case (migrating from one instance to another using the same software) no transcoding or conversion should be necessary.

#### How do you keep my old instance from impersonating me?

ActivityPub actors already have public/private keypairs and any activities delivered to other servers have to be signed with your private key, which is then validated by the recipients. As part of the migration, the new host can regenerate your keys, so anything the old instance forges and tries to publish as you will fail validation.

#### Why not DIDs?

ATProto uses DIDs, rather than URIs, for identifiers. DIDs seem interesting, if quite complicated. The requirement of the ActivityPub spec that identifier URIs’ authorities belong to “their originating server” does not seem to preclude using DIDs as AP identifiers. The primary advantage DIDs confer is that they let you migrate between not just hosts/PDS’s but usernames: he same underlying DID can be updated to refer to @my.domain from @someone.bsky.social.

This does solve the caveat mentioned earlier, that the shared identity resolver has to be treated as infrastructure and be above moderation decisions. But, if the goal is to move the existing ecosystem towards portable identity in a reasonably expendient manner—and I believe that is the goal—adopting DIDs in the short term is unnecessary.

#### Moderating Actions Against Hosts

A very good point brought up in reply to this post was that since, right now, a domain/host/instance are all one and the same, they serve as a very useful target for moderation actions, but portable identity seems to interfere with that. If the moderators of a certain instance condone bad behavior from one person, another instance can take action against that entire instance, rather than just the individual, on the reasonable assumption the moderators will permit similar behavior from other people. But adding the layer of indirection I described makes it much harder to take such actions. Where as now it’s clear that @alice@example.com and @bob@example.com are hosted at the same place, if they used their own domains—say, @alice@alices.place and @bob@bob.online—it’s no longer self-evident that they’re hosted, and thus moderated, at the same place.

First off, I think that portable identity inherently makes this sort of moderation against the host less useful. Making it easier for everyone to up and move hosts definitionally makes it easier for bad actors to do the same. Taking moderation action against example.com has less utility if it’s easy for everyone hosted there to relocate.

But, that said, I think this model is still possible—and for one method we need only look to email, where identities (email addresses) are already separate from hosts (SMTP servers). Hosts could require that inbound activities specify the host that they originate from, not just the actor, and then reject activities from blocked domains. To prevent the originating host from being spoofed, the host would sign the message with a private key and then the recipient would then validate the signature against the originating host’s public key looked up from DNS.

The ideal is something that SwiftUI gets right: the environment. You put any values you want in, you can read them at any point in the view hierarchy and changes automatically propagate down. Unfortunately, UIKit has no similar mechanism. Or does it?

UIKit does have a way of defining colors that react to certain changes in their environment—specifically, anything in the trait collection. This works by providing a closure to UIColor which the framework runs when it needs the concrete color. But it’s just an ordinary closure, so why can’t you base the decision on something else, such as the user’s preferences?

If you did so, you could just use your dynamic colors everywhere, and they’d use the right colors based on the selected theme. And when the user’s preferences changed, you’d just need to tell the system the trait collection changed, and it would handle re-resolving all the dynamic colors.

The word “just” there is doing a lot of heavy lifting, though. I initially went spelunking through UIKitCore to try and find a way of forcing a dynamic color update, and found a promising lead. There’s a very attractively named -[UITraitCollection hasDifferentColorAppearanceComparedToTraitCollection:] method, which sounded exactly like what I was after. Alas, swizzling it was to no avail. The code path the system used when the appearance changed seem to go straight to an internal, non-Objective-C (and thus, not swizzlable) function __UITraitCollectionTraitChangesAlteredEffectiveColorAppearance.

But, in doing all this, I found another approach that seemed even better. UITraitCollection has a _clientDefinedTraits dictionary. And, as the name implies, this goes a long way towards the ideal of the SwiftUI environment in UIKit.

To be clear: this has significant drawbacks. It’s relying on private API that could change at any time. I’m only comfortable doing so because I’m careful to catch Objective-C exceptions whenever I’m accessing it and I’m only using it for a small feature: a preference for switching between the regular iOS pure-black dark mode style, and a non-pure-black one. The failure mode is that the app is still in dark mode, but slightly darker than intended. If themes were a more central aspect of my app, I wouldn’t want to rely on this.

With a simple extension on UITraitCollection, you can make it look like any other property:

extension UITraitCollection {
    var pureBlackDarkMode: Bool {
        get {
            (value(forKey: "_clientDefinedTraits") as? [String: Any])?["tusker_usePureBlackDarkMode"] as? Bool ?? true
        }
        set {
            var dict = value(forKey: "_clientDefinedTraits") as? [String: Any] ?? [:]
            dict["tusker_usePureBlackDarkMode"] = newValue
            setValue(dict, forKey: "_clientDefinedTraits")
        }
    }
    
    convenience init(pureBlackDarkMode: Bool) {
        self.init()
        self.pureBlackDarkMode = pureBlackDarkMode
    }
}

And a dynamic color can read it just like any other trait:

extension UIColor {
    static let appBackground = UIColor { traitCollection in
        if case .dark = traitCollection.userInterfaceStyle,
           !traitCollection.pureBlackDarkMode {
            return UIColor(hue: 230/360, saturation: 23/100, brightness: 10/100, alpha: 1)
        } else {
            return .systemBackground
        }
    }
}

Applying it is fairly straightforward: you just use setOverrideTraitCollection(_:forChild:) on a container view controller that contains the app UI. The only wrinkle is that this means the custom trait doesn’t propagate to modally-presented view controllers. Presentation is always performed by the window’s root VC, and so it doesn’t inherit the child’s traits.

But fret not, for another little bit of private API saves the day: UIWindow._rootPresentationController is actually responsible for the presentation, and since it’s a UIPresentationController, you can use the regular, public overrideTraitCollection API. When setting up the window, or when the user’s preference changes, adjust the override collection and it will apply to presented VCs.

if let rootPresentationController = window.value(forKey: "_rootPresentationController") as? UIPresentationController {
    rootPresentationController.overrideTraitCollection = UITraitCollection(...)
}

I’ll end by reiterating that this is all a giant hack and echoing Christian’s sentiment that hopefully iOS 16 17 will introduce a proper way of doing this.

Update: Hell yeah

There are unfortunately few options for parsing and syntax highlighting Swift from Rust code. There does exist a Swift grammar for Tree Sitter, which I use for the rest of the new version of my blog, but using it is incredibly slow—just parsing the highlight query took upwards of 5 seconds. What’s more, it doesn’t produce especially good highlighting results. Since Swift accounts for a substantial portion of what I write about on here, that wasn’t tenable.

The best Swift highlighter I know of is John Sundell’s Splash library. But, since it’s written in Swift, using it from my new blog engine is rather more complicated than ideal.

There is an existing package called swift-rs, however it doesn’t work for this use case. It relies in no small part on Objective-C and its runtime for transferring values across the FFI boundary. My blog is hosted on a Linux machine, where no such runtime is present, so just using that crate is a no-go.

First off is the Swift package. This can be fairly simple, since it just exposes a single function annotated with @_cdecl. The only complication is that, since the Swift function is exposed to C, its type signature must be expressible in C. That means no complex Swift types, like String. But since the goal of this is syntax highlighting, passing a string across the FFI boundary is integral. So, strings are passed as a pointer to the underlying byte buffer and a length.

@_cdecl("highlight_swift")
public func highlight(codePtr: UnsafePointer<UInt8>, codeLen: UInt64, htmlLenPtr: UnsafeMutablePointer<UInt64>) -> UnsafeMutablePointer<UInt8> {
}

Reading the input is accomplished by turning the base pointer and length into a buffer pointer, turning that into a Data, and finally into a String. Unfortunately, there are no zero-copy initializers^[3], so this always copies its input. Being in a Rust mindset, I really wanted to get rid of this copy, but there doesn’t seem to be an obvious way, and at the end of the day, it’s not actually a problem.

let buf = UnsafeBufferPointer(start: codePtr, count: Int(codeLen))
let data = Data(buffer: buf)
let code = String(data: data, encoding: .utf8)!

You may notice that the code string length is being passed into the function as an unsigned 64-bit integer, and then being converted to an Int, which may not be capable of representing the value. But, since this is just a syntax highlighter for my blog, there’s absolutely no chance of it ever being used to highlight a string longer than 2⁶³-1 bytes.

The actual highlighting I’ll skip, you can refer to the documentation for Splash. Once that’s done, though, the output needs to be sent back to Rust somehow. Again, since the function signature needs to be compatible with C, it returns a pointer to a byte buffer containing the UTF-8 encoded string. It also sets a length pointer provided by the caller to the length in bytes of the output.

var html = // ...
let outPtr = UnsafeMutableBufferPointer<UInt8>.allocate(capacity: html.utf8.count)
_ = html.withUTF8 { buf in
    buf.copyBytes(to: outPtr, count: buf.count)
}
htmlLenPtr.pointee = UInt64(outPtr.count)
return outPtr.baseAddress!

Note that the html string is declared as variable, since withUTF8 may mutate it if the backing storage is not already contiguous.

The Swift code allocates a buffer of the appropriate length, and copies the data into it. By itself, this would leak, but the Rust code on the other side of the FFI boundary will take ownership of it and then deallocate it as usual when the string is dropped.

Invoking the Swift function on the Rust is is fairly straightforward. First the external function needs to be declared:

extern "C" {
    fn highlight_swift(code_ptr: *const u8, code_len: u64, html_len_ptr: *mut u64) -> *mut u8;
}

Then, there’s a little wrapper function that provides a more Rust-y interface, rather than the actual client having to deal with raw pointers:

pub fn highlight(code: &str) -> String {
    unsafe {
        let mut html_len: u64 = 0;
        let html_ptr = highlight_swift(code.as_ptr(), code.len() as u64, &mut html_len);
        String::from_raw_parts(html_ptr, html_len as usize, html_len as usize)
    }
}

String::from_raw_parts takes the base pointer, the length, and the buffer capacity and produces an owned String that uses that buffer as its storage, with the given length and capacity. This does require that the buffer be managed by the same allocator as Rust’s global one, but here everything is using the system malloc, so it’s safe.

Next, comes actually building this thing. First, the Swift package product needs to be changed to be a static library:

let package = Package(
    // ...
    products: [
        .library(
            name: "highlight-swift",
            type: .static,
            targets: ["highlight-swift"]
        )
    ],
    // ...
)

Then, comes a whole bunch of stuff in the build.rs script of the Rust wrapper crate. Before I get into it, I want to note that much of this is based off the work of the swift-rs project.

First comes a bunch of stuff to get the macOS build working. This part is cribbed from swift-rs, albeit simplified to only do exactly what I need. It needs to emit instructions for the Rust compiler to link against the Swift standard library as well as compile the Swift package and link against it too.

fn main() {
    link_swift();
    link_swift_package("highlight-swift", "./highlight-swift/");
}

fn link_swift() {
    let swift_target_info = get_swift_target_info();

    swift_target_info
        .paths
        .runtime_library_paths
        .iter()
        .for_each(|path| {
            println!("cargo:rustc-link-search=native={}", path);
        });
}

fn link_swift_package(package_name: &str, package_root: &str) {
    let profile = env::var("PROFILE").unwrap();

    if !Command::new("swift")
        .args(&["build", "-c", &profile])
        .current_dir(package_root)
        .status()
        .unwrap()
        .success()
    {
        panic!("Failed to compile swift package {}", package_name);
    }

    let swift_target_info = get_swift_target_info();
    
    println!("cargo:rustc-link-search=native={}.build/{}/{}", package_root, swift_target_info.unversioned_triple, profile);
    println!("cargo:rustc-link-lib=static={}", package_name);
}

This relies on another supporting function from swift-rs, get_swift_target_info, which parses the output of swift -print-target-info to get information about the current target and location of the Swift stdlib. Note that this also requires serde and serde_json to be added to the [build-dependencies] section of Cargo.toml.

fn get_swift_target_info() -> SwiftTarget {
    let swift_target_info_str = Command::new("swift")
        .args(&["-print-target-info"])
        .output()
        .unwrap()
        .stdout;
    serde_json::from_slice(&swift_target_info_str).unwrap()
}

#[derive(Deserialize)]
struct SwiftTarget {
    target: SwiftTargetInfo,
    paths: SwiftPaths,
}

#[derive(Deserialize)]
struct SwiftTargetInfo {
    unversioned_triple: String,
    #[serde(rename = "librariesRequireRPath")]
    libraries_require_rpath: bool,
}

#[derive(Deserialize)]
struct SwiftPaths {
    runtime_library_paths: Vec<String>,
    runtime_resource_path: String,
}

This is enough to get everything to compile, link, and run on macOS. But unsurprisingly, running on Linux won’t yet work. The macOS dynamic linker, dyld, is doing a lot of heavy lifting: at runtime it will dynamically link in the Swift standard library and make sure the Swift runtime is initialized. But the same is not the case on Linux, so the build script also needs to tell Rust which libraries to link against.

First, the Swift standard library paths need to be included in the Rust target’s rpath, so that the dynamic libraries there will be located at runtime.

fn main() {
    // ...
    let target = get_swift_target_info();
    if target.target.unversioned_triple.contains("linux") {
        target.paths.runtime_library_paths.iter().for_each(|path| {
            println!("cargo:rustc-link-arg=-Wl,-rpath={}", path);
        });
    }
}

Then come the instructions to link against the various components of the Swift standard library. Not all of these are likely necessary, but rather than try to figure out which were, I opted to just link against all of of the .sos in the /usr/lib/swift/linux directory inside the Swift install.

fn main() {
    // ...
    if target.target.unversioned_triple.contains("linux") {
        // ...
        println!("cargo:rustc-link-lib=dylib=BlocksRuntime");
        println!("cargo:rustc-link-lib=dylib=dispatch");
        // etc.
    }
}

Finally, the swiftrt.o object file, which actually initializes the runtime (needed for just about any non-trivial Swift code).

fn main() {
    // ...
    if target.target.unversioned_triple.contains("linux") {
        // ...
        let arch = env::var("CARGO_CFG_TARGET_ARCH").unwrawp();
        println!("cargo:rustc-link-arg={}/linux/{}/swiftrt.o", &target.paths.runtime_resource_path, arch);
    }
}

I’m doing this by just passing the object file as a rustc link argument, since the linker will interpret it properly, but I’m not sure if there’s a better way of telling rustc I want this object file included. Unfortunately, this method means that swiftrt.o isn’t included when this crate is compiled into another one (and nor is the rpath linker flag used by the outer binary). So, in my actual blog crate, there’s some very similar code.

And, at long last, that is enough to run the program on Linux, with the Rust code calling into Swift and getting the result back as expected. If you want to see the entire project, it can be found on my Gitea.

I am very excited to announce that after almost four and a half years of development, Tusker, my iOS app for Mastodon is now available on the App Store!

If you follow my blog or follow me on the fediverse, you’ve no doubt heard me talk at length about it before, so I’ll spare you the details here. Suffice it to say that Tusker is a completely native iOS app that supports many of the latest features of both Mastodon and iOS.

This is not the end of development, there are still lots of features I plan to add—including push notifications and the ability to edit posts. If you’re already in the beta via TestFlight, you’re more than welcome to remain there. It will continue to get beta updates with new features and bugfixes ahead of the App Store releases.

The fundamental architecture of my site is unchanged from the last rewrite. All of the HTML pages are generated up front and written to disk. The HTTP server can then handle any ActivityPub-specific requests and fall back to serving files straight from disk.

i look forward to finishing this rewrite and then being able to sit back and enjoy... *checks notes* the exact same website i had before

Because this project was undertaken with the deliberate goal of using Rust more, I let myself spend more time bikeshedding and working on pieces that I ordinarily would have ignored or left to 3rd party libraries. One of those was spending probably too much time writing a bunch of code to slugify post titles—that is, turning titles with lots of punctuation and things into a nice and URL-safe format. It handles a bunch of pet peeves I have when I look at URLs on other websites (e.g., non-ASCII characters geting blindly replaced resulting in long sequences of hyphens), even if those are highly unlikely to ever arise here.

Another component I spent a great deal of time working on was the Markdown to HTML rendering. I’m using the pulldown-cmark crate, which handles a great deal for me, but not quite everything. In the previous implementation of my blog, I was using the markdown-it package which has some other little niceties to make the generated HTML better, in addition to the custom plugins I was using for the Markdown decorations you see if you’re reading this on my blog itself. I have custom code for handling the link and heading decorations, as before. I also override how footnote definitions are generated, to make them all appear at the end of the HTML rather appearing in the same locations they’re defined in the Markdown. As part of the changes to footnote definitions, I also generate backlinks which go from the footnote back to where it was referenced, to make reading them a bit easier.

## Syntax Highlighting

The previous, Node.js implementation of my blog used highlight.js for syntax highlighting. This works decently well, but it doesn’t produce the most accurate highlighting since it’s essentially a big pile of regexes rather than parsing the syntax. Now, I’m using Tree Sitter which actually parses the language and so highlighting ends up being more accurate.

Unfortunately, software being what it is, this is not always the case. The syntax highlighting for Swift with the best available Tree Sitter grammar is substantially worse than the highlight.js results. It routinely misses keywords, misinterprets variables as functions, and—to top all that off—something about how the highlight query is structured is incredibly slow for Tree Sitter to load. It more than doubles the time it takes to generate my blog, from about 0.5 seconds when skipping Swift highlighting to 1.3s when using tree-sitter-swift.

So, because I’ve never met a problem I couldn’t yak-shave, I decided the solution was to use John Sundell’s Splash library for highlighting Swift snippets. A number of Swift blogs I follow use it, and it seems to produce very good results. But, of course, it’s written in Swift, so I need some way of accessing it from Rust. This was a little bit of an ordeal, and it ended up being very easy, then very difficult, then easy, then confusing, and finally not too bad. The details of how exactly everything works and what I went through are a subject for another time, but if you want to see how it works, the source code for the Rust/Swift bridge is available here.

## ActivityPub

One of the big features from the last time I rewrote my blog was the ActivityPub integration. Almost nobody uses it, but I think it’s a cool feature and so I wanted to keep it. I’m using the activitystreams crate, and what I’ve learned is that statically typed languages are maybe not so great for writing AP implementations. Lots of properties can be multiple different types, so you can’t directly read anything, you have to check that it is in fact the type you expect. This does make for a more correct implementation, but it ends up being a big pain in the ass.

ActivityPub support was undoubtedly the part of this project that took the most time. Just about all of the static generator was complete within a couple weeks. The AP support dragged on over the next 6 months because it was so unpleasant (both for the aforementioned reasons, and just that dealing with all the quirks of different AP implementations is a pain).

But, now that it’s all done, I’m pretty happy with where it is. The ActivityPub support is mostly unchanged. Blog posts are still AP Article objects that you can interact with and comment on, and you can still follow the blog AP actor from Mastodon/etc. to get new posts to show up in your home feed. I did make a couple small quality-of-life changes:

1. If you reply to a blog post with a non-public post, you’ll get an automated reply back telling you that’s not supported. The purpose of replying to a blog post is to make comments show up, and I don’t want to display things that people didn’t intend to be public. If you want to send a private comment, you can message me directly, rather than the blog itself.
2. When there are new comments, the blog will automatically send me a notification (via AP, of course) with a digest of new posts. Previously, I had to manually check if there were any comments (if you ever commented and I never noticed it, sorry).

## Misc

The only other notable change is the addition of the TV section, which is an archive of the various long-running commentary threads I’ve written on Mastodon.

This post isn’t going to be a detailed guide or anything, just a collection of some mildly interesting things I learned.

Before I started this, I was already aware that Live Activity updates had a 4KB limit on the size of the dynamic state. So, the first thing I worked on was encoding the video into a smaller format. Because this particular video is black and white, I encoded it as 1 bit-per-pixel. Each frame is also complete, so the widget doesn’t need access to any previous frames to display the current one.

Since I wanted to display it in the Dynamic Island, the video is also scaled down to be very small, 60 × 45 pixels—one eighth the resolution of the original. This is a convenient size because each row of pixels can be encoded as a single UInt64. The entire frame comes out to 45 × 8 = 360 bytes, which is plenty small.^[1]

The whole video is encoded when the app starts up, which takes about 8 seconds. That’s faster than real time (the video is 3m39s), so it could be done during playback, but doing it ahead of time is fast enough that I didn’t feel like putting in the work to optimize a shitpost.

The widget can then unpack the encoded frame into a bitmap that can be turned into an image.

Adding the Live Activity wasn’t difficult—the API is wonderfully straightforward—but, alas, updating it was not so.

While the app was in the foreground, ActivityKit would log a message whenever I asked it to update the activity. But, when the app went into the background, those messages were no longer logged—even though my code was still running and requesting updates. Interestingly, the app is considered to be in the foreground if you open Notification Center while in the app, and so the activity can be updated, which is how this demo came about:

I scratched my head at the issue of background updates for a while, and tried a couple things to no avail, until I attached Console.app to my phone and filtered for “activity”. At which point, I saw a bunch of messages like these from sessionkitd (which is the system daemon that manages live activities):

Apps playing background audio seem to be completely forbidden from updating Live Activities. The only possible reason for this I can imagine is to prevent apps from making their own now playing activities, rather than relying on the system one. I don’t know why this is the case, but whatever, back to trying to find workarounds.

At this point, I downloaded the most recent iOS 16 IPSW for the iPhone 14 Pro. After extracting the dyld shared cache from the image (using this helpful tool) I started poking around to try and find the code responsible for deciding if an app is allowed to update its activities. Opening the dyld shared cache in Hopper and searching for “session” revealed several promising-sounding frameworks: SessionCore, SessionFoundation, and SessionKit.

SessionCore turned out to be the one containing that log message. But unfortunately, it’s written in Swift and I am not good enough at reverse engineering to decipher what it’s actually doing. I did, however, manage to find a couple tidbits just be looking through the strings in the binary:

1. An entitlement named com.apple.private.sessionkit.backgroundAudioUpdater

It wasn’t of much use to me, but if someone ever manages to jailbreak these devices, you could have some fun with it.

2. A log message reading “Process looks like a navigation app and can update activity”

This looked more promising because the phrasing “looks like” suggests to me that it’s just using a heuristic, rather than determining what the app is for certain. I tried to trick it by adding entitlements for MapKit and location, tracking the user’s location while in the background, and trying various audio session categories that seemed more map-app like. But this effort was to no avail, sessionkitd still forbade me from updating the activity in the background.

At this point, I gave up on getting audio working and just settled for playing the video in the Dynamic Island. I had to scrap the previous implementation of detecting new frames because it used AVPlayer and is therefore incompatible with updating in the background. But, since I have an array of frames, playing it back myself is as simple as using a timer to emit a new one every 1/30th of a second.

With that, I was finally able to play back the video in the island:

You may notice that in both attempts the video appears somewhat blurry. This is becuase iOS animates any changes to the Live Activity view. As with all widgets, the SwiftUI view tree is serialized and then deserialized and displayed in a separate process, so you don’t have direct control over the animation. There is a new ContentTransition API that the Live Activity docs say should work, but unfortunately the identity transition “which indicates that content changes should’t animate” has no effect on the activity.

Just having the video in the island was pretty satisfying to see. But, I still really wanted to see the video playing in the island with the audio.

Zhuowei suggested using the system Music app to play back the sound while my app controlled the video. This worked, though it means the app is no longer entirely self-contained. You can use the MPMusicPlayerController.systemMusicPlayer to control the system Music app, and playing back a particular track is simple enough:

let player = MPMusicPlayerController.systemMusicPlayer
let query = MPMediaQuery.songs()
query.addFilterPredicate(MPMediaPropertyPredicate(value: "badapple", forProperty: MPMediaItemPropertyTitle))
player.setQueue(with: query)
do {
	try await player.prepareToPlay()
	player.play()
} catch {
	// if the song doesn't exist, ignore it
}

Annoyingly, the only way of getting the track into the Music app on my phone was by disabling iCloud Music Library and syncing it from my Mac. Why iCloud needs to be disabled to let you manually sync files, I do not know—especially seeing as the manually-synced tracks remain on the device even after iCloud is turned back on.

And this, it turns out, is impossible to record because apparently the Music app mutes itself during screen recordings (using the builtin Control Center method, or attaching to QuickTime on a Mac). So, you’ll just have to take my word for it.

This was a fun, if utterly pointless, endeavor. If you want to see the code or run it yourself, it’s available here.

Using LiveView Native lets avoid duplicating business logic on the frontend and save time on implementing dedicated APIs for native apps. The iOS client can be integrated into any existing app by using a single SwiftUI view, so it’s easy to adopt it for just a single screen at a time.

You can find the documentation^[1] for the Swift package here, including a step-by-step tutorial which walks you through building a complete app with LiveView Native.

We’ve also developed a simple chat app which was used by the attendees of ElixirConf this year, and serves as a complete example of a LiveView Native app.

I’m very excited to see what people build with it.

(A tribute^[1] to the seemingly ended webshit weekly series from n-gate.)

### To uncover a deepfake video call, ask the caller to turn sideways

#### August 8, 2022 (comments)

Metaphysic (a company that seeks to “empower individuals” with artificial intelligence “content generation” (read: plagiarism laundering) tools) determines that relatively few images of people in big datasets are from a completely in profile. As such, deepfake tools have poor results when compositing someone’s face onto a profile view of a subject. Hackernews helpfully notes that this is merely a present limitation of deepfake and face alignment neural networks, and that within a few short years, these dark days of deepfake detectability will be behind us. Other Hackernews propose a series of solutions that will surely not be defeated by further advancements in deepfake tools.

### An incident impacting 5M accounts and private information on Twitter

#### August 9, 2022 (comments)

Twitter (business model: “Uber for bad takes”) informs the public that a flaw in their code let anyone discover which account, if any, a particular email address or phone number belonged to. Again. They assure everyone that they take privacy Very Seriously and that only they are allowed to use that information for illicit purposes. One Hackernews considers the possibility of enumerating all 10 billion phone numbers. A number of Hackernews also note that Twitter helpfully victim blames suggests their users simply not use a publicly-known phone number to protect against the company’s incompetence.

### Instagram can track anything you do on any website in their in-app browser

#### August 10, 2022 (comments)

Meta™ (business model: “Uber for antitrust complaints”) decides that not only are they entitled to write down everything you do on their websites apps metaspaces, but they are also entitled to spy on you whenever you try to leave them. Hackernews are confused about why Meta™ doesn’t simply use the APIs provided by Apple (business model: “Uber for UI frameworks”) for showing web views without spyware. A conversation ensues about whether Apple should neuter in-app, faux-Safari browsers and further clamp down on the already nigh nonexistent 3rd-party browser ecosystem (another topic of frequent consternation). Another sub-thread raises the alarm that in-app web browsers allow access to *shock*, *awe* The Internet.

### A 17-year-old designed a novel synchronous reluctance motor

#### August 11, 2022 (comments)

A high school student improves upon an electric motor design that doesn’t use rare-earth magnets. Hackernews bitterly resents that they weren’t child prodigies and tries to nitpick the student’s work into meaningless-ness. Other Hackernews conclude that maybe the kids are alright.

### Arrest of suspected developr of Tornado Cash

#### August 12, 2022 (comments)

The Dutch Fiscal Information and Investigation Service (business model: “Uber for stopping financial crimes”) arrests a man believed to be the developer of the Ethereum tumbler Tornado Cash (business model: “Uber for committing financial crimes”). Hackernews is very concerned about their future prospects if all of a sudden governments are arresting people who build software designed to let people commit crimes. One subthread devolves into arguments about gun rights in the United States (business model: “Uber for racial inequality”) and others into general fearmongering about the end of privacy as we know it.

### I hacked my car

#### August 13, 2022 (comments)

In which the author finds a series of vulnerabilities that should be embarassing for a company with a 36B USD market cap, culminating in finding the private key used to sign their car’s firmware on the internet—an engineer having evidently reused it from a tutorial. At over 3000 words, most Hackernews can’t be bothered with reading it and, as such, the comments are a barren wasteland. Hackernews mostly has complaints about their own cars. Another Hackernews does a casual racism and slights the engineering ability of an entire continent (but it’s definitely okay because he’s personally had bad experiences with those people).

### Oasis: Small statically-linked Linux system

#### August 14, 2022 (comments)

Some developers have come up with a Linux (business model: “Uber for FOSS dweebs”) distribution that will be even more annoying to use than the usual musl-based ones. Half of Hackernews rails against dynamic linking and the other half rails against static linking. Compromise is on no one’s mind; this can only end in war. Only one Hackernews is excited about any other potential merit of the project (namely that it boots a few seconds faster than their current distro of choice).

The first part of my implementation that needed to change is how I get which link is being tapped. I have a function called getLinkAtPoint(_:) that takes a CGPoint in the coordinate space of the view and tries to find the link at that point. To update it, almost the entire old body of the function is wrapped in an if statement that checks if TextKit 2 is available:

func getLinkAtPoint(_ point: CGPoint) -> (URL, NSRange)? {
	let pointInTextContainer = CGPoint(x: point.x - textContainerInset.left, y: point.y - textContainerInset.top)
	if #available(iOS 16.0, *),
	   let textLayoutManager = self.textLayoutManager {
		// ...
	} else {
		// ...
	}
}

Note that I fall back to the TextKit 1 path if the app’s not running on iOS 16 or the NSTextLayoutManager is not available. Even on iOS 16, a text view may still fall back to TextKit 1 if the old API is used—in which case, the TextKit 2 stack may be swapped out for TextKit 1. In my testing this can happen occasionally even if you’re never using the TextKit 1 API yourself, meaning something in the framework is accessing it (though this may simply be a beta bug).

When TextKit 2 is available, there are several steps we need to go through to get the attributes at a point.

First, we get the text layout fragment that the layout manager has for the point in the coordinate space of the text container. The documentation is sparse, but in my testing, layout fragments correspond to paragraphs.

guard let fragment = textLayoutManager.textLayoutFragment(for: pointInTextContainer) else {
	return nil
}

From there, we get the line fragment (corresponding to a visual line of text) that contains our point. To get the line fragment, there is no builtin helper method, so we just go through the layout fragment’s textLineFragments array until we find the one that matches. For each line fragment, we check if its typographic bounds contain the target point converted to the layout fragment’s coordinate space.

let pointInLayoutFragment = CGPoint(x: pointInTextContainer.x - fragment.layoutFragmentFrame.minX, y: pointInTextContainer.y - fragment.layoutFragmentFrame.minY)
guard let let lineFragment = fragment.textLineFragments.first(where: { lineFragment in
		  lineFragment.typographicBounds.contains(pointInLayoutFragment)
	 }) else {
	return nil
}

If there’s no matching layout or line fragment, that means the given location is on a piece of text and therefore there’s no link, so the method returns nil.

After that, we can get the tapped character index by using the characterIndex(for:) method with the target point converted to the line fragment’s coordinate space.

let pointInLineFragment = CGPoint(x: pointInLayoutFragment.x - lineFragment.typographicBounds.minX, y: pointInLayoutFragment.y - lineFragment.typographicBounds.minY)
let charIndex = lineFragment.characterIndex(for: pointInLineFragment)

And then we can use the line fragment’s attributed string to lookup the attribute:

var range = NSRange()
guard let link = lineFragment.attributedString.attribute(.link, at: charIndex, longestEffectiveRange: &range, in: lineFragment.attributedString.fullRange) as? URL else {
	return nil
}

let textLayoutFragmentStart = textLayoutManager.offset(from: textLayoutManager.documentRange.location, to: fragment.rangeInElement.location)
let rangeInSelf = NSRange(location: range.location + textLayoutFragmentStart, length: range.length)
return (link, rangeInSelf)

One important thing to note is that the line fragment’s attributedString property is an entirely separate string from the text view’s atttributed string. So the return value of characterIndex and the longest effective range have indices into the substring. The rest of my code expects the return value to be a range in the index-space of the full string, so I need to convert it by adding the offset between the beginning of the document and the beginning of the line fragment’s substring.

For the legacy TextKit 1 path, I use the characterIndex(for:in:fractionOfDistanceBetweenInsertionPoints:) method on the layout manager to get the character index and then look up the attribute at that location. I won’t go into detail in that code here, since it’s more straightforward—and lots of other examples can be found online.

Next up: context menu previews. The vast majority of the code is unchanged, all that needs to be done is changing how we get the rects spanned by a range in the text.

In the contextMenuInteraction(_:previewForHighlightingMenuWithConfiguration:) method, rather than always using the TextKit 1 API, we again check if TextKit 2 is available, and if so, use that:

var textLineRects = [CGRect]()
if #available(iOS 16.0),
   let textLayoutManager = self.textLayoutManager {
	let contentManager = textLayoutManager.contentManager!
	guard let startLoc = contentManager.location(contentManager.documentRange.location, offsetBy: range.location),
	      let endLoc = contentManager.location(startLoc, offsetBy: range.length),
		  let textRange = NSTextRange(location: startLoc, end: endLoc) else {
		return nil
	}
	textLayoutManager.enumerateTextSegments(in: textRange, type: .standard, options: .rangeNotRequired) { _, rect, _, _ in
		textLineRects.append(rect)
		return true
	}
} else {
	let notFoundRange = NSRange(location: NSNotFound, length: 0)
	self.layoutManager.enumerateEnclosingRects(forGlyphRange: linkRange,
											   withinSelectedGlyphRange: notFoundRange,
											   in: self.textContainer) { (rect, stop) in
		textLineRects.append(rect)
	}
}

In the TextKit 2 path, we get the NSTextContentManager and force-unwrap it (as far as I can tell, this is never nil and is only optional because it’s weak).

We use the content manager to convert the NSRange of the link that’s being previewed into an NSTextRange (a range of NSTextLocations, which are opaque objects that represent locations in the document however the content manager sees fit). I’m not sure in what circumstances these calls could fail, but if any of them do, we return nil and let the framework use the default preview.

With that, we can call enumerateTextSegments to get the bounding rectangles of the text segments. Since these are in the coordinate space of the text layout manager, there’s nothing further we need to do, so we can just add them to the textLineRects array. And from the block, we return true to continue enumerating. One minor thing to note is that we can pass the .rangeNotRequired option to tell the framework to skip calculating text ranges for every segment since we don’t need them.

From there, the code is exactly the same as last time.

And with those changes in place, I can use my app without any warnings about text views falling back to TextKit 1 and the accompanying visual artifacts.

Did you know that with macOS Ventura, Clarus the Dogcow has at long last returned home? Recently, while doing something else, I accidentally hit Cmd+Shift+P which opened the Page Setup dialog. I was greeted, surprisingly, with a new high-resolution version of the classic Clarus icon that I’d never seen before. I looked at it briefly, and then closed the dialog and went back to whatever I was doing before. I had assumed that because I’d been in a 3rd-party app at the time, that the Clarus icon was just some easter egg the developer had left. But a little while later, I got to thinking. What were the chances that someone went to the trouble of customizing the Page Setup dialog, of all things, just for an easter egg? Zero, it turns out. That dialog shows Clarus on the page preview in every app.

I don’t have a Monterey machine to test it at the moment (I, er, accidentally updated my laptop to the beta), but I believe this is a new change with Ventura.

Update: I installed Monterey in a virtual machine to check, and, indeed, the Page Setup dialog there bears no sign of Clarus.

The next step, then—having been thoroughly nerd-sniped by this—was to figure out where the icon was coming from and if I could pull it out of whatever nook it was hidden in.

The first stop was NSPageLayout, the panel object that is responsible for displaying the panel. It was unlikely that the class would actually contain the implementation of the panel, but it was at least a starting point.

In order to actually look at the disassembled implementation of this AppKit class, I needed the actual AppKit framework binary. Since macOS Big Sur, all system framework binaries are stored merged in the dyld shared cache, rather than in separate files. But, I need them as separate files in order to actually inspect them.

Since the last time I wrote about this, a couple things have changed. Before, I built the Apple dyld_shared_cache_util from one of the periodic dyld source dumps. This is annoying because you have to make a bunch of changes to the source code to get it to compile outside of an Apple-internal environment. It also may break whenever there’s an OS update. So, I’ve switched to using this utility which uses the dyld_extractor.bundle that ships with Xcode. The other difference since before is a minor one: the dyld shared cache has moved. Wheras before it was in /System/Library/dyld/, in the Ventura beta it’s moved to /System/Cryptexes/OS/System/Library/dyld/ (the Cryptex seems to be part of the Rapid Security Response feature Apple announced).

With the shared cache extracted, I could load the AppKit binary into Hopper (I had to disable the Objective-C analysis, otherwise the app crashed when trying to load the binary) and start poking around. I searched for the NSPageLayout class that I’m interested in, and looked at the runModalWithPrintInfo: method, since that sounded like a good candidate for something that would lead to the bulk of the implementation. And, indeed, it was. The method appears to be a fairly simple wrapper around the PMPrepare... function that sounds like it lives in a separate private framework.

The next step was figuring out where that prepare function is actually implemented. Running otool -L on the AppKit binary doesn’t reveal anything obviously useful, but in the PrivateFrameworks directory extracted from the dyld shared cache, there’s something called PrintingPrivate.framework, which sounds promising. Opening it up in Hopper, I saw that this is indeed the framework I was looking for.

Looking at the implementation of the prepare function, what immediately jumps out is the call to _LoadAndGetPrintingUIBundle. This seems to be yet another layer of indirection with the actual thing implemented in a different bundle. There’s also a call in the else branch to the similarly-named _LoadAndGetPrintCocoaUIBundle, but let’s start with the first one in hopes that it’s more common.

The implementation of that function goes through another helper function and it ends up loading a PrintingUI.bundle plugin from inside the PrintingPrivate framework bundle. This one isn’t part of the dyld shared cache, so I can just open it right up in Hopper without any fuss.

If you look for the function PrintingPrivate calls, it turns out it winds up in a method on PMPageSetupController. This sounds promising, let’s see what else that class can do.

What’s this? A method called updateClarus? Could it be? Have we finally reached it?

Yes! Clarus, I’m coming! One method that sounds particularly encouraging is -[PMPageSetupController setClarusImageView:]. If I can find out what’s setting the image view, maybe that’ll lead to where it’s being configured with the image.

Unfortunately, the setter for that property isn’t referenced anywhere in the PrintingUI binary. Nor is the getter. I was stuck here for a while, until I realized that the setter not being called anywhere was probably a sign that the UI was defined in a Nib and that an outlet was added from Interface Builder, even though it was never used.

Sure enough, in the plugin bundle’s resources, there is a PMPageSetup.nib. And if the page setup UI is defined in a Nib, and Clarus is being shown in a image view, the image itself is probably located in the asset catalog.

Using the system assetutil program, one can list all of the files in a compiled asset catalog. And sure enough, there she is:

$ assetutil --info /System/Library/PrivateFrameworks/PrintingPrivate.framework/Versions/A/Plugins/PrintingUI.bundle/Contents/Resources/Assets.car | grep -i clarus
    "Name" : "Clarus",
    "RenditionName" : "ClarusSmooth2.pdf",
    "Name" : "Clarus",
    "RenditionName" : "ClarusSmooth2.pdf",
    "Name" : "Clarus",
    "RenditionName" : "ClarusSmooth2.pdf",

To actually extract the image from the asset catalog, I needed to use a third-party tool. acextract worked perfectly on the first try, though it did need couple of additional @imports to compile on Ventura since Foundation no-longer re-exports CoreGraphics.

And with that, I finally gazed upon the 512 × 512px beauty that is Smooth Clarus:

The version shown here I’ve added a white background to, so it’s not invisible in dark mode. The original image has a transparent background.

Lastly, if you’re writing a Mac app and would like to hide Clarus somewhere, you can load the bundle yourself and then pull the image out like so:

let bundle = Bundle(path: "/System/Library/PrivateFrameworks/PrintingPrivate.framework/Versions/A/Plugins/PrintingUI.bundle")!
try! bundle.loadAndReturnError()
let image = bundle.image(forResource: "Clarus")

I’m not sure if the Mac App Store would consider that using private SPI, so use it at your own risk.

It would be very cool to see Clarus return as an SF Symbol some day. If hundreds of icons for various Apple products can go in, so too can everyone’s favorite dogcow.

## Part 1: Type Theory is for Chumps

I spent a while thinking about what I wanted the type system to look like—I do want some level of static typing, I know that much—but it got to the point where I was tired of thinking about it and just wanted to get back to writing code. So, lo and behold, the world’s simplest type system:

#[derive(Debug, PartialEq, Clone, Copy)]
enum Type {
	Integer,
	Boolean,
	String,
}

impl Type {
	fn is_assignable_to(&self, other: &Type) -> bool {
		self == other
	}
}

Then, in the Context, rather than variables just being a map of names to Values, the map now stores VariableDecls:

struct VariableDecl {
	variable_type: Type,
	value: Value,
}

So variable declaration and lookup now goes through a simple helper in the function that creates the VariableDecl.

For now, types at variable declarations are optional at parse time since I haven’t touched type inference yet and I didn’t want to go back and update a bunch of unit tests. They are, however, inferred at evaluation time, if one wasn’t specified.

fn parse_statement<'a, I: Iterator<Item = &'a Token>>(it: &mut Peekable<'a, I>) -> Option<Statement> {
	// ...
	let node = match token {
		Token::Let => {
			let name: String;
			if let Some(Token::Identifier(s)) = it.peek() {
				name = s.clone();
				it.next();
			} else {
				panic!("expected identifier after let");
			}
			let mut variable_type = None;
			if let Some(Token::Colon) = it.peek() {
				it.next();
				variable_type = Some(parse_type().expect("type after colon in variable declaration"));
			}
			expect_token!(it, Equals, "equals in variable declaration");
			let value = parse_expression(it).expect("initial value in variable declaration");
			Some(Statement::Declare {
				name,
				variable_type,
				value,
			})
		}
		// ...
	};
	// ...
}

The parse_type function is super simple, so I won’t go over it—it just converts a the tokens for string/int/bool into their respective Types. I call expect on the result of that type and then again wrap it in a Some, which seems redundant, because if whatever followed the colon wasn’t a type, there’s a syntax error and I don’t want to continue.

Actually evaluating the variable declaration is still pretty straightforward, though it now checks that the type the initialization expression evaluated to matches the declared type:

fn eval_declare_variable(
    name: &str,
    mutable: bool,
    variable_type: &Option<Type>,
    value: &Node,
    context: &ContextRef,
) {
    let val = eval_expr(value, context);
    let variable_type = match variable_type {
        Some(declared) => {
            assert!(
                val.value_type().is_assignable_to(declared),
                "variable value type is not assignable to declared type"
            );
            *declared
        }
        None => val.value_type(),
    };
    context
        .borrow_mut()
        .declare_variable(name, mutable, variable_type, val);
}

## Part 2: Variable Variables

The other bit I added was mutable variables, so that I could write a small program that did something non-trivial.

To do this, I changed the VariableDecl struct I showed above to hold a ValueStorage rather than a Value directly.

ValueStorage is an enum with variants for mutable and immutable variables. Immutables variables simply own their Value. Mutable ones, though, wrap it in a RefCell so that it can be mutated.

enum ValueStorage {
	Immutable(Value),
	Mutable(RefCell<Value>),
}

Setting the value is straightforward, but getting them is a bit annoying because Value isn’t Copy, since it may own a string. So, there are a couple of helper functions: one to access the borrowed value and one to clone it.

impl ValueStorage {
    fn set(&self, value: Value) {
        match self {
            ValueStorage::Immutable(_) => panic!("cannot set immutable variable"),
            ValueStorage::Mutable(cell) => {
                *cell.borrow_mut() = value;
            }
        }
    }

    fn with_value<R, F: FnOnce(&Value) -> R>(&self, f: F) -> R {
        match self {
            ValueStorage::Immutable(val) => f(&val),
            ValueStorage::Mutable(cell) => f(&cell.borrow()),
        }
    }

    fn clone_value(&self) -> Value {
        self.with_value(|v| v.clone())
    }
}

This works, but isn’t ideal. At some point, the complex Value types should probably changed to reference-counted so, even if they’re still not copy-able, cloning doesn’t always involve an allocation.

Lexing and parsing I won’t go into detail on, since it’s trivial. There’s a new for var and whether a declaration starts with that or let controls the mutability.

Setting variables isn’t complicated either: when parsing a statement, if there’s an equals sign after an identifier, that turns into a SetVariable which is evaluated simply by calling the aforementioned set function on the ValueStorage for that variable.

And with that, I can write a little fibonacci program:

$ cat fib.toy
var a = 0
var b = 1
var i = 0
while (i < 10) {
	print("iteration: " + toString(i) + ", a: " + toString(a));
	let tmp = a
	a = b
	b = tmp + a
	i = i + 1
}

$ cargo r -- fib.toy
iteration: 0, a: 0
iteration: 1, a: 1
iteration: 2, a: 1
iteration: 3, a: 2
iteration: 4, a: 3
iteration: 5, a: 5
iteration: 6, a: 8
iteration: 7, a: 13
iteration: 8, a: 21
iteration: 9, a: 34

I also added a small CLI using structopt so I didn’t have to keep writing code inside a string in main.rs.

Ultimately I did take the route of turning my framework into a Swift Package. I revisited it because I noticed in another project that local packages inside the same folder as the main project worked perfectly fine. The only difference I found was that the project where it worked used only an .xcodeproj, whereas Tusker used an .xcworkspace. So, I deleted the (for unrelated reasons, no longer necessary) workspace and found that, after quitting and relaunching Xcode, the local package worked perfectly fine.

I briefly started writing a feedback report, but upon further testing I found that xcworkspaces in general weren’t the problem—a new project and workspace worked fine. So, I gave up trying to reproduce it and assumed there was just something weird about the 3.5 year old workspace.

## Installation

The installation process went very smoothly. The installer utility the Asahi team built functioned perfectly. The one pain point, however, was when shrinking the default partition to make room for the new Linux one, my whole machine locked up for about three minutes. No input was registered and the entire screen stopped updating. The installer does warn you about this beforehand, but it was kind of nerve-wracking nonetheless. After that, the installation went perfectly smoothly, and I’m now running the Asahi Linux Desktop (a modified version of Arch Linux) on my M1 Max MacBook Pro.

## State of Linux

Overall, Linux runs quite well natively on the M1 Max. Despite the alpha state, I haven’t (yet) run into any issues the Asahi team didn’t warn about.

So far, almost all of the software I’e tried to install has worked. The sole exception was Rust, as the prebuilt version in the Arch repo uses jemalloc, which does not support the 16K page size of the M1 family. But, building Rust from source^[1] got it working fine.

The biggest gap in Asahi right now is the lack of drivers for the chip’s GPU. This means the machine uses software rendering (llvmpipe), which is a great deal slower than proper GPU rendering would be. However, the M1 Max is fast enough for software rendering to be usable for most tasks. Even watching 1080p youtube in Firefox was fine, and glxgears runs at about 700 FPS. Although moving the cursor seems to be a smooth 60 FPS^[2], everything else appears to be capped at 30. This could be a configuration issue, but I don’t know enough to figure it out (and cursory googling doesn’t reveal anything helpful).

Another as-of-yet unsupported feature on Linux is CPU scaling, meaning all the CPU cores run at their full clock speed continuously, consuming all that extra power. I haven’t had very much time to use it on battery, but in spite of this, the battery life seems usable (probably no worse than my old Intel MBP).

This has been a fairly pleasant experience. Though, after I finish writing this post, I’ll be returning to macOS. A good deal of what I do is macOS-specific, not to mention that I enjoy having audio output.

The work of the Asahi team so far is incredibly impressive. What’s more, some of their efforts, such as the m1n1 bootloader, are going towards supporting other operating systems as well as Linux. Maybe I should try OpenBSD next…

Adding the package to the app went perfectly well. I added it in Xcode, set the framework and app to depend on it, and then got to building. Everything worked and ran perfectly normally. But, when the time came to publish a new build to TestFlight, the issues started appearing.

Upon uploading, App Store Connect returned an error telling me that the framework for the Swift Package I’d added wasn’t code signed. This was surprising for a couple reasons: first, Swift Packages are generally statically linked (meaning they’re compiled directly into the binary that uses them) rather than shipping as separate, dynamically-linked frameworks. Second, my Xcode project is setup to automatically handle code signing. Why would it be skipping the framework?

The answer is that it wasn’t. The framework for the framework for the package was getting signed perfectly fine. Just not the right one.

It seems having multiple targets that depend on a Swift package causes Xcode to dynamically link it. As with other frameworks, the framework for the package gets built and embedded in the Frameworks/ folder of the app that depends on it.

But the app isn’t the only thing that depends on it. The package framework was also getting embedded inside my framework before it was in turn being embedded in the app.

Tusker.app
└── Frameworks
   ├── Pachyderm.framework
   │  └── Frameworks
   │      └── WebURL.framework
   └── WebURL.framework

Xcode was properly signing the app’s frameworks, it was not signing the nested frameworks. Hence the App Store Connect error.

“No problem,” I naively thought, “I’ll just add a Run Script build phase to codesign the nested one myself.” Yes problem. Turns out App Store Connect entirely rejects nested frameworks, even if they’re correctly signed.

So, I changed the script to entirely delete the nested Frameworks directory (this is fine at runtime because the runpath search paths includes the top-level Frameworks dir), which finally convinced App Store Connect to accept my build.

if [ "$(ls "$BUILT_PRODUCTS_DIR/Tusker.app/Frameworks/Pachyderm.framework/Frameworks/")" -ne "WebURL.framework" ]; then
	echo "error: unexpected framework inside Pachyderm, make sure it's embedded directly in the app"
    exit 1
fi
rm -rf "$BUILT_PRODUCTS_DIR/Tusker.app/Frameworks/Pachyderm.framework/Frameworks/"

You might think that’s where this story ends, but, sadly, it’s not. I noticed when downloading the new TestFlight build that the app was up to 25 megabytes in size. That might not sound like much, but the previous version was around 5MB and I hadn’t added anything that should have caused a quintupling in size.

Looking at the archive and comparing it to a previous one, it was clear that almost all of the increase was coming from the package framework.

I’d previously tried out this Swift package in a small test app, so I went back to compare its size to the new version of Tusker. The test project was nowhere near as big—just two megabytes.

The sole relevant difference between the two projects, as far as I can tell, is whether the Swift package is linked statically or dynamically. My best guess for the app size difference is that when the package is linked dynamically, dead code elimination can’t happen across module boundaries. Anything declared as public—or used by something public—must be kept, because you don’t know which parts of it the ultimate consumer needs. When linked statically, public-but-unused code can be stripped, which can result in significant size savings if you’re not using the entire API surface of a package.

## Addendum

I tried two (2) methods for getting Xcode to statically link everything, in hopes of bringing the binary size back down.

The first method was changing my own framework from being, well, a framework to a Swift package. In theory, this should mean that everything gets statically linked into just the app binary. This should be straightforward, but I want the framework/package code to live in the same Git repo alongside the app, as no one else uses it and versioning it separately is a pain. Xcode… does not like this.

You can create a new package in the same repo as an existing project from Xcode no problem. While doing so, you can add it to the existing xcworkspace without objection. But when you try to add the package as a dependency of the app, it just fails silently.

I can click the “Add Local” button in the add package window, I can select the package directory, and click the add button. And then nothing happens. The package doesn’t show up in the “Swift Packages” tab of the project nor under the dependencies of the target to which I added it. So, compilation just fails because the module is missing.

After abandoning that idea, the other, similarly unsuccessful, tactic I tried was encapsulating all of the usages of the package’s types within my framework in an opaque type and removing the dependency on the package from the app target. This was in the hopes that the package would be statically linked into the framework and have all the unnecessary bits stripped.

That did not work. I don’t know why. There seems to be very little visibility (read: none at all) into how Xcode chooses to static versus dynamic linking for Swift packages.

That’s where I gave up, so if you have any better ideas, please let me know. At the end of the day, I don’t have the energy to spend more time fighting Xcode over 20 megabytes. Oh well. I should probably throw a report into the void that is Feedback Assistant.

Update: As of April 2022, I’ve resolved this issue.

Some time ago, a bug was filed against WebKit because setting scrollIndicatorStyle on a web view’s scroll view was broken on iOS 15. The fix for this bug landed in iOS 15.4 and it subtly changed the behavior of WKScrollView when it comes to the indicator style.

The bug was fixed by tracking whether the web view client has overriden the scroll indicator style and, if so, blocking the web view from resetting it internally. Unfortunately, it does this by checking if the new indicator style is not .default. So, even if you set it to .default to make it automatically switch based on system appearance, the scroll view will interpret that to mean the indicator style hasn’t been overriden and continue erroneously setting it based on background color (or, in my case, the non-opaqueness of the web view).

The solution is simple, if annoying. You need to check the current user interface style and select the appropriate scroll indicator style yourself.

override func viewWillAppear(_ animated: Bool) {
	super.viewWillAppear(animated)
	updateScrollIndicatorStyle()
}
override func traitCollectionDidChange(_ previousTraitCollection: UITraitCollection?) {
	super.traitCollectionDidChange(previousTraitCollection)
	updateScrollIndicatorStyle()
}
private func updateScrollIndicatorStyle() {
	guard #available(iOS 15.4, *) else {
		// different workaround pre-iOS 15.4
		return
	}
	if traitCollection.userInterfaceStyle == .dark {
		webView.scrollView.indicatorStyle = .white
	} else {
		webView.scrollView.indicatorStyle = .black
	}
}

And, if you, like me were previously using the old, swizzling workaround, you need to disable in on iOS 15.4. If the old workaround remains active, studiously setting the indicator style to .default whenever WebKit would override it, it would merely undo all of our hard work.

Getting a Rust library compiled into a form that it could be used from Swift didn’t turn out to be as complicated as I expected, but both Apple Silicon and Mac Catalyst introduced fun wrinkles.

This blog post from Mozilla was helpful in getting started, but things have changed somewhat in the years since it was written.

The first thing you need to do is install the appropriate targets for the Rust toolchain to let it build targeting iOS devices.

aarch64-apple-ios works for all actual devices. Additionally, to build for the iOS Simulator, you also need the aarch64-apple-ios-sim target (if you’re on Apple Silicon) or x86_64-apple-ios (for Intel Macs).

$ rustup target add aarch64-apple-ios aarch64-apple-ios-sim x86_64-apple-ios

To build a Rust project, you need a library crate with the crate-type set to staticlib so that it can be statically linked by the iOS app. The Rust library also needs an API that’s callable from C, i.e. using #[no_mangle] and extern "C" (outside the scope of this post). Fortunately for me, lol-html already includes such an API.

Building for iOS is done by running cargo build with the appropriate --target option. For example:

$ cargo build --release --target aarch64-apple-ios-sim

With the Rust library built, the next step is configuring Xcode to use it. In the app target’s build settings, the Header Search Paths needs to be updated to include the path to the C headers that correspond to the C-API that the Rust library exposes. In my case, that’s lol-html/c-api/include/.

That’ll get it working if you want to call it from C or Objective-C code in your project. To make it accessible from Swift, you need to add a bridging header that imports whatever library headers you need. For lol-html, there’s only lol_html.h. This will make Rust library’s functions directly accessible from all of the app target’s Swift files.

This is enough to compile it successfully, but to actually link the output into a runnable app, the we need to tell the linker where to find the library.

With a normal library, you could add the static library .a to the Xcode target’s “Frameworks, Libraries, and Embedded Content”. But, because Cargo puts the build products in separate directories depending on which platform it targets (e.g., target/aarch64-apple-ios/release/liblolhtml.a), we need to do it slightly differently. Just adding one of the liblolhtml.a’s to the Xcode target would make the linker try to always link against that specific one regardless of which platform the iOS app is building for. Instead, I modified the “Other Linker Flags” build setting to include -llolhtml and then update the “Library Search Paths” settings on a per-platform basis to tell it 1) that it needs to link against something called liblolhtml.a and 2) where exactly that file can be found.

Configuring the Library Search Paths build setting is kind of annoying, because the Xcode UI doesn’t fully match what the .pbxproj file can actually describe. Clicking the plus button next to a build setting in Xcode lets you pick for which SDKs the setting value applies. But we also need to narrow that down to specific architectures, because the Intel and Apple Silicon simulator builds need different versions of the library.

The easiest way I’ve found to do this is to go into the Build Settings tab of the Xcode target, find Library Search Paths, expand it, and click the little plus button next to each of Debug and Release. (If you click on the “Any Architecture | Any SDK” dropdown, you’ll see what I mean about not being able to actually specify the architecture from the UI.)

Then, open the project.pbxproj file in a text editor. I recommend closing the Xcode proejct before making any changes to this file. Search for the newly added line starting with "LIBRARY_SEARCH_PATHS[arch=*]" and replace it with the following. There will be two occurrences of that line (for the debug and releaes configurations) and both need to be replaced.

"LIBRARY_SEARCH_PATHS[sdk=iphoneos*]" = "$(PROJECT_DIR)/lol-html/c-api/target/aarch64-apple-ios/release/";
"LIBRARY_SEARCH_PATHS[sdk=iphonesimulator*][arch=arm64]" = "$(PROJECT_DIR)/lol-html/c-api/target/aarch64-apple-ios-sim/release/";
"LIBRARY_SEARCH_PATHS[sdk=iphonesimulator*][arch=x86_64]" = "$(PROJECT_DIR)/lol-html/c-api/target/x86_64-apple-ios/release/";

You’ll need to substitute the lol-html/c-api part for the actual path to the library you’re using. This will tell Xcode to to use the aarch64-apple-ios version for all actual iOS device targets, and the appropriate simulator version depending on the architecture.

After that, you should be able to re-open the project in Xcode and see all the configurations you added in Build Settings.

With that, you should be able to use the Rust library from your Swift code and successfully build and run your app in both the simulator and on a real device.

## Mac Catalyst

My first attempt at getting Catalyst builds to work was just by using the normal Mac targets for the Rust library (e.g., aarch64-apple-darwin). But that results in a link error when Xcode builds the app, because it considers binaries built for Catalyst to be distinct targets from regular macOS.

The separate Rust targets for Catalyst are aarch64-apple-ios-macabi and x86_64-apple-ios-macabi for ARM and Intel respectively. As of writing, these are tier 3 targets, which means the Rust project doesn’t provide official builds. This, in turn, means to use them you have to build the standard library from source yourself.

Doing so requires a Rust Nightly feature, build-std, to let Cargo include the standard library in the crate graph for compilation. So, with Nightly installed (rustup toolchain install nightly) and the std source downloaded (rustup component add rust-src --toolchain-nightly), you can run the following command to build for a specific target with the standard library built from source:

$ cargo +nightly build -Z build-std=std,panic_abort --release --target aarch64-apple-ios-macabi

This separate set of platform/arch combinations requires another set of additions to the Xcode project file, in the sample place as before:

"LIBRARY_SEARCH_PATHS[sdk=macosx*][arch=arm64]" = "$(PROJECT_DIR)/lol-html/c-api/target/aarch64-apple-ios-macabi/release";
"LIBRARY_SEARCH_PATHS[sdk=macosx*][arch=x86_64]" = "$(PROJECT_DIR)/lol-html/c-api/target/x86_64-apple-ios-macabi/release";

With that added, the build setting should have values configured for iOS, ARM and Intel Simulators, and ARM and Intel Catalyst:

I initially thought handling universal builds (i.e., combining arm64 and x86_64 into one binary) of the Catalyst app would be complicated, like I would have to lipo them together myself, but it turned out to be entirely painless. Just having the built static libraries for both architectures present in their expected locations is enough. Xcode’s build process takes care of linking each architecture of the app with the respective version of the Rust library and then combining those into one universal package.

## Build Script

Keeping track of all of those Rust build targets and making sure to rebuild the right ones if anything changes is rather annoying, so I wrote a little script for Xcode to run to take care of it.

It uses the environment variables provided by Xcode to figure out which platform and architecture(s) are being targeted and build the appropriate Rust targets.

pushd "$PROJECT_DIR/lol-html/c-api/"

build() {
    echo "Building lol-html for target: $1"

    ~/.cargo/bin/cargo build --release --target $1
}

build_std() {
    echo "Building lol-html with std for target: $1"
    
    ~/.cargo/bin/cargo +nightly build -Z build-std=panic_abort,std --release --target $1
}

if [ "$PLATFORM_NAME" == "iphonesimulator" ]; then
    if [ "$ARCHS" == "arm64" ]; then
        build "aarch64-apple-ios-sim"
    elif [ "$ARCHS" == "x86_64" ]; then
        build "x86_64-apple-ios"
    else
        echo "error: unknown value for \$ARCHS"
        exit 1
    fi
elif [ "$PLATFORM_NAME" == "iphoneos" ]; then
    build "aarch64-apple-ios"
elif [ "$PLATFORM_NAME" == "macosx" ]; then
    if grep -q "arm64" <<< "$ARCHS"; then
        build_std "aarch64-apple-ios-macabi"
    fi
    if grep -q "x86_64" <<< "$ARCHS"; then
        build_std "x86_64-apple-ios-macabi"
    fi
else
    echo "error: unknown value for \$PLATFORM_NAME"
    exit 1
fi

One thing to note is that when building the universal Mac target, $ARCHS has the value arm64 x86_64. So I check whether the string contains the target architecture, rather than strictly equals, and don’t use elif in the Mac branch so that both architectures are built.

I have it configured to not bother with any of the dependency analysis stuff, because Cargo takes care of only actually rebuilding if something’s changed and when nothing has, the time it takes is negligible so running on every incremental build is fine.

With the script added to Build Phases in Xcode (for some reason it needs to come not just before Link Binary with Libraries but also before Compile Sources), I can run cargo clean in the Rust project directory and then seamlessly build and run from Xcode.

First off: This article assumes you’re using MultiMC and already have the ARM Zulu 8 JDK installed^[1].

There are guides on the internet that tell you to download some precompiled libraries and run Minecraft through a wrapper script, but doing that doesn’t sit right with me. Didn’t your mother ever tell you not to download executables from strangers on the internet^[2]? So, I wanted to actually compile LWJGL 2 and its natives from source myself.

You can find the source code for my modified version of LWJGL 2 here (and you can compare against upstream to see I haven’t made any malicious changes).

If you’re interested, here’s a brief overview of the changes that were necessary. If not, skip ahead.

First, there are a bunch of changes to the Ant build.xml. javah was replaced with javac -h, source/target versions less than 1.7 aren’t supported when compiling with a Java 17 JDK. In the build.xml for the native library, the big changes are the macOS SDK being in a different place and that we’re compiling against the Zulu JVM, not the Apple one.

In MacOSXSysImplementation.java, a since-removed class was being used to ask the OS to open URLs. That’s been replaced with a JNI function (see org_lwjgl_MacOSXSysImplementation.m).

In MemoryUtilSun.java, a accessor implementation that was using a removed Sun-internal class was removed (it was only used as a fallback, so it wasn’t replaced with anything).

Some applet code that was using a removed Java archive format was commented out (irrelevant because the applet build is disabled altogether).

Lastly in Java-land, there were a bunch of casts from java.nio.ByteBuffer to Buffer added in places where methods ByteBuffer inherits from Buffer (e.g., flip) are being called. This is because, in between Java 8 and 17, ByteBuffer itself got overriden implementations and so trying to use a LWJGL jar built against Java 17 on Java 8 would result in a NoSuchMethodError.

In the native code, there are a few more changes.

First, an JAWT_MacOSXDrawingSurfaceInfo struct is defined. It’s used for the NSView-backed drawing mode that Minecraft uses on Mac. This was previously defined in the JDK headers, however it’s no longer present. An old version of the Apple JVM headers say that it is being removed, however it is clearly still implemented internally (fortuntely for us).

The other significant change is a bunch of AppKit calls being wrapped in dispatch_sync blocks since they were previously being called from background threads, which AppKit Does Not Like. This solution is rather less than ideal. I suspect dispatch_sync is part of the significant performance delta between LWJGL3 versions and earlier ones, because it blocks the render thread until the work on the main thread completes. I tried using the -XstartOnFirstThread JVM argument so that the Minecraft client thread would be the same as the AppKit main thread, however that only caused more problems.

Finally, high DPI mode is disabled altogether. I spent some time trying to get it working, but whenever I’d launch the game it would only render at 50% scale. And frankly, I don’t care enough about high DPI mode to spend even more time debugging it.

To get Minecraft up and running, there’s actually a second library that we also need to compile for arm64: jinput. My fork’s source code can be found here (compare against upstream).

The changes for this one are much simpler, thankfully. A bunch of stuff that we don’t need is disabled in the various maven files.

Then, one (1) unused import for a no-longer-extant Java stdlib class was removed.

Lastly, the build.xml for the macOS platform specific module was changed similar to the LWJGL one to make it use the macOS SDK in the correct location and link agains the Zulu 8 JDK rather than the system JavaVM framework.

## Building Everything

You’ll need Ant and Maven installed if you don’t have them already. You also need to have a current version of Xcode downloaded.

### LWJGL

1. Clone the repo: git clone https://github.com/shadowfacts/lwjgl-arm64.git
2. Run ant generate-all to generate some required Java files from templates.
3. Run ant jars to build the LWJGL jar
4. Run ant compile_native to build the native libraries
5. In the libs/macosx/ folder inside your LWJGL clone, select liblwjgl.dylib and openal.dylib in Finder and right-click and select Compress. Then rename the created Archive.zip to lwjgl-platform-natives-osx.jar

### jinput

1. Clone the repo: git clone https://github.com/shadowfacts/jinput-arm64.git
2. Run mvn package to build everything

## Setup MultiMC

In MultiMC, create an instance of whichever pre-LWJGL3 version you want and make sure it’s configured to use the ARM Java 8 you installed. Then, in the Version section of the Edit Instance window, click the Open Libraries button in the sidebar. To the folder that opens, copy the lwjgl-platform-natives-osx.jar you created earlier. Then, copy osx-plugin-2.0.10-SNAPSHOT-natives-osx.jar from inside the plugins/OSX/target folder of the jinput directory as well.

Next, in the Version section of the Edit Instance window, select LWJGL 2 and click the Customize and then Edit buttons in the sidebar.

The file that opens needs to be modified to point to local versions of the libraries we compiled. Replace its contents with the JSON below.

{
    "formatVersion": 1,
    "libraries": [
        {
            "name": "net.java.jinput:jinput-platform:2.0.10-SNAPSHOT",
            "natives": {
                "linux": "natives-linux",
                "osx": "natives-osx",
                "windows": "natives-windows"
            },
            "MMC-hint": "local",
			"MMC-filename": "osx-plugin-2.0.10-SNAPSHOT-natives-osx.jar"
        },
        {
            "downloads": {
                "artifact": {
                    "sha1": "39c7796b469a600f72380316f6b1f11db6c2c7c4",
                    "size": 208338,
                    "url": "https://libraries.minecraft.net/net/java/jinput/jinput/2.0.5/jinput-2.0.5.jar"
                }
            },
            "name": "net.java.jinput:jinput:2.0.5"
        },
        {
            "downloads": {
                "artifact": {
                    "sha1": "e12fe1fda814bd348c1579329c86943d2cd3c6a6",
                    "size": 7508,
                    "url": "https://libraries.minecraft.net/net/java/jutils/jutils/1.0.0/jutils-1.0.0.jar"
                }
            },
            "name": "net.java.jutils:jutils:1.0.0"
        },
        {
            "name": "org.lwjgl.lwjgl:lwjgl-platform:2.9.4-nightly-20150209",
            "natives": {
                "linux": "natives-linux",
                "osx": "natives-osx",
                "windows": "natives-windows"
            },
            "MMC-hint": "local",
			"MMC-filename": "lwjgl-platform-natives-osx.jar"
        },
        {
            "name": "org.lwjgl.lwjgl:lwjgl:2.9.4-nightly-20150209",
            "MMC-hint": "local",
            "MMC-filename": "lwjgl.jar"
        },
        {
            "downloads": {
                "artifact": {
                    "sha1": "d51a7c040a721d13efdfbd34f8b257b2df882ad0",
                    "size": 173887,
                    "url": "https://libraries.minecraft.net/org/lwjgl/lwjgl/lwjgl_util/2.9.4-nightly-20150209/lwjgl_util-2.9.4-nightly-20150209.jar"
                }
            },
            "name": "org.lwjgl.lwjgl:lwjgl_util:2.9.4-nightly-20150209"
        }
    ],
    "name": "LWJGL 2",
    "releaseTime": "2017-04-05T13:58:01+00:00",
    "type": "release",
    "uid": "org.lwjgl",
    "version": "2.9.4-nightly-20150209",
    "volatile": true
}

From there, you should be able to launch Minecraft natively on ARM:

Here’s a stupid bug I ran into recently: if you’ve got a WKWebView in an iOS app and it shows something that’s not a normal webpage^[1], the scroll indicator appearance switching doesn’t work. What I mean by that is, while the indicators appear correctly (with a dark color) when the system is in light mode, they do not take on a light color when dark mode is enabled. This renders the scroll indicators invisible against dark backgrounds, which can be annoying if you’re using the web view to display potentially lengthy content.

Let’s say you’ve got a web view with some content that you want to match the system color scheme. The simplest way to do that is to set the web view’s background color to .systemBackground and add the following to the page’s stylesheet (to make the default text color change with the theme):

:root {
    color-scheme: light dark;
}

If you haven’t changed the page’s background-color in CSS, this will correctly make both the background and text colors follow the system theme. Unfortunately, it has no effect on the scroll indicator. Fixing that, it turns out, is rather involved.

The obvious thing to do would be to just set the indicatorStyle to .default on the web view’s internal scroll view, which should theoretically make the indicators automatically adjust to the color of the content inside the scroll view. Try this, however, and you’ll find that it does not work.

Regardless of where you set the property (in viewDidLoad, viewWillAppear, etc.) it appears to never be respected. Launching the app and checking webView.scrollView.indicatorStyle when paused in the view debugger reveals why: it’s been changed to .black:

(lldb) po [(UIScrollView*)0x7fcc9e817000 indicatorStyle]
UIScrollViewIndicatorStyleBlack

Because WebKit—including, surprisingly, some the iOS platform-specific parts of it—is open source, we can go poking through it and try to figure out why this is happening. Looking for files with names like WKWebView in the WebKit source code reveals the promising-sounding WKWebViewIOS.mm. Searching in that file for indicatorStyle reveals this lovely method:

- (void)_updateScrollViewBackground
{
    auto newScrollViewBackgroundColor = scrollViewBackgroundColor(self, AllowPageBackgroundColorOverride::Yes);
    if (_scrollViewBackgroundColor != newScrollViewBackgroundColor) {
        _scrollViewBackgroundColor = newScrollViewBackgroundColor;

        auto uiBackgroundColor = adoptNS([[UIColor alloc] initWithCGColor:cachedCGColor(newScrollViewBackgroundColor)]);
        [_scrollView setBackgroundColor:uiBackgroundColor.get()];
    }

    // Update the indicator style based on the lightness/darkness of the background color.
    auto newPageBackgroundColor = scrollViewBackgroundColor(self, AllowPageBackgroundColorOverride::No);
    if (newPageBackgroundColor.lightness() <= .5f && newPageBackgroundColor.isVisible())
        [_scrollView setIndicatorStyle:UIScrollViewIndicatorStyleWhite];
    else
        [_scrollView setIndicatorStyle:UIScrollViewIndicatorStyleBlack];
}

“Update the indicator style based on the lightness/darkness of the background color.” Ah. That explains it. It’s examining the background color and using its lightness to explicitly set the scroll view’s indicator style to either white or black.

And why’s it overriding the .default that we set? Well, that method is called from (among other places) _didCommitLayerTree, which, as I understand it, is called whenever the remote WebKit process sends the in-process web view a new layer tree to display—so basically continuously.

Knowing that WebKit is using the page’s background color to set the indicator style, the answer seems simple, right? Just set the page’s background color in CSS to match the current color scheme. Wrong: setting background-color: black; on the body does not alter the scroll indicator color.

And why is that? It’s because scrollViewBackgroundColor ignores all other factors and returns a transparent black color if the web view is non-opaque. And since the isVisible method returns false for fully transparent colors, the scroll indicator style is being set to black.

Now, this is where the minor iOS crimes come in. If you can’t make the framework do what you want, change (read: swizzle) the framework.

So, in the didFinishLaunching app delegate callback, I swizzle the _updateScrollViewBackground method of WKWebView. My swizzled version calls the original implementation and then sets the scroll indicator mode back to .default, superseding whatever WebKit changed it to. And this finally makes the scroll indicator visible in dark mode.

private func swizzleWKWebView() {
	let selector = Selector(("_updateScrollViewBackground"))
	var originalIMP: IMP?
	let imp = imp_implementationWithBlock({ (self: WKWebView) in
		if let originalIMP = originalIMP {
			let original = unsafeBitCast(originalIMP, to: (@convention(c) (WKWebView, Selector) -> Void).self)
			original(self, selector)
		}
		
		self.scrollView.indicatorStyle = .default
		
	} as (@convention(block) (WKWebView) -> Void))
	originalIMP = class_replaceMethod(WKWebView.self, selector, imp, "v@:")
	if originalIMP == nil {
		os_log(.error, "Missing originalIMP for -[WKWebView _updateScrollViewBackground], did WebKit change?")
	}
}

A couple things to note about this code:

originalIMP, is optional because class_replaceMethod returns nil if the method does not already exist, though it still adds our new implementation. If this happens, I log a message because it probably means that WebKit has changed and hopefully this hack is no longer necessary (or that it may need updating).

The unsafeBitCast is necessary because an IMP is in fact a C function pointer but it’s imported into Swift as an OpaquePointer.

The Swift closure is cast to an Objective-C block because, although imp_implementationWithBlock is imported into Swift as taking a parameter of type Any, what it really needs, as the name implies, is a block.

The “v@:” string is the Objective-C type encoding of the method’s signature (it returns void, takes the self instance, and the selector for the method being called).

And with that annoying workaround, the scroll indicator is finally visible in dark mode (and updates correctly when switching color schemes). Given that WKWebView gives some consideration to the system color scheme, I’m inclined to think this is a bug, so hopefully it’ll be fixed eventually. Alternatively, as I noted, that part of WebKit is in fact open source, so an intrepid developer could fix it themself…

Before I get into the details, some context: My previous primary computer was a 16” Intel MacBook Pro (8 cores, 32 GB of memory, and a Radeon 5500M). I got it almost immediately when it came out in the fall of 2019, coming from the original 2012 15” Retina MacBook Pro. When I first got it, it was the best (for definitions of best relating to the specifics I care about, primarily CPU performance) laptop Apple had made. What’s more it was the first laptop to get rid of the accursed butterfly keyboard. I had zero complaints about it when I first got it, and for a good while afterwards.

But then, last winter, shortly after they came out, I used an M1 Mac mini (8-core, 16 GB RAM) for a couple months. For all I’d fawned over the 2019 MBP, the M1 Mac mini (as well as other people’s reviews of the corresponding laptops) gave me a newfound disappointment of all of my laptop’s shortcomings. Well, most of the issues I had actually stem from one particular shortcoming: the Intel processor. It was by no means a slouch, it was just pitifully constrained by the thermal design of the laptop. Any sustained workload would cause it to drop down to the base clock—though not throttle below, an improvement over previous generations—and leave a good deal of performance on the table (particularly when playing games, with the GPU also dumping heat into the shared cooler). The abysmal battery life and incredible noise and heat that went along with it didn’t do anything to help (a laptop whose processor routinely runs at 100°C cannot rightfully be called a laptop). The M1, by contrast, had none of those issues. In addition to having better single-core performance, it was almost comically more power-efficient. At the end of my M1 review, I said that I was incredibly excited to see what the future of the new Mac architecture held, and I was not disappointed.

## Hardware

First off, the most important part: SoC performance. This machine handily beats my old laptop in literally everything I do.

I could list a bunch of artificial benchmarks for you oooh and ahhh at, but that’s not generally representative of what I actually use it for.

A full release build of Tusker, my iOS app for Mastodon, takes about 94 seconds on my Intel laptop. It’s 44% faster on the M1 Max, taking 53 seconds. Debug builds with incremental compilation see a similar improvement. And, as was the case with the M1, where the single-core performance really shines is in reducing the feedback loop between making a code change and being able to see that reflected in the running app.

As with the Mac mini, everything just feels faster. No doubt some of that is a placebo, but not entirely. I can put my old and new laptops side by side and launch the same app, and the Apple Silicon one will appear on screen several seconds sooner. This snappiness extends to within apps too, especially ones using Catalyst which always felt a little bit unresponsive before.

The other incredible improvement Apple Silicon brings is the power and thermal efficiency. It’s not quite as miraculous as the M1 Mini, whose fans I could never get to spin up, but it’s still a vast improvement. I have to be pushing both the CPU and the GPU fairly hard (e.g., by playing a graphically intensive game) in order to get the fans to be audible. Ordinary, CPU-focused workloads don’t cause the fans to ramp up and barely make the laptop feel warm. I once accidentally left a script running that was consuming 100% CPU all day because the fans weren’t there to signal that something was amiss.

Compare that to the Intel MBP whose fans sound like they’re about to take flight if you so much as look at the machine wrong (seriously, the fans spun up to audible levels and the chassis felt burning hot while it was just in target disk mode).

This has big ramifications on the battery front. When I’d be using my Intel laptop on the go and not be expecting to charge for a while, I’d use Turbo Boost Switcher to disable Intel Turbo Boost and limit the CPU clock speed to the base level. This sligtly degrades performance, but substantially improves battery life. The Apple Silicon laptop, by contrast, achieves in normal mode the battery life that the Intel machine did with turbo disabled. If I were to put this into Low Power Mode (which, as I understand it, partly works by limiting processor speed), I don’t even know how long it would last.

### Display

The display on this computer is great. Having had a high-refresh rate external monitor for several years, my expectation was that the 120Hz support would be the thing I’d enjoy the most. But, that hasn’t actually been the case. Sure, 120Hz is great, but over the past couple months, I’ve been using my laptop undocked more frequently, and what I’ve come to really appreciate is the true retina pixel density.

If you don’t know, Apple laptops starting with the 2016 MacBook Pro have used non-integer scaling factors. That is, by default they ran at point resolutions which were more than half of the pixel resolution in each dimension. So, a software pixel mapped to some fraction of a hardware pixel, meaning everything had to be imprecisely scaled before actually going to the panel. People have been complaining about this for years, and I’d always dismissed it because I never observed the issue. But, in hindsight, that’s because the vast majority of my laptop usage was with it docked to an external monitor and peripherals. In that scenario, the laptop’s builtin display ends up physically far enough away from my eyes that I don’t perceive any blurriness. But, since I’ve been using this laptop more as an actual laptop—bringing the screen a good foot or two closer to my eyes—I’ve noticed that text is undeniably crisper.

All that said, 120Hz is of course also great. Scrolling and mousing around and animations all feel smoother, because they are. In particular the trackpad feels even more responsive and natural. And the mini-LED backlight makes HDR content look great. Seriously, if I were watching a movie or something on my computer, I’d prefer to watch it on the smaller builtin display than my big external monitor just for the vastly better contrast and black levels. Some people have complained about haloing (when you can see the backlight zone illuminated because there’s a small, bright object against a dark background), but I’ve never noticed except when deliberately looking for it by moving the cursor around on a black screen.

The one very minor complaint I do have is that ghosting on the display seems noticeably worse than my external monitor. I don’t have any way of objectively testing this, but moving the mouse cursor around seems to leave a longer trail. But again, I only actually notice that when I’m looking for it. In normal usage, and even in playing games, it isn’t apparent.

And, lastly about the display, let’s talk about the notch, since everyone needs to have an opinion on it. The short version is I don’t give a crap about it. The longer version is I really do not give a crap about it. Since the first week I had the computer, this is the only time I’ve given it any thought. It sits in the middle of the menu bar where nothing’s visible anyway (helped by having the 16” rather than the 14”, where big menus or many menubar items are more likely to overflow their respective halves), so it’s never once caused a problem for me.

### Hardware Miscellany

MagSafe is wonderful, I’m very happy it’s back. I get a little spark of joy when I walk up to my laptop and I see the little green dot on the connector. I pulled out my old laptop the other day to begin the process of erasing it before it can be sold, and because it had been sitting for so long, the battery was completely dead. I plugged in a USB-C charger and experienced a mild flash of annoyance that I had know way of knowing whether it was delivering power, other than to wait several minutes for the machine to boot up.

While I didn’t hate the Touch Bar as much as some people, I never found it to be better than plain old function keys. Nonetheless, I’m perfectly happy that it’s gone. My stupid minor gripe about the Touch Bar was that, when I’m using my computer docked with an external monitor and keyboard, the Touch Bar would remain on and active. That doesn’t sound so bad, but it becomes an annoyance as I interact with apps and see the software buttons on the Touch Bar changing and flashing in the corner of my eye. The removal of the Touch Bar has dealt with that annoyance and has made absolutely no difference to my productivity when using the laptop on its own, so I’m happy.

The hardware changes with this machine can be divided into two categories: Apple Silicon-related and not. The non-Apple Silicon changes by themselves are fairly small, but they represent a marked quality-of-life improvement when just using the computer.

## Software

With the M1 Mac mini, I had both ARM-native Homebrew and Intel-under-Rosetta Homebrew installed, in case I needed to install tools from brew that only ran under Rosetta. This time, that’s been entirely unnecessary^[2]. The one Rosetta package I installed last time, iPerf, now runs natively. In general, I’ve had to use Rosetta for far fewer things than I expected (with the notable exception of games), and even when I have, it’s been impressively stable and performant.

### Games

Video games are where the Apple Silicon software story gets complicated. Gaming on the Mac has always been a tenuous proposition and ARM has added a whole set of fun, new complications.

Here’s my rule of thumb for guessing how well a game will run on macOS: If it’s compiled for ARM or it uses Metal directly (not through a translation layer like OpenGL^[3] or MoltenVK), it’ll probably run great. If not, all bets are off. If a game’s either built for ARM or uses Metal, chances are someone has put at least some effort into getting it to work on Macs, so letting it have the most powerful CPU and GPU that’s ever been in a Mac laptop (which come close to being the most powerful in a Mac, period) gives it a really good chance of running well.

Rise of the Tomb Raider and Shadow of the Tomb Raider both run under Rosetta and use Metal and they run shockingly well. Each can manage a fairly consistent 60 FPS at 1440p on high graphical settings. The HDR support in Shadow of the Tomb Raider even worked with the built-in display. It wasn’t quite the difference in visual fidelity that I expected, but I was pleasantly surprised it worked at all.

Minecraft runs well, once you’ve got a JVM installed that’s built for ARM and the LWJGL natives swapped out with ones compiled for ARM, since the set Minecraft ships isn’t. (These instructions explain how to replace the native libs when using MultiMC^[4]. Though I use the Temurin JDK, not whatever’s in Homebrew.)

Cities: Skylines doesn’t see much of a graphical difference, but does benefit from the faster CPU. My old laptop can’t simulate the most recent city I built at 3x speed without things like vehicle movement appearing noticeably jerky whereas the Apple Silicon one can handle it. That said, I haven’t spent enough time playing C:S on it to know if the ceiling (that is to say, how much farther I could grow my city before I encountered performance issues) is substantially higher.

I haven’t tried many others, but I’m fairly confident that anything that previously ran on macOS and isn’t too graphically demanding will run well. That includes games like Hades and Into the Breach.

### Software Miscellany

Beyond games, 1Password 6 remains the only app I regularly use that needs Rosetta and it continues to work flawlessly.

Being able to run iOS apps natively is awesome, even if I don’t use it terribly often. The iOS app I use most on the Mac is Overcast, my preferred podcast player. It’s quite nice to be able to get notifications when new episodes are available right on my computer, rather than having to check my phone, as well as being able to listen without using the web interface.

Electron apps that are compiled for ARM, such as Spotify, are more responsive too. But it’s mostly a reflection of the performance/efficiency of Apple Silicon that it’s able to compensate for the bloat that is Electron. However, when it comes to Electron/browser-based apps that aren’t compiled for ARM (cough Steam cough), things aren’t as good. Steam does run and functions properly, but interacting with anything it uses embedded Chromium for (most of the application) is painfully slow and unresponsive.

One of my few complaints about the M1 Mac mini was resolved with the release of macOS Monterey: Apple Silicon Macs can now use DDC commands to control external displays. Not being able to control the of my external monitors was never a huge issue, but it was a persistent inconvenience that I’m glad has been resolved.

## Conclusion

Overall, this is a fantastic computer. Apple Silicon means it’s vastly faster and more efficient than any previous Mac laptop. As with last year, I’m impresed how much software is already native—just a year and a half into the Mac’s ARM transition—and how well Rosetta 2 works for software that isn’t. Beyond Apple Silicon, this laptop is an upgrade in every single way over the few preceding generations which felt like a big regression. Two laptops ago, I was using the 7.5 year old 2012 Retina MacBook Pro: the first laptop of a new generation of MacBooks. I’m hopeful that with all these long-standing issues resolved, this machine will last a similarly long time.

## Watching for USB Device Changes

Some cursory googling quickly led me to the IOKit documentation, which describes an IODeviceManager that sounds exactly like what I wanted. Unfortunately, the docs are rather lacking—there’s not a single example of how to actually set it up (and it’s not helped by the fact that it’s all old CoreFoundation style, nor that the Swift overlay is very poor).

But, with a combination of the docs and the open source ManyMouse project^[1], I managed to get it working.

manager = IOHIDManagerCreate(kCFAllocatorDefault, 0, /* kIOHIDManagerOptionNone */)

First, you need to create a manager using IOHIDManagerCreate. It takes an allocator and the options to use. I’m using the literal 0 here because the constants for the options are not imported into Swift. You also need to retain the manager for as long as you want to observe changes, so I’m storing it here in an instance var on my app delegate.

After that, you tell the manager what sorts of devices you care about. You do this by creating a dictionary that matches against the properties of a device.

var dict = IOServiceMatching(kIOHIDDeviceKey)! as! [String: Any]
dict[kIOHIDDeviceUsagePageKey] = kHIDPage_GenericDesktop
dict[kIOHIDDeviceUsageKey] = kHIDUsage_GD_Mouse
IOHIDManagerSetDeviceMatching(manager, dict as CFDictionary)

The IOServiceMatching function takes the name of a service type, for which we pass the redundantly named HID device key. It returns a CFDictionary, which I convert into a Swift dictionary, so that I can set properties on it more easily than dealing with CFDictionarySetValue from Swift. The filter is further refined by limiting it to the USB Generic Desktop usage page^[2], and specifically the mouse usage (the USB HID spec makes no differentiation between kinds of pointing devices, they’re all just mice), before setting the matching dictionary on the manager.

After that, we register callbacks for device addition and removal and add the manager to the run loop.

IOHIDManagerRegisterDeviceMatchingCallback(manager, hidDeviceAdded(context:result:sender:device:), nil)
IOHIDManagerRegisterDeviceRemovalCallback(manager, hidDeviceRemoved(context:result:sender:device:), nil)

IOHIDManagerScheduleWithRunLoop(manager, CFRunLoopGetCurrent(), CFRunLoopMode.commonModes.rawValue)

One important thing to note about the callbacks is that, since this is all old CoreFoundation-style code and not Objective-C, they’re not blocks, they’re C function pointers. Specifically, IOHIDDeviceCallbacks. This means that, when providing one from Swift, the value can either be a global function or a closure that does not capture anything. In either case, this requirement is because a C function pointer needs to point to a specific point in our binary, which can’t encode additional information like captures or the method receiver. To help alleviate this, the callback registration functions take a third parameter, a void pointer to some context that will be provided to the function when it’s called. All of the contextual information I need in the callbacks can be stored on the app delegate, though, so I don’t bother with trying to make the void *context shenanigans play nicely with Swift.

The callback functions receive an IOHIDDevice, the functions for which appear to only be documented in the headers. The first thing I do in the callbacks is get the name and usage for the device:

func hidDeviceAdded(context: UnsafeMutableRawPointer?, result: IOReturn, sender: UnsafeMutableRawPointer?, device: IOHIDDevice) {
    guard let name = IOHIDDeviceGetProperty(device, kIOHIDProductKey as CFString) as? String,
          let usage = IOHIDDeviceGetProperty(device, kIOHIDPrimaryUsageKey as CFString) as? UInt32 else {
        fatalError()
    }
}

The get property function returns an optional CFTypeRef so we have to try and cast it to the appropriate type.

The usage we need because, even though the manager’s filter is set to only allow Mouse devices through, the callback is still sometimes invoked with a device of type kHIDUsage_GD_SystemControl for reasons that I can’t discern. So, as a precaution, I silently ignore devices which don’t have the right usage:

func hidDeviceAdded(context: UnsafeMutableRawPointer?, result: IOReturn, sender: UnsafeMutableRawPointer?, device: IOHIDDevice) {
	// ...
    guard usage == kHIDUsage_GD_Mouse else { return }
}

## Determining Device Types

The next task is determining whether the given device is a mouse or trackpad. Unfortunately, as I mentioned, the USB HID spec doesn’t differentiate between mice and trackpads, and I couldn’t find any IOHIDDevice properties that did either. So, we have to come up with our own heuristics based on the information we do have access. Here’s where it gets really dumb:

func deviceNameIsProbablyTrackpad(_ name: String) -> Bool {
    return name.lowercased().contains("trackpad")
}

Yep. I figure this should cover most (at least Apple ones, which do call themselves “trackpad“s). And what makes something a mouse? Well, if it’s not a trackpad. Yes, it’s a bit ridiculous, but it works well enough.

func hidDeviceAdded(context: UnsafeMutableRawPointer?, result: IOReturn, sender: UnsafeMutableRawPointer?, device: IOHIDDevice) {
    // ...
    let delegate = NSApp.delegate as! AppDelegate
    if deviceNameIsProbablyTrackpad(name) {
        delegate.trackpadCount += 1
    } else {
        delegate.mouseCount += 1
    }
}

We track the actual counts of trackpads and mice rather than just whether one is connected or not, because in the device removed callback, it saves have to re-enumerate all connected devices in case there were multiple of one type connected and only one was removed.

The device removed callback does pretty much the same thing, just subtracting instead of adding 1 to the respective counts.

## Automatically Switching Scroll Direction

After the counts are updated, the delegate is notified that devices have changed and told to update the scroll direction. This is done by sending void to a PasstroughSubject on the app delegate. I use Combine rather than just calling a method on the app delegate because, when the IOHIDManager is initially added, it fires the added callback a whole bunch of times for every device that’s already connected. Additionally, when a USB hub is added/removed we get callbacks for all of the individual devices and I don’t want to flip the scroll direction a bunch of times. So, when the app delegate listens to the subject, it uses Combine’s debouncing to only fire every 0.1 seconds at most.

Actually changing the scroll direction is next, which requires figuring out what direction to change it to, based on the current device counts. This has a slight complication: laptops.

On a laptop, the trackpad is always connected, so if we were to switch to natural scrolling whenever a trackpad was connected, that would be useless. Similarly, for desktops, you can imagine a case where a mouse is always connected, so just switching to normal when a mouse is present would be similarly ineffectual in some cases. The solution? Doing both, and having a preference to let the user choose.

private func updateDirectionForAutoMode() {
    switch Preferences.autoMode {
	case .disabled:
        return

	case .normalWhenMousePresent:
        setDirection(mouseCount > 0 ? .normal : .natural)

	case .naturalWhenTrackpadPresent:
        setDirection(trackpadCount > 0 ? .natural : .normal)
    }
}

This way, on a laptop you can set it to “normal when mouse present” and on a desktop with an always-connected mouse you can set it to “natural when trackpad present”.

I’ve been using this updated version of ScrollSwitcher for about a week now, and it’s worked flawlessly. I haven’t had to open System Preferences or even click the menu bar icon. If you’re interested, the full source code for the app can be found here: https://git.shadowfacts.net/shadowfacts/ScrollSwitcher.

The other option—natural scrolling—flips this, so as your fingers move down on the trackpad, the content moves down and viewport moves up, and similarly for rotating the mouse wheel. When using a mouse, this feels to most people obviously backwards. But this setting doesn’t exist without reason. It’s transposing the paradigm of touchscreens on to the trackpad, where your finger remains pinned to the point in the content where you first touched. You move the content directly, rather than moving the viewport.

This generally isn’t a big deal; most people just find the mode they prefer, change the setting, and then never touch it again. But what if you prefer both?

Why might you want both options for this preference? Well, I use my laptop both docked and undocked. When it’s at my desk and connected to a monitor, I also use a keyboard and mouse, for which I want normal scrolling. But when it’s undocked and I’m using the trackpad, I vastly prefer natural scrolling.

Unfortunately, macOS only has one setting shared between both mice and trackpads. Yes, despite appearing in two different places in System Preferences (under both Trackpad and Mouse), with the implication that they’re two different settings, both checkboxes are backed by the same underlying value. So, we need some workaround.

Since the preference can’t be different for mouse and trackpad, I at least want some way of toggling it quickly, so that when I dock/undock my laptop, I can correct it quickly. A menu bar app seemed like the perfect fit, as I don’t want something cluttering up my dock and there’s almost no UI (but I do want more than just a global keyboard shortcut).

A bit of cursory googling reveals that the actual preference is stored in a global user default, com.apple.swipescrolldirection. It’s a boolean value, and true means natural scrolling is enabled. Reading it is easy, but unfortunately setting it is a bit more complicated. Just using defaults write -g com.apple.swipescrolldirection -bool YES on the command line does not change the actual value—although when you open the Mouse preferences pane^[1], you can see the checkbox state has indeed opened. Places on the internet mention this and say you need to log out and back in to correct the input behavior. But, that isn’t necessary when you change in System Preferences, so clearly something more is going on.

To try and figure out what else System Preferences was doing, I launched Console.app and started streaming log messages. I toggled the scroll direction checkbox a bunch of times in both the Mouse and Trackpad pref panes before stopping log streaming. Searching the logs for “mouse” and “trackpad” revealed nothing useful, but searching for “scroll” turned up one beautiful nugget of information:

SwipeScrollDirectionDidChangeNotification. Sure sounds like the name of a notification that would be fired to inform other parts of the OS about the change.

With at least an inkling of the direction I needed to go in, I started building the app with the initial goal of displaying the current scrolling mode. I created a new Mac app in Xcode and set the LSUIElement key in the Info.plist to YES, hiding the app from the dock. I also removed the Xcode-created storyboard and then created a system status bar item with a dummy icon.

## Fun Problem #1: Storyboards, or the lack thereof

Here is where I encountered the first (small) problem: the menu item never appeared. My code that created the menu item in applicationDidFinishLaunching was being hit, as evidenced by a breakpoint, and the menu item was being created, but it just didn’t show up. I recalled noticing that the docs of NSStatusBar.statusItem(withLength:) mentioned that the NSStatusItem had to be retained by the caller, otherwise it would be removed when deallocated. But that shouldn’t have been the problem, as I was storing it in a property on my app delegate. Unless…

It turns out using @main on your app delegate does not strongly retain it unless there’s a storyboard, which I had deleted. To fix it, I had to replace the @main with a custom main.swift which creates the NSApplication instance and strongly retains the delegate.

## Reading the Scroll Direction

With the menu item now displaying, it was time to read the current scroll direction. The most direct mapping from what I did on the command line would be to use Foundation’s Process to actually run the defaults command and then examine the output. But, because the value we’re after is stored in the global domain, it should be—and indeed is—accessible to us directly through UserDefaults.

let defaultsKey = "com.apple.swipescrolldirection"

enum Direction: Int, Equatable {
    case normal, natural

    static var current: Direction {
        let naturalEnabled = UserDefaults.standard.bool(forKey: defaultssKey)
        return naturalEnabled ? .natural : .normal
    }
}

Then, using the current direction, I could set the menu bar item’s icon. SF Symbols’ scroll.filled for natural scrolling and scroll for normal.

## Setting the Scroll Direction

Setting the scroll direction via the command line isn’t too bad. I just construct a Process, configure it to run defaults with the right arguments, and then launch it:

private func setDirection(_ new: Direction) {
    let proc = Process()
    proc.launchPath = "/usr/bin/defaults"
    let newVal = new == .normal ? "NO" : "YES"
    proc.arguments = ["write", "-g", "com.apple.swipescrolldirection", "-bool", newVal]
    proc.launch()
    proc.waitUntilExit()

    if proc.terminationStatus != 0 {
		fatalError("uh oh, exit code: \(proc.terminationStatus)")
    }
}

With that wired up to run when the menu item was clicked, I tried toggling the scroll direction. Unfortunately, it failed, because defaults didn’t run successfully. But, it had at least printed an error message:

[User Defaults] Couldn't write values for keys (
    "com.apple.swipescrolldirection"
) in CFPrefsPlistSource<0x6000017a1200> (Domain: kCFPreferencesAnyApplication, User: kCFPreferencesCurrentUser, ByHost: No, Container: (null), Contents Need Refresh: Yes): setting preferences outside an application's container requires user-preference-write or file-write-data sandbox access

Everyone’s favorite sandbox, snatching defeat from the jaws of victory.

## Fun Problem #2: Sandboxing

Unfortunately, both of the mentioned entitlements seem to be private (the only mentions of them I can find on the internet are from people running into some Catalyst error). So, I needed to disable sandboxing for this app altogether.

Disabling sandboxing turned out to be annoyingly confusing. Most of the documentation you can find on the internet is outdated and says you can simple flip the “App Sandbox” switch in the Capabilities section of the Xcode project. Slight problem: as of Xcode 13 (if not earlier), that switch no longer exists. And flat out removing the App Sandbox entry from Signing & Capabilities did not disable it, the same error occurred. Setting App Sandbox to NO in the entitlements file was similarly ineffective.

After banging my head against this for a while (it seems like this has not been discussed on the internet recently enough to be of any help), I looked in target’s Build Settings. Where I found the “Enable App Sandbox” flag—and it was set to Yes. Setting it to No finally fixed the issue, and the default was actually getting set.

The updated value could successfully be read from the app itself, as well as from outside. And that left me where I got stuck before on the command line: the preference was being updated, but nothing else on the system was aware of the change.

## Into the Caves

I knew the name of the notification I needed to fire, but not what to do with it—the normal NotificationCenter only works in-process, doesn’t it? I decided the best course of action was to go spelunking through the System Preferences binary to try and figure out what it was doing. But not actually the System Preferences binary: there’s a separate binary for each preference pane. A little bit of poking around the filesystem led me to the /System/Library/PreferencePanes/ directory where all the builtin ones live. Mouse.prefPane looked exactly like what I wanted. Opening it in Hopper, I could search the strings for the notification name. Following the references to the string back through the CFString led me to the -[MouseController awakeFromNib] method.

Looking at the disassembly, we can see exactly what it’s doing:

It’s adding an observer to NSDistributedNotificationCenter’s defaultCenter for the SwipeScrollDirectionDidChangeNotification—the notification name I saw earlier in the Console. The other place it’s referenced from ([MTMouseScrollGesture initWithDictionary:andReadPreferences:]) is doing the same thing: adding an observer. So, it looks like this notification isn’t what triggers the actual change to the actual scroll input handling deep in the guts of the system. If that were the case, I’d expect to see the preference pane send the notification, not just receive it.

But, it still may be useful. Looking at the implementation of the _swipeScrollDirectionDidChangeNotification: method it’s setting as the callback for the action, we can see that it’s probably updating the checkbox value. That setState: call sure seems like it’s on the NSButton used for the natural scrolling checkbox.

NSDistributedNotificationCenter is described as being like the regular notification center but for inter-process communication, which sounds like what we want. It has pretty much the same API as the regular one, so we can just send that notification when we change the scroll mode.

extension Notification.Name {
    static let swipeScrollDirectionDidChangeNotification = Notification.Name(rawValue: "SwipeScrollDirectionDidChangeNotification")
}

private func setDirection(_ new: Direction) {
    let proc = Process()
    // omitted
    DistributedNotificationCenter.default().postNotificationName(.swipeScrollDirectionDidChangeNotification, object: nil, userInfo: nil, deliverImmediately: false)
}

With that in place, clicking the menu bar item both sets the default and causes the System Preferences UI to update to match. Going the other way is similarly easy, since, although I couldn’t find it in Mouse.prefPane, something is emitting that notification when the value changes. I just call addObserver and register myself for the notification and update the icon when it’s received.

## Back Into the Caves

That’s all well and good, but clicking the menu item still doesn’t actually change what happens when you move two fingers on the trackpad. It clearly works when clicking the checkbox in System Preferences, so there must be something else it’s doing that we’re not. Internally, this feature seems to be consistently referred to as the “swipe scroll direction” (even though it affects non-swipe scrolling), so, back in Hopper, we can search for procedures named like that. There’s one that immediately looks promising setSwipeScrollDirection, that just delegates to an externally implemented _setSwipeScrollDirection.

Looking at the references to the function, I saw it was called by the -[MouseController scrollingBehavior:] setter. That seems like the function that I wanted, but since it was implemented elsewhere, I had no idea what parameters it took. So, where’s it implemented?

I used otool -L to print all the frameworks the prefpane was linked against, and then started guessing.

$ otool -L /System/Library/PreferencePanes/Mouse.prefPane/Contents/MacOS/Mouse
/System/Library/PreferencePanes/Mouse.prefPane/Contents/MacOS/Mouse:
	/System/Library/Frameworks/PreferencePanes.framework/Versions/A/PreferencePanes (compatibility version 1.0.0, current version 1.0.0)
# rest omitted

Actually getting the framework binaries is a bit tricky, since, starting with macOS Big Sur, the binaries are only present in the dyld shared cache. The process is a little bit annoying, but not terribly complicated. This article by Jeff Johnson explains how to build dyld_shared_cache_util, which you can use to transform the shared cache back into a directory with all the framework binaries.

$ dyld_shared_cache_util -extract ~/Desktop/Libraries/ /System/Library/dyld/dyld_shared_cache_x86_64

It took a couple guesses, but I found that the _setSwipeScrollingDirection function is defined in PreferencePanesSupport.framework.

Hopper thinks it takes an int, but we can clearly see the parameter’s being used as a bool. rcx is initialized to kCFBooleanFalse and set to kCFBooleanTrue if the parameter is true, and that’s the value being passed to CFPreferencesSetValue. Perfect.

Now—finally—that should be everything we need.

Back in the Xcode project, I added a bridging header that defines the externally implemented function and lets me call it from Swift.

#import <stdbool.h>
extern void setSwipeScrollDirection(bool direction);

Then, from Swift, we can simply call the function.

private func setDirection(_ new: Direction) {
    let proc = Process()
    // omitted
    DistributedNotificationCenter.default().postNotificationName(.swipeScrollDirectionDidChangeNotification, object: nil, userInfo: nil, deliverImmediately: false)
    setSwipeScrollDirection(new == .natural)
}

Lastly, in order to make the extern function actually go to the right place, the app needs to be linked against /System/Library/PrivateFrameworks/PreferencePanesSupport.framework. And with that, clicking the menu item toggles the preference and immediately updates the user input behavior.

I can’t really take a screen recording of that, so you’ll have to take my word that it works.

If you’re interested in the complete code, it can be found here. It’s not currently packaged for distribution, but you can build and run it yourself. Because it needs the sandbox disabled, it won’t ever been in the App Store, but at some point I might slap an app icon on it and published a notarized, built version. So, if anyone’s interested, let me know.

As it currently exists, the app—which I’m calling ScrollSwitcher—covers 90% of my needs. I don’t generally dock/undock more than a one or twice a day, so just being able to click a menu bar item is plenty fast. That said, I may still extend it for “fun”. One obvious improvement would be automatically changing the state when an external mouse is connected/disconnected. That shouldn’t be too hard, right? Right?

The app I built is a little TOTP app that works exactly the way I want, because Authy is bad^[1]. And it plays to SwiftUI’s strengths pretty well. It’s got a straightforward UI, with very few custom interface elements. The data model is simple and how it interacts with the interface is clearly delineated.

For the most part, writing the app has been a good experience. The vast majority of it I wrote in about 20 hours worth of work spread over four days. As of writing this, it’s about 1700 lines of Swift. The app itself is open source here, so if you want to see the codebase that I’m rambling on about, you can take a look.

The process was generally enjoyable. It’s what everyone says about SwiftUI: you can iterate incredibly quickly, there’s vastly less boilerplate than UIKit, and it’s much easier to build simple, reusable views.

This was also the first project where I actually managed to use the Xcode Previews feature. Although it always seems slick in WWDC demos, I was never able to get it to work reliably. But this time was different. The fact that Xcode pauses previews whenever something outside of the view body changes remains annoying, but at least while I was only modifying the body it did update reasonably quickly. I guess it really just works best for smaller projects.

Building almost all of the UI was easy. The design is simple enough that it doesn’t have to use any fancy tricks or workarounds—not a GeometryReader in sight. Plumbing up the model was similarly painless, both for editing and displaying data (aside from some oddities involving timing and refreshing codes).

But, here’s where the complaints start. Although the layout and design were uncomplicated, building some of the interactions were not. While working on it, SwiftUI felt incredibly powerful and uncomfortably restrictive simultaneously. For example:

Want to make give a context menu action the destructive style? Ok. Want to make a nested menu destructive? Nope.

Want to drag and drop one view onto another view? Go ahead. How about dropping that view onto one that’s inside a List? Woah there.

Making a text field focused programatically? Sure. And what about making it focused immediately on appearance? Not so fast.

To be clear, many of the issues I’ve encountered with SwiftUI are almost certainly bugs that will (hopefully) be fixed eventually. But where this differs from UIKit is that the architecture of UIKit lets you fix and workaround things yourself, whereas SwiftUI operates as a black box.

To use the same examples, but in UIKit: to make a context menu element destructive or not is exactly the same regardless of whether it has children, adding a drop interaction to a UIView is the same inside a table view as it is outside, and focusing a text field uses the same method no matter when in the lifecycle it’s called.

I don’t think this is just because I have more experience with UIKit. It’s not that there are lots of UIKit tricks I know to make this work, in constrast with SwiftUI. There is a fundamental difference between the two frameworks. The delta between doing any of the things I mentioned in one context versus in the other is zero. Because of the fundamental design of UIKit, all of these things basically just work. Yes, implementation wise, the framework engineers may still need to take special care to ensure that they work. But, from an API design point of view, nothing different is required.

The impression I get from Apple is that SwiftUI wants to make hard things easy. That’s a great goal, but currently it comes at the expense of making easy things complicated. And every time I do anything substantial with SwiftUI, I constantly feel this tension. SwiftUI both empowers me and hinders me, every step of the way.

I don’t know what the solution is, if there is one (part of me thinks these sorts of issues are intrinsic to declarative tools), but I really hope there is one. When it works, I really enjoy SwiftUI and I want to be able to enjoy it more without running into unresolvable framework issues quite so frequently.

The gist of the issue is that when connecting to a certain host, the connection would hang indefinitely. The issue was 100% reproducible in my app, both in the simulator and on an actual device, in all manner of network conditions. What led me to believe that it was an issue with my implementation was that the problem only happened with a single host (I suspect the incompatibility was with whatever server software was being used) and the fact that I could not reproduce the issue with any other client.

My initial attempts at debugging were fruitless. Every handler and callback I had was just never getting called. There was no error that was being swallowed or anything, just silence. Eventually I set a breakpoint in the handleInput method of my NWProtocolFramerImplementation.

Stepping through^[1], I saw the status code get parsed successfully. Then it moved on to parsing the response meta, a string of up to 1024 bytes followed by a carriage return and line feed.

Next, I paused inside the parseInput closure that’s responsible for parsing the meta string. The buffer that I received was 11 bytes long. I was immediately wary because I’d seen that number before in this context—it’s the length of text/gemini. Indeed, printing out String(bytes: buffer, encoding: .utf8) revealed that it was indeed the meta string I was expecting. But it was just the meta string, not including the CR/LF pair that’s supposed to follow it.

Continuing to step through, I saw the closure return 0, indicating that no bytes were consumed, because the CRLF was missing. After that, the main body of the handleInput method would see that the meta wasn’t found and would itself return the number of bytes it expected to receive before it should be invoked again. It does this by adding 1 to the length of the buffer from which meta parsing failed. So, the next time handleInput is called by the framework, there should be at least 12 bytes available.

After hitting resume, the debugger trapped again in the parseInput closure for the meta. I checked buffer.count and found… 11 bytes. I know handleInput returned 12, so why did I still only have 11 bytes?

The realization I came to, after puzzling over this for a couple days, is that parseInput behaves a little weirdly, though in a way that’s technically correct. It seems that even if more data is available in some internal buffer of the framer’s—which we know there must be because handleInput was invoked again after we returned 12 earlier—we won’t receive all of it. It’s not wrong, 11 bytes is indeed at least 2 and no more than 1026, but it’s certainly unintuitive.

To verify this behavior, I tweaked my code to call parseInput with a minimum length of 13 instead of 2 the second attempt to parse meta. Lo and behold, it worked. The buffer now had 13 bytes of data: the full meta string and the CR/LF.

And that explains the user-facing issue that led to all this. A few attempts at parsing would fail, but then the server would stop sending data because there was nothing left, so handleInput would never be called, leaving the Gemini request in a waiting state forever.

So, to properly fix the issue what needs to happen is we have to be smarter about the minimumIncompleteLength on subsequent attempts to parse the metadata. To do this, I saved the length of the last buffer for which meta parsing was attempted as an instance variable in the framer implementation. With that, we can determine how many bytes we actually need before trying again.

class GeminiProtocol: NWProtocolFramerImplementation {
	// ...
	private var lastAttemptedMetaLength: Int? = nil
	// ...

	func handleInput(framer: NWProtocolFramer.Instance) -> Int {
		// ...
		if tempMeta == nil {
			let min: Int
			if let lastAttemptedMetaLength = lastAttemptedMetaLength {
				min = lastAttemptedMetaLength + 1
			} else {
				min = 2
			}
			_ = framer.parseInput(minimumIncompleteLength: min, maximumLength: 1024 + 2) { (buffer, isComplete) -> Int in
				guard let buffer = buffer else { return }
				self.lastAttemptedMetaLength = buffer.count
				// ...
			}
		}
		guard let meta = tempMeta else {
			if let attempted = self.lastAttemptedMetaLength {
				return attempted + 1
			} else {
				return 2
			}
		}
		// ...
	}

	// ...
}

The minimum incomplete length still defaults to 2 because on the first attempt to parse, we don’t have previous attempt info and the only thing we know is that there must be a carriage return and line feed (as best as I can interpret it, the Gemini spec doesn’t say that there must be meta text, so it could be zero bytes).

With that, the original issue is finally fixed and requests to the problematic server complete successfully.

My best guess for why this was happening in the first place and only in such specific circumstances is this: the specific server software being used was sending the meta text over the wire separately from the CRLF. This could mean they arrive at my device separately and are thus stored by in two separate “chunks”. Then, when parseInput is called, Network.framework simply starts looking through the stored chunks in order. And, since the first chunk is longer than minimumIncompleteLength, it’s the only one that’s returned.

There’s a note in the Network.framework header comments^[2] for nw_framer_parse_input that leads me to believe this. It says, with regard to the temp_buffer parameter:

If it is NULL, the buffer provided in the completion will not copy unless a copy is required to provide the minimum bytes as a contiguous buffer.

The possibility of a copy being needed to form a contiguous buffer implies that there could be discontiguous data, which lines up with my “chunks” hypothesis and would explain the behavior I observed.

Anyhow, if you’re interested, you can find the current version of my Gemini client implementation (as of this post) here.

The main issue is that of lexical scope, which is where the context (i.e., what variables are accessible) at each point in the program is defined by where in the original source code it is.

Let’s say you have some code:

let a = 1
if (condition) {
	print(a)
	let b = 2
}
print(b)

Entering the body of the if statement starts a new scope in which variables defined in the any encompassing scope can be accessed, but not vice versa. a, defined in the outer scope, can be read from the inner scope, but b, defined in the inner scope, cannot be accessed from the outer scope.

What this means for me is that, in the evaluator, it’s not just enough to have access to the current scope. All the parent scopes are also needed.

There are a couple ways I could approach this. One way would be to have something like a vector of contexts, where the last element is the current context. Accessing a parent context would just mean walking backwards through the vector. And to enter a new scope, you’d construct a new context and push it onto the vector, evaluate whatever you wanted in the new scope, and then remove it from the vector afterwards. This would work, but it risks needing to replace the vector’s backing storage every time a context is entered. It’s probably a premature optimization, but I decided to take a different approach to avoid the issue.

Another way of doing it is effectively building a singly-linked list, where each Context stores an optional reference to its parent. But a simple reference isn’t enough. The Context struct would need a generic lifetime parameter in order to know how long the reference to its parent lives for. And in order to specify what type the reference refers to, we would need to be able to know how long the parent context’s parent lives for. And in order to spell that type out we’d have to know how long the parent’s parent’s parent—you get the idea.

The solution I came up with was to wrap the context struct in an Rc, a reference-counted pointer. So, instead of each context having a direct reference to its parent, it owns an Rc that’s backed by the same storage. Though, that’s not quite enough, because the context needs to be mutable so code can do things like set variables. For that reason, it’s actually an Rc<RefCell<Context>>. I understand this pattern of interior mutability is common practice in Rust, but coming from languages where this sort of things is handled transparently, it’s one of the weirder things I’ve encountered.

Now, on to how this is actually implemented. It’s pretty simple. The Context struct stores the Rc I described above, but inside an Option so that the root context can have None as its parent.

struct Context {
	parent: Option<Rc<RefCell<Context>>>,
	variables: HashMap<String, Value>,
}

Then, instead of the various eval functions taking a reference directly to the context, they get a reference to the Rc-wrapped context instead.

fn eval_binary_op(left: &Node, op: &BinaryOp, right: &Node, context: &Rc<RefCell<Context>>) -> Value {
	let left_value = eval_expr(left, context);
	// ...
}

The constructors for Context also change a bit. There’s one that doesn’t have a parent, called root, in addition to new which does.

impl Context {
	fn root() -> Self {
		Self {
			parent: None,
			variables: HashMap::new(),
		}
	}

	fn new(parent: Rc<RefCell<Context>>) -> Self {
		Self {
			parent: Some(parent),
			variables: HashMap::new(),
		}
	}
}

Unlike the evaluation functions, Context::new takes an owned Rc, avoiding the infinite-lifetimes problem from earlier. This requires that, when constructing a new context, we just need to clone the existing Rc.

fn eval_if(condition: &Node, body: &[Statement], context: &Rc<RefCell<Context>>) {
	let body_context = Context::new(Rc::clone(context));
	let body_context_ref = Rc::new(RefCell::new(body_context));
}

After the new context is constructed, it too is wrapped in a RefCell and Rc for when the condition and body are evaluated. This is a little bit unweidly, but hey, it works.

Actually evaluating the if is simple enough that I won’t bother going through it in detail. It just evaluates the condition expression (confirming that it’s a boolean; there shall be no implicit conversions!) and, if it’s true, evaluates each of the statements in the body.

fn main() {
	let code = r#"let a = 1; if (a == 1) { dbg(a); }"#;
	let tokens = tokenize(&code);
	let statements = parse(&tokens);
	eval(&statements);
}

$ cargo run
Integer(1)

First off, the lexer now lexes a couple new things: the let keyword and the equals sign.

When the parser tries to parse a statement and sees a let token, it knows it’s looking at a variable declaration. After the let token it expects to find a identifier (the variable name), an equals sign, and then an expression for the initial value of the variable. The variable name and the initial value expression then make up a Declare AST node.

fn parse_statement<'a, I: Iterator<Item = &'a Token>>(it: &mut Peekable<'a, I>) -> Option<Statement> {
	// ...
	let node = match token {
		Token::Let => {
			let name: String;
			if let Some(Token::Identifier(s)) = it.peek() {
				name = s.clone();
				it.next();
			} else {
				panic!("expected identifier after let");
			}
			expect_token!(it, Equals, "expected equals after identifier after let");
			let value = parse_expression(it).expect("initial value in let statement");
			Some(Statement::Declare { name, value })
		}
		// ...
	};
	// ...
}

expect_token! is a simple macro I wrote to handle expecting to see a specific token in the stream and panicking if it’s not there, since that’s a pattern that was coming up frequently:

macro_rules! expect_token {
	($stream:ident, $token:ident, $msg:expr) => {
		if let Some(Token::$token) = $stream.peek() {
			$stream.next();
		} else {
			panic!($msg);
		}
	};
}

Next, to actually evaluate variable declarations, the evaulator needs to have some concept of a context. Right now, every expression can be evaluated without any external state. But, when a variable is declared, we want it to be accessible later on in the code, so there needs to be somewhere to store that information.

For now, the only information we need is the map of variable names to their values.

struct Context {
	variables: HashMap<String, Value>,
}

There are also a few methods for Context, one to construct a new context and one to declare a variable with an initial value.

impl Context {
	fn new() -> Self { 
		Self {
			variables: HashMap::new(),
		}
	}

	fn declare_variable(&mut self, name: &str, value: Value) {
		if self.variables.contains_key(name) {
			panic!("cannot re-declare variable {}", name);
		} else {
			self.variables.insert(name.into(), value);
		}
	}
}

Every eval_ function has also changed to take a reference to the current context^[1] and the main eval function creates a context before evaluating each statement.

With that, declaration statements can be evaluated just by calling the declare_variable method on the context:

fn eval_declare_variable(name: &str, value: &Node, context: &mut Context) {
	let val = eval_expr(value, context);
	context.declare_variable(name, val);
}

And we can actually set and read variables now^[2]:

fn main() {
	let code = "let foo = 1; dbg(foo)";
	let tokens = tokenize(&code);
	let statements = parse(&tokens);
	eval(&statements);
}

$ cargo run
Integer(1)

In the AST, there’s a new top-level type: Statement. For now, the only type of statement is one that contains an expression and nothing else.

enum Statement {
	Expr(Node),
}

The top level parse function has also changed to reflect this. It now returns a vector of statements, instead of a single expression node. The do_parse function continues to work exactly as it has, but is renamed parse_expression to since that’s what it’s actually doing.

fn parse(tokens: &[Token]) -> Vec<Statement> {
	let mut it = tokens.iter().peekable();
	let mut statements: Vec<Statement> = vec![];
	while let Some(_) = it.peek() {
		match parse_statement(&mut it) {
			Some(statement) => statements.push(statement),
			None => (),
		}
	}
	statements
}

The parse_statement function does exactly what the name suggests.

fn parse_statement<'a, I: Iterator<Item = &'a Token>>(it: &mut Peekable<'a, I>) -> Option<Statement> {
	if it.peek().is_none() {
		return None;
	}

	let node = parse_expression(it).map(|node| Statement::Expr(node));
	node
}

With that in place, parsing multiple statements is easy. The only change is that, after successfully parsing a statement, we need to consume a semicolon if there is one. Then, the parse loop will continue and the next statement can be parsed.

fn parse_statement<'a, I: Iterator<Item = &'a Token>>(it: &mut Peekable<'a, I>) -> Option<Statement> {
	// ...
	match it.peek() {
		Some(Token::Semicolon) => {
			it.next();
		}
		Some(tok) => {
			panic!("unexpected token {:?} after statement", tok);
		}
		None => (),
	}
	
	node
}

I intend to make semicolons optional and allow newline-delimited statements, but that is more complicated and will have to wait for another time. For now, this is good enough:

fn main() {
	let tokens = tokenize("1 + 2; foo();");
	print("statements: {:?}", parse(&tokens));
}

$ cargo run
statements: [
	Expr(
		BinaryOp {
			left: Integer(1),
			op: Add,
			right: Integer(2),
		},
	),
	Expr(
		Call {
			name: "foo",
			params: [],
		},
	),
]

First step: lexing.

There are two new token types: identifier and comma.

enum Token {
	// ...
	Identifier(String),
	Comma,
}

The comma is just a single comma character. The identifier is a sequence of characters that represent the name of a variable or function. An identifier starts with a letter (either lower or uppercase) and is followed by any number of letters, digits, and underscores.

The main tokenize function checks if it’s looking at a letter, and, if so, calls the parse_identifier function. parse_identifier simply accumulates as many valid identifier characters as there are and wraps them up in a token.

fn parse_identifier<I: Iterator<Item = char>>(it: &mut Peekable<I>) -> Option<Token> {
	let chars = take_while_peek(it, |c| {
		LOWERCASE_LETTERS.contains(c)
			|| UPPERCASE_LETTERS.contains(c)
			|| DIGITS.contains(c)
			|| *c == '_'
	});
	if chars.is_empty() {
		None
	} else {
		let s = String::from_iter(chars);
		Some(Token::Identifier(s))
	}
}

The next step is parsing.

There are two new kinds of AST nodes: lookup nodes and function call nodes. The only data lookup nodes store is the name of the variable they refer to. Function call nodes store the nodes for their parameters, in addition to the function name.

When parsing an expression, an identifier token results in either a lookup or function call node, depending on whether it’s followed by a left-paren.

fn do_parse<'a, I: Iterator<Item = &'a Token>>(it: &mut Peekable<I>) -> Option<Node> {
	// ...
	let mut node: Node = match it.peek().unwrap() {
		// ...
		Token::Identifier(value) => {
			it.next();
			match it.peek() {
				Some(Token::LeftParen) => Node::Call {
					name: value.clone(),
					params: parse_function_params(it),
				}
				_ => Node::Lookup {
					name: value.clone(),
				},
			}
		}
	};
}

Actually parsing function parameters is left to another function. After consuming the opening parenthesis, it checks if the next token is the closing right-paren. If it is, the right-paren is consumed and an empty vector is returned for the paramters.

If it isn’t, the function enters a loop in which it parses a parameter expression and then expects to find either a comma or right-paren. If there’s a comma, it’s consumed and it moves on to the next iteration of the loop. If it’s a closing parenthesis, it too is consumed and then the loop is exited and the parameter list returned. Upon encountering any other token, it panics.

fn parse_function_params<'a, I: Iterator<Item = &'a Token>>(it: &mut Peekable<I>) -> Vec<Node> {
	it.next(); // consume left paren
	if let Some(Token::RightParen) = it.peek() {
		it.next();
		vec![]
	} else {
		let mut params = vec![];
		loop {
			let param_node = do_parse(it).expect("function parameter");
			params.push(param_node);
			match it.peek() {
				Some(Token::Comma) => {
					it.next();
				}
				Some(Token::RightParen) => {
					it.next();
					break;
				}
				tok => {
					panic!("unexpected token {:?} after function parameter", tok);
				}
			}
		}
		params
	}
}

And lastly, to make this work correctly, the comma token is added to the list of expression-end tokens.

With that, parsing function calls and variable lookups is possible:

fn main() {
	let tokens = tokenize("foo(bar)");
	if let node = parse(&tokens) {
		println!("{:?}", node);
	}
}

$ cargo run
Call {
	name: "foo",
	params: [
		Lookup {
			name: "bar",
		},
	],
}

As a reminder, the code currently looks like this:

fn combine_with_binary_operator(left: Node, token: &Token, right: Node) -> Node {
	let op: BinaryOp = match token {
		// ...
	};

	if should_group_left(&op, &right) {
		if let Node::BinaryOp {
			left: right_left,
			op: right_op,
			right: right_right,
		} {
			Node::BinaryOp {
				left: Box::new(Node::BinaryOp {
					left: Box::new(left),
					op,
					right: right_left,
				}),
				op: right_op,
				right: right_right,
			}
		} else {
			panic!();
		}
	} else {
		Node::BinaryOp {
			left: Box::new(left),
			op,
			right: Box::new(right),
		}
	}
}

fn should_group_left(left_op: &BinaryOp, right: &Node) -> bool {
	match right {
		Node::BinaryOp { op: right_op, .. } => {
			right_op.precedence() < left_op.precedence()
				|| (right_op.precedence() == left_op.precedence()
					&& left_op.associativity() == Associativity::Left)
		}
		_ => false,
	}
}

See that panic!() in the else branch? The compiler thinks it (or some return value) is necessary there, because the pattern match could fail. But as the programmer, I know better. I know that if we’re in the true branch of the outer if statement, then should_group_left returned true and the pattern match can never fail.

This is why I just call panic! without even a message: because I know that code is unreachable.

But it would be even better not to have it at all.

Basically, what I want the should_group_left function to do is pattern match on the right node, and if it meets the conditions for being left-grouped, to get the values inside the right binary operator node out without having to do another pattern match.

The best solution I was able to come up with was changing should_group_left to take ownership of the right node and return a Result<(Box<Node>, BinaryOp, Box<Node>), Node>^[1]. If it returns an Ok result, all the values are available. If it returns an “error”, ownership of the right node is returned back to the caller.

fn should_group_left(
	left_op: &BinaryOp,
	right: Node,
) -> Result<(Box<Node>, BinaryOp, Box<Node>), Node> {
	match right {
		Node::BinaryOp {
			left: right_left,
			op: right_op,
			right: right_right,
		} => {
			let should_group = // ...
			if should_group {
				Ok((right_left, right_op, right_right))
			} else {
				Err(Node::BinaryOp {
					left: right_left,
					op: right_op,
					right: right_right,
				})
			}
		}
		_ => Err(right),
	}
}

Even this isn’t ideal, because in the else branch in the first match arm, I still need to reconstruct the original right node, since it’s been moved by the destructuring. I spent a while playing around with using references for various things in this function, but ultimately couldn’t come up with anything better than this. If you have any ideas, let me know.

At any rate, at the call site in combine_with_binary_operator, this works pretty well:

fn combine_with_binary_operator(left: Node, token: &Token, right: Node) -> Node {
	let op = match token {
		// ...
	};

	match should_group_left(&op, right) {
		Ok((right_left, right_op, right_right)) => Node::BinaryOp {
			left: Box::new(Node::BinaryOp {
				left: Box::new(left),
				op,
				right: right_left,
			}),
			op: right_op,
			right: right_right,
		},
		Err(right) => Node::BinaryOp {
			left: Box::new(left),
			op,
			right: Box::new(right),
		},
	}
}

To actually parse the group from the token stream, in the parse_expression function I look for a left paren at the beginning of an expression, and call parse_group if one is found.

fn parse_expression<'a, I: Iterator<Item = &'a Token>>(it: &mut Peekable<I>) -> Option<Node> {
	// ...
	let mut node: Node = match it.peek().unwrap() {
		// ...
		Token::LeftParen => parse_group(it).unwrap(),
	}
	// ...
}

The parse_group function is also pretty simple. It consumes the left paren and then calls parse_expression to parse what’s inside the parentheses. Afterwards, assuming it’s found something, it consumes the right paren and returns a new Group node (which has just one field, another boxed Node that’s its content.

fn parse_group<'a, I: Iterator<Item = &'a Token>>(it: &mut Peekable<I>) -> Option<Node> {
	match it.peek() {
		Some(Token::LeftParen) => (),
		_ => return None,
	}

	it.next();

	if let Some(inner) = parse_expression(it) {
		match it.peek() {
			Some(Token::RightParen) => (),
			tok => panic!("expected closing parenthesis after group, got {:?}", tok),
		}
		it.next();
		Some(Node::Group {
			node: Box::new(inner),
		})
	} else {
		panic!("expected expression inside group");
	}
}

This looks pretty good, but trying to run it and parse an expression like (1) will crash the program. Specifically, it’ll panic with a message saying unexpected token: RightParen.

At first, this was pretty confusing. Shouldn’t the right paren be consumed the parse_group function? Running with RUST_BACKTRACE=1 reveals what the problem actually is.

It’s panicking inside the recursive call to parse_expression coming from parse_group, before that function even has a chance to cosume the right paren. Specifically, parse_expression is seeing a token after the first part of the expression and is trying to combine it with the existing node and failing because a right paren is not a binary operator.

What should happen is that parse_expression should see the paren following the expression, realize that the expression is over, and not do anything with it. That way, when the recursive parse_expression returns, parse_group will be able to consume the right paren as it expects.

To do this, there’s a constant list of tokens which are considered to end the current expression. Then, in parse_expression, in addition to checking if the next token after an expression is a binary operator, we can check if the token is an expression end. And if so, avoid panicking.

const EXPRESSION_END_TOKENS: &[Token] = &[Token::RightParen];

fn parse_expression<'a, T: Iterator<Item = &'a Token>>(it: &mut Peekable<T>) -> Option<Node> {
	// ...
	if let Some(next) = it.peek() {
		if is_binary_operator_token(next) {
			// ...
		} else if EXPRESSION_END_TOKENS.contains(next) {
			// no-op
		} else {
			panic!("unexpected token: {:?}", next);
		}
	}

	Some(node)
}

And now it can parse grouped expressions:

fn main() {
	let tokens = tokenize("(1)");
	if let node = parse(&tokens) {
		println!("node: {:#?}", &node);
	}
}

$ cargo run
node: Group {
	node: Integer(1),
}

(I won’t bother discussing evaluating groups because it’s trivial.)

My initial attempt at this was a simple modification to the original parse_number function from the lexer. Instead of stopping when it encounters a non-digit character, I changed it to continuing collecting characters when it encounters a decimal point for the first time.

fn parse_number<T: Iterator<Item = char>>(it: &mut T) -> Option<Token> {
	let mut found_decimal_point = false;
	let digits = it.take_while(|c| {
		if DIGITS.contains(c) {
			true
		} else if *c == '.' && !found_decimal_point {
			found_decimal_point = true;
			true
		} else {
			false
		}
	});
	// ...
}

This seemed to work, and produced tokens like Float(1.2) for the input “1.2”. But then I tried it with the string “1.2.3”, to make sure that lexing would fail when it encountered a dot after the number literal. But it didn’t. It failed because it didn’t expect the ‘3’ character. The dot seemed to vanish into thin air.

I came up with a simpler test case^[1]:

fn main() {
	let is = vec![1, 2, 3, 4];
	let it = is.iter().peekable();
	let taken = take_some(it);
	println!("taken: {:?}, next: {:?}", taken, it.peek());
}

fn take_some<I: Iterator<Item = i32>>(&mut it: Peekable<I>) -> Vec<i32> {
	it.take_while(|i| **i < 3).collect()
}

To my surprise, it printed taken: [1, 2], next: Some(4). This time the 3 disappeared.

I inquired about this behavior on the fediverse, and learned that I missed a key line of the docs for the take_while method. Before it invokes the closure you passed in, it calls next() on the underlying iterator in order to actually have an item for the closure to test. So, it ends up consuming the first element for which the closure returns false.

I would have expected it to use the peek() method on peekable iterators to avoid this, but I guess not. No matter, a peeking version is easy to implement:

fn take_while_peek<I, P>(peekable: &mut Peekable<I>, mut predicate: P) -> Vec<I::Item> 
where
	I: Iterator,
	P: FnMut(&I::Item) -> bool,
{
	let mut vec: Vec<I::Item> = vec![];
	while let Some(it) = peekable.peek() {
		if predicate(it) {
			vec.push(peekable.next().unwrap());
		} else {
			break;
		}
	}
	vec
}

I can then switch to using the new function in parse_number_literal and it no longer consumes extra characters.

fn parse_number<T: Iterator<Item = char>>(it: &mut T) -> Option<Token> {
	let mut found_decimal_point = false;
	let digits = take_while_peek(it, |c| {
		// ...
	});
	// ...
}

Currently, an expression like 2 * 3 + 4 will be parsed as if the 3 + 4 was grouped together, meaning the evaluation would ultimately result in 14 instead of the expected 10. This is what the AST for that expression currently looks like:

  *
 / \
2   +
   / \
  3   4

But, the multiplication operator should actually have a higher pririty and therefore be deeper in the node tree so that it’s evaluated first.

Another closely related issue is associativity. Whereas operator precedence governs implicit grouping behavior when there are operators of different precedences (like addition and multiplication), operator associativity defines how implicit grouping works for multiple operators of the same precedence (or multiple of the same operator).

Looking at the AST for an expression like “1 - 2 - 3”, you can see the same issue is present as above:

  -
 / \
1   -
   / \
  2   3

In both of these cases, what the parser needs to do is the same. It needs to implicitly group the middle node with the left node, rather than the right one. This will result in node trees that look like this:

    +           -
   / \         / \
  *   4       -   3
 / \         / \
2   3       1   2

To accomplish this, I added precedence and associativity enums as well as methods on the BinaryOp enum to get each operation’s specific values so that when the parser is parsing, it can make a decision about how to group things based on this information.

The Precedence enum has a derived implementation of the PartialOrd trait, meaning the cases are ordered from least to greatest in the same order they’re written in the code, so that the precedence values can be compared directly with operators like <. Addition/subtraction and multiplication/division currently share precedences. Also, every operator currently has left associativity.

enum BinaryOp {
	Add,
	Subtract,
	Multiply,
	Divide,
}
#[derive(PartialEq, PartialOrd)]
enum Precedence {
	AddSub,
	MulDiv
}
#[derive(PartialEq)]
enum Associativity {
	Left,
	Right,
}
impl BinaryOp {
	fn precedence(&self) -> Precedence {
		match self {
			BinaryOp::Add | BinaryOp::Subtract => Precedence::AddSub,
			BinaryOp::Multiply | BinaryOp::Divide => Precedence::MulDiv,
		}
	}
	fn associativity(&self) -> Associativity {
		Associativity::Left
	}
}

In the do_parse function, things have been changed up. First, there’s a separate function for checking if the token that follows the first token in the expression should combine with the first node (i.e., is a binary operator):

fn is_binary_operator_token(token: &Token) -> bool {
	if let Token::Plus | Token::Minus | Token::Asterisk | Token::Slash = token {
		true
	} else {
		false
	}
}

So instead of matching on individual tokens, do_parse just calls that function. If the next token is a binary operator, it consumes the operator token, calls do_parse recursively to get the right-hand node and then calls another function to combine the two nodes.

fn do_parse<'a, T: Iterator<Item = &'a Token>>(it: &mut Peekable<T>) -> Option<Node> {
	// ...
	if let Some(next) = it.peek() {
		if is_binary_operator_token(next) {
			let operator_token = it.next().unwrap();
			let right = do_parse(it).expect("expression after binary operator");
			node = combine_with_binary_operator(node, operator_token, right);
		} else {
			panic!("unexpected token: {:?}", next);
		}
	}

	Some(node)
}

But, before I get to the combine_with_binary_operator function, there’s another function that decides whether a binary operator should be grouped to the left with another node by following the rule I described earlier.

fn should_group_left(left_op: &BinaryOp, right: &Node) -> bool {
	match right {
		Node::BinaryOp { op: right_op, .. } => {
			right_op.precedence() < left_op.precedence()
				|| (right_op.precedence() == left_op.precedence()
					&& left_op.associativity() == Associativity::Left)
		}
		_ => false,
	}
}

The combine_with_binary_operator function can then use this (after converting the binary operator token into a BinaryOp value) to decide what it should do.

fn combine_with_binary_operator(left: Node, token: &Token, right: Node) -> Node {
	let op: BinaryOp = match token {
		// ...
	};

	if should_group_left(&op, &right) {
		if let Node::BinaryOp {
			left: right_left,
			op: right_op,
			right: right_right,
		} {
			Node::BinaryOp {
				left: Box::new(Node::BinaryOp {
					left: Box::new(left),
					op,
					right: right_left,
				}),
				op: right_op,
				right: right_right,
			}
		} else {
			panic!();
		}
	} else {
		Node::BinaryOp {
			left: Box::new(left),
			op,
			right: Box::new(right),
		}
	}
}

If there are two binary operators and it does need to be grouped to the left, it performs the same transformation I described above, constructing a new outer binary operator node and the new left-hand inner node. Diagramatically, the transformation looks like this (where uppercase letters are the binary operator nodes and lowercase letters are values):

Original expression: x A y B z

  A               B
 / \             / \
x   B     ->    A   z
   / \         / \
  y   z       x   y

If it does not need to be grouped left, the function simply creates a new binary operator node, leaving the left and right sides as-is.

And, after adding the new operators to the eval_binary_op function, it can now correctly compute simple arithmetic expressions!

fn main() {
	let tokens = tokenize("2 * 3 + 4");
	if let node = parse(&tokens) {
		println!("result: ", eval(&node));
	}
}

$ cargo run
result: Integer(10)

First, there needs to be something to actually store values during the evaluation process. For this, I used yet another enum. It only has one case for now because we can currently only lex and parse integer values and one arithmetic operator.

enum Value {
	Integer(i64),
}

There’s also a helper function to extract the underlying integer from a value in places where we’re certain it’s going to be an integer:

impl Value {
	fn expect_integer(&self, &msg) -> i64 {
		match self {
			Value::Integer(n) => *n,
			_ => panic!("{}", msg),
		}
	}
}

The compiler warns about the unreachable match arm, but it’ll be useful once there are more types of values. (Once again, actual error reporting will wait.)

The actual evaulation starts in the eval function which takes a reference to the node to evaluate and returns a Value representing its result.

For integer nodes, the value of the AST node is wrapped in a Value and returned directly. For binary operator (i.e. addition) nodes the left- and right-hand values are extracted and another function is called to perform the operation.

fn eval(node: &Node) -> Value {
	match node {
		Node::Integer(n) => Value::Integer(*n),
		Node::BinaryOp { left, right } => eval_binary_op(left, right),
	}
}

This eval_binary_op function takes each of the nodes and calls eval with it. By doing this, it recurses through the the AST evaluating each node in a depth-first manner. It then turns each value into an integer (panicking if either isn’t what it expects) and returns a new Value with the values added together.

fn eval_binary_op(left: &Node, right: &Node) -> Value {
	let left = eval(left).expect_integer("left hand side of binary operator must be an integer");
	let right = eval(right).expect_integer("right hand side of binary operator must be an integer");
	Value::Integer(left + right)
}

And with that surpisingly small amount of code, I’ve got a very dumb calculator that can perform arbitrary additions:

fn main() {
	let tokens = tokenize("1 + 2 + 3");
	if let Some(node) = parse(tokens) {
		println!("result: {:?}", eval(&node));
	}
}

$ cargo run
result: Integer(6)

Next time, I’ll add some more operators and actually get around to operator precedence.

The actual nodes in the AST are represented as an enum with a different case for each node type. The individual node types could also be structs that share a common trait, but so far the enum works fine.

enum Node {
	Integer(i64),
	BinaryOp {
		left: Box<Node>,
		right: Box<Node>,
	},
}

The BinaryOp node doesn’t have an operator type for now because the only supported one is addition, but it’ll be added in the future. Also, the operands of the binary operator are both boxed (stored on the heap) because otherwise the Node type would be recursive and its size not knowable at compile-time.

There’s a helper function that takes a slice of tokens and calls another function that does all of the actual work of parsing.

The parser is a simple recursive descent parser, so the function that does the actual parsing mutably borrows (for the same reasons as the lexer) an iterator that iterates over tokens. The actual iterator it receives is a peekable one, with the underlying iterator type specified with a generic.

I haven’t the faintest idea why the lifetime is needed here, because I still haven’t read that chapter of the book. But hey, adding it fixed the compiler error and it seems to work fine.

fn do_parse<'a, T: Iterator<Item = &'a Token>>(it: &mut std::iter::Peekable<T>) -> Option<Node> {
}

The do_parse function first ensures that there is a node (otherwise it returns None) and then constructs the appropriate node based on the type of the token. For now, that means just integer literals. If there’s any other type of token, it panics because it can’t be turned into a node (again, actual error reporting will come later).

fn do_parse<'a, T: Iterator<Item = &'a Token>>(it: &mut std::iter::Peekable<T>) -> Option<Node> {
	if it.peek().is_none() {
		return None;
	}

	let mut node: Node = match it.next().unwrap() {
		Token::Integer(n) => Node::Integer(n),
		tok => panic!("unexpected token: {:?}", tok),
	};
}

After that, it checks if there are any tokens remaining. If there aren’t, the node is returned as-is.

If there is another token, and it’s a plus sign, it creates a new binary operator node, making the previously-created node to the left operand. To get the right operand, it calls do_parse (putting the recursion into recursive descent) to create a node from the remaining tokens (if it doesn’t find a right-hand side token, it simply panics). If the next token is of any other type, it’s not able to combine it with the existing node into a new node, so it panics.

fn do_parse<'a, T: Iterator<Item = &'a Token>>(it: &mut std::iter::Peekable<T>) -> Option<Node> {
	// ...
	match it.next() {
		Some(Token::Plus) => {
			node = Node::BinaryOp {
				left: Box::new(node),
				right: Box::new(do_parse(it).expect("expression after binary operator")),
			}
		}
		Some(tok) => panic!("unexpected token: {:?}", tok),
		None => (),
	}

	Some(node)
}

And with that, it can parse a simple expression!

fn main() {
	let tokens = tokenize("12 + 34 + 56");
	let node = parse(tokens);
	println!("node: {:#?}", node);
}

$ cargo run
node: Some(
    BinaryOp {
        left: Integer(12),
        right: BinaryOp {
            left: Integer(34),
            right: Integer(56),
        },
    },
)

The eagle-eyed may notice that while we have parsed the expression, we have not parsed it correctly. What’s missing is operator precedence and associativity, but that will have to wait for next time.

I’ve decided to represent tokens themselves as an enum because there are a bunch of cases without any data (e.g., a plus sign is always just a plus sign) and some with (e.g., a number literal token has a value).

When I was reading the Rust book, I was excited to see Rust enums have associated values. Enums with associated data is one of my favorite features of Swift, and I’m glad to see Rust has them too.

#[derive(Debug)]
enum Token {
	Integer(u64),
	Plus,
}

For now, I’m only starting with integer literals and the plus sign. More tokens will come once I’ve got basic lexing and parsing in place.

Most of the work of turning a string into tokens is done in the drumroll please… tokenize function.

It creates an initially empty vector of tokens, and starts iterating through characters in the string.

Single character tokens like plus are the easiest to handle. If the current character matches that of a token, consume the character (by advancing the iterator) and add the appropriate token.

fn tokenize(s: &str) -> Vec<Token> {
	let mut tokens: Vec<Token> = Vec::new();

	let mut it = s.chars().peekable();
	while let Some(c) = it.peek() {
		if *c == '+' {
			it.next();
			tokens.push(Token::Plus);
		}
	}

	tokens
}

Already I’ve encountered a Rust thing. Inside the while loop, c is a reference to a char, so in order to check its value, you have to dereference it. I had expected that you’d be able to compare a value of some type to a reference of that same type (with the language deref’ing the reference automatically), but I can see how forcing the programmer to be explicit about it makes sense.

Next, to parse numbers literals, I check if the current character is in the digit range:

const DIGITS: RangeInclusive<char> = '0'..='9';

fn tokenize(s: &str) -> Vec<Token> {
	// ...
	while let Some(c) = it.peek() {
		// ...
		} else if DIGITS.contains(c) {
			let n = parse_number(&mut it).unwrap();
			tokens.push(Token::Integer(n));
		}
	}
}

You may note that even though the integer token takes a signed integer, I’m totally ignoring the possibility of negative number literals. That’s because they’ll be implemented in the parser along with the unary minus operator.

If the character is indeed a digit, a separate function is called to parse the entire number. This is the first thing that I’ve encountered for which mutable borrows are quite nice. The parse_number function operates on the same data as the tokenize function, it needs to start wherever tokenize left off, and it needs to tell tokenize how much it advanced by. Mutably borrowing the iterator has exactly these properties.

fn parse_number<T: Iterator<Item = char>>(it: &mut T) -> Option<i64> {
	let digits = it.take_while(|c| DIGITS.contains(c));
	let s: String = digits.collect();
	s.parse().ok()
}

Writing the declaration of parse_number was a bit rough, I’ll admit. I haven’t read the generics chapter of the book yet, so I stumbled through a number of compiler error messages (which are very detailed!) until I arrived at this. Having emerged victorious, what I ended up with makes sense though. The specific type of the iterator doesn’t matter, but it still needs to be known at the call site during compilation.

The actual implementation takes as many digits as there are from the iterator, turns them into a string, and parses it into an integer. It returns an optional because the parse method could fail if the string there were no digit characters at the beginning of the string (this will never happen in the one case I’m calling it, but in case it’s reused in the future and I forget…).

Lastly, the tokenize function also ignores whitespace just by calling it.next() whenever it encounters a whitespace char.

And with that, we can tokenize simple inputs:

fn main() {
	println!("tokens: {:#?}", tokenize("12 + 34"));
}

$ cargo run
tokens: [Integer(12), Plus, Integer(34)]

In that vein, I’ve been thinking a lot lately about a building a small programming language. It’s something that I find interesting, and Rust is a good fit for it. It’s also something that lets me start out relatively small and simple, but still always have opportunities for growing more complex.

This post, and the posts that follow it are going to be a sort of experiment. Ordinarily, with an ongoing project, I would just post about it on the fediverse. This time, I’m going to try to write blog posts about it. I expect I’ll continue to post in-the-moment stuff on the fediverse, but I also want to write more semi-regular posts about how it’s going overall. They probably won’t be super detailed, and they certainly won’t be tutorials, but they will be more detailed and more explanatory than much of what I post on the fediverse. And, as with what I would otherwise write on the fediverse, these posts aren’t going to be perfect. I’m not an expert at this; there are going to be mistakes, and I’m going to end up backtracking. But I figure that’s slightly more interesting to read about than if I did everything correctly on the first try.

With that, a few thoughts about the programming language I intend to build:

The biggest thing is that I don’t have a super comprehensive plan. I have some general ideas about what I want to do (and how to do it), but mostly I’m just going to see what happens. I don’t really need to use the end product for anything, I just want to have some fun building it.

The language is going to be interpreted. Building a compiler from scratch is not something I’m interested in (for now), and writing just a frontend for, say, LLVM, doesn’t seem terribly interesting either.

It will be weakly typed, because building a good, sound type system is more complicated than I want to deal with. There may be some static typing features, but it won’t be a focus.

It’s going to be mostly imperative. I’ll probably build more functional-language features at some point, but it’s going to be imperative first.

I’m also going to write everything by hand. No parser generators or combinator libraries. On a related note, there isn’t going to be a formal grammar or spec. The definition of the language will be whatever the parser does.

And with that, here goes nothing.

$ cargo new toylang

In the beginning, there was nothing. Then I started displaying images and almost immediately realized there would need to be some amount of caching. Otherwise, just scrolling down a little bit and then back up would result in images being re-loaded that were present mere seconds ago.

The very first implementation was super simple. It was basically just a dictionary of image URLs to the Data for the image at that URL. This fulfilled the primary goals of being 1) super easy to build and 2) mostly working for the simplest of use cases. But, the blog post doesn’t end here, so clearly there are some issues remaining.

Now, back to the implementation. The first strategy has an obvious issue: memory usage will grow indefinitely. Luckily, there’s a builtin solution for this: NSCache. NSCache is essentially a dictionary that works with the OS to automatically remove its contents when the system needs memory for something else.

This worked well enough for a little while, but there’s another fairly low-hanging optimization. Because URLs aren’t reused, images can be cached for a very long time. Even across app launches, if the cache were persisted to disk. Enter the Cache library. It provides memory- and disk-based caches (the memory one just wraps NSCache). While needing to load things from disk is relatively rare (because once an object is loaded from the on-disk cache, it will be kept in the in-memory cache), it’s still a nice improvement during app launch and for the eventuality that Tusker is asked by the system to give back some memory.

This setup served me fairly well, and (aside from bugfixes) the image caching architecture went untouched for a while. Until I started worked on improving the app’s behavior in degraded network conditions.

When running with the Network Link Conditioner in a super low-bandwidth preset, I launched the app to see what would happen. After a few API requests, all the posts loaded. But none of the images yet (I had purged the on-disk cache in order to test this scenario). Then the user avatars started loading in, one by one. Even for the same user.

The next optimization, then, is obvious. Why many request when few do trick? So, whenever something needs to load an image, instead of only checking if the URL already exists in the cache, I can also check whether there are any in-flight requests for that URL. If there are, then instead of starting a new request, the completion handler just gets tacked on to the existing request. With this in place, when you launch the app under poor network conditions, every instance of a specific user’s avatar will load in simultaneously with the net outcome being that the app overall is finished loading sooner.

The network request batching mechanism also has one more feature. When something calls it to either kickoff a network request or add a completion handler to one that’s already running, it receives back an object (called Request in my code, because that’s what they are from the API consumer’s point-of-view) which can be used to cancel the request. This is so that, if, say, a table view cell is reused, the requests for the cell’s old data can be cancelled. But because the actual network requests are batched together, calling the cancel method on the request object doesn’t necessarily cancel the underlying request (what I call a RequestGroup). The individual completion handler for the “cancelled” request will be removed, but the actual URL request won’t be cancelled if there are still other active handlers.

There’s also one more feature of the batching system. In some cases (primarily table view prefetching) it’s useful to pre-warm the cache. This can either be by just loading something from disk, or by starting a network request for the image (either the request will finish by the time the data is needed, in which case the image will be in the in-memory cache or it will still be in-progress, in which case the completion handler that actually wants the data will be added to the request group). For this, there are also completion handler-less requests. They are also part of the RequestGroup and contribute to keeping the underlying network request alive. Cancelling a callback-less request is trivial because, without the completion handler, each one that belongs to the same URL is identical.

And this was how caching worked in Tusker for almost a year and a half. But, of course, this couldn’t last forever. A few months ago, I was doing a bunch of profiling and optimizing to try to improve scroll view performance and reduce animation hitches.

The first thing I noticed was that while I was just scrolling through the timeline, there was a lot of time being spent in syscalls in the main thread. The syscalls were open, stat, and fstat and they were being called from NSURL’s initFileURLWithPath: initializer. This method was being called with the cache key (which in my case was the URL to the remote image) in order to check if the key string has a file extension so that the extension can be used for the locally cached file. It was being called very frequently because in order to check if an image exists in the disk cache, it needs to check if there’s a file on-disk at the path derived from the cache key, which includes the potential file extension of the key.

Another thing the initFileURLWithPath: initializer does is, if the path does not end with a slash, determine if it represents a directory by querying the filesystem. Since that initializer was also used to construct the final path to the cached file on-disk, it was doing even more pointless work. Because the cache is the only thing writing to that directory and all it’s writing are files, it should never need to ask the filesystem.

There were a couple super low-hanging optimizations here:

First was using NSString’s pathExtension property instead of turning the cache key into an NSURL to get the same property. The NSString property merely interprets the string as a file path, rather than hitting the disk, so it can be much faster.

The second thing was, as the documentation suggests, using the initFileURLWithPath:isDirectory: initializer instead. It allows you to specify yourself whether the path is to a directory or not, bypassing the filesystem query.

I sent both of these improvements upstream, because they were super simple and resulted in a nice performance improvement for free. But, while I was waiting for my changes to be merged, I came up with another optimization. This one was complex enough (though still not very) that I didn’t feel like sending it upstream, so I finally decided to just write my own copy of the library^[2] so I could make whatever changes I wanted.

To avoid having to do disk I/O just to check if something is cached, I added a dictionary of cache keys to file states. The file state is an enum with three cases: exists, does not exist, and unknown. When the disk cache is created, the file state dictionary is empty, so the state for every key is effectively unknown. With this, when the disk cache is asked whether there is an object for a certain key, it can first consult its internal dictionary. If file state is exists or does not exist, then no filesystem query takes place. If the state is unknown, it asks the OS whether the file exists and saves the result to the dictionary, so the request can be avoided next time. The methods for adding to/removing from the cache can then also update the dictionary to avoid potential future filesystem queries.

Combined with the improvements I’d sent to the upstream library, this eliminated almost all of the syscalls from the scrolling hot path. Sadly though, scrolling performance, while better, still wasn’t what I had hoped.

The next thing I realized was that I was being incredibly ineffecient with how images were decoded from raw data.

This WWDC session from 2018 explains that although UIImage looks like a fairly simple model object, there’s more going on under the covers that can work to our advantage, if we let it.

The UIImage instance itself is what owns the decoded bitmap of the image. So when a UIImage is used multiple times, the PNG/JPEG/etc. only needs to be decoded once.

But, in both the memory and disk caches, I was only storing the data that came back from the network request. This meant that every time something needed to display an image, it would have to re-decode it from the original format into a bitmap the system could display directly. This showed up in profiles of the app as a bunch of time being spent in the ImageIO functions being called by internal UIKit code.

To fix this, I changed the in-memory cache to store only UIImage objects^[3], which only decode the original data once and share a single bitmap across every usage. The first time an image is retrieved from the network (or loaded from disk), a UIImage is constructed for it and stored in the memory cache. This resulted in a significant performance improvement. When running on an iPhone 6s (the device I use for performance testing), scrolling felt noticeably smoother. Additionally, this has the very nice added benefit of reducing memory consumption by a good deal.

We can still go one step farther with caching image objects, though. The aforementioned WWDC talk also mentions that the size of the bitmap stored internally by each UIImage is proportional to the dimensions of the input image, not to the size of the view it’s being displayed in. This is because if the same image is shown in multiple views of different sizes, it wants to retain as much information as possible so the image looks as good as it can. Another key effect of using larger-than-necessary images is that the render server needs to do more work to scale down those images to the actual display size. By doing that ourselves, ahead of time, we can keep it from repeatedly doing extra work.

This strategy is a reasonable default, but we, the app developer, know better. Depending on the category of image, it may only be shown at one particular size. In my case, user avatars are almost always shown at a resolution no larger than 50pt × 50pt. So, instead of keeping a bunch of full size bitmaps around, when creating the image that’s going to be cached in-memory, we can use CoreGraphics to downscale the input image to a maximum dimension of 50 points^[4]. And, because the original image data is still cached on disk, if the user goes to a screen in the app where user avatars are displayed larger than usual, we can just load the original data. This is relatively uncommon compared to just scrolling through the timeline, so the slight performance hit here is a worthwhile tradeoff for the improvement in the more common case.

Before we reach the end, there’s one final bit of image caching Tusker does. Some time last year, I added an accessibility/digital wellness preference which changes the app to only display images in grayscale. I use the CoreImage framework to actually do this conversion^[5]. CoreImage is GPU-accelerated and so is reasonably speedy, but it still adds a not-insignificant amount of time, which can be felt on slower devices. To try and mitigate this, the images are also cached post-grayscale conversion.

And that finally brings us to how image caching in Tusker works today. It started out very simple, and the underlying concepts largely haven’t changed, there’s just been a steady series of improvements. As with most things related to caching, what seemed initially to be a simple problem got progressively more and more complex. And, though there are a lot of moving parts, the system overall works quite well. Images are no longer the bottleneck in scrolling performance, except in the rarest of cases (like using grayscale images on the oldest supported devices. Either of those individually are fine, but together they’re just too much). And, memory usage overall is substantially reduced making the app a better platform citizen.

I’ll use Twitter as an example, because I’m very familiar with it. At a high level, what does Twitter want you to do? It wants you to keep using Twitter. Why? So it can show you more ads. But, the platform needs some way of figuring out what ads to show you, because being able to target types of ads at people who are more likely to be interested lets Twitter charge higher prices to advertisers. Twitter doesn’t just want you to be spending time on the platform, it wants you to interact and be engaged.

The most common interaction people have when just browsing Twitter is clicking the Like button. Clicking the Like button also serves as a useful signal to Twitter’s ad targeting algorithms that you probably have some interest in the topics of whichever tweet you liked.

So, Twitter has an action it wants you to perform. But it doesn’t want to tell you to do it, just for you to get into the habit of taking the action in the regular course of using the platform. The same problem game designers are faced with. And Twitter uses the same techniques.

As part of the design of a video game, you need to get the player to do certain things that will lead them to progress through the game. But, you don’t want to just throw a wall of text in their face to explain everything in great detail, you want to be more subtle about it. So, you design the game so that eventually the player will try the thing you want them to do. Then, when the player does the Good Thing, you signal your affirmation and say, “Yes, good job!” But you have to be subtle about it. You want the game to really feel fun for the player, not to give the impression that it’s coddling them. Instead, you use little celebratory cues that the player will perceive without even thinking about. These could be little animations, particle effects, screen shakes, or even auditory cues. There are lots of possibilities, but the key component is that celebrations don’t have to be thought about by the user.

When you click the Like button on a tweet, a few things happen: the heart button itself turns solid red, a small particle effect plays and the number of likes rolls up. The same techniques game designers use. The animation is eye-catching without being distracting and the like count increasing lets you subconsciously connect the action you just took to the effect it had.

I can’t know if Twitter does it with the deliberate intent of making users form habits, but I can’t help but feel like that is a consequence, even if a small one.

The first thing I did after setting up my account was to install Firefox. When the M1 Macs first became available there weren’t yet native versions of big applications like Firefox and Chrome available, but fortunately by the time I got mine, a beta version of Firefox that was recompiled for ARM was available.

Just playing around, visiting a few websites, I was happy to see everything appeared to be working. You might think this would be a given, but when I tried Firefox on the macOS 11 at the beginning of the beta cycle (on my Intel machine, even), video playback was almost completely non-functional.

With Firefox reinstalled, I started signing back into a bunch of my accounts. With long, random passwords, this is rather tedious, so before proceeding I wanted to install my password manager. For reasons that don’t bear getting into, I use an old version of 1Password 6, not the most recent. Of course, this means that it hasn’t been recompiled to be a Universal app; it only runs on Intel. Fortunately, Rosetta comes to the rescue. After copying the .app from another computer, I double clicked it to launch it, just like any other app, after which the OS presented a dialog asking if I wanted to install Rosetta.

Presumably it’s shipped separately and not bundled with the base OS because of the bandwidth requirements of shipping an x86_64 version of every system library that not everyone may need. Additional copies of system libraries are needed because within a single process only one architecture can be used: either everything must be Intel or everything must be Apple Silicon.

Using Rosetta for 1Password worked perfectly. Aside from a slight delay when launching the app for the first time (due to Rosetta pre-translating the Intel binary into ARM). Once it launched, everything worked exactly as I expect. Even IPC between 1Password under Rosetta and the natively-running Firefox extension works flawlessly.

Other Intel-only apps I’ve tried have also worked similarly well. IINA, a video player built on top of mpv, works perfectly. Inkscape, in my brief testing, also works as well as it does on Intel, which is to say, it functions but is not particularly pleasant (fortunately Inkscape some time ago released a native macOS version that no longer depends on XQuartz, removing a potential barrier). I’m also currently running MacVim under Rosetta (in which I’m writing this post) and it’s working without issue. coc.nvim is a Vim plugin which hosts Node.js extensions that provide autocompletion and LSP support. The part of the plugin that runs inside Vim runs, of course, through Rosetta. Node, which I installed through Homebrew, supports ARM natively, and, as with 1Password, IPC accross the architecture boundary works perfectly. The only other significant app I tested under Rosetta is the game Sayonara Wild Hearts. The version in the App Store is Intel-only, and it’s built with Unity 2018.4.7f1, which was released in August of 2019. It works flawlessly running at 1440p @ 144Hz. I can’t say the exact framerate, because the game doesn’t have a counter built in and I’m too lazy to find a standalone macOS one, but it’s definitely well above 60FPS and runs very smoothly, without any noticable input latency or hitching.

The only performance issue I encountered with Rosetta was using an older version of Node.js that was not compiled to run natively. Performance when running under Rosetta was significantly worse than both running natively on the M1 and natively on my Intel laptop. I tried to run npm install but called it quits after almost 5 minutes and switched to a newer version of Node that was compiled for ARM. Presumably this significant performance difference is due to Node’s usage of the V8 VM, which does just-in-time compilation. This requires Rosetta to constantly re-translate x86_64 code into ARM, rather than being able to translate everything ahead-of-time and then execute natively, without interruption.

Of the Mac apps I’ve tried, the sole app that hasn’t worked is MonitorControl, an app that uses the Display Data Channel to control the backlight brightness on monitors that macOS itself does not support. Although the app is running under Rosetta, that isn’t the problem. It seems that the system APIs used to access the DDC either aren’t implemented on ARM or aren’t supported by the driver for Apple’s own GPU. Hopefully this is merely an oversight that can be fixed in a future macOS update.

One of the first things I did upon receiving the machine was following these popular instructions to install Homebrew. I installed two copies of it: one running natively on ARM, installed in /opt/homebrew/, and the other forced to run in x86_64 under Rosetta in the classic location. There’s also a shell alias to let me easily access the Rosetta version by running ibrew which wraps the Homebrew command in arch -x86_64.

I expected the native version of Homebrew to be fairly unstable, and that I would have to fallback on the Intel version frequently. In a pleasant surprise, that has not been the case. Almost everything I’ve tried to install with Homebrew has worked perfectly with the ARM version, even those packages which themselves are x86 only and run under Rosetta. In fact, the only thing that hasn’t worked with native Homebrew has been iPerf.

$ brew list --formula | wc -l
	87
$ ibrew list --formula | wc -l
	1

Up to this point, everything I’d done had been not so different from setting up any previous Mac. The first thing I did that really gave me a sense of how much faster the M1 is was installing Xcode. The non-App Store version of Xcode is distributed in an ~11GB .xip file, which is a compressed, cryptographically-signed format that is notoriously slow to extract. I didn’t time it, but when extracting Xcode on my Intel laptop, I’d start the extraction process and then go back to doing something else for 10 to 15 minutes. On the M1, it took between 5 and 10 minutes. Still not terribly quick, but a substantial improvement. With Xcode installed, I was eager to start playing around with my iOS projects.

Tusker, my Mastodon app for iOS, is easily the biggest iOS project I have, at about 21 thousand lines of Swift code and a couple dozen .xib files (which surprisingly are the source of a significant fraction of the compile time). On my Intel laptop, which is an 8-core 16“ MacBook Pro (with 32GB of RAM, double that of the mini), a clean build in debug mode takes 47.9 seconds. On the Mac mini, the same takes 29.1 seconds, an incredible 39% improvement. When performing a full archive of the app, which builds it in release mode with all optimizations enabled, my Intel machine takes 83.9 seconds and the M1 takes 59.9 seconds, making for a 28% improvement.

It’s difficult to overstate how impressive this is. With Intel having long since fallen off the curve of Moore’s law, a 30 to 40 percent in a single (customer-facing) hardware generation is incredible.

While developing, I don’t actually spend that much time doing full rebuilds, though. So, how about a slightly more practical example of why this is such a big deal? One thing I’ve previously talked about (as have many others) as making perhaps the single biggest difference in the enjoyablity of programming is the speed of the feedback loop. That is to say, how fast is it from the time when I make a change to my code and hit the Build & Run button to the time at which I can actually see the effect of that change. The smaller the window of time in between those two, the less time there is for me to get distracted, the faster I can figure out whether the change has done what I desired, and the faster I can move on to making the next change. To test this, again using Tusker, I built and launched the app, made a 1-line trivial change to the project (adding view.backgroundColor = .red to a certain view controller), and then rebuilt and launched the app once more. This was more difficult to time accurately, but the M1 was about twice as fast as the Intel machine (~5 seconds down from ~10). Because of this, as I’ve continued to use this computer, iOS development is the task for which it’s made the largest difference in my day-to-day activities.

I ran a few more programming-related tests aside from iOS development, because that’s not all I do. When I first got the machine, I was still in the midst of Advent of Code. The puzzle I had most recently completed was day 11, which was implenting a variation of Conway’s Game of Life. I wrote my solution in Elixir on my laptop, and it worked perfeclty well but was bottlenecked by the fact that looking up the n-th element of a list in Elixir is an O(n) operation. I thought these would make for a decent improvised Elixir benchmark. On my Intel MBP, my solution for part 1 of the puzzle ran in 5.9 seconds and part 2 in 8.0 seconds. With BEAM (the VM used by Erlang/Elixir) running natively on the M1, those times came down to 2.6 seconds and 3.5 seconds respectively, or over twice as fast in both cases.

Out of curiosity (I didn’t expect to actually be able to use this machine for work, mainly because of the memory limitation), I also tried compiling the front-end code for the project I work on at my job. It’s composed of approximately 190 thousand lines in total of TypeScript, Vue, and Sass. Here the performance improvement was less impressive, though not insignificant. Compiling everything from scratch takes 85 seconds on my laptop and 69 seconds on the Mac mini. Additionally, when compiling from scratch, the fans in my laptop spin up and become quite loud (not to mention the chasis gets unpleasantly hot) about 60 seconds in, whereas the Mac mini remains silent for the entire duration. While running Webpack in watch mode and making a trivial change (inserting a console.log call in a Vue component) the time it takes for Webpack to finish recompiling is 19 seconds on my laptop and just 11 on the M1. I’m not certain, but I suspect at least part of the reason the performance improvement with Webpack is so much less than with other tasks, particularly during a clean build, is because Node.js is entirely single-threaded.

As for the hardware itself? It’s fantastic.

Like I went over, performance is great; it easily beats my 16“ Intel MacBook Pro in every task I used it for. The other remarkable thing is the noise. It has a single fan, but if you told me it was completely fanless, I would believe you. Not once have I heard the fan turn on. The environment I’m using it in isn’t incredibly quiet, but it’s really not that loud. The only way I can tell the fan in this computer is spinning is if I put my hand behind the rear exhaust vent or if I press my ear up against its surface.

Unlike on the laptop variants of the M1 machine, where the port selection is a measly two USB-C ports, the mini also has an Ethernet port, two type-A USB ports, a HDMI output, and a separate AC input. On a laptop, the I/O would be very limiting; trying to connect to an external desk setup would require some adapters or a big, fancy (not to mention expensive) Thunderbolt dock. But the mini has enough ports that I can directly connect everything from my setup that I’d normally connect to my laptop (granted, I do use my monitor’s built-in USB hub for both machines).

I’ve had only a few issues I’ve experienced that may be attributable to hardware. The first is that when I’m playing back audio to my Bluetooth headphones, they periodically cut out for a fraction of a second before resuming audio playback. I regularly use these same headphones with my Intel Mac on which I haven’t experienced this issue while running either Catalina or Big Sur. The other issue is that when the Mac mini is connected to my 1440p/144Hz display, it loses the 144Hz setting almost every time I wake it from sleep. Odly, just changing the refresh rate while the resolution is set to “Default for display” does nothing. I have to first change the resolution to be scaled down, and then back to the default before changing the refresh rate actually takes effect. The final issue, which has only happened once in the past month, is that when I woke the computer up from sleep, it had stopped outputting video over the HDMI port. It was still sending video over a USB-C to DisplayPort adapter, but not HDMI. Unplugging and reconnecting the HDMI cable didn’t fix the issue, nor did power cycling the monitor. I had to fully restart the Mac mini to resolve the issue.

But, all in all, for a product that’s the first generation of both a new (to macOS) CPU and GPU architecture, this has been an phenomenally good experience. I am incredibly eager to see both what a higher-performance variant of the M1 looks like and future generations of this architecture.

First, a overview of the anatomy of an MP3 file. As we went through in the last post, an MP3 file may optionally start with an ID3 tag containing metadata about the track. After that comes the meat of the file: a sequence of MP3 frames. Each frame contains a specific amount of encoded audio data (the actual amount is governed by a number of properties encoded in the frame header). The total duration of the file is simply the sum of the durations of the individual frames. Each MP3 frame starts with a marker sequence of 11 1-bits followed by three bytes of flags describing the contents of the frame and then the audio data itself.

Based on this information, many posts on various internet forums suggest inspecting the first frame to find its bitrate, and then dividing the bit size of the entire file by the bitrate to get the total duration. There are a few problems with this though. The biggest is that whole MP3 files can generally be divided into two groups: constant bitrate (CBR) files and variable bitrate (VBR) ones. Bitrate refers to the number of bits used to represent audio data for a certain time interval. As the name suggests, files encoded with a constant bitrate use the same number of bits per second to represent audio data throughout the entire file. For the naive length estimation method, this is okay (though not perfect, because it doesn’t account for the frame headers and any potential unused space in between frames). In variable bitrate MP3s though, each frame can have a different bitrate, which allows the encoder to work more space-efficiently (because portions of the audio that are less complex can be encoded at a lower bitrate). Because of this, the naive estimation doesn’t work at all (unless, by coincidence, the bitrate of the first frame happens to be close to the average bitrate for the entire file). In order to accurately get the duration for a VBR file, we need to go through every single frame in the file and sum their individual durations. So that’s what we’re gonna do.

The overall structure of the MP3 decoder is going to be fairly similar to the ID3 one. We can even take advantage of the existing ID3 decoder to skip over the ID3 tag at the beginning of the file, thereby avoiding any false syncs (the parse_tag function needs to be amended to return the remaining binary data after the tag in order to do this). From there, it’s simply a matter of scanning through the file looking for the magic sequence of 11 1-bits that mark the frame synchronization point and repeating that until we reach the end of the file.

def get_mp3_duration(data) when is_binary(data) do
  {_, rest} = ID3.parse_tag(data)
  parse_frame(rest, 0, 0, 0)
end

The parse_frame function takes several arguments in addition to the data. These are the accumulated duration, the number of frames parsed so far, and the byte offset in the file. These last two aren’t strictly needed for parsing the file, but come in very useful if you have to debug any issues. The function has several different cases. The first looks for the sync bits at the start of the binary and, if it finds it, parses the frame header to caclulate the duration, adds it to the accumulator, and recurses. The next case skips a byte from the beginning of the binary and then recurses. And the final case handles an empty binary and simply returns the accumulated duration.

def parse_frame(
	  <<
	    0xFF::size(8),
		0b111::size(3),
		version_bits::size(2),
		layer_bits::size(2),
		_protected::size(1),
		bitrate_index::size(4),
		sampling_rate_index::size(2),
		padding::size(1),
		_private::size(1),
		_channel_mode_index::size(2),
		_mode_extension::size(2),
		_copyright::size(1),
		_original::size(1),
		_emphasis::size(2),
		_rest::binary
      >>,
      acc,
      frame_count,
      offset
    ) do
end

def parse_frame(<<_::size(8), rest::binary>>, acc, frame_count, offset) do
  parse_frame(rest, acc, frame_count, offset + 1)
end

def parse_frame(<<>>, acc, _frame_count, _offset) do
  acc
end

The main implementation of the parse_frames function isn’t too complicated. It’s just getting a bunch of numbers out of lookup-tables and doing a bit of math.

The first thing we need to know is what kind of frame we are looking at. MP3 frames are divided two ways, by the version of the frame and the layer of the frame. In the header, there are two fields, each two bits wide, that indicate the version and layer. But not every combination of those bits are valid. There are only three versions (and version 2.5 is technically an unnoficial addition at that) and three layers. We can use a couple functions to look up atoms representing the different versions/layers, since it’s more convenient than having to use the raw numeric values in other places. We also return the :invalid atom for version 0b01 and layer 0b00 respectively, so that if we enocunter one when parsing a frame, we can immediately stop and look for the next sync point.

defp lookup_version(0b00), do: :version25
defp lookup_version(0b01), do: :invalid
defp lookup_version(0b10), do: :version2
defp lookup_version(0b11), do: :version1

defp lookup_layer(0b00), do: :invalid
defp lookup_layer(0b01), do: :layer3
defp lookup_layer(0b10), do: :layer2
defp lookup_layer(0b11), do: :layer1

The next piece of information we need is the sampling rate which is the frequency (in Hertz) with respect to time at which individual audio samples are taken. In the header, it’s also represented by two bits. As before, we pattern match in the function definition to find the actual sampling rate from the index, and return :invalid if the index is not permitted.

defp lookup_sampling_rate(_version, 0b11), do: :invalid
defp lookup_sampling_rate(:version1, 0b00), do: 44100
defp lookup_sampling_rate(:version1, 0b01), do: 48000
defp lookup_sampling_rate(:version1, 0b10), do: 32000
defp lookup_sampling_rate(:version2, 0b00), do: 22050
defp lookup_sampling_rate(:version2, 0b01), do: 24000
defp lookup_sampling_rate(:version2, 0b10), do: 16000
defp lookup_sampling_rate(:version25, 0b00), do: 11025
defp lookup_sampling_rate(:version25, 0b01), do: 12000
defp lookup_sampling_rate(:version25, 0b10), do: 8000

The last piece of information we need from the header is the bitrate, or the number of bits that are used to represent a single second (or, in our case, the number of kilobits, for simplicity’s sake). The header has a four bit field that represent which index in the lookup table we should use to find the bitrate. But that’s not all the information that’s necessary. In order to know which lookup table to use, we also need the version and layer of the frame. For each version and layer combination there is a different set of bitrates that the frame may use.

So, the lookup_bitrate function will need to take three parameters: the version, layer, and bitrate index. First off, indices 0 and 15 are reserved by the spec, so we can just return the :invalid atom regardless of the version or layer. For the other version/layer combinations, we simply look up the index in the appropriate list. A couple things to note are that in version 2, layers 2 and 3 use the same bitrates, and all layers for version 2.5 use the same bitrates as version 2.

@v1_l1_bitrates [:invalid, 32, 64, 96, 128, 160, 192, 224, 256, 288, 320, 352, 384, 416, 448, :invalid]
@v1_l2_bitrates [:invalid, 32, 48, 56, 64, 80, 96, 112, 128, 160, 192, 224, 256, 320, 384, :invalid]
@v1_l3_bitrates [:invalid, 32, 40, 48, 56, 64, 80, 96, 112, 128, 160, 192, 224, 256, 320, :invalid]
@v2_l1_bitrates [:invalid, 32, 48, 56, 64, 80, 96, 112, 128, 144, 160, 176, 192, 224, 256, :invalid]
@v2_l2_l3_bitrates [:invalid, 8, 16, 24, 32, 40, 48, 56, 64, 80, 96, 112, 128, 144, 160, :invalid]

defp lookup_bitrate(_version, _layer, 0), do: :invalid
defp lookup_bitrate(_version, _layer, 0xF), do: :invalid
defp lookup_bitrate(:version1, :layer1, index), do: Enum.at(@v1_l1_bitrates, index)
defp lookup_bitrate(:version1, :layer2, index), do: Enum.at(@v1_l2_bitrates, index)
defp lookup_bitrate(:version1, :layer3, index), do: Enum.at(@v1_l3_bitrates, index)
defp lookup_bitrate(v, :layer1, index) when v in [:version2, :version25], do: Enum.at(@v2_l1_bitrates, index)
defp lookup_bitrate(v, l, index) when v in [:version2, :version25] and l in [:layer2, :layer3], do: Enum.at(@v2_l2_l3_bitrates, index)

One could do some fancy metaprogramming to generate a function case for each version/layer/index combination to avoid the Enum.at call at runtime and avoid some of the code repetition, but this is perfectly fine.

With those four functions implemented, we can return to the body of the main parse_frame implementation.

def parse_frame(...) do
  with version when version != :invalid <- lookup_version(version_bits),
       layer when layer != :invalid <- lookup_layer(layer_bits),
       sampling_rate when sampling_rate != :invalid <- lookup_sampling_rate(version, sampling_rate_index),
       bitrate when bitrate != :invalid <- lookup_bitrate(version, layer, bitrate_index) do
  else
    _ ->
      <<_::size(8), rest::binary>> = data
      parse_frame(rest, acc, frame_count, offset + 1)
  end
end

We call the individual lookup functions for each of the pieces of data we need from the header. Using Elixir’s with statement lets us pattern match on a bunch of values together. If any of the functions return :invalid, the pattern match will fail and it will fall through to the else part of the with statement that matches anything. If that happens, we skip the first byte from the binary and recurse, looking for the next potential sync point.

Inside the main body of the with statement, we need to find the number of samples in the frame.

def parse_frame(...) do
  with ... do
    samples = lookup_samples_per_frame(version, layer)
  else
    ...
  end
end