Shipping and staying up

A game is not finished when it works on the developer’s laptop. This last part follows a change from an editor to a player’s phone, and covers the pieces that keep that journey safe: automated checks that play the game, and a small server stack that can lose a machine and carry on.

Tests that go further than tests

Unit tests (about nine hundred, one fast suite) cover the rules from Parts 2 and 3. But a lot of what goes wrong in a game is not a rule. It is a button that overflows a small screen, a memory leak after twenty in-game years, or a menu a screen reader cannot name. So the pipeline has three more layers.

• Lint and types. The linter allows zero warnings. The type checker runs over the pure modules with no errors allowed, and over everything else with a ratchet: the error count may go down, never up. It lets us adopt types in a big old codebase one corner at a time without a flag day.
• The smoke test. A real Chrome opens the built game, builds a town, runs years, opens every management page at phone, landscape and desktop width, presses the buttons, and checks that nothing overflows or throws. An accessibility engine then scans each page for serious problems such as contrast, names and roles.
• The soak test. A town is played for fifteen to twenty years while a monitor watches memory, page elements, scene objects, the size of the saved city and the speed of the game. If any of them keeps growing, or the graphics card is left holding geometry nobody uses, the build fails. This catches the slow leaks that only a long game session would ever find.

// the idea behind the soak test, in miniature
const before = measure();                  // heap, DOM nodes, scene objects, save size
for (let year = 0; year < YEARS; year++) playOneYear();
const after = measure();

for (const key of Object.keys(before)) {
  const growth = after[key] / before[key];
  if (growth > TOLERANCE[key]) fail(key + ' grew ' + growth.toFixed(2) + 'x over ' + YEARS + ' years');
}

A pre-commit hook runs the quick half of these checks before any commit leaves the machine. The full set runs in continuous integration, with a real Postgres database attached so that the tests for running several server copies against one database are run, not skipped.

The server: small on purpose

The server is plain Node with no web framework and one runtime dependency, the database driver. It does the things the browser cannot: sharing cities, keeping accounts for those who want one, anonymous analytics, the sponsor system and a partner portal. Everything it knows is stored in Postgres; it writes nothing to its own disk.

That last sentence is the key design choice. A server that keeps nothing of its own can be copied, replaced or restarted at will, because the copy has nothing to lose.

Releasing without a gap

With two identical copies behind the proxy, a new release is a rolling restart: take one copy out of service, replace it, wait until it reports healthy, then do the other. On a stop signal a copy first tells the proxy it is leaving, lets requests that are already running finish, and then exits.

We do not take that on trust. A test script builds the whole stack in containers on a laptop, sends steady traffic through it (around three hundred requests a second of page loads and play events), and then abuses it:

We are honest about the limits too. Two copies on one small machine share its processor, so this setup buys resilience, not capacity. Real extra capacity needs a bigger machine or a second one, and the proxy and the database are still single points of failure until they are doubled as well. Writing the limits down next to the results is part of the result.

Fast, from a long way away

Many of our players are far from the server, and a round trip across continents can cost over a second before the first byte arrives. Two ideas fix most of it.

• Fingerprints. The landing page names each of its files with a short hash of the file’s content. A changed file gets a new address, so nothing old is ever shown beside something new, and an unchanged file can be kept by the browser for a year. A returning visitor then asks the server for none of the page’s files.
• A short edge cache for the two big pages. The landing page and the game are the same for every visitor, so the CDN may hold them for about a minute while browsers are still told to check every time. A release reaches players within minutes, and the typical first byte fell from about 1.2 seconds to a fraction of that.

// make a file's address depend on its content
function fingerprint(file) {
  return createHash('sha256').update(fs.readFileSync(file)).digest('base64url').slice(0, 10);
}
html = html.replace(/(\/landing\/)([\w\-\/.]+\.(?:css|js|jpe?g|png))/g,
  (m, dir, rel) => dir + rel + '?v=' + fingerprint(path.join(ROOT, rel)));
// then: Cache-Control: public, max-age=31536000, immutable   for any address that carries ?v=

Watching, kindly

Once the game is out, we want to know what real phones do to it. Crashes in players’ games are reported so that we can fix what actually happens rather than what we imagine, under the privacy choices set out in the privacy policy. A small watcher on the server records processor, memory, disk, whether the game answers and how old the newest backup is, and raises its hand when a number looks wrong.

What we learned

1. Test the experience, not only the code: a real browser, real screen sizes, real years of play.
2. Make servers boring. Stateless copies and one source of truth are what make restarts safe.
3. Do not trust a resilience claim you have not tried to break. Kill a copy and count the failures.
4. Write down what you do not have. It is more useful than the list of what you do.

One more part

That is the machinery that carries the game: one file, a testable kernel, a visible simulation, an adaptive renderer, a soundtrack that keeps time, data that is never believed, and a pipeline that plays the game before you do. One part is left, and it is the one that makes the town green: how trees, flowers and water are grown from a seed.