Infrastructure as SaaS

Ampbase is an OpAMP control plane for the agents running on your hosts — Fluent Bit, Vector, Telegraf, OTel Collector, coding agents, and Tetragon — managed from one place with versioned configs and flag-driven rollouts. There's a 14-day trial and SSO on every plan.

In the last post I said each customer gets a bucket of their own, and that the credentials which reach one customer’s data cannot address anyone else’s. Now we’ll dive deep into the Compute. TL;DR every organization on Ampbase runs on its own Fly.io app, on its own private network, with its own secrets, its own Vault keys, and its own machines. When I think about “Compute” or “Cloud Infrastructure”, what I’m really concerned with is blast radius, identity, network access, reliability, and finally the actual unit of compute. And for me, well designed “Compute” makes each of those easy to reason about, operationally simple, and hard to use wrong (i.e. you shouldn’t have to ask “am I holding it wrong”).

What I always loved about helping build Fly.io is everything can be done in an afternoon with flyctl. However, we’re building a SaaS product, so we took the primitives available in the Machines API, paired them with open source tools, and composed a system to manage org apps as “Compute”. Like most first versions, it was less than ideal but it worked and let us see and feel the product. Taking a step back, it was pretty obvious what we were reinventing and the blocker was clear. I was at Fly.io for the whole life of the community fly-apps provider, including its archiving, so I knew what was and was not out there. It was time for some NIH: we had to write a Fly provider. Our provider is now public on the Terraform and OpenTofu registries. We’ll show some snippets of the HCL that runs in production (with more examples in the repo) throughout the post.

What a customer gets #

Fly.io in 20 seconds, if you haven’t used it: a Fly organization holds billing, members, and API tokens (not to be confused with an Ampbase organization, which is one of our customers). Inside it, an app is a group of Machines, Firecracker microVMs you drive with the Machines API or flyctl, and every Machine in an app gets that app’s secrets. Apps talk over a private network, and Flycast gives an app a private address on it. Fly.io essentials does it better than I can.

An Ampbase organization is a Fly app with a deterministic name containing a ULID. Everything else is named after it, so there is no lookup table between the Fly app, the bucket, and any other piece of infrastructure needing a name. Inside the app are multiple machines serving the OpAMP server the customer’s agents connect to. It’s on a private network inside a customer-only Fly organization with selective network access back to the Ampbase control plane. Around it, the rest of what provisioning creates, and how the pieces feed each other:

 tigris_org_provision ──▶ fly_secrets ──▶ fly_machines
          │                                          │
          ▼                                          │
 tigris_access_key                                   │
 tigris_bucket_lifecycle    ┌─ network {ulid} ───────▼──────────────┐
                            │  org fly_app {ulid}                   │
 vault_transit_keys         │    fly_ip flycast  ◀── control plane  │
 vault tigris role ────────▶│                                       │
                            │  intel fly_app {ulid}                 │
                            │                                       │
                            │  fly_ips  ──▶ control plane           │
                            └───────────────────────────────────────┘

Yea, we see it too, there’s another app in the box. When it came to building the analytics system, which we call intel, compute for the daily intelligence run could be ephemeral and didn’t need the same permissions as the org app. So it gets an app of its own on the same network. The only credential it holds is one it leases from Vault using its own machine identity and a single presigned POST endpoint for storing results in the control plane bucket.

There are obviously other ways we could do this but we like the pattern we chose because the isolation is based on access needs and is repeatable across what we hope will be hundreds, then thousands of orgs.

Where the boundaries are #

 Fly org: control plane                      Tigris, as a partner
 ┌──────────────────────────────────┐        ┌────────────────────┐
 │ amp    vault    clickhouse       │───────▶│ directory bucket   │
 │ each on its own network          │        │   tofu state       │
 └───────┬──────────────────▲───────┘        │ tofu bucket        │
         │ fly-replay,      │ flycast out,   │   tofu state       │
         │ flycast in       │ to the rest    └────────────────────┘
 ┌───────▼──────────────────┴───────┐
 │ Fly org: customers-only          │        ┌────────────────────┐
 │ ┌─ network A ─────────────────┐  │        │ Tigris org A       │
 │ │ org A                       │──┼───────▶│ bucket w/ keys     │
 │ │ intel A                     │  │        └────────────────────┘
 │ └─────────────────────────────┘  │        ┌────────────────────┐
 │ ┌─ network B ─────────────────┐  │        │ Tigris org B       │
 │ │ org B                       │──┼───────▶│ bucket w/ keys     │
 │ │ intel B                     │  │        └────────────────────┘
 │ └─────────────────────────────┘  │
 └──────────────────────────────────┘

We like personal boundaries so it’s no surprise we lean heavily on technical boundaries here. Each network gives us control to decide what can access what and in what direction. Orgs have no access to each other. And each is granted specific access from its network back to certain control plane networks. Thankfully, the Tigris Partner API provides the boundaries to storage so we don’t have to think about it at all.

The customer apps live in a separate Fly organization from the control plane primarily to limit the blast radius. A leaked control-plane API token authorizes destroy, scale, and secrets operations against every app in its org, and if the customer apps were in that org, a leak in a CI job would be a leak of every customer’s data plane. No system is without compromise or exposure so the goal is to reduce the risk when possible. The residual risk is real: a leak of the provisioning token is still a multi-tenant event. The split moved the boundary and our goal is to remove it.

The MVP before the customers #

The first provisioner was “simple” Go. It created the app, allocated the IP, called the Tigris Partner API for the bucket, set the secrets, created the machine, polled for health, and cleaned up on failure. Then it needed a reconciler, so it grew a state classifier with nine classes, a blue-green replacement worker with four resume cases for when the process died mid-replacement, tiered rollout tracking, and a cursor for sweeping the fleet in batches. About 2,700 lines, and it worked (mostly).

Imperative is great for quick iterations (one could even argue “bash would do just fine”). However, not so great for ensuring consistency across infrastructure that is constantly evolving. The pattern is well trodden: describe desired state, read actual state, compute a diff, apply the corrections, handle partial failure. We had written a worse Terraform, by hand, for one module.

I kinda knew this would be the eventual direction and the question was really about when to make the move. Once the core infrastructure was in place, the per-org infrastructure became one HCL template, rendered with per-org variables, applied by OpenTofu from inside our own worker process. The classifier became tofu plan. The replacement worker became an in-place machine update that settles whether the machine was started, stopped, or suspended. Partial failure recovery became “apply is idempotent, run it again.” What was left in the control plane: the rollout policy that decides when an org may receive a new image, and the pool that decides whether to provision one at all. Enter our first (and probably not our last) NIH open source repo.

The provider #

Our Fly provider manages most of what the Machines API exposes: apps, machines, IPs, secrets, volumes, and certificates. The client underneath it is generated from Fly’s own OpenAPI document, and reused throughout our codebase.

What separates it from a thin wrapper is the behaviors we need for the provider as well as my own knowledge of how the system works (or at least used to). A fly_machine is not “created” when the API returns an ID. The provider waits until the machine reports started and its health check passes. Without that wait the apply finishes while the console is still booting, the pool counts the org as ready, and a customer can be handed an org whose OpAMP server is not answering yet, so their first agent connection fails.

Fly never returns a secret’s value, only a digest, so a provider cannot tell from the API whether the value it last sent is current. Thankfully, the answer is in Terraform’s provider framework: the value is write-only, never in plan or state, and rotation is driven by a version of your choosing.

resource "fly_secret" "aws_access_key" {
  app              = fly_app.org.name
  name             = "AWS_ACCESS_KEY_ID"
  value_wo         = tigris_org_provision.org.access_key_id
  value_wo_version = tigris_org_provision.org.access_key_id
}

resource "fly_secret" "aws_secret_key" {
  app              = fly_app.org.name
  name             = "AWS_SECRET_ACCESS_KEY"
  value_wo         = tigris_org_provision.org.secret_access_key
  value_wo_version = tigris_org_provision.org.access_key_id
}

There are a few different patterns for choosing the version.

• a non-secret value is its own version (and should arguably be a normal environment variable).
• if it comes with an identifier, use it: a new secret means a new ID.
• otherwise, hash the secret.

The secrets live in one template and the machine in another, so knowing a machine needs to be updated requires plumbing the output into the min_secrets_version.

output "secrets_version" {
  value = max(
    fly_secret.aws_endpoint.version,
    fly_secret.aws_access_key.version,
    fly_secret.aws_secret_key.version,
    fly_secret.aws_region.version,
  )
}

resource "fly_machine" "org" {
  count = length(local.machine_regions)
  app   = local.org_app_name
  image = var.console_image

  min_secrets_version = var.secrets_version
}

Fly secret changes are not seen by running machines so they are “staged” until a machine is created/updated. That alone isn’t enough, though. A rotation on its own changes nothing in the machine’s plan, so no update runs. And when an update does run for some other reason, secret replication across regions is asynchronous, so the call can land before the new value does and the machine boots on the old one with nothing in the plan to say so. min_secrets_version covers both: the rotation becomes a change to the machine, and Fly won’t start it on anything older than the version you named.

The cross-org grants in the diagram are declared the same way. Flycast lets you allocate a private address on an app that a different org’s network is permitted to route to, so each org carries a fly_ip per control-plane service its network needs to reach:

resource "fly_ip" "vault_grant" {
  provider   = fly.ampbase
  app        = local.vault_app
  type       = "private_v6"
  network    = local.org_network
  org_slug   = var.fly_org
  depends_on = [fly_app.org]
}

Read that carefully, because it doesn’t matter how often I see it, it always takes me a minute to reason about the flow. The resource is created under the control-plane provider, on the control-plane’s Vault app, but the network it names is the customer org’s, and the org_slug is the customer org. The grant targets the network, not the app, which is why the intel app inherits the Vault grant by joining it and can fetch its per-run bucket credential with no secret set on it.

Attenuating Fly’s tokens #

The boundaries section left a loose end: the token that provisions customer apps is org-wide, and leaking it is the kind of press we’d like to avoid. Creating apps, allocating cross-org grants, and calling the Partner API need authority wider than one org, so it became its own control template primarily used at signup and rarely reconciled. The machines running the OpAMP server use a second template that only needs its own org app’s scope, carries no credentials, and is constantly being reconciled. So how could we constrain the token for all that constant work? Macaroons, and I’m super thankful for the work Thomas and Ben (but mostly Ben :P) did to make macaroons accessible to customers.

I could never do macaroons justice so I’ll focus on what’s been useful when building Ampbase. The token is an HMAC chain, and each caveat extends the chain by one link. Anyone holding a token can add a caveat, and nobody can remove one, because removing a link would break every signature after it. Only minting the root needs Fly’s key. So the holder of an org-level token can hand out narrower tokens all day without an API call, and Fly’s verifier checks every caveat on the way in, with caveats of the same kind intersecting.

 fly tokens create org <customers org>       org-level, long-lived
   + a third-party caveat we author           inert without a discharge
   = the parent, written to Vault
        │
        │ per run, offline, no call to Fly
        ▼
 child = parent + Apps{org app} + ValidityWindow{20 minutes}
        │
        │ + a discharge the worker mints under a key Vault never sees
        ▼
 FLY_API_TOKEN = FlyV1 <child>,<Fly's auth discharge>,<ours>

So we’ve got two caveats: an Apps caveat naming the org’s app with every action, plus read on the “shell” Apps we use for images used on every org, and a validity window:

apps := resset.ResourceSet[uint64, resset.Action]{appID: resset.ActionAll}
// pulls are the "shell" Apps containing only an image so a manifest fetch
// is a read of the repository's app
for _, id := range pulls {
    apps[id] = pullMask
}
return p.attenuate(window, &resset.IfPresent{
    Ifs:  macaroon.NewCaveatSet(&flyio.Apps{Apps: apps}),
    Else: resset.ActionNone,
})

We decoded one of Fly’s own deploy tokens and found IfPresent{[Apps, FeatureSet{builder, wg}], Else: read}; ours is the same thing narrowed by one step. The Apps caveat keys on the app’s numeric internal ID, not its name. And a machine’s image is resolved with the run’s token, so a child that named only the org’s app could not pull the console image. That’s what the read on the “shell” apps is for.

The window is twenty minutes: fifteen for the run’s own deadline and five for a machine to boot and reach the API. It opens thirty seconds early, which is the allowance Fly’s own discharges carry, so a verifier whose clock trails ours does not refuse a fresh token. When it closes, the next Machines call gets a 401, tofu exits nonzero with state written for whatever it had applied, and the next pass re-plans under a fresh child.

In an ideal world, we’d be able to scope the control template to say “can create App named XYZ” but no Apps caveat can name something that does not exist yet so it’s scoped by action instead for the whole org with delete removed. Now a provisioning run can create and configure anywhere in the customers org, but it can only delete inside its own org’s apps. The destroy side is the inverse: it names the two apps (console + intel) and carries read and delete.

If you decide to drink the macaroon koolaid, plan to try and fail because the API documents no per-operation scope. A Flycast allocation was refused at rwcd and at rwcC and accepted at rwcdC, and a secret set was refused at rwcC, which is how we landed on every action for the org’s own apps and everything but delete everywhere else.

Third-party caveats

The third-party caveat is for custody. Fly already does this to itself: every token it issues carries a caveat that only Fly’s authentication service can discharge, so the token on its own is, in their words, useless. We put one more such caveat on the parent, with a location under our own domain and a key that is a Fly secret on the worker and never enters Vault. Now the parent in Vault, and every child minted from it, is inert bytes until the worker mints a discharge for that run, bound to the parent’s tail so one discharge serves every child. Fly’s verifier accepts a discharge for a location and key it has never seen, and refuses the same token without one. Why does any of this matter? A leaked Vault snapshot or backup holds no usable credential for the customers org, and a run cannot outlive the window the worker set for it. It does nothing against a live compromise of the worker, which obviously has the key because it’s doing the work.

Whenever we all talk about tokens, one of the most irritating parts is revocation because it’s where stateless becomes stateful so a core part of the design is the best revocation is no revocation. Fly revokes by nonce, and a child carries its parent’s nonce, so a child dies with its parent and only with its parent. With a twenty-minute validity window and no Vault lease renewal, the exposure radius is contained and short enough to simply wait until it’s invalid. We considered server-minted deploy tokens (which Fly can revoke by ID) but adding the API call and needing to track per run for such a short period feels like unnecessary complexity.

A machine per run #

The tofu run machine is self-destroying, scoped specifically to the job, and in an isolated Fly App with nothing else. We give it a deterministic name which is tied to the Vault claim (we use OIDC for all the things), so its policy reads exactly one role naming exactly one org’s app. Any state arrives and leaves by presigned URL, so it holds no storage credential of any kind and its exit code is the plan result. The full workflow is driven by an fsm library born out of the internals of flyd. Its origin is scoped to only one process on one host so state was simple. We couldn’t use it in that manner so we worked through adding object storage as a supported backend because Object Storage Is All You Need. These machines’ plans and applies, and an org’s destroy, all run as fsm workflows, which gives us a durable execution environment to manage the fleet of customer apps across regions. If you’re about to say “just use Temporal”, you’re not wrong, and we likely will at some point.

Why we decided to open source it now #

The provider has been the piece under every org for months now, and we’ve hit (and filed off) most of the rough edges you run into doing a few hundred plans and applies a day: the waits, write-only secrets, a machine that got destroyed before it ever ran but still read as running, and container behavior we had documented wrong ourselves. It’s core to the platform and it’s stopped changing much, so it was time for it to leave our repo.

It also let us delete something annoying. While the provider lived in our tree, every operator’s machine needed a hand-filled mirror directory, and a mirror silently shadows the lock file and the version pin, so a stale copy would still validate green. Publishing it as ampbase-io/fly on both registries made all of that go away. Every root module commits a lock file, CI validates each one against the registry provider, and the worker image builds the provider from the same Go module the pool imports.

The other reason is that AI coding agents are really good at writing Terraform, even for stuff that will never see a second apply. A spike, or a machine stood up to answer one question about how Fly behaves and torn down right after. Half of what the examples say was “measured” was measured exactly that way. With the provider on the registry, real examples, and the schema docs the registry renders, you can point an agent at it instead of setting everything up by hand.

A provider without examples isn’t worth much, and the only examples I’d trust are the ones we actually run. So the examples are lifted from production, and CI validates them against the provider built from the same commit: a ClickHouse cluster with a Keeper quorum, a Vault HA cluster adopted from machines flyctl created (with the image built inside the apply), a public surface with a certificate gate, and the three secret version patterns from earlier. Each README calls out what we measured against Fly and what we assumed.

Same as the storage post, what we wanted was for the boundary between customers to come from the platform, so it’s hard to hold wrong, and for the next per-customer resource to just be a resource block and an apply. The provider is what got us there, so have at it.