This might seem like bait but I assure you it isn't - I reach for MySQL by default over postgres - but why did you choose MySQL for the state database? In terms of HA and synchronous clustering it's a good solution but unless I'm misreading it seems like you only use it for a key value store and for an append only log, neither of which are necessarily the problems you think "MySQL" for.
Given what seems to be a pretty fashionable stack (kubernetes, using etcd directly, cloud first, IaC and custom database technology) I'm interested what a company like that reaches for MySQL. Was the MySQL cluster already deployed for other reasons?
Yeah, we've had a PlanetScale cluster since the early days that powers the dashboard experience (turbopuffer.com/dashboard). That's a fairly standard webapp (customers, orgs, users, API keys, namespace stats) that benefits from the relational model.
When we built the cplane it was natural to just store the cplane ops in MySQL too. cplane ops belong to clusters which belong to customers, so it's actually rather nice to be able to foreign key it all together [0].
[0] Well, not actually enforced foreign keys, because Vitess, but you get the idea. It lets us apply settings at the customer level that flow to all ops for that customer's clusters.
I was part of the team that proposed this manner of operation (Kubernetes Operator)[1]/controller and it's immensely cool to see a competent team make what seems like an incredible one that really scales the business.
I had always assumed databases would be the silo that had the most benefits for an Operator. There are a few good Postgres Operators, but not many outside of that which actually do the day 2 operations.
thanks rob! It does help that we are backed by object storage, and we try to make architecture choices that keep the system simple and safe to roll out at any time.
Do you collect metrics from those BYOC clusters? If so, I’m assuming you push them, not pull them? That would seem to follow the same direction as the cluster operator config syncs (i.e. the direction in which connections are made, not in which data flows).
Yep, all pushed based directly from cluster! It helps that most of the observability ecosystem is converging on this? Even Prometheus, while pull-based inside the cluster, can remote-write to external storage, and that's the shape we'd pick even in that world. (we use datadog right now)
This might seem like bait but I assure you it isn't - I reach for MySQL by default over postgres - but why did you choose MySQL for the state database? In terms of HA and synchronous clustering it's a good solution but unless I'm misreading it seems like you only use it for a key value store and for an append only log, neither of which are necessarily the problems you think "MySQL" for.
Given what seems to be a pretty fashionable stack (kubernetes, using etcd directly, cloud first, IaC and custom database technology) I'm interested what a company like that reaches for MySQL. Was the MySQL cluster already deployed for other reasons?
Yeah, we've had a PlanetScale cluster since the early days that powers the dashboard experience (turbopuffer.com/dashboard). That's a fairly standard webapp (customers, orgs, users, API keys, namespace stats) that benefits from the relational model.
When we built the cplane it was natural to just store the cplane ops in MySQL too. cplane ops belong to clusters which belong to customers, so it's actually rather nice to be able to foreign key it all together [0].
[0] Well, not actually enforced foreign keys, because Vitess, but you get the idea. It lets us apply settings at the customer level that flow to all ops for that customer's clusters.
I've been following Turbopuffer for a while; IIRC the founder talked about scaling MySQL at Shopify. So might just be operational familiarity.
I was part of the team that proposed this manner of operation (Kubernetes Operator)[1]/controller and it's immensely cool to see a competent team make what seems like an incredible one that really scales the business.
I had always assumed databases would be the silo that had the most benefits for an Operator. There are a few good Postgres Operators, but not many outside of that which actually do the day 2 operations.
[1]: https://operatorhub.io/what-is-an-operator
thanks rob! It does help that we are backed by object storage, and we try to make architecture choices that keep the system simple and safe to roll out at any time.
Interesting read. A simpler way to manage the data plane.
Curious to understand if you also use OTEL layer to understand what is happening in your BYOC cluster ?
Yes, the database is instrumented with OTel, but Datadog (via local agent) is our primary sink.
Do you collect metrics from those BYOC clusters? If so, I’m assuming you push them, not pull them? That would seem to follow the same direction as the cluster operator config syncs (i.e. the direction in which connections are made, not in which data flows).
Yep, all pushed based directly from cluster! It helps that most of the observability ecosystem is converging on this? Even Prometheus, while pull-based inside the cluster, can remote-write to external storage, and that's the shape we'd pick even in that world. (we use datadog right now)
Everything but self hosted, is it?
[dead]
[dead]
[flagged]