It's probably intuitive to anybody who's tried to do this the hard way, but if your target is people without that experience, you might want to dumb it down even more. So this is from your readme:
What is that URI supposed to be? Is that where the thing will eventually get served from and I have to run this on the server, or synchronize an output directory to the server, or is that going to do some sort of magic upload itself?
Totally agree here. The current target is people who are familiar with the process, and we plan to write more documentation as we start thinking more about accessibility to a broader audience. We actually ran into this issue ourselves while building the tool - the Debian documentation is all there, but not the easiest to approach. These docs are one of the rough edges (sorry!) since we wanted to get this out earlier for feedback.
To answer your direct question: on self-hosted repositories, the URI right now is just a string identifier. We are working right now on a cloud hosted service, and there the URI will help us figure out which repository to actually serve.
One of the tools we tried was Aptly! A couple issues we had with Aptly:
1. Aptly needs to rebuild the entire repository before it can do any changes. One of our customers builds packages for every Linux OS, and every architecture, and each release contains 10-something separately packaged tools, and they do regular releases. This means their repository of packages is _huge_. When they try to do small maintenance actions like removing a package that has a bug, they need to rebuild their entire repository to do this, which takes upwards of 30 minutes. On Attune, since we have a centralized control plane, it takes about 10 seconds.
2. Aptly doesn't have concurrency controls. This makes it annoying to run in CI. Imagine you're setting up a CI pipeline that runs on every commit and publishes a package. You need to make sure you don't have two different instances of Aptly running at the same time, because their updates will clobber each other if they publish at the same time. Can you ensure that your CI only ever runs one publish at a time? This is easy in some setups, but surprisingly hard in GitHub Actions - concurrency keys don't quite work the way you'd expect them to.
3. Aptly uses its storage as its source of truth for the current state of packages. This is a great design choice for simplicity! However, if your CI job dies halfway through during an Aptly publish, this can leave your repository in a corrupted (in this case, half-updated) state. To recover from this, you need to rebuild the repository state... which you can't, because the ground truth is contained in the storage which was just corrupted! We mitigate this issue by tracking metadata in a separate transactional database so CI crashes don't cause corruption.
These are some examples of little cuts that build up over time with all of the tools we've found so far. Lots of the open source tooling is great at low scale, but the cuts start to build up once you are pushing a meaningful amount of packages to production.
Maybe I'm wrong, but aren't apt repositories just files in a web root? There are many scripts in the Debian ecosystem to generate and maintain such web directories. Maybe the actual package building would be interesting, but I don't know about just "we'll host your .deb files and keep the index updated"
You are correct. The reason we built this tool is because all of the scripts we found had sharp edges when we tried to use them in production at scale (I talk about some of them here: https://news.ycombinator.com/item?id=43730136).
Most of these scripts were designed for a world where there was A Blessed Deployment Machine that acted as its own de facto centralized control plane. We're designed for a newer world where publishing is just another piece of your CI, so you need more features to handle concurrency control, distributed signing, incremental index rebuilds, etc.
Our use case is customers who (1) want to use a managed cloud hosting service that we provide, but (2) are not willing to give us their signing keys. Our design allows them to keep all of their signing keys local to their environment.
To our knowledge, we have not found another provider who supports both of these requirements. It's not some amazing technical innovation, but it is one of those annoying paper cuts that builds up with all the others.
That's true! The Debian and Ubuntu folks are also experts at this. In our experience, the sharp edges generally affect teams that don't have a lot of in-house expertise in this, and where release engineering is not a core engineering competency that they want to invest in.
Yeah, solving this locally for one repository definitely isn't that hard at all. Most of the features we're building become useful when you're trying to build CI integrations for a larger team while also complying with enterprise security requirements (e.g. audit logging, HSM key protections, etc.).
As a FOSS project which publishes debs, it's a real hassle using current tooling. The tools are heavily biased towards the needs of distro maintainers, forcing projects into overly complex workflows which are overkill for a small numbers of packages. Features that would benefit smaller repositories with more frequent updates are limited or missing.
The other options is to DIY without tooling (essentially, write custom tooling as scripts or whatever), which is messy and full of pitfalls to the inexperienced.
Yup, this was similar to our experience as well. If publishing is still a source of toil for you, feel free to reach out at eliza@attunehq.com - we'd love to see if we can do any dev work or hosting pro bono for your open source projects.
I'm not familiar with mise and ubi, but these look like they're tools that the end user runs to install binaries. In contrast, we're building something that sets up a server so that end users can use their existing `apt-get install`.
Yeah, multi-distro and multi-arch is something on our roadmap! We're especially interested in automating the more annoying OS-specific parts of releasing, like MacOS notarization.
Excited to try. I ran a reprepo for a couple of years & it wasn't the worst to run, but not fun either. My confidence in efficiently publishing packages was super low, never improved much.
It also mandated running a web server (configuring my nginx to point at the repo filesystem), not very batteries included.
Thanks for trying, and sorry in advance for the rough edges :)
We're also working on a hosted service! If you'd like a sneak peek, send us a message at founders@attunehq.com (or email me directly at eliza@attunehq.com). I'm happy to talk about your specific needs and see if we can build something for them.
(And yes, it is Rust. I keep trying to find projects where I get to stretch my Haskell wings again, but unfortunately I keep working on things with silly requirements such as "easily understandable performance" and "a cross-compilation story" and "not making my collaborators sit through another monad tutorial".)
We don't plan to do that right now. The CLI isn't that complicated, and the choice for that was driven more by Go's excellent cross-compilation story than anything else.
It would be a fun thing to do if we had the resources to get equally good cross-compilation in Rust, but we're focused on building functionality right now.
It's probably intuitive to anybody who's tried to do this the hard way, but if your target is people without that experience, you might want to dumb it down even more. So this is from your readme:
And I think I know what `-d bookworm` must be, but I've no clue what -r or -c are doing in the second command.Also, from your post here (and why are the quick start steps different here?),
What is that URI supposed to be? Is that where the thing will eventually get served from and I have to run this on the server, or synchronize an output directory to the server, or is that going to do some sort of magic upload itself?Totally agree here. The current target is people who are familiar with the process, and we plan to write more documentation as we start thinking more about accessibility to a broader audience. We actually ran into this issue ourselves while building the tool - the Debian documentation is all there, but not the easiest to approach. These docs are one of the rough edges (sorry!) since we wanted to get this out earlier for feedback.
To answer your direct question: on self-hosted repositories, the URI right now is just a string identifier. We are working right now on a cloud hosted service, and there the URI will help us figure out which repository to actually serve.
Lots more options listed on the Debian wiki, I've added Attune to the list.
https://wiki.debian.org/DebianRepository/Setup
Thanks, appreciate it!
How is this different to aptly, especially if one already has object storage?
One of the tools we tried was Aptly! A couple issues we had with Aptly:
1. Aptly needs to rebuild the entire repository before it can do any changes. One of our customers builds packages for every Linux OS, and every architecture, and each release contains 10-something separately packaged tools, and they do regular releases. This means their repository of packages is _huge_. When they try to do small maintenance actions like removing a package that has a bug, they need to rebuild their entire repository to do this, which takes upwards of 30 minutes. On Attune, since we have a centralized control plane, it takes about 10 seconds.
2. Aptly doesn't have concurrency controls. This makes it annoying to run in CI. Imagine you're setting up a CI pipeline that runs on every commit and publishes a package. You need to make sure you don't have two different instances of Aptly running at the same time, because their updates will clobber each other if they publish at the same time. Can you ensure that your CI only ever runs one publish at a time? This is easy in some setups, but surprisingly hard in GitHub Actions - concurrency keys don't quite work the way you'd expect them to.
3. Aptly uses its storage as its source of truth for the current state of packages. This is a great design choice for simplicity! However, if your CI job dies halfway through during an Aptly publish, this can leave your repository in a corrupted (in this case, half-updated) state. To recover from this, you need to rebuild the repository state... which you can't, because the ground truth is contained in the storage which was just corrupted! We mitigate this issue by tracking metadata in a separate transactional database so CI crashes don't cause corruption.
These are some examples of little cuts that build up over time with all of the tools we've found so far. Lots of the open source tooling is great at low scale, but the cuts start to build up once you are pushing a meaningful amount of packages to production.
Thanks for the detailed response!
Maybe I'm wrong, but aren't apt repositories just files in a web root? There are many scripts in the Debian ecosystem to generate and maintain such web directories. Maybe the actual package building would be interesting, but I don't know about just "we'll host your .deb files and keep the index updated"
You are correct. The reason we built this tool is because all of the scripts we found had sharp edges when we tried to use them in production at scale (I talk about some of them here: https://news.ycombinator.com/item?id=43730136).
Most of these scripts were designed for a world where there was A Blessed Deployment Machine that acted as its own de facto centralized control plane. We're designed for a newer world where publishing is just another piece of your CI, so you need more features to handle concurrency control, distributed signing, incremental index rebuilds, etc.
What is an example of something that would require "distributed signing"?
Our use case is customers who (1) want to use a managed cloud hosting service that we provide, but (2) are not willing to give us their signing keys. Our design allows them to keep all of their signing keys local to their environment.
To our knowledge, we have not found another provider who supports both of these requirements. It's not some amazing technical innovation, but it is one of those annoying paper cuts that builds up with all the others.
> when we tried to use them in production at scale
Debian and Ubuntu have been using them in production and "at scale" for decades. What are the "sharp edges" that you're trying to solve?
That's true! The Debian and Ubuntu folks are also experts at this. In our experience, the sharp edges generally affect teams that don't have a lot of in-house expertise in this, and where release engineering is not a core engineering competency that they want to invest in.
Signing an unsigned apt repository is about three gpg commands though. I don't see how this kind of thing requires a whole separate product.
Yeah, solving this locally for one repository definitely isn't that hard at all. Most of the features we're building become useful when you're trying to build CI integrations for a larger team while also complying with enterprise security requirements (e.g. audit logging, HSM key protections, etc.).
As a FOSS project which publishes debs, it's a real hassle using current tooling. The tools are heavily biased towards the needs of distro maintainers, forcing projects into overly complex workflows which are overkill for a small numbers of packages. Features that would benefit smaller repositories with more frequent updates are limited or missing.
The other options is to DIY without tooling (essentially, write custom tooling as scripts or whatever), which is messy and full of pitfalls to the inexperienced.
Yup, this was similar to our experience as well. If publishing is still a source of toil for you, feel free to reach out at eliza@attunehq.com - we'd love to see if we can do any dev work or hosting pro bono for your open source projects.
Curious how different is this from mise and ubi? https://github.com/jdx/mise
I have a few pain points with that for installing cua (https://github.com/trycua/cua/issues/27), so if it can remove the initial friction happy to chat!
I've never heard of mise used for creating and managing APT repositories. Where does that figure in?
I'm not familiar with mise and ubi, but these look like they're tools that the end user runs to install binaries. In contrast, we're building something that sets up a server so that end users can use their existing `apt-get install`.
Looks neat!
What would be really valuable for us would be future support for other distros/repos.
A more unified interface for synchronized publishing across say dep/rpm/archlinux/alpine when we have multi-distro packages to build and publish.
Yeah, multi-distro and multi-arch is something on our roadmap! We're especially interested in automating the more annoying OS-specific parts of releasing, like MacOS notarization.
Happy to test this out, because I'm really unhappy with aptly quirks
very cool stuff!
Yes, it's Rust ;). Sign of quality to me.
Excited to try. I ran a reprepo for a couple of years & it wasn't the worst to run, but not fun either. My confidence in efficiently publishing packages was super low, never improved much.
It also mandated running a web server (configuring my nginx to point at the repo filesystem), not very batteries included.
https://wiki.debian.org/DebianRepository/SetupWithReprepro
Thanks for trying, and sorry in advance for the rough edges :)
We're also working on a hosted service! If you'd like a sneak peek, send us a message at founders@attunehq.com (or email me directly at eliza@attunehq.com). I'm happy to talk about your specific needs and see if we can build something for them.
(And yes, it is Rust. I keep trying to find projects where I get to stretch my Haskell wings again, but unfortunately I keep working on things with silly requirements such as "easily understandable performance" and "a cross-compilation story" and "not making my collaborators sit through another monad tutorial".)
Will you be rewriting the Golang part in Rust?
We don't plan to do that right now. The CLI isn't that complicated, and the choice for that was driven more by Go's excellent cross-compilation story than anything else.
It would be a fun thing to do if we had the resources to get equally good cross-compilation in Rust, but we're focused on building functionality right now.
It looks like Golang to me?
Edit: the CLI part is Golang and the rest(?) is Rust.
.. for the repo github pages can be used as well, although the disgust when pushing huge .deb files up there is real.