Nice move and nice write-up! There has been so much propaganda around serverless, so we need to hear more of these voices.
I moved away from FreeBSD to Debian for hosting my things because the process/daemon management was too tricky. You seem to have figured out a good solution, but I wanted something simpler like PM2 for automatic process management/restarting/logs. Unfortunately PM2 has an issue [1] that makes it unworkable with FreeBSD. It would be so nice if FreeBSD had a smooth, more declarative way of managing processes.
> I moved away from FreeBSD to Debian for hosting my things because the process/daemon management was too tricky.
It indeed is tricky. To be honest, I wasn't "put off" by it because I've been using BSDs and old-style Linux startup systems for almost 30 years now... but the lack of abstraction shows, and I don't think it's great.
The daemon(8) wrapper is neat to integrate pre-existing servers into rc.d, but I do not fancy having to deal with that "by hand" nor to create a shell script to manage my own service (related from a few years ago: https://jmmv.dev/2020/08/rcd-libexec-etc.html) nor to have something entirely separate to manage log rotation.
As much hate as systemd gets, I do think being declarative (and doing so in a DSL that's not a programming language) and having a true process "supervisor" is a better model. BUT, as I mentioned in this article, I also like the "no churn" of the BSDs because what I learned and refined over ~30 years is still similar to this day and that I won't be bitten by surprises.
Not GP, but I do prefer the very direct control you get with rcctl (OpenBSD), openrc (Alpine),... Systemd often feels like autoconf. It's needed when you really want the capabilities, otherwise the opaqueness and complexity feels very much cumbersome when you're dealing with a simple service.
I do like the Unix way of having different components handling different tasks instead of having different things which are entangled with each other. It encourages transparency.
Even with daemon(8), PID files and the lack of process supervision might be my least favorite aspect of FreeBSD, an OS I like overall. Not long ago, I wanted to avoid running a custom service that way on a fresh FreeBSD server. After researching my options, I found an adequate solution in the daemontools family. I'd heard of daemontools but hadn't paid much attention to it.
My service has been managed by runit and, most recently, nitro (https://github.com/leahneukirchen/nitro). Both have run as the service's user. They supervise the process and handle logging. I have found the design of daemontools and its derivatives runit and nitro elegant; it lives up to the reputation.
I have been using daemontools for managing my services on FreeBSD servers that have run 24/7 for almost a quarter of century, with down times of an hour or so only at intervals of a few years of continuous running, whenever I made a hardware upgrade or a complete OS replacement (by passing to another major version of FreeBSD).
Now there are several daemontools derivatives that bring it more up-to-date, but even the ancient original version did most of one would need for reliable service management.
> As much hate as systemd gets, I do think being declarative (and doing so in a DSL that's not a programming language) and having a true process "supervisor" is a better model.
I've been playing with dinit for a bit now; it combines a lot of the nice advantages of systemd with a finite scope and being portable across OSs.
I remember having same problem with Linux, tried to compensate with Monit but it never just worked quite right.
Systemd might add a bunch of unnecessary complexity in places, but for a sysadmin it's a fucking blessing. Just write one simple file, set binary, user, done, add some limitations and dependencies if you want to be fancy, set auto restart to true and It Just Works.
Can even set some memory limits if you want to be fancy so any memory leak won't get you into too much trouble but just gets the service restarted
One thing that strikes me about the author's experience is how unusable Azure appears to be in all this. Like, this hits somewhat close to me as I also run a couple web projects, also written in Rust, on a serverless architecture, and frankly couldn't imagine wanting to spend the time to run them any other way. AWS Lambda for compute, and Dynamo for DB, have a perpetual free tier and it's been running practically without me touching it for years.
The AWS Rust SDK also seemed very mature to me when I was using it ... compare to:
>the only option for a database instance with a free plan was [...] serverless Microsoft SQL Server (MSSQL)
>the MSSQL connection required TLS and this hadn’t been implemented in the sqlx connector I chose to use for my Rust-based functions. I wasted two weeks implementing TLS support in sqlx
That is insane. Not to mention the later bit with sudden, unexplained availability and the only hint that it might to be related to a _future_ deprecation? Like, imagine if this were a critical service for you. Professional malpractice on Microsoft's part.
This isn't really the main point of the article, and I did find the stuff on self-hosting interesting, but it does seem like this could have been avoided if Azure had lived up to its peers.
At $DAYJOB, we used to have a few internal use Azure Functions (which we’ve migrated to AWS Lambda). My favourite Azure weirdness was a warning message that said some Blob Storage-related thing was about to expire. We hadn’t set up any Blob Storage, the Azure Functions .NET deployment thing we used did. And apparently, there was an encryption key or something with an expiry of 1 year. It seems that Microsoft did not expect code to just work and not need to be redeployed for that long.
There's not enough information to come to a conclusion as to what the 503s originated from. There is enough information for manufactured outrage, of course.
I don't see why one would want to run in-the-clear over the Azure network for SQL connections.
The author was doing hobby projects. Granted, hobby projects should run on any platform, but Azure seems to have less of that free tier you can get elsewhere.
> There's not enough information to come to a conclusion as to what the 503s originated from.
It's the MS way. I remember writing some code for MS graph and wondering why the hell it isn't working, I'm doing near same thing that doc example did. Nonsense generic error message. Left for the day, ran code next morning, everything works fine.
No message saying "hey the thing behind it is down", just error that was generic enough that it could be my inputs being wrong.
The author did not make any statements that would lead to the conclusion the issue is lack of information provided by the platform, simply they do not state any troubleshooting steps taken.
The author chose a library that didn't support TLS, written by a company that isn't Microsoft. Terrible devex indeed but hard to see why that's the fault of the platform that required TLS.
I have a personal project and I used a tiny Azure 'dev/test' postgres database like OP mentioned and it went down for a day with no explanation. Restoring from back ups did not work. Now Auzre have a new DB service in preview called Horizon which I'd hope is more reliable.
Before that I used stackgres on kubernetes which was awful. I hear the cloud native pg operator is okay though?
Another problem I had was Bitnami yanking some helm charts and images I had used for a Redis cluster.
I'm now thinking something like Supabase or Convex may be the way to go for personal projects. Any experiences?
I'm using CNPG on my rinky-dinky 3-node k8s homelab, and it's been pretty good. I had a situation where my object storage ran out of space (again: homelab) which caused backpressure on db log backups; which stopped the pg databases. It correctly recovered automatically once I cleared up the root cause without much intervention on the CNPG side.
Interesting. So the premise of the article is wrong, where he says he only gets $300 a year. Apparently he even works in Azure. Would have thought he would get more given the role.
Which means that you need a Visual Studio Enterprise license that costs $500 a month, unless you can get it though partnership programs. I take it OP doesn't have an Enterprise license.
This coincides with my experience. At first, Azure (AWS, ...) were simple enough to get started in no time. Later, the service creation "wizards" started becoming progressively bloated, getting full of vendor-specific details I could not care less, but was required to pay attention to in order to get a minimally running service.
Needless to say that those details weren't rewarding at all. Knowing them just served the ego of specific vendors, who were more than happy to pull a rug under your feet with deprecations, migrations, and "required actions" I had to manually follow in order to keep the services just running.
Enough is enough. One sunny day in 2021, I started to migrate the infrastructure using a garage inspired approach. Dockerfile became a breath of fresh air, a relieve after the years of convoluted dictatorships. No more dependencies on ever-changing whims of individual cloud vendors, no rug pulls. Just you, your services, and freedom.
I had no prerogative of keeping my servers near me, instead I found a good home at fly.io. I still use Azure, this time not as a swiss-knife almighty cloud, but as an interchangeable commodity supplier (storage).
in my experience, Azure functions are best used by .NET executables and powershell scripts. GCP functions are what I recommend for anyone entering serverless (from limited experience of course).
I would not upload software directly to servers and change the configuration files to make it work. I rather use containers so I can be sure it will run on different servers with ease.
Nice move and nice write-up! There has been so much propaganda around serverless, so we need to hear more of these voices.
I moved away from FreeBSD to Debian for hosting my things because the process/daemon management was too tricky. You seem to have figured out a good solution, but I wanted something simpler like PM2 for automatic process management/restarting/logs. Unfortunately PM2 has an issue [1] that makes it unworkable with FreeBSD. It would be so nice if FreeBSD had a smooth, more declarative way of managing processes.
1. https://github.com/Unitech/pm2/issues/5718
Thanks!
> I moved away from FreeBSD to Debian for hosting my things because the process/daemon management was too tricky.
It indeed is tricky. To be honest, I wasn't "put off" by it because I've been using BSDs and old-style Linux startup systems for almost 30 years now... but the lack of abstraction shows, and I don't think it's great.
The daemon(8) wrapper is neat to integrate pre-existing servers into rc.d, but I do not fancy having to deal with that "by hand" nor to create a shell script to manage my own service (related from a few years ago: https://jmmv.dev/2020/08/rcd-libexec-etc.html) nor to have something entirely separate to manage log rotation.
As much hate as systemd gets, I do think being declarative (and doing so in a DSL that's not a programming language) and having a true process "supervisor" is a better model. BUT, as I mentioned in this article, I also like the "no churn" of the BSDs because what I learned and refined over ~30 years is still similar to this day and that I won't be bitten by surprises.
Not GP, but I do prefer the very direct control you get with rcctl (OpenBSD), openrc (Alpine),... Systemd often feels like autoconf. It's needed when you really want the capabilities, otherwise the opaqueness and complexity feels very much cumbersome when you're dealing with a simple service.
I do like the Unix way of having different components handling different tasks instead of having different things which are entangled with each other. It encourages transparency.
Even with daemon(8), PID files and the lack of process supervision might be my least favorite aspect of FreeBSD, an OS I like overall. Not long ago, I wanted to avoid running a custom service that way on a fresh FreeBSD server. After researching my options, I found an adequate solution in the daemontools family. I'd heard of daemontools but hadn't paid much attention to it.
My service has been managed by runit and, most recently, nitro (https://github.com/leahneukirchen/nitro). Both have run as the service's user. They supervise the process and handle logging. I have found the design of daemontools and its derivatives runit and nitro elegant; it lives up to the reputation.
I have been using daemontools for managing my services on FreeBSD servers that have run 24/7 for almost a quarter of century, with down times of an hour or so only at intervals of a few years of continuous running, whenever I made a hardware upgrade or a complete OS replacement (by passing to another major version of FreeBSD).
Now there are several daemontools derivatives that bring it more up-to-date, but even the ancient original version did most of one would need for reliable service management.
> As much hate as systemd gets, I do think being declarative (and doing so in a DSL that's not a programming language) and having a true process "supervisor" is a better model.
I've been playing with dinit for a bit now; it combines a lot of the nice advantages of systemd with a finite scope and being portable across OSs.
I remember having same problem with Linux, tried to compensate with Monit but it never just worked quite right.
Systemd might add a bunch of unnecessary complexity in places, but for a sysadmin it's a fucking blessing. Just write one simple file, set binary, user, done, add some limitations and dependencies if you want to be fancy, set auto restart to true and It Just Works.
Can even set some memory limits if you want to be fancy so any memory leak won't get you into too much trouble but just gets the service restarted
One thing that strikes me about the author's experience is how unusable Azure appears to be in all this. Like, this hits somewhat close to me as I also run a couple web projects, also written in Rust, on a serverless architecture, and frankly couldn't imagine wanting to spend the time to run them any other way. AWS Lambda for compute, and Dynamo for DB, have a perpetual free tier and it's been running practically without me touching it for years.
The AWS Rust SDK also seemed very mature to me when I was using it ... compare to:
>the only option for a database instance with a free plan was [...] serverless Microsoft SQL Server (MSSQL)
>the MSSQL connection required TLS and this hadn’t been implemented in the sqlx connector I chose to use for my Rust-based functions. I wasted two weeks implementing TLS support in sqlx
That is insane. Not to mention the later bit with sudden, unexplained availability and the only hint that it might to be related to a _future_ deprecation? Like, imagine if this were a critical service for you. Professional malpractice on Microsoft's part.
This isn't really the main point of the article, and I did find the stuff on self-hosting interesting, but it does seem like this could have been avoided if Azure had lived up to its peers.
At $DAYJOB, we used to have a few internal use Azure Functions (which we’ve migrated to AWS Lambda). My favourite Azure weirdness was a warning message that said some Blob Storage-related thing was about to expire. We hadn’t set up any Blob Storage, the Azure Functions .NET deployment thing we used did. And apparently, there was an encryption key or something with an expiry of 1 year. It seems that Microsoft did not expect code to just work and not need to be redeployed for that long.
Azure Functions use Azure Storage to store its metadata.
There's not enough information to come to a conclusion as to what the 503s originated from. There is enough information for manufactured outrage, of course.
I don't see why one would want to run in-the-clear over the Azure network for SQL connections.
The author was doing hobby projects. Granted, hobby projects should run on any platform, but Azure seems to have less of that free tier you can get elsewhere.
> There's not enough information to come to a conclusion as to what the 503s originated from.
It's the MS way. I remember writing some code for MS graph and wondering why the hell it isn't working, I'm doing near same thing that doc example did. Nonsense generic error message. Left for the day, ran code next morning, everything works fine.
No message saying "hey the thing behind it is down", just error that was generic enough that it could be my inputs being wrong.
The author did not make any statements that would lead to the conclusion the issue is lack of information provided by the platform, simply they do not state any troubleshooting steps taken.
Oh for sure you want to use TLS, what's insane is the developer experience of needing to implement it yourself.
The author chose a library that didn't support TLS, written by a company that isn't Microsoft. Terrible devex indeed but hard to see why that's the fault of the platform that required TLS.
I've seen these mysterious and unexplained 503s occur
Solution:
1. Delete Function App
2. Deploy again
3. Profit?
I have a personal project and I used a tiny Azure 'dev/test' postgres database like OP mentioned and it went down for a day with no explanation. Restoring from back ups did not work. Now Auzre have a new DB service in preview called Horizon which I'd hope is more reliable. Before that I used stackgres on kubernetes which was awful. I hear the cloud native pg operator is okay though? Another problem I had was Bitnami yanking some helm charts and images I had used for a Redis cluster.
I'm now thinking something like Supabase or Convex may be the way to go for personal projects. Any experiences?
I'm using CNPG on my rinky-dinky 3-node k8s homelab, and it's been pretty good. I had a situation where my object storage ran out of space (again: homelab) which caused backpressure on db log backups; which stopped the pg databases. It correctly recovered automatically once I cleared up the root cause without much intervention on the CNPG side.
Interesting they don't grant you enough credit as a Microsoft employee to dog food your own products. $300 a year seems a tiny amount.
I never worked at Amazon but understand they give you generally open ended access for learning/testing?
Surely there plenty of business value in getting someone like the author to ensure Rust applications run smoothly with all their products.
For the 5 years I've been at Microsoft it's always been $150 monthly.
Interesting. So the premise of the article is wrong, where he says he only gets $300 a year. Apparently he even works in Azure. Would have thought he would get more given the role.
Yes, it's the standard Visual Studio Enterprise credit that you get with the license - 150/month
https://azure.microsoft.com/en-us/pricing/member-offers/cred...
Which means that you need a Visual Studio Enterprise license that costs $500 a month, unless you can get it though partnership programs. I take it OP doesn't have an Enterprise license.
This coincides with my experience. At first, Azure (AWS, ...) were simple enough to get started in no time. Later, the service creation "wizards" started becoming progressively bloated, getting full of vendor-specific details I could not care less, but was required to pay attention to in order to get a minimally running service.
Needless to say that those details weren't rewarding at all. Knowing them just served the ego of specific vendors, who were more than happy to pull a rug under your feet with deprecations, migrations, and "required actions" I had to manually follow in order to keep the services just running.
Enough is enough. One sunny day in 2021, I started to migrate the infrastructure using a garage inspired approach. Dockerfile became a breath of fresh air, a relieve after the years of convoluted dictatorships. No more dependencies on ever-changing whims of individual cloud vendors, no rug pulls. Just you, your services, and freedom.
I had no prerogative of keeping my servers near me, instead I found a good home at fly.io. I still use Azure, this time not as a swiss-knife almighty cloud, but as an interchangeable commodity supplier (storage).
in my experience, Azure functions are best used by .NET executables and powershell scripts. GCP functions are what I recommend for anyone entering serverless (from limited experience of course).
I would not upload software directly to servers and change the configuration files to make it work. I rather use containers so I can be sure it will run on different servers with ease.