S3: "Block Public Access is now enabled by default on new buckets."
On the one hand, this is obviously the right decision. The number of giant data breeches caused by incorrectly configured S3 buckets is enormous.
But... every year or so I find myself wanting to create an S3 bucket with public read access to I can serve files out of it. And every time I need to do that I find something has changed and my old recipe doesn't work any more and I have to figure it out again from scratch!
The thing to keep in mind with the "Block Public Access" setting is that is a redundancy built in to save people from making really big mistakes.
Even if you have a terrible and permissive bucket policy or ACLs (legacy but still around) configured for the S3 bucket, if you have Block Public Access turned on - it won't matter. It still won't allow public access to the objects within.
If you turn it off but you have a well scoped and ironclad bucket policy - you're still good! The bucket policy will dictate who, if anyone, has access. Of course, you have to make sure nobody inadvertantly modifies that bucket policy over time, or adds an IAM role with access, or modifies the trust policy for an existing IAM role that has access, and so on.
My understanding is that there isn't actually any "overriding" in the sense of two rules conflicting and one of them having to "win" and take effect. I think it's more that an enabled rule always is in effect, but it might overlap with another rule, in which case removing one of them still won't remove the restrictions on the area of overlap. It's possible I'm reading too much into your choice of words, but it does sound like there's a chance that the confusion is stemming from an incorrect assumption of how various permissions interact.
That being said, there's certain a lot more that could into making a system like that easier for developers. One thing that springs to mind is tooling that can describe what rules are currently in effect that limit (or grant, depending on the model) permissions for something. That would make it more clear when there are overlapping rules that affect the permissions of something, which in turn would make it much more clear why something is still not accessible from a given context despite one of the rules being removed.
If one rule explicitly restricts access and another explicitly grants access, which one is in effect? Do restrictions override grants? Does a grant to GroupOne override a restriction to GroupAlpha when the authenticated use in is both groups? Do rules set by GodAdmin override rules set by AngelAdmin?
It's possible I'm making the exact mistake that the article describes and relying on outdated information, but my understanding is that pretty much all of the rules are actually permissions rather than restrictions. "Block public access" is an unfortunate exception to this, and I suspect that it's probably just a poorly named inversion of an "allow public access" permission. You're 100% right that modeling permissions like this requires having everything in the same "direction", i.e. either all permissions or all restrictions.
After thinking about this sort of thing a lot when designing a system for something sort of similar to this (at a much smaller scale, but with the intent to define it in a way that could be extended to define new types of rules for a given set of resources), I feel pretty strongly that the best way for a system like this to work from the protectives of security, ease of implementation, and intuitiveness for users are all aligned in requiring every rule to explicitly be defined as a permission rather than representing any of them as restrictions (both in how they're presented to the user and how they're modeled under the hood). With this model, veryifing whether an action is allowed can be implemented by mapping an action to the set of accesses (or mutations, as the case may be) it would perform, and then checking that each of them has a rule present that allows it. This makes it much easier to figure out whether something is allowed or not, and there's plenty of room for quality of life things to help users understand the system (e.g. being able to easily show a user what rules pertain to a given resource with essentially the same lookup that you'd need to do when verifying an action in it). My sense is that this is actually not far from how AWS permissions are implemented under the hood, but they completely fail at the user-facing side of this by making it much harder than it needs to be to discover where to define the rules for something (and by extension, where to find the rules currently in effect for it).
They don't really override each other but they act like stacked barriers, like a garage door blocking access to an open or closed car. Access is granted if every relevant layer allows it.
You say this, someone challenges you, now you're on the defensive during an interview and everyone has a bad taste in their mouth. Yeah, that's how it goes.
That's just the taste of iron from the blood after the duel. But this is completely normal after a formal challenge! Companies want real cyberwarriors, and the old (lame) rockstar ninjas that they hired 10 years ago are very prone to issuing these.
I just stick CloudFront in front of those buckets. You don't need to expose the bucket at all then and can point it at a canonical hostname in your DNS.
That’s definitely the “correct” way of doing things if you’re writing infra professionally. But I do also get that more casual users might prefer not to incur the additional costs nor complexity of having CloudFront in front. Though at that point, one could reasonably ask if S3 is the right choice for causal users.
S3 + cloudfront is also incredibly popular so you can just find recipes for automating that in any technology you want, Terraform, ansible, plain bash scripts, Cloudformation (god forbid)
It's designed to be a declarative DSL, but then you have to do all sorts of filters and maps in any group of resources and suddenly you are programming in yaml with both hands tied behind your back
Yeah it’s just terrible. If Amazon knew what was good they’d just replace it with almost anything else. Heck just got all in on terraform and call it a day.
It does compile down to Azure Resource Manager's json DSL, so in that way close to Troposphere I guess, only both sides are official and not just some rando project that happens to emit yaml/json
The implementation, of course, is ... very Azure, so I don't mean to praise using it, merely that it's a better idea than rawdogging json
Meh. The CDK doesn’t look terrible. It’s still not ideal. But even if this compiles to a mess of CF it’s still better than writing CF by hand and that’s only because CF is so bad to begin with.
As for "go all in on terraform," I pray to all that is holy every night that terraform rots in the hell that spawned it. And that's not even getting into the rug pull parts, I mean the very idea of
1. I need a goddamn CLI to run it (versus giving someone a URL they can load in their tenant and have running resources afterward)
1. the goddamn CLI mandates live cloud credentials, but then stright-up never uses them to check a goddamn thing it intends to do to my cloud control plane
You may say "running 'plan' does" and I can offer 50+ examples clearly demonstrating that it does not catch the most facepalm of bugs
1. related to that, having a state file that believes it knows what exists in the world is just ludicrous and pain made manifest
1. a tool that thinks nuking things is an appropriate fix ... whew. Although I guess in our new LLM world, saying such things makes me the old person who should get onboard the "nothing matters" train
There's a lot wrong with Terraform but I don't think you're being at all fair with your specific critisims here:
> 1. I need a goddamn CLI to run it (versus giving someone a URL they can load in their tenant and have running resources afterward)
CloudFormation is the only IaC that supports "running as a URL" and that's only because it's an AWS native solution. And CloudFormation is a hell of a lot more painful to write and slower to iterate on. So you're not any better off for using CF.
What usually happens with TF is you'd build a deploy pipeline. Thus you can test via the CLI then deploy via CI/CD. So you're not limited to just the CLI. But personally, I don't see the CLI as a limitation.
> the goddamn CLI mandates live cloud credentials, but then stright-up never uses them to check a goddamn thing it intends to do to my cloud control plane
All IaC requires live cloud credentials. It would be impossible for them to work without live credentials ;)
Terraform does do a lot of checking. I do agree there is a lot that the plan misses though. That's definitely frustrating. But it's a side effect of cloud vendors having arbitrary conditions that are hard to define and forever changing. You run into the same problem with any tool you'd use to provision. Heck, even manually deploying stuff from the web console sometimes takes a couple of tweaks to get right.
> 1. related to that, having a state file that believes it knows what exists in the world is just ludicrous and pain made manifest
This is a very strange complaint. Having a state file is the bare minimum any IaC NEEDS for it to be considered a viable option. If you don't like IaC tracking state then you're really little better off than managing resources manually.
> a tool that thinks nuking things is an appropriate fix ... whew.
This is grossly unfair. Terraform only destroys resources when:
1. you remove those resources from the source. Which is sensible because you're telling Terraform you no longer want those resources
2. when you make a change that AWS doesn't support doing on live resources. Thus the limitation isn't Terraform, it is AWS
In either scenario, the destroy is explicit in the plan and expected behaviour.
> All IaC requires live cloud credentials. It would be impossible for them to work without live credentials ;)
Did you read the rest of the sentence? I said it's the worst of both worlds: I can't run "plan" without live creds, but then it doesn't use them to check jack shit. Also, to circle back to our CF and Bicep discussion, no, I don't need cloud creds to write code for those stacks - I need only creds to apply them
I don't need a state file for CF nor Bicep. Mysterious about that, huh?
> Incorrect, ARM does too, they even have a much nicer icon for one click "Deploy to Azure"
That’s Azure, not AWS. My point was to have “one click” HTTP installs you need native integration with the cloud vendor. For Azure it’s the clusterfuck that is Bicep. For AWS it’s the clusterfuck that is CF
> I don't need a state file for CF nor Bicep.
CF does have a state file, it’s just hidden from view.
And bicep is shit precisely because it doesn’t track state. In fact the lack of a state file is the main complain against bicep and thus the biggest thing holding it back from wider adoption — despite being endorsed by Microsoft Azure.
All Terraform does is build a DAG, compare it with the current state file and pass the changes down to the provider so it can translate to the correct sequence of interactions with the upstream API. Most of your criticism boils down to limitations of the cloud provider API and/or Terraform provider quality. It won't check for naming collision for instance, it assumes you know what you are doing.
Regarding HCL, I respect their decision to keep the language minimal, and for all it's worth you can go very, very far with the language expressions and using modules to abstract some logic, but I think it's a fair criticism for the language not to support custom functions and higher level abstractions.
I believe the usual uninformed thinking is "terraform exists outside of AWS, so I can move off of AWS" versus "we have used CF or Bicep, now we're stuck" kind of deal
Which is to say both of you are correct, but OP was highlighting the improper expectations of "if we write in TF, sure it sucks balls but we can then just pivot to $other_cloud" not realizing it's untrue and now you've used a rusty paintbrush as a screwdriver
Last time I tried to use CF, the third party IAC tools were faster to release new features than the functionality of CF itself. (Like Terraform would support some S3 bucket feature when creating a bucket, but CF did not).
I'm not sure if that's changed recently, I've stopped using it.
I have been on the terraform side for 7 years-ish.
eksctl just really impressed me with its eks management, specifically managed node groups & cluster add-ons, over terraform.
that uses cloudformation under the hood. so i gave it a try, and it’s awesome. combine with github actions and you have your IAC automation.
nice web interface for others to check stacks status, events for debugging and associated resources that were created.
oh, ever destroy some legacy complex (or not that complex) aws shit in terraform? it’s not going to be smooth. site to site connections, network interfaces, subnets, peering connections, associated resources… oh, my.
so far cloudformation has been good at destroying, but i haven’t tested that with massive legacy infra yet.
but i am happily converted tf>cf.
and will happily use both alongside each other as needed.
Because its an old early IaC language, but it works and lots depends on it, so instead of dumping or retooling it, AWS keeps it around as a compilation target, while pushing other solutions (years ago, the SAM transform on top of it, more recently CDK) as the main thing for people to actually use directly.
> Yeah holy crap why is cloud formation so terrible?
I can't confirm it, but I suspect that it was always meant to be a sales tool.
Every AWS announcement blog has a "just copy this JSON blob, and paste it $here to get your own copy of the toy demo we used to demonstrate in this announcement blog" vibe to it.
I do get where you're coming from, but I don't agree. With the CF+S3 combo you now need to choose which sharing mode to work with S3 (there are several different ways you can link CF to S3). Then you have the wider configuration of CF to manage too. And that's before you account for any caching issues you might run into when debugging your site.
If you know what you're doing, as it sounds like you and I do, then all of this is very easy to get set up (but then aren't most things easy when you already know how? hehe). However we are talking about people who aren't comfortable with vanilla S3, so throwing another service into the mix isn't going to make things easier for them.
It's actually incredibly cheap. I think our software distribution costs, in the account I run, are around $2.00 a month. That's pushing out several thousand MSI packages a day.
S3 is actually quite expensive compared to the competition for both storage costs and egress costs. At a previous start-up, we had terrabytes of data on S3 and it was our second largest cost (after GPUs) and by some margin.
For small scale stuff,
S3s storage and egress charges are unlikely to be impactful. But it doesn’t mean they’re cheap relative to the competition.
There are also ways you can reduce S3 costs, but then you're trading the costs received from AWS with the costs of hiring competent DevOps. Either way, you pay.
Really? If I remember correctly: My Static website served from S3 + CF + R53 by about 0.67$ / mo, 0.5 being R53 from that, 0.16 being CF, S3 being 0.01 for my page.
BTW: Is GitHub Page still free for custom domains? (I don't know the EULA)
- signed url's in case you want a session base files download
- default public files, for e.g. a static site.
You can also map a domain (sub-domain) to Cloudfront with a CNAME record and serve the files via your own domain.
Cloudfront distributions are also CDN based. This way you serve files local to the users location, thus increasing the speed of your site.
For lower to mid range traffic, cloudfront with s3 is cheaper as the network cost of cloudfront is cheaper. But for large network traffic, cloudfront cost can balloon very fast. But in those scenarios S3 costs are prohibitive too!
Not always that simple - for example if you want to automatically load /foo/index.html when the browser requests /foo/ you'll need to either use the web serving feature of S3 (bucket can't be private) or set up some lambda at edge or similar fiddly shenanigans.
Back when GPT4 was the new hotness, I dumped the markdown text from the Azure documentation GitHub repo into a vector index and wrapped a chatbot around it. That way, I got answers based on the latest documentation instead of a year-old LLM model's fuzzy memory.
I now have the daunting challenge of deploying an Azure Kubernetes cluster with... shudder... Windows Server containers on top. There's a mile-long list of deprecations and missing features that were fixed just "last week" (or whatever). That is just too much work to keep up with for mere humans.
I'm thinking of doing the same kind of customised chatbot but with a scheduled daily script that pulls the latest doco commits, and the Azure blogs, and the open GitHub issue tickets in the relevant projects and dumps all of that directly into the chat context.
I'm going to roll up my sleeves next week and actually do that.
Then, then, I'm going to ask the wizard in the machine how to make this madness work.
You're braver than me if you're willing to trust the LLM here - fine if you're ready to properly review all the relevant docs once you have code in hand, but there are some very expensive risks otherwise.
This is LLM as semantic search- so it's way way easier to start from the basic example code and google to confirm that it's correct than it is to read the docs from scratch and piece together the basic example code. Especially for things like configurations and permissions.
Sure, if you do that second part of verifying it. If you just get the LLM to spit it out then yolo it into production it is going to make you sad at some point.
However on AWS the difference between "generally working the way it should and not working the way it should" can be a 30,000$ cloud bill racked up in a few hours with EC2 going full speed ahead mining bitcoin.
For those high stakes cases maybe you can be more careful. You can still use an LLM to search and get references to the appropriate place and do your own verification.
But for low stakes LLM works just fine - not everything is going to blow up to a 30,000 bill.
In fact I'll take the complete opposite stance - verifying your design with an LLM will help you _save_ money more often than not. It knows things you don't and has awareness of concepts that you might have not even read about.
Well, the "accidentally making the S3 bucket public" scenario would be a good one. If you review carefully with full understanding of what e.g. all your policies are doing then great, no problem.
If you don't do that will you necessarily notice that you accidentally leaked customer data to the world?
The problem isn't the LLM it's assuming its output is correct just the same as assuming Stack Overflow answers are correct without verifying/understanding them.
I agree but its about the extent. I'm willing to accept the risk of ocassionally making S3 public but getting things done much faster, much like I don't meticulously read documentation when I can get the answer from stackoverflow.
If you are comparing with stackoverflow then I guess we are on the same page - most people are fine with taking stuff from stackoverflow and it doesn't count as "brave".
I think anyone who just copies and pastes from SO is indeed "brave" for pretty much exactly the same reason.
> I'm willing to accept the risk of ocassionally making S3 public
This is definitely where we diverge. I'm generally working with stuff that legally cannot be exposed - with hefty compliance fines on the horizon if we fuck up.
The thing is that you can now ask the LLM for links and you can ask it to break down why it thinks a piece of code, for example, protects the bucket from being public. Things that are easy to verify against the actual docs.
I feel like this workflow is still less time, easier and less error prone than digging out the exact right syntax from the AWS docs.
I honestly don't mind that you have to jump through hurdles to make your bucket publically available and that it's annoying. That to me seems like a feature, not a bug
>In EC2, you can now change security groups and IAM roles without shutting the instance down to do it.
Hasn't it been this way for many years?
>Spot instances used to be much more of a bidding war / marketplace.
Yeah because there's no bidding any more at all, which is great because you don't get those super high spikes as availability drops and only the ones who bid super high to ensure they wouldn't be priced out are able to get them.
>You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots.
This one was a nightmare and it took ages to convince some of my more pig headed coworkers in the past that they didn't need to do it any more. The funniest part is that they were storing their data as millions and millions of 10-100kb files, so the S3 backend scaling wasn't the thing bottlenecking performance anyway!
>Originally Lambda had a 5 minute timeout and didn’t support container images. Now you can run them for up to 15 minutes, use Docker images, use shared storage with EFS, give them up to 10GB of RAM (for which CPU scales accordingly and invisibly), and give /tmp up to 10GB of storage instead of just half a gig.
This was/is killer. It used to be such a pain to have to manage pyarrow's package size if I wanted a Python Lambda function that used it. One thing I'll add that took me an embarrassingly long time to realize is that your Python global scope is actually persisted, not just the /tmp directory.
> You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots.
Sorry, this is absolutely still the case if you want to scale throughput beyond the few thousand IOPS a single shard can serve. S3 will automatically reshard your key space, but if your keys are sequential (eg leading timestamp) all your writes will still hit the same shard.
Re: SG, yeah I wasnt doing any cloud stuff when that was the case. Never had to restart anything for an SG change and this must be at least 5-6 years..
> Glacier restores are also no longer painfully slow.
I had a theory (based on no evidence I'm aware of except knowing how Amazon operates) that the original Glacier service operated out of an Amazon fulfillment center somewhere. When you put it a request for your data, a picker would go to a shelf, pick up some removable media, take it back, and slot it into a drive in a rack.
This, BTW, is how tape backups on timesharing machines used to work once upon a time. You'd put in a request for a tape and the operator in the machine room would have to go get it from a shelf and mount it on the tape drive.
Which is basically exactly what you described but the picker is a robot.
Data requests go into a queue; when your request comes up, the robot looks up the data you requested, finds the tape and the offset, fetches the tape and inserts it into the drive, fast-forwards it to the offset, reads the file to temporary storage, rewinds the tape, ejects it, and puts it back. The latency of offline storage is in fetching/replacing the casette and in forwarding/rewinding the tape, plus waiting for an available drive.
Realistically, the systems probably fetch the next request from the queue, look up the tape it's on, and then process every request from that tape so they're not swapping the same tape in and out twenty times for twenty requests.
For truly write once read never data tape is the optimal storage method. It is exactly what the LTO standard was designed to do and it does it very well. You can be confident that you will be able to read every bit of data from a 30 year old tape, probably even 50 years old. It has the lowest bit error rate of any technology I am aware of. LTO-9 is better than 1 uncorrectable bit error in 10^20 user bits, which is 1 bit error in 12.5 exabytes. There is also the substantial advantage that tapes on a shelf are completely immune to ransomware. As a sysadmin I get that warm fuzzy feeling when critical data is backed up on a good LTO tape library.
As someone who does tape recovery on very very old tape I largely concur with this with a couple of caveats.
1. Do not encrypt your tapes if you want the data back in 30/50 years. We have had so many companies lose encryption keys and turn their tapes into paperweights because the company they bought out 17 years ago had poor key management.
2. The typical failure case on tape is physical damage not bit errors. This can be via blunt force trauma (i.e. dropping, or sometimes crushing) or via poor storage (i.e. mould/mildew).
3. Not all tape formats are created equal. I have seen far higher failure rates on tape formats that are repeatedly accessed, updated, ejected, than your old style write once, read none pattern.
Call it bad luck, but I’ve never had a fully successful restore. Drives eat tapes, drives are damaged and write bad data, robot arms die or malfunction.
Tapes have NEVER worked for me.
SANs and remote disk though, rock solid.
That said, I don’t miss any of that stuff, gimme S3 any day :)
You do realized that that isn't normal at all? LTO tape is still used by thousands of companies to backup many exabytes of data. I know it once saved Google from permanent loss of gmail data from a bug. You should really get a refund for your tape drives.
By the time it's hard to get a compatible LTO drive, I'd be very suspicious of a mothballed drive working either. If you want reliable long term storage you're going to have to update it every couple decades.
I can't talk about it, but I've yet to see an accurate guess at how Glacier was originally designed. I think I'm in safe territory to say Glacier operated out of the same data centers as every other AWS service.
It's been a long time, and features launched since I left make clear some changes have happened, but I'll still tread a little carefully (though no one probably cares there anymore):
One of the most crucial things to do in all walks of engineering and product management is to learn how to manage the customer expectations. If you say customers can only upload 10 images, and then allow them to upload 12, they will come to expect that you will always let them upload 12. Sometimes it's really valuable to manage expectations so that you give yourself space for future changes that you may want to make. It's a lot easier to go from supporting 10 images to 20, than the reverse.
Im like 90% sure ive seen folks (unofficially) disclose the original storage and API decisions over the years, in roughly accurate terms. Personally I think the multi dimensional striping/erasure code ideas are way more interesting than the “its just a tape library” speculation/arguments. That and the real lessons learned around product differentiation as supporting technologies converge.
I signed NDAs. I wish Glacier was more open about their history, because it's honestly interesting, and they have a number of notable innovations in how they approach things.
I think folks have missed what I think would have been clever about the implentation I (apparently) dreamt up. It's not that "it's just a tape library", it's that it would have used the existing FC and picker infrastructure that Amazon had already built, with some racks containing drives for removable media. I was thinking that it would not have been some special facility purely for Glacier, but rather one or more regular FCs would just have had some shelves with Glacier media (not necessarily tapes).
Then the existing pickers would get special instructions on their handhelds: Go get item number NNNN from Row/shelf/bin X/Y/Z and take it to [machine-M] and slot it in, etc.
They would definitely be using rubies robots given how uniform hard drives are. The only reason warehouses still have humans is that heterogeneity (different sizes, different textures, different squishiness, etc).
I'll add: When doing instance to instance communication (in the same AZ) always use private ips. If you use public ip routing (even the same AZ) this is charged as regional data transfer.
Even worse, if you run self hosted NAT instance(s) don't use a EIP attached to them. Just use a auto-assigned public IP (no EIP).
NAT instance with EIP
- AWS routes it through the public AWS network infrastructure (hairpinning).
- You get charged $0.01/GB regional data transfer, even if in the same AZ.
NAT instance with auto-assigned public IP (no EIP)
- Traffic routes through the NAT instance’s private IP, not its public IP.
- No regional data transfer fee — because all traffic stays within the private VPC network.
- auto-assigned public IP may change if the instance is shutdown or re-created so have automations to handle that. Though you should be using the network interface ID reference in your VPC routing tables.
I think there is more of us who kind of degenerated from doing it the AWS way - API Gateway, serverless lambdas mess around with IAM roles until it works, ... - to - Give me EC2 / LightSail VPS instance maybe an S3 bucket let's set domain through Route53 and go away with the rest of your orchestrion AWS.
At what point is AWS worth using over other compute competitors when you’re using them as a storage bucket + VPS. They’re wholly more expensive at that point. Why not go with a more traditional but rock solid VPS provider?
I have the opposite philosophy for what it’s worth: if we are going to pay for AWS I want to use it correctly, but maximally. So for instance if I can offload N thing to Amazon and it’s appropriate to do so, it’s preferable. Step Functions, lambda, DynamoDB etc, over time, have come to supplant their alternatives and its overall more efficient and cost effective.
That said, I strongly believe developers don’t do enough consideration as to how to maximize vendor usage in an optimal way
> That said, I strongly believe developers don’t do enough consideration as to how to maximize vendor usage in an optimal way
Because it's not straightforward. 1) You need to have general knowledge of AWS services and their strong and weak points to be able to choose the optimal one for the task, 2) you need to have good knowledge of the chosen service (like DynamoDB or Step Functions) to be able to use it optimally; being mediocre at it is often not enough, 3) local testing is often a challenge or plain impossible, you often have to do all testing on a dev account on AWS infra.
AWS can be used in a different, cost effective, way.
It can be used as a middle-ground capable of serving the existing business, while building towards a cloud agnostic future.
The good AWS services (s3, ec2, acm, ssm, r53, RDS, metadata, IAM, and E/A/NLBs) are actually good, even if they are a concern in terms of tracking their billing changes.
If you architect with these primitives, you are not beholden to any cloud provider, and can cut over traffic to a non AWS provider as soon as you’re done with your work.
Let me explain why we’re not talking about an 80/20 split.
There’s no reason to treat something like a route53 record, or security group rule, in the same way that you treat the creation of IAM Policies/Roles and their associated attachments.
If you create a common interface for your engineers/auditors, using real primitives like the idea of a firewall rule, you’ve made it easy for everyone to avoid learning the idiosyncrasies of each deployment target, and feel empowered to write their own merge requests, or review the intended state of a given deployment target.
If you need to do something provider-specific, make a provider-specific module.
I agree that using them as a VPS provider is a mistake.
If you don't use the E(lasticity) of EC2, you're burning cash.
For prod workloads, if you can go from 1 to 10 instances during an average day, that's interesting.
If you have 3 instances running 24/7/365, go somewhere else.
For dev workloads, being able to spin instances in a matter of seconds is a bliss.
I installed the wrong version of a package on my instance? I just terminate it, wait for the auto-scaling group to pop a fresh new one a start again.
No need to waste my time trying to clean my mess on the previous instance.
You speak about Step Functions as an efficient and cost effective service from AWS, and I must admit that it's one that I avoid as much as I can...
Given the absolute mess that it is to setup/maintain, and that you completely lock yourself in AWS with this, I never pick it to do anything.
I'd rather have a containerized workflow engine running on ECS, even though I miss on the few nice features that SF offers within AWS.
The approach I try to have is:
- business logic should be cloud agnostic
- infra should swallow all the provider's pills it needs to be as efficient as possible
In practice I found this to be more burden than it’s worth. I have yet to work somewhere that is on Azure, GCP or AWS and actually switch between clouds. I am sure it happens, but is it really that common?
I instead think of these platforms as a marriage, you’re going to settle in one and do your best to never divorce
Part of my job is to do migrations for customers, so, to me at least, it's not uncommon.
Using all the bells and whistles of a provider and being locked-in is one thing.
But the other big issue is that, as service providers, they can (and some of them did more often than not) stop providing some services or changing them in a way that forces you to make big changes in your app to keep it running on this service.
Whereas, if you build your app in a agnostic way, they can stop or change what they want, you either don't rely on those services heavily enough for the changes required to be huge, or you can just deploy elsewhere, with another flavor of the same service.
Let's say you have a legacy Java app that works only with a library that is not maintained. If you don't want to bear the cost of rewriting with a new and maintained library, you can keep the app running, knowing the risks and taking the necessary steps to protect you against it.
Whereas if your app relies heavily on DynamoDB's API and they decide to drop the service completely, the only way to keep the app running is to rewrite everything for a similar service, or to find a service relying on the same API elsewhere.
Because the compartmentalization of business duties means that devs are fighting uphill against the wind to sign a deal
with a new vendor for something. It's business bikeshedding, as soon as you open the door to a new vendor everyone, especially finance,
has opinions and you might end up stuck with a vendor you didn't want. Or you can use the pre-approved money furnace and just ship.
There are entire industries that have largely de-volved their clouds primarily for footprint flexibility (not all AWS services are in all regions) and billing consistency.
Honestly just having to manage IAM is such a time-suck that the way I've explained it to people is that we've traded the time we used to spend administering systems for time spent just managing permissions, and IAM is so obtuse that it comes out as a net loss.
There's a sweet spot somewhere in between raw VPSes and insanely detailed least-privilege serverless setups that I'm trying to revert to. Fargate isn't unmanageable as a candidate, not sure it's The One yet but I'm going to try moving more workloads to it to find out.
Usually I write some IaC to automate this tedium so I only have to go through the IAM setup pain once. Now if requirements change, that's an entirely different story...
So the problem when you combine IAC with CI/CD is that the role assumed by the CI agent needs privileges to deploy things, so you need a bootstrap config to set up what it needs. If you have a mandate to go least-privilege, then that needs to include only the permissions strictly needed by the current deployable. So, no "s3:*", you need each one listed.
So far so good, you can do this with a bootstrap script that you only need to run at project setup.
If you also have a mandate (effectively) to go fully serverless, then as your project evolves and you add functionality, what you find is that most interesting changes use something new in the platform. So you're not getting away with running the bootstrap script once. You're updating it and running it for almost every change. And you can't tell in advance what permissions you're going to need, because (especially if you're on terraform) there's apparently no documentation connecting the resources you want to manage and the permissions needed to do so. So you try to deploy your change, IAM pops an error or two, you try to figure out what permissions you need to add to the bootstrap script, you run it (fixing it when it breaks at this point), you try deploying again, IAM pops another couple of errors, and then you're in a grind cycle which you can't predict the length of - and you need to get to the end of it before you can even test your feature, because fully serverless means you can't run your application locally (and getting management to pay for the pro localstack licence is a dead end). At some point it won't be clear why IAM is complaining, because the error you get makes no sense whatsoever, so at that point it's off to support to find out a day later that ah, yes, you can't use an assumed role just there, it's got to be an actual role, and no, that's not written down anywhere, you've just got to know it, so you need to redesign how you're using the roles completely, and right about this point is when I usually want to buy a farm, raise goats, and get way too into oil painting, instead of whatever this insane waste of life is.
> I understand your situation is a bit unique, where you are unable to log in to your AWS account without an MFA device, but you also can't order an MFA device without being able to log in. This is a scenario that is not directly covered in our standard operating procedures.
The best course of action would be for you to contact AWS Support directly. They will be able to review your specific case and provide guidance on how to obtain an MFA device to regain access to your account. The support team may have alternative options or processes they can walk you through to resolve this issue.
Please submit a support request, and one of our agents will be happy to assist you further. You can access the support request form here: https://console.aws.amazon.com/support/home
You know what's still stupid? That if you have an S3 bucket in the same region as your VPC that you will get billed on your NAT Gateway to send data out to the public internet and right back in to the same datacenter. There is simply no reason to not default that behavior to opt out vs opt in (via a VPC endpoint) beyond AWS profiting off of people's lack of knowledge in this realm. The amount of people who would want the current opt-in behavior is... if not zero, infinitesimally small.
It's a design that is secure by default. If you have no NAT gateway and no VPC Gateway Endpoint for S3 (and no other means of Internet egress) then workloads cannot access S3. Networking should be closed by default, and it is. If the user sets up things they don't understand (like NAT gateways), that's on them. Managed NAT gateways are not the only option for Internet egress and users are responsible for the networks they build on top of AWS's primitives (and yes, it is indeed important to remember that they are primitives, this is an IaaS, not a PaaS).
Fine for when you have no NAT gateway and have a subnet with truly no egress allowed. But if you're adding a NAT gateway, it's crazy that you need to setup the gateway endpoint for S3/DDB separately. And even crazier that you have to pay for private links per AWS service endpoint.
There's very real differences between NAT gateways and VPC Gateway Endpoints.
NAT gateways are not purely hands-off, you can attach additional IP addresses to NAT gateways to help them scale to supporting more instances behind the NAT gateway, which is a fundamental part of how NAT gateways work in network architectures, because of the limit on the number of ports that can be opened through a single IP address. When you use a VPC Gateway Endpoint then it doesn't use up ports or IP addresses attached to a NAT gateway at all. And what about metering? If you pay per GB for traffic passing through the NAT gateway, but I guess not for traffic to an implicit built-in S3 gateway, so do you expect AWS to show you different meters for billed and not-billed traffic, but performance still depends on the sum total of the traffic (S3 and Internet egress) passing through it? How is that not confusing?
It's also besides the point that not all NAT gateways are used for Internet egress, indeed there are many enterprise networks where there are nested layers of private networks where NAT gateways help deal with overlapping private IP CIDR ranges. In such cases, having some kind of implicit built-in S3 gateway violates assumptions about how network traffic is controlled and routed, since the assumption is for the traffic to be completely private. So even if it was supported, it would need to be disabled by default (for secure defaults), and you're right back at the equivalent situation you have today, where the VPC Gateway Endpoint is a separate resource to be configured.
Not to mention that VPC Gateway Endpoints allow you to define policy on the gateway describing what may pass through, e.g. permitting read-only traffic through the endpoint but not writes. Not sure how you expect that to work with NAT gateways. This is something that AWS and Azure have very similar implementatoons for that work really well, whereas GCP only permits configuring such controls at the Organization level (!)
They are just completely different networking tools for completely different purposes. I expect closed-by-default secure defaults. I expect AWS to expose the power of different networking implements to me because these are low-level building blocks. Because they are low-level building blocks, I expect for there to be footguns and for the user to be held responsible for correct configuration.
Again, you are dealing with low-level primitives. You can provision an EC2 VM with multiple GPUs at high cost and use it to host nginx. That is not a correct configuration. There are much cheaper ways available to you. It's ridiculous to imply that AWS shouldn't send you a higher bill because you didn't use the GPUs or that AWS shouldn't offer instances with GPUs because they are more expensive. You, the user, are responsible for building a correct configuration with the low-level primitives that have been made available to you! If it's too much then feel free to move up the stack and host your workloads on a PaaS instead.
It being low level is not an excuse for systems that lead people down the wrong path.
And the traffic never even reaches the public internet. There's a mismatch between what the billing is supposedly for and what it's actually applied to.
> do you expect AWS to show you different meters for billed and not-billed traffic, but performance still depends on the sum total of the traffic (S3 and Internet egress) passing through it?
Yes.
> How is that not confusing?
That's how network ports work. They only go so fast, and you can be charged based on destination. I don't see the issue.
> It's also besides the point that not all NAT gateways are used for Internet egress
Okay, if two NAT gateways talk to each other it also should not have egress fees.
> some kind of implicit built-in S3 gateway violates assumptions
So don't do that. Checking if the traffic will leave the datacenter doesn't need such a thing.
AWS VPCs are secure by default, which means no traffic traverses their boundaries unless you intentionally enable it.
There are many IaC libraries, including the standard CloudFormation VPC template and CDK VPC class, that can create them automatically if you so choose. I suspect the same is also true of commonly-used Terraform templates.
As others have pointed out, this is by design. If VPCs have access to AWS resources (such as S3, DynamoDB, etc), an otherwise locked down VPC can still have data leaks to those services, including to other AWS accounts.
It's a convenience VS security argument, though the documentation could be better (including via AWS recommended settings if it sees you using S3).
I've been testing our PrivateLink connectivity at work in the past few weeks. This means I've been creating and destroying a bunch of VPCs to test the functionality. The flow in the AWS console when you select the "VPC and more" wizard does have an S3 Gateway enabled by default
If it were opt-out someone would accidentally leave it on and eventually realize that entire systems had been accidentally "backed up" and exfiltrated to S3.
What? The same is possible whether it's opt-in or opt-out. It's just that if you have the gateway as opt-out you wouldn't also have this problem AND a massive AWS bill. You would just have this problem.
The bad situation is if you created a VPC with no internet access but the hypothetical automatic VPC endpoint still let instances access S3. Then a compromised instance has a vector for data exfiltration.
"The door is locked, so instead of suggesting to the end user that they should unlock the door with this key that we know how to give the end user deterministically, we instead tell them to drive across town and back on our toll roads and collect money from it"
Speaking solely on my own behalf: I don't know a single person at AWS (and I know a lot of them) who wants to mislead customers into spending more money than they need to. I remember a time before Gateway Endpoints existed, and customers (including me at the time) were spending tons of money passing traffic through pricey NAT Gateways to S3. S3 Gateway Endpoints saved them money.
Clearly you guys are aware of the problem though. I mean, every time this thing happens there's probably a ticket. I've personally filed one myself years ago when it happened to me. So why has the behavior not changed? You don't have to give up security to remove this footgun, it's possible to remove it and still make it an opt-in action for security purposes.
Having experienced the joy of setting up VPC, subnets and PrivateLink endpoints the whole thing just seems absurd.
They spent the effort of branding private VPC endpoints "PrivateLink". Maybe it took some engineering effort on their part, but it should be the default out of the box, and an entirely unremarkable feature.
In fact, I think if you have private subnets, the only way to use S3 etc is Private Link (correct me if I'm wrong).
VPC endpoints in general should be free and enabled by default. That you need to pay extra to reach AWS' own API endpoints from your VPC feels egregious.
The original private endpoints implementation required meaningful work from the service teams (ec2 networking, s3, & ddb). It also changed how the "front end" API servers handled requests and how their infrastructure was deployed (at the time?). The newer LB/ENI style privatelink abstracts away _most_ of that "per service" implementation effort at the cost of more per-request/connection work fromthe virtual network. Hence why theres more support from other services, and it includes a cost.
> People who are price insensitive will not invest the time to fix it
This just sounds like a polite way of saying "we're taking peoples' money in exchange for nothing of value, and we can get away with it because they don't know any better".
That's fascinating! I hadn't found that in the documentation; everything seems to steer people towards PrivateLink, not gateway endpoints.
Would you recommend using VPC Gateway even on a public VPC that has an Internet gateway (note: not a NAT gateway)? Or only on a private VPC or one with a NAT gateway?
I recommend S3 Gateways for all VPCs that need to access S3, even those that already have routes to the Internet. Plus they eliminate the need for NAT Gateway traversal for requests that originate from private subnets.
It's a much more direct/efficient connection from the EC2 instance to the S3 storage servers through the virtual network layer. It reduces the network path/length through the AWS network _and_ removes the number of virtual network functions/servers (ala "LB") that your connections will traverse.
If you're saying "other services should offer VPC Endpoints," I am 100% on-board. One should never have to traverse the Internet to contact any AWS control plane
Both interface endpoints and gateway endpoints are also called VPC endpoints. The former get distinct IP addresses in your VPC subnets while the latter get distinct entries in your VPC routing tables. They are even created with the same API call: https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_C...
I doubt the traffic ever actually leaves AWS. Assuming it does make it all the way out to their edge routers, the destination ASN will still be one of their own. Not that the pricing will reflect this, of course.
The other problem with (interface) VPC endpoints is that they eat up IP addresses. Every service/region permutation needs a separate IP address drawn from your subnets. Immaterial if you're using IPv6, but can be quite limiting if you're using IPv4.
There were still a couple of services/features that choked on IPv6 last time I looked (1.5-2 years ago) but it works with most things and they do seem to be making progress on the others.
If you had an ALB inside the VPC that routed the requests to something that lives inside the VPC, which called the AWS PutObject api on the bucket, would that still be the case?
Some good stuff here. I wish AWS would just focus on these boring, but ultimately important, things that they’re good at instead of all the current distractions trying to play catch up on “AI.” AWS leadership missed the boat there big time, but that’s OK.
Ultimately AWS doesn’t have the right leadership or talent to be good at GenAI, but they do (or at least used to) have decent core engineers. I’d like to see them get back to basics and focus there. Right now leadership seems panicked about GenAI and is just throwing random stuff at the wall desperately trying to get something to stick. Thats really annoying to customers.
They continue to have large teams working on core stuff. It’s just that they’re working at such a low level (like high perf virtualized networking on their custom network cards) that most people don’t hear about it or care that much.
> You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots.
From my understanding, I don't think this is completely accurate. But, to be fair, AWS doesn't really document this very well.
From my (informal) conversations with AWS engineers a few months ago, it works approximately like this (modulo some details I'm sure the engineers didn't really want to share):
S3 requests scale based on something called a 'partition'. Partitions form automatically based on the smallest common prefixes among objects in your bucket, and how many requests objects with that prefix receive. And the bucket starts out with a single partition.
So as an example, if you have a bucket with objects "2025-08-20/foo.txt" and "2025-08-19/foo.txt", the smallest common prefix is "2" (or maybe it considers the root as the generator partition, I don't actually know). (As a reminder, a / in an object key has no special significance in S3 -- it's just another character. There are no "sub-directories"). Therefore a partition forms based on that prefix. You start with a single partition.
Now if the object "2025-08-20/foo.txt" suddenly receives a ton of requests, what you'll see happen is S3 throttle those requests for approximately 30-60 minutes. That's the amount of time it takes for a new partition to form. In this case, the smallest common prefix for "2025-08-20/foo.txt" is "2025-08-2". So a 2nd partition forms for that prefix. (Again, the details here may not be fully accurate, but this is the example conveyed to me). Once the partition forms, you're good to go.
But the key issue here with the above situation is you have to wait for that warm up time. So if you have some workload generating or reading a ton of small objects, that workload may get throttled for a non-trivial amount of time until partitions can form. If the workload is sensitive to multi-minute latency, then that's basically an outage condition.
The way around this is that you can submit an AWS support ticket and have them pre-generate partitions for you before your workload actually goes live. Or you could simulate load to generate the partitions. But obviously, neither of these is ideal. Ideally, you should just really not try and store billions of tiny objects and expect unlimited scalability and no latency. For example, you could use some kind of caching layer in front of S3.
Yep, this is still a thing. In the past year I’ve been throttled due to hot partitions. They’ve improved the partitioning so you hit it less, but if you scale too fast you will get limited.
Hit it when building an iceberg Lakehouse using pre existing data. Using object prefixes fixed the issue.
This is my understanding too, and this is particularly problematic for workloads that are read/write heavy on very recent data. When partitioning by a date or by an auto-incrementing id, you still run into the same issue.
Ex: your prefix is /id=12345. S3, under the hood, generates partitions named `/id=` and `/id=1`. Now, your id rolls over to `/id=20000`. All read/write activity on `/id=2xxxx` falls back to the original partition. Now, on rollover, you end up with read contention.
For any high-throughput workloads with unevenly distributed reads, you are best off using some element of randomness, or some evenly distributed partition key, at the root of your path.
They did this to stop people from overloading us-east-1a.
It was fine, until there started to be ways of wiring up networks between accounts (eg PrivateLink endpoint services) and you had to figure out which AZ was which so you could be sure you were mapping to the the same AZs in each account.
I built a whole methodology for mapping this out across dozens of AWS accounts, and built lookup tables for our internal infrastructure… and then AWS added the zone ID to AZ metadata so that we could just look it up directly instead.
It was for spreading load out. If someone was managing resources in a bunch of accounts and always defaulted to, say, 1b, AWS randomized what AZs corresponded to what datacenter segments to avoid hot spots.
The canonical AZ naming was provided because, I bet, they realized that the users who needed canonical AZ identifiers were rarely the same users that were causing hot spots via always picking the same AZ.
Almost everyone went with 1a, every time. It causes significant issues for all sorts of reasons, especially considering the latency target for network connections between data centres in an AD
>VPC peering used to be annoying; now there are better options like Transit Gateway, VPC sharing between accounts, resource sharing between accounts, and Cloud WAN.
I think there is some nuance needed here. If you ask support to partition your bucket then they will be a bit annoying if you ask for specific partition points and the first part of the prefix is not randomised. They tried to push me to refactor the bucket first to randomise the beginning of the prefix, but eventually they did it.
The auto partitioning is different. It can isolate hot prefixes on its own and can intelligently pick the partition points. Problem is the process is slow and you can be throttled for more than a day before it kicks in.
They can do this with manual partitioning indeed. I've done it before, but it's not ideal because the auto partitioner will scale beyond almost anything AWS will give you with manual partitioning unless you have 24/7 workloads.
> you can be throttled for more than a day before it kicks in
I expect that this would depends on your use case. If you are dropping content you need to scale out to tons of readers, that is absolutely the case. If you are dropping tons of content with well distributed reads, then the auto partitioner is The Way.
It would've been nice if each of those claims in the article also linked to either the relevant announcement or to the documentation. If I'm interested in any of these headline items, I'd like to learn more.
I don't believe AWS offers permalinks, so it would only help until they rolled over the next documentation release :-(
They actually used to have the upstream docs in GitHub, and that was super nice for giving permalinks but also building the docs locally in a non-pdf-single-file setup. Pour one out, I guess
TAMs are super hit and miss. We’ve had great ones (hi Nick!) and not so great ones.
($7-10M/mo customer AWS spend, support is a complicated sliding scale % of that, gogo ES!). Non-ES at smaller customers has been universally useless, except at quota increases.
Every AWS update can potentially affect your SOC 2 or HIPAA compliance posture. I've seen companies fail audits because they assumed their security configurations were still current.
The cloud moves fast. Compliance processes need to keep up. Manual annual reviews aren't enough when your infrastructure is changing constantly.
This is also why we built automated compliance monitoring - because what worked last quarter might not work today.
Also S3 related: the bucket owner can now be configured as the object owner no matter where the object originated. In the past this was exceedingly painful if you wanted to allow one account contribute objects to a bucket in another account. You could do the initial contribution, but the contributor always owned the object, and you couldn't delegate access to a third account.
This article was a relief. I’m always a tiny bit worried Amazon will change some thing drastically and I’ll have to migrate. I’ve had an ec2 instance running since 2013. It requires effectively zero maintenance. So I am glad there were no surprises in this article. Thanks OP.
A "Catch me up" on AWS (and for that matter other large platforms) would be very useful for many folks.
Ideally it should be a stream of important updates that can be interactively filtered by time-range. For example, if I have not been actively consuming AWS updates firehose for last 18 months, I should be able to "summarize" that length of updates.
Why this is not already a feature of "What's New" section of AWS and other platforms -- I dont know. Waiting to be built -- either by OEM or by the Community.
I played a lot of DOTA2 in the past and I've often thought that big tech could learn something from Valve's patch notes. Especially in the context of process changes, stuff you should know, etc. Expecting folk to read a series of lengthy emails/blog posts to stay up to date is unrealistic.
I haven’t used AWS in the last 5 years. Is IPv6 still somewhat of an issue? I remember some services not supporting it at all and making it impossible to manage as a IPv6-only network.
We enabled ipv6 for our APIs at work. Nothing broke immediately, but we've had a steady stream of unreachable host errors related to ipv6 since then.
Turns out there're many incorrect implementations of Happy Eyeballs that cancel the ipv4 connection attempts after the timeout, and then switch to trying the AAAA records and subsequently throwing a "Cannot reach host" error. For context, in Happy Eyeballs you're supposed to continue trying both network families in parallel.
This only impacts our customers who live far away from the region they're accessing, however, and there's usually a workaround - in Node you can force the network family to be v4 for instance
The amazonaws.com domain endpoints did not introduce ipv6/AAAA directly is (mostly) due to access control. For better or worse there are a lot of "v4 centric" IAM statements, like aws:SourceIp, in identity/resource/bucket policies. Introducing a new v6 value is going to break all of those existing policies with either unexpected DENYs or, worse, ALLOWs. Thats a pretty poor customer experience to unexpectedly break your existing infrastructure or compromise your access control intentions.
AWS _could_ have audited every potential IAM policy and run a MASSIVE outreach campaign, but something as simple as increasing (opaque!) instance ID length was a multi year effort. And introducing backwards compatibility on a _per policy_ basis is its own infinite security & UX yak shaving exercise as well.
So thats why you have opt-in usage of v6/dualstack in the client/SDK/endpoint name.
I have a preempt-able workload for which I could use Spot instances or Savings Plans.
Does anyone have experience running Spot in 2025? If you were to start over, would you keep using Spot?
- I observe with pricing that Spot is cheaper
- I am running on three different architectures, which should limit Spot unavailability
- I've been running about 50 Spot EC2 instances for a month without issue. I'm debating turning it on for many more instances
In terms of cost, from cheapest to most expensive:
1. Spot with autoscaling to adjust to demand and a savings plan that covers the ~75th percentile scale
2. On-demand with RIs (RIs will definitely die some day)
3. On-demand with savings-plans (More flexible but more expensive than RIs)
3. Spot
4. On-demand
I definitely recommend spot instances. If you're greenfielding a new service and you're not tied to AWS, some other providers have hilariously cheap spot markets - see http://spot.rackspace.com/. If you're using AWS, definitely auto-scaling spot with savings plans are the way to go. If you're using Kubernetes, the AWS Karpenter project (https://karpenter.sh/) has mechanisms for determining the cheapest spot price among a set of requirements.
Overall tho, in my experience, ec2 is always pretty far down the list of AWS costs. S3, RDS, Redshift, etc wind up being a bigger bill in almost all past-early-stage startups.
To "me, too" this, it's not like that AWS spot instance just go "poof," they do actually warn you (my recollection is 60s in advance of the TerminateInstance call), and so a resiliency plane on top of the workloads (such as the cited Kubernetes) can make that a decided "non-event". Shout out to the reverse uptime crew, a subset of Chaos Engineering
I just saw Weird Al in concert, and one of my favorite songs of his is "Everything You Know is Wrong." This is the AWS version of that song! Nice work Corey!
> Once upon a time Glacier was its own service that had nothing to do with S3. If you look closely (hi, billing data!) you can see vestiges of how this used to be, before the S3 team absorbed it as a series of storage classes.
Your assumption holds if they still use tape. But this paragraph hints at it not being tape anymore. The eternal battle between tape versus drive backup takes another turn.
I am also assuming that Amazon intends for the Deep Archive tier to be a profitable offering. At $0.00099/gb-month, I don't see how it could be anything other than tape.
I wonder if it's where old S3 hard drives go to die? Presumably AWS have the world's single largest collection of used storage devices - if you RAID them up you can probably get reliable performance out of them for Glacier?
Not quite. Hardware with customer data or corp IP (eg any sort of storage or nvram) doesnt leave the "red zone"[1] without being destroyed. And reusing EOL hardware is a nightmare of failure rates and consistency issues. Its usually more cost effective to scrap the entire rack once its depreciated, or potentially at the 4-5 year mark at most. More revenue is generated by replacing the entire rack with new hardware that will make better use of the monthly recurring cost (MRC) of that rack position/power whips/etc.
I still don't know if it's possible to make it profitable with old drives in this kind of arrangement, especially if we intend to hit their crazy durability figures. The cost of keeping drives spinning is low, but is double-digit margin % in this context. You can't leave drives unpowered in a warehouse for years on end and say you have 11+ nines of durability.
Unpowered in a warehouse is a huge latency problem.
For storage especially we now build enough redundancy into systems that we don't have to jump on every fault. That reduces the chance of human error when trying to address it, and pushing the hardware harder during recovery (resilvering, catching up in a distributed concensus system, etc).
When the entire box gets taken out of the rack due to hitting max faults, then you can piece out the machine and recycle parts that are still good.
You could in theory ship them all off to the backend of nowhere, but it seems that Glacier is all the places where AWS data centers are, so it's not that. But Glacier being durable storage, with a low expectation of data out versus data in, they could and probably are cutting the aggregate bandwidth to the bone.
How good do your power backups have to be to power a pure Glacier server room? Can you use much cheaper in-rack switches? Can you use old in-rack switches from the m5i era?
Also most of the use cases they mention involve linear reads, which has its own recipe book for optimization. Including caching just enough of each file on fast media to hide the slow lookup time for the rest of the stream.
Little's Law would absolutely kill you in any other context but we are linear write, orders of magnitude fewer reads here. You have hardware sitting around waiting for a request. "Orders of magnitude" is the space where interesting solutions can live.
Only if you have low redundancy. RAIDZ is better about this isn’t it? And Backblaze goes a lot farther. They just decommission the rack when it hits the limit for failed disks, and the files on the cluster are stored on m of n racks, so adding a rack and “resilvering” doesn’t even require scanning the entire cluster, just m/n of it.
My understanding is some AWS products (e.g. RDS) need very fast disks with lots of IOPS. To get the IOPS, though, you have to buy +++X TB sized SSDs, far more storage space than RDS actually needs. This doesn't fully utilize the underlying hardware, you are left with lots of remaining storage space but no IOPS. It's perfect for Glacier.
The disks for Glacier cost $0 because you already have them.
Since ~2014 or so the constraint on all HDD based storage has been IOPs/throughput/queue time. Shortly after that we started seeing "minimum" device sizes that were so large as to be challenging to productively use their total capacity. Glacier type retrieval is also nice in that you have much more room for "best effort" scheduling and queuing compared to "real time" request like S3:PutObject.
Last I was aware flash/nvme storage didnt have quite the same problem, due to orers of magnitude improved access times and parallelism. But you can combine the two in a kind of distributed reimplementation of access tiering (behind a single consistent API or block interface).
There’s a really old trick with HDDs where you buy a big disc and then allocate less than half of it. There’s more throughput on the first half of the disk, more tracks per cylinder so fewer seeks, and never having to read half the disk reduces the worst case seek time. All increase IOPs.
But then what do you do with the other half of the disk? If you access it when the machine isn’t dormant you lose most of these benefits.
For deep storage you have two problems. Time to access the files, and resources to locate the files. In a distributed file store there’s the potential for chatty access or large memory footprints for directory structures. You might need an elaborate system to locate file 54325 if you’re doing some consistent hashing thing, but the customer has no clue what 54325 is. They want the birthday party video. So they still need a directory structure even if you can avoid it.
One way to know is to see if new tape products exist, indicating ongoing development. As of May, 2025, LTO-10 is available, offering 30TB/75TB (raw/compressed) storage per cartridge. Street price is a bit under $300 each. Two manufacturers are extant: Fujifilm and IBM.
Generally speaking this isn't something Amazon S3 customers need to worry about - as others have said, S3 will automatically scale index performance over time based on load. The challenge primarily comes when customers need large bursts of requests within a namespace that hasn't had a chance to scale - that's when balancing your workload over randomized prefixes is helpful.
FWIW The optimal way we were told was to partition our data was to do this:
010111/some/file.jpg.
Where `010111/` is a random binary string which will please both the automatic partitioning (503s => partition) and manual partitioning you could ask AWS. Please as in the cardinality of partitions grows slower at each characters vs prefixes like `az9trm/`.
We were told that the later version makes manual partitioning a challenge because as soon as you reach two characters you've already created 36x36 partitions (1,296).
The issue with that: your keys are no more meaningful if you're relying on S3 to have "folders" by tenants for example (customer1/..).
I have had the same experience within the last 18 months. The storage team came back to me and asked me to spread my ultra high throughput write workload across 52 (A-Za-z) prefixes and then they pre-partitioned the bucket for me.
S3 will automatically do this over time now, but I think there are/were edge cases still. I definitely hit one and experienced throttling at peak load until we made the change.
By the way, that happens quite frequently. I regularly ask them about new AWS technologies or recent changes, and most of the time they are not aware. They usually say they will call back later after doing some research.
It was always there but it required much more activity to get it done (document your use case & traffic levels and then work with your TAM to get the limit changed).
I just started working with a vendor who has a service behind API Gateway. It is a bit slow(!) and times out at 30 seconds. I've since modified my requests to chunk subsets of the whole dataset, to keep things under the timeout.
Has this changed? Is 30 secs the new or the old timeout?
For our VM solution, we get around this by hot staging VM's. As soon as one customer stops theirs, we reset everything and start it up again. To the end user, our compute seems to be instantly available. Unless of course, we run out.
> DynamoDB now supports empty values for non-key String and Binary attributes in DynamoDB tables. Empty value support gives you greater flexibility to use attributes for a broader set of use cases without having to transform such attributes before sending them to DynamoDB. List, Map, and Set data types also support empty String and Binary values.
Lots of this seems to boil down to: AWS shipped something that was barely usable, but then iterated.
That's a reasonable approach, but the fact this post exists shows that this practice is a reputational risk. By all means do this if you think it's the right thing to do, but be aware that first impressions matter and will stick for a long time.
S3: "Block Public Access is now enabled by default on new buckets."
On the one hand, this is obviously the right decision. The number of giant data breeches caused by incorrectly configured S3 buckets is enormous.
But... every year or so I find myself wanting to create an S3 bucket with public read access to I can serve files out of it. And every time I need to do that I find something has changed and my old recipe doesn't work any more and I have to figure it out again from scratch!
The thing to keep in mind with the "Block Public Access" setting is that is a redundancy built in to save people from making really big mistakes.
Even if you have a terrible and permissive bucket policy or ACLs (legacy but still around) configured for the S3 bucket, if you have Block Public Access turned on - it won't matter. It still won't allow public access to the objects within.
If you turn it off but you have a well scoped and ironclad bucket policy - you're still good! The bucket policy will dictate who, if anyone, has access. Of course, you have to make sure nobody inadvertantly modifies that bucket policy over time, or adds an IAM role with access, or modifies the trust policy for an existing IAM role that has access, and so on.
I think this is the key of why I find it confusing: I need a very clear diagram showing which rules override which other rules.
My understanding is that there isn't actually any "overriding" in the sense of two rules conflicting and one of them having to "win" and take effect. I think it's more that an enabled rule always is in effect, but it might overlap with another rule, in which case removing one of them still won't remove the restrictions on the area of overlap. It's possible I'm reading too much into your choice of words, but it does sound like there's a chance that the confusion is stemming from an incorrect assumption of how various permissions interact.
That being said, there's certain a lot more that could into making a system like that easier for developers. One thing that springs to mind is tooling that can describe what rules are currently in effect that limit (or grant, depending on the model) permissions for something. That would make it more clear when there are overlapping rules that affect the permissions of something, which in turn would make it much more clear why something is still not accessible from a given context despite one of the rules being removed.
If one rule explicitly restricts access and another explicitly grants access, which one is in effect? Do restrictions override grants? Does a grant to GroupOne override a restriction to GroupAlpha when the authenticated use in is both groups? Do rules set by GodAdmin override rules set by AngelAdmin?
It's possible I'm making the exact mistake that the article describes and relying on outdated information, but my understanding is that pretty much all of the rules are actually permissions rather than restrictions. "Block public access" is an unfortunate exception to this, and I suspect that it's probably just a poorly named inversion of an "allow public access" permission. You're 100% right that modeling permissions like this requires having everything in the same "direction", i.e. either all permissions or all restrictions.
After thinking about this sort of thing a lot when designing a system for something sort of similar to this (at a much smaller scale, but with the intent to define it in a way that could be extended to define new types of rules for a given set of resources), I feel pretty strongly that the best way for a system like this to work from the protectives of security, ease of implementation, and intuitiveness for users are all aligned in requiring every rule to explicitly be defined as a permission rather than representing any of them as restrictions (both in how they're presented to the user and how they're modeled under the hood). With this model, veryifing whether an action is allowed can be implemented by mapping an action to the set of accesses (or mutations, as the case may be) it would perform, and then checking that each of them has a rule present that allows it. This makes it much easier to figure out whether something is allowed or not, and there's plenty of room for quality of life things to help users understand the system (e.g. being able to easily show a user what rules pertain to a given resource with essentially the same lookup that you'd need to do when verifying an action in it). My sense is that this is actually not far from how AWS permissions are implemented under the hood, but they completely fail at the user-facing side of this by making it much harder than it needs to be to discover where to define the rules for something (and by extension, where to find the rules currently in effect for it).
They don't really override each other but they act like stacked barriers, like a garage door blocking access to an open or closed car. Access is granted if every relevant layer allows it.
This sort of thing drives me nuts in interviews, when people are like, are you familiar with such-and-such technology?
Yeah, what month?
If you're aware of changes, then explain that there were changes over time, that's it
You seem to be lacking the experience of what actually happens in interviews.
You say this, someone challenges you, now you're on the defensive during an interview and everyone has a bad taste in their mouth. Yeah, that's how it goes.
That's just the taste of iron from the blood after the duel. But this is completely normal after a formal challenge! Companies want real cyberwarriors, and the old (lame) rockstar ninjas that they hired 10 years ago are very prone to issuing these.
I don’t want to go to war, I just want a quiet house in the mountains and a career that allows me to think about things.
Amen.
I just stick CloudFront in front of those buckets. You don't need to expose the bucket at all then and can point it at a canonical hostname in your DNS.
That’s definitely the “correct” way of doing things if you’re writing infra professionally. But I do also get that more casual users might prefer not to incur the additional costs nor complexity of having CloudFront in front. Though at that point, one could reasonably ask if S3 is the right choice for causal users.
S3 + cloudfront is also incredibly popular so you can just find recipes for automating that in any technology you want, Terraform, ansible, plain bash scripts, Cloudformation (god forbid)
Yeah holy crap why is cloud formation so terrible?
It's designed to be a declarative DSL, but then you have to do all sorts of filters and maps in any group of resources and suddenly you are programming in yaml with both hands tied behind your back
Yeah it’s just terrible. If Amazon knew what was good they’d just replace it with almost anything else. Heck just got all in on terraform and call it a day.
This may be heresy in an AWS thread, but as a concept Bicep actually isn't terrible: https://github.com/Azure/bicep/blob/v0.37.4/src/Bicep.Cli.E2...
It does compile down to Azure Resource Manager's json DSL, so in that way close to Troposphere I guess, only both sides are official and not just some rando project that happens to emit yaml/json
The implementation, of course, is ... very Azure, so I don't mean to praise using it, merely that it's a better idea than rawdogging json
I’ve heard so many bad things about bicep on Azure that I’m not convinced it’s an upgrade over TF.
The syntax does look nicer but sadly that’s just a superficial improvement.
They do contribute to the AWS provider for Terraform.
Also that have CDK which is a framework for writing IaC in Java/TypeScript, Go, Python, etc.
Meh. The CDK doesn’t look terrible. It’s still not ideal. But even if this compiles to a mess of CF it’s still better than writing CF by hand and that’s only because CF is so bad to begin with.
https://dev.to/kelvinskell/getting-started-with-aws-cdk-in-p...
As for "go all in on terraform," I pray to all that is holy every night that terraform rots in the hell that spawned it. And that's not even getting into the rug pull parts, I mean the very idea of
1. I need a goddamn CLI to run it (versus giving someone a URL they can load in their tenant and have running resources afterward)
1. the goddamn CLI mandates live cloud credentials, but then stright-up never uses them to check a goddamn thing it intends to do to my cloud control plane
You may say "running 'plan' does" and I can offer 50+ examples clearly demonstrating that it does not catch the most facepalm of bugs
1. related to that, having a state file that believes it knows what exists in the world is just ludicrous and pain made manifest
1. a tool that thinks nuking things is an appropriate fix ... whew. Although I guess in our new LLM world, saying such things makes me the old person who should get onboard the "nothing matters" train
and the language is a dumpster, imho
There's a lot wrong with Terraform but I don't think you're being at all fair with your specific critisims here:
> 1. I need a goddamn CLI to run it (versus giving someone a URL they can load in their tenant and have running resources afterward)
CloudFormation is the only IaC that supports "running as a URL" and that's only because it's an AWS native solution. And CloudFormation is a hell of a lot more painful to write and slower to iterate on. So you're not any better off for using CF.
What usually happens with TF is you'd build a deploy pipeline. Thus you can test via the CLI then deploy via CI/CD. So you're not limited to just the CLI. But personally, I don't see the CLI as a limitation.
> the goddamn CLI mandates live cloud credentials, but then stright-up never uses them to check a goddamn thing it intends to do to my cloud control plane
All IaC requires live cloud credentials. It would be impossible for them to work without live credentials ;)
Terraform does do a lot of checking. I do agree there is a lot that the plan misses though. That's definitely frustrating. But it's a side effect of cloud vendors having arbitrary conditions that are hard to define and forever changing. You run into the same problem with any tool you'd use to provision. Heck, even manually deploying stuff from the web console sometimes takes a couple of tweaks to get right.
> 1. related to that, having a state file that believes it knows what exists in the world is just ludicrous and pain made manifest
This is a very strange complaint. Having a state file is the bare minimum any IaC NEEDS for it to be considered a viable option. If you don't like IaC tracking state then you're really little better off than managing resources manually.
> a tool that thinks nuking things is an appropriate fix ... whew.
This is grossly unfair. Terraform only destroys resources when:
1. you remove those resources from the source. Which is sensible because you're telling Terraform you no longer want those resources
2. when you make a change that AWS doesn't support doing on live resources. Thus the limitation isn't Terraform, it is AWS
In either scenario, the destroy is explicit in the plan and expected behaviour.
> CloudFormation is the only IaC that supports "running as a URL"
Incorrect, ARM does too, they even have a much nicer icon for one click "Deploy to Azure" <https://learn.microsoft.com/en-us/azure/azure-resource-manag...> and as a concrete example (or whole repo of them): <https://github.com/Azure/azure-quickstart-templates/tree/2db...>
> All IaC requires live cloud credentials. It would be impossible for them to work without live credentials ;)
Did you read the rest of the sentence? I said it's the worst of both worlds: I can't run "plan" without live creds, but then it doesn't use them to check jack shit. Also, to circle back to our CF and Bicep discussion, no, I don't need cloud creds to write code for those stacks - I need only creds to apply them
I don't need a state file for CF nor Bicep. Mysterious about that, huh?
> Incorrect, ARM does too, they even have a much nicer icon for one click "Deploy to Azure"
That’s Azure, not AWS. My point was to have “one click” HTTP installs you need native integration with the cloud vendor. For Azure it’s the clusterfuck that is Bicep. For AWS it’s the clusterfuck that is CF
> I don't need a state file for CF nor Bicep.
CF does have a state file, it’s just hidden from view.
And bicep is shit precisely because it doesn’t track state. In fact the lack of a state file is the main complain against bicep and thus the biggest thing holding it back from wider adoption — despite being endorsed by Microsoft Azure.
All Terraform does is build a DAG, compare it with the current state file and pass the changes down to the provider so it can translate to the correct sequence of interactions with the upstream API. Most of your criticism boils down to limitations of the cloud provider API and/or Terraform provider quality. It won't check for naming collision for instance, it assumes you know what you are doing.
Regarding HCL, I respect their decision to keep the language minimal, and for all it's worth you can go very, very far with the language expressions and using modules to abstract some logic, but I think it's a fair criticism for the language not to support custom functions and higher level abstractions.
Amen, and I would add to that list “no, just because you use terraform doesn’t mean you can simply switch between cloud providers”.
Is there any IaC solutions where you can “simply switch between cloud providers”?
This isn’t a limitation of TF, it’s an intended consequence of cloud vendor lock in
I believe the usual uninformed thinking is "terraform exists outside of AWS, so I can move off of AWS" versus "we have used CF or Bicep, now we're stuck" kind of deal
Which is to say both of you are correct, but OP was highlighting the improper expectations of "if we write in TF, sure it sucks balls but we can then just pivot to $other_cloud" not realizing it's untrue and now you've used a rusty paintbrush as a screwdriver
I don’t think that expectation exists with anyone with even the slightest understanding of IaC and systems.
But maybe I’ve just been blessed to work with people who aren’t complete idiots?
Isn’t that what CDK was for?
Last time I tried to use CF, the third party IAC tools were faster to release new features than the functionality of CF itself. (Like Terraform would support some S3 bucket feature when creating a bucket, but CF did not).
I'm not sure if that's changed recently, I've stopped using it.
I have been on the terraform side for 7 years-ish.
eksctl just really impressed me with its eks management, specifically managed node groups & cluster add-ons, over terraform.
that uses cloudformation under the hood. so i gave it a try, and it’s awesome. combine with github actions and you have your IAC automation.
nice web interface for others to check stacks status, events for debugging and associated resources that were created.
oh, ever destroy some legacy complex (or not that complex) aws shit in terraform? it’s not going to be smooth. site to site connections, network interfaces, subnets, peering connections, associated resources… oh, my.
so far cloudformation has been good at destroying, but i haven’t tested that with massive legacy infra yet.
but i am happily converted tf>cf.
and will happily use both alongside each other as needed.
Because its an old early IaC language, but it works and lots depends on it, so instead of dumping or retooling it, AWS keeps it around as a compilation target, while pushing other solutions (years ago, the SAM transform on top of it, more recently CDK) as the main thing for people to actually use directly.
> Yeah holy crap why is cloud formation so terrible?
I can't confirm it, but I suspect that it was always meant to be a sales tool.
Every AWS announcement blog has a "just copy this JSON blob, and paste it $here to get your own copy of the toy demo we used to demonstrate in this announcement blog" vibe to it.
I'd argue putting CloudFront on top of S3 is less complex than getting the permissions and static sharing setup right on S3 itself.
I do get where you're coming from, but I don't agree. With the CF+S3 combo you now need to choose which sharing mode to work with S3 (there are several different ways you can link CF to S3). Then you have the wider configuration of CF to manage too. And that's before you account for any caching issues you might run into when debugging your site.
If you know what you're doing, as it sounds like you and I do, then all of this is very easy to get set up (but then aren't most things easy when you already know how? hehe). However we are talking about people who aren't comfortable with vanilla S3, so throwing another service into the mix isn't going to make things easier for them.
It's actually incredibly cheap. I think our software distribution costs, in the account I run, are around $2.00 a month. That's pushing out several thousand MSI packages a day.
S3 is actually quite expensive compared to the competition for both storage costs and egress costs. At a previous start-up, we had terrabytes of data on S3 and it was our second largest cost (after GPUs) and by some margin.
For small scale stuff, S3s storage and egress charges are unlikely to be impactful. But it doesn’t mean they’re cheap relative to the competition.
There are also ways you can reduce S3 costs, but then you're trading the costs received from AWS with the costs of hiring competent DevOps. Either way, you pay.
With CloudFront?
>S3 is the right choice for causal users.
It's so simple for storing and serving a static website.
Are there good and cheap alternatives?
Yeah, your classic web hoster. Just today I uploaded a static website to one via FTP.
Really? If I remember correctly: My Static website served from S3 + CF + R53 by about 0.67$ / mo, 0.5 being R53 from that, 0.16 being CF, S3 being 0.01 for my page.
BTW: Is GitHub Page still free for custom domains? (I don't know the EULA)
GitHub Pages are still free but commercial websites are forbidden.
For the sake of understanding, can you explain why putting CloudFront in front of the buckets helps?
Cloudfront allows you to map your S3 with both
- signed url's in case you want a session base files download
- default public files, for e.g. a static site.
You can also map a domain (sub-domain) to Cloudfront with a CNAME record and serve the files via your own domain.
Cloudfront distributions are also CDN based. This way you serve files local to the users location, thus increasing the speed of your site.
For lower to mid range traffic, cloudfront with s3 is cheaper as the network cost of cloudfront is cheaper. But for large network traffic, cloudfront cost can balloon very fast. But in those scenarios S3 costs are prohibitive too!
Not always that simple - for example if you want to automatically load /foo/index.html when the browser requests /foo/ you'll need to either use the web serving feature of S3 (bucket can't be private) or set up some lambda at edge or similar fiddly shenanigans.
I’m getting deja vu, didn’t they already do this like 10 years ago because people kept leaving their buckets wide open?
This is exactly what I use LLMs for. To just read the docs for me and pull out the base level demo code that's buried in all the AWS documentation.
Once I have that I can also ask it for the custom tweaks I need.
Back when GPT4 was the new hotness, I dumped the markdown text from the Azure documentation GitHub repo into a vector index and wrapped a chatbot around it. That way, I got answers based on the latest documentation instead of a year-old LLM model's fuzzy memory.
I now have the daunting challenge of deploying an Azure Kubernetes cluster with... shudder... Windows Server containers on top. There's a mile-long list of deprecations and missing features that were fixed just "last week" (or whatever). That is just too much work to keep up with for mere humans.
I'm thinking of doing the same kind of customised chatbot but with a scheduled daily script that pulls the latest doco commits, and the Azure blogs, and the open GitHub issue tickets in the relevant projects and dumps all of that directly into the chat context.
I'm going to roll up my sleeves next week and actually do that.
Then, then, I'm going to ask the wizard in the machine how to make this madness work.
Pray for me.
I just want a service that does this. Pulls in the latest docs into a vector db with a chat or front-end. Not the windows containers bit.
This could not possibly go wrong...
You're braver than me if you're willing to trust the LLM here - fine if you're ready to properly review all the relevant docs once you have code in hand, but there are some very expensive risks otherwise.
This is LLM as semantic search- so it's way way easier to start from the basic example code and google to confirm that it's correct than it is to read the docs from scratch and piece together the basic example code. Especially for things like configurations and permissions.
Sure, if you do that second part of verifying it. If you just get the LLM to spit it out then yolo it into production it is going to make you sad at some point.
There’s nothing brave in this. It generally works the way it should and even if it doesn’t - you just go back to see what went wrong.
I take code from stack overflow all the time and there’s like a 90% chance it can work. What’s the difference here?
However on AWS the difference between "generally working the way it should and not working the way it should" can be a 30,000$ cloud bill racked up in a few hours with EC2 going full speed ahead mining bitcoin.
For those high stakes cases maybe you can be more careful. You can still use an LLM to search and get references to the appropriate place and do your own verification.
But for low stakes LLM works just fine - not everything is going to blow up to a 30,000 bill.
In fact I'll take the complete opposite stance - verifying your design with an LLM will help you _save_ money more often than not. It knows things you don't and has awareness of concepts that you might have not even read about.
Well, the "accidentally making the S3 bucket public" scenario would be a good one. If you review carefully with full understanding of what e.g. all your policies are doing then great, no problem.
If you don't do that will you necessarily notice that you accidentally leaked customer data to the world?
The problem isn't the LLM it's assuming its output is correct just the same as assuming Stack Overflow answers are correct without verifying/understanding them.
I agree but its about the extent. I'm willing to accept the risk of ocassionally making S3 public but getting things done much faster, much like I don't meticulously read documentation when I can get the answer from stackoverflow.
If you are comparing with stackoverflow then I guess we are on the same page - most people are fine with taking stuff from stackoverflow and it doesn't count as "brave".
I think anyone who just copies and pastes from SO is indeed "brave" for pretty much exactly the same reason.
> I'm willing to accept the risk of ocassionally making S3 public
This is definitely where we diverge. I'm generally working with stuff that legally cannot be exposed - with hefty compliance fines on the horizon if we fuck up.
That's fair - I would definitely use stackoverflow liberally and dive into documentation when situation demands it.
The thing is that you can now ask the LLM for links and you can ask it to break down why it thinks a piece of code, for example, protects the bucket from being public. Things that are easy to verify against the actual docs.
I feel like this workflow is still less time, easier and less error prone than digging out the exact right syntax from the AWS docs.
They'll teach you how for $250 and a certification test...
I honestly don't mind that you have to jump through hurdles to make your bucket publically available and that it's annoying. That to me seems like a feature, not a bug
I think the OPs objection is not that hurdles exist but that they move them every time you try and run the track.
Sure... but last time I needed to jump through those hurdles I lost nearly an hour to them!
I'm still not sure I know how to do it if I need to again.
>In EC2, you can now change security groups and IAM roles without shutting the instance down to do it.
Hasn't it been this way for many years?
>Spot instances used to be much more of a bidding war / marketplace.
Yeah because there's no bidding any more at all, which is great because you don't get those super high spikes as availability drops and only the ones who bid super high to ensure they wouldn't be priced out are able to get them.
>You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots.
This one was a nightmare and it took ages to convince some of my more pig headed coworkers in the past that they didn't need to do it any more. The funniest part is that they were storing their data as millions and millions of 10-100kb files, so the S3 backend scaling wasn't the thing bottlenecking performance anyway!
>Originally Lambda had a 5 minute timeout and didn’t support container images. Now you can run them for up to 15 minutes, use Docker images, use shared storage with EFS, give them up to 10GB of RAM (for which CPU scales accordingly and invisibly), and give /tmp up to 10GB of storage instead of just half a gig.
This was/is killer. It used to be such a pain to have to manage pyarrow's package size if I wanted a Python Lambda function that used it. One thing I'll add that took me an embarrassingly long time to realize is that your Python global scope is actually persisted, not just the /tmp directory.
> You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots.
Sorry, this is absolutely still the case if you want to scale throughput beyond the few thousand IOPS a single shard can serve. S3 will automatically reshard your key space, but if your keys are sequential (eg leading timestamp) all your writes will still hit the same shard.
Source: direct conversations with AWS teams.
Re: SG, yeah I wasnt doing any cloud stuff when that was the case. Never had to restart anything for an SG change and this must be at least 5-6 years..
IAM Role change is more recent though
> Glacier restores are also no longer painfully slow.
I had a theory (based on no evidence I'm aware of except knowing how Amazon operates) that the original Glacier service operated out of an Amazon fulfillment center somewhere. When you put it a request for your data, a picker would go to a shelf, pick up some removable media, take it back, and slot it into a drive in a rack.
This, BTW, is how tape backups on timesharing machines used to work once upon a time. You'd put in a request for a tape and the operator in the machine room would have to go get it from a shelf and mount it on the tape drive.
The most likely explanation is that they used a tape robot, such as the one seen here:
https://www.reddit.com/r/DataHoarder/comments/12um0ga/the_ro...
Which is basically exactly what you described but the picker is a robot.
Data requests go into a queue; when your request comes up, the robot looks up the data you requested, finds the tape and the offset, fetches the tape and inserts it into the drive, fast-forwards it to the offset, reads the file to temporary storage, rewinds the tape, ejects it, and puts it back. The latency of offline storage is in fetching/replacing the casette and in forwarding/rewinding the tape, plus waiting for an available drive.
Realistically, the systems probably fetch the next request from the queue, look up the tape it's on, and then process every request from that tape so they're not swapping the same tape in and out twenty times for twenty requests.
I've read very definitive discussions on here that Glacier never used tape. It has always been powered off hard disks.
For truly write once read never data tape is the optimal storage method. It is exactly what the LTO standard was designed to do and it does it very well. You can be confident that you will be able to read every bit of data from a 30 year old tape, probably even 50 years old. It has the lowest bit error rate of any technology I am aware of. LTO-9 is better than 1 uncorrectable bit error in 10^20 user bits, which is 1 bit error in 12.5 exabytes. There is also the substantial advantage that tapes on a shelf are completely immune to ransomware. As a sysadmin I get that warm fuzzy feeling when critical data is backed up on a good LTO tape library.
As someone who does tape recovery on very very old tape I largely concur with this with a couple of caveats.
1. Do not encrypt your tapes if you want the data back in 30/50 years. We have had so many companies lose encryption keys and turn their tapes into paperweights because the company they bought out 17 years ago had poor key management.
2. The typical failure case on tape is physical damage not bit errors. This can be via blunt force trauma (i.e. dropping, or sometimes crushing) or via poor storage (i.e. mould/mildew).
3. Not all tape formats are created equal. I have seen far higher failure rates on tape formats that are repeatedly accessed, updated, ejected, than your old style write once, read none pattern.
Call it bad luck, but I’ve never had a fully successful restore. Drives eat tapes, drives are damaged and write bad data, robot arms die or malfunction. Tapes have NEVER worked for me. SANs and remote disk though, rock solid.
That said, I don’t miss any of that stuff, gimme S3 any day :)
You do realized that that isn't normal at all? LTO tape is still used by thousands of companies to backup many exabytes of data. I know it once saved Google from permanent loss of gmail data from a bug. You should really get a refund for your tape drives.
Aren't LTO formats only backward compatible with the immediate prior version?
They can write to one version back and read two version back. for really long term data storage you have to also store the read/write hardware.
By the time it's hard to get a compatible LTO drive, I'd be very suspicious of a mothballed drive working either. If you want reliable long term storage you're going to have to update it every couple decades.
That's... interesting. I wonder what the wear-and-tear on an HDD is to spin it up/power it back down again.
I can't talk about it, but I've yet to see an accurate guess at how Glacier was originally designed. I think I'm in safe territory to say Glacier operated out of the same data centers as every other AWS service.
It's been a long time, and features launched since I left make clear some changes have happened, but I'll still tread a little carefully (though no one probably cares there anymore):
One of the most crucial things to do in all walks of engineering and product management is to learn how to manage the customer expectations. If you say customers can only upload 10 images, and then allow them to upload 12, they will come to expect that you will always let them upload 12. Sometimes it's really valuable to manage expectations so that you give yourself space for future changes that you may want to make. It's a lot easier to go from supporting 10 images to 20, than the reverse.
Im like 90% sure ive seen folks (unofficially) disclose the original storage and API decisions over the years, in roughly accurate terms. Personally I think the multi dimensional striping/erasure code ideas are way more interesting than the “its just a tape library” speculation/arguments. That and the real lessons learned around product differentiation as supporting technologies converge.
> I can't talk about it, but I've yet to see an accurate guess at how Glacier was originally designed.
It feels odd that this is some sort of secret. Why can't you talk about it?
I signed NDAs. I wish Glacier was more open about their history, because it's honestly interesting, and they have a number of notable innovations in how they approach things.
Well assuming your NDA is a reasonable length I hope you talk about it later.
(And if Amazon is making unreasonable length NDAs I hope they lose a lot of money over it.)
..oh. That's clever. Thanks for posting this.
I think folks have missed what I think would have been clever about the implentation I (apparently) dreamt up. It's not that "it's just a tape library", it's that it would have used the existing FC and picker infrastructure that Amazon had already built, with some racks containing drives for removable media. I was thinking that it would not have been some special facility purely for Glacier, but rather one or more regular FCs would just have had some shelves with Glacier media (not necessarily tapes).
Then the existing pickers would get special instructions on their handhelds: Go get item number NNNN from Row/shelf/bin X/Y/Z and take it to [machine-M] and slot it in, etc.
Yeah, but they've been robotic for decades since.
They would definitely be using rubies robots given how uniform hard drives are. The only reason warehouses still have humans is that heterogeneity (different sizes, different textures, different squishiness, etc).
I'll add: When doing instance to instance communication (in the same AZ) always use private ips. If you use public ip routing (even the same AZ) this is charged as regional data transfer.
Even worse, if you run self hosted NAT instance(s) don't use a EIP attached to them. Just use a auto-assigned public IP (no EIP).
> You get charged $0.01/GB regional data transfer, even if in the same AZ.
My understanding is that transfer gets charged on both sides as well. So if you own both sides you'll pay $0.02/GB.
Is transfer into the instance not always free anymore?
I think there is more of us who kind of degenerated from doing it the AWS way - API Gateway, serverless lambdas mess around with IAM roles until it works, ... - to - Give me EC2 / LightSail VPS instance maybe an S3 bucket let's set domain through Route53 and go away with the rest of your orchestrion AWS.
At what point is AWS worth using over other compute competitors when you’re using them as a storage bucket + VPS. They’re wholly more expensive at that point. Why not go with a more traditional but rock solid VPS provider?
I have the opposite philosophy for what it’s worth: if we are going to pay for AWS I want to use it correctly, but maximally. So for instance if I can offload N thing to Amazon and it’s appropriate to do so, it’s preferable. Step Functions, lambda, DynamoDB etc, over time, have come to supplant their alternatives and its overall more efficient and cost effective.
That said, I strongly believe developers don’t do enough consideration as to how to maximize vendor usage in an optimal way
Your management will frequently be strangely happier to waste money on AWS, unfortunately.
Truly a marketing success.
> That said, I strongly believe developers don’t do enough consideration as to how to maximize vendor usage in an optimal way
Because it's not straightforward. 1) You need to have general knowledge of AWS services and their strong and weak points to be able to choose the optimal one for the task, 2) you need to have good knowledge of the chosen service (like DynamoDB or Step Functions) to be able to use it optimally; being mediocre at it is often not enough, 3) local testing is often a challenge or plain impossible, you often have to do all testing on a dev account on AWS infra.
Most work isn’t greenfield.
AWS can be used in a different, cost effective, way.
It can be used as a middle-ground capable of serving the existing business, while building towards a cloud agnostic future.
The good AWS services (s3, ec2, acm, ssm, r53, RDS, metadata, IAM, and E/A/NLBs) are actually good, even if they are a concern in terms of tracking their billing changes.
If you architect with these primitives, you are not beholden to any cloud provider, and can cut over traffic to a non AWS provider as soon as you’re done with your work.
Of that list, watch out since IAM != IAM != IAM, so "cloud agnostic" is that famous 80/20 split
Some stuff is going to be provider-specific.
Let me explain why we’re not talking about an 80/20 split.
There’s no reason to treat something like a route53 record, or security group rule, in the same way that you treat the creation of IAM Policies/Roles and their associated attachments.
If you create a common interface for your engineers/auditors, using real primitives like the idea of a firewall rule, you’ve made it easy for everyone to avoid learning the idiosyncrasies of each deployment target, and feel empowered to write their own merge requests, or review the intended state of a given deployment target.
If you need to do something provider-specific, make a provider-specific module.
I agree that using them as a VPS provider is a mistake.
If you don't use the E(lasticity) of EC2, you're burning cash.
For prod workloads, if you can go from 1 to 10 instances during an average day, that's interesting. If you have 3 instances running 24/7/365, go somewhere else.
For dev workloads, being able to spin instances in a matter of seconds is a bliss. I installed the wrong version of a package on my instance? I just terminate it, wait for the auto-scaling group to pop a fresh new one a start again. No need to waste my time trying to clean my mess on the previous instance.
You speak about Step Functions as an efficient and cost effective service from AWS, and I must admit that it's one that I avoid as much as I can... Given the absolute mess that it is to setup/maintain, and that you completely lock yourself in AWS with this, I never pick it to do anything. I'd rather have a containerized workflow engine running on ECS, even though I miss on the few nice features that SF offers within AWS.
The approach I try to have is:
- business logic should be cloud agnostic
- infra should swallow all the provider's pills it needs to be as efficient as possible
>business logic should be cloud agnostic
In practice I found this to be more burden than it’s worth. I have yet to work somewhere that is on Azure, GCP or AWS and actually switch between clouds. I am sure it happens, but is it really that common?
I instead think of these platforms as a marriage, you’re going to settle in one and do your best to never divorce
Part of my job is to do migrations for customers, so, to me at least, it's not uncommon.
Using all the bells and whistles of a provider and being locked-in is one thing. But the other big issue is that, as service providers, they can (and some of them did more often than not) stop providing some services or changing them in a way that forces you to make big changes in your app to keep it running on this service.
Whereas, if you build your app in a agnostic way, they can stop or change what they want, you either don't rely on those services heavily enough for the changes required to be huge, or you can just deploy elsewhere, with another flavor of the same service.
Let's say you have a legacy Java app that works only with a library that is not maintained. If you don't want to bear the cost of rewriting with a new and maintained library, you can keep the app running, knowing the risks and taking the necessary steps to protect you against it.
Whereas if your app relies heavily on DynamoDB's API and they decide to drop the service completely, the only way to keep the app running is to rewrite everything for a similar service, or to find a service relying on the same API elsewhere.
Because the compartmentalization of business duties means that devs are fighting uphill against the wind to sign a deal with a new vendor for something. It's business bikeshedding, as soon as you open the door to a new vendor everyone, especially finance, has opinions and you might end up stuck with a vendor you didn't want. Or you can use the pre-approved money furnace and just ship.
There are entire industries that have largely de-volved their clouds primarily for footprint flexibility (not all AWS services are in all regions) and billing consistency.
Honestly just having to manage IAM is such a time-suck that the way I've explained it to people is that we've traded the time we used to spend administering systems for time spent just managing permissions, and IAM is so obtuse that it comes out as a net loss.
There's a sweet spot somewhere in between raw VPSes and insanely detailed least-privilege serverless setups that I'm trying to revert to. Fargate isn't unmanageable as a candidate, not sure it's The One yet but I'm going to try moving more workloads to it to find out.
Usually I write some IaC to automate this tedium so I only have to go through the IAM setup pain once. Now if requirements change, that's an entirely different story...
So the problem when you combine IAC with CI/CD is that the role assumed by the CI agent needs privileges to deploy things, so you need a bootstrap config to set up what it needs. If you have a mandate to go least-privilege, then that needs to include only the permissions strictly needed by the current deployable. So, no "s3:*", you need each one listed.
So far so good, you can do this with a bootstrap script that you only need to run at project setup.
If you also have a mandate (effectively) to go fully serverless, then as your project evolves and you add functionality, what you find is that most interesting changes use something new in the platform. So you're not getting away with running the bootstrap script once. You're updating it and running it for almost every change. And you can't tell in advance what permissions you're going to need, because (especially if you're on terraform) there's apparently no documentation connecting the resources you want to manage and the permissions needed to do so. So you try to deploy your change, IAM pops an error or two, you try to figure out what permissions you need to add to the bootstrap script, you run it (fixing it when it breaks at this point), you try deploying again, IAM pops another couple of errors, and then you're in a grind cycle which you can't predict the length of - and you need to get to the end of it before you can even test your feature, because fully serverless means you can't run your application locally (and getting management to pay for the pro localstack licence is a dead end). At some point it won't be clear why IAM is complaining, because the error you get makes no sense whatsoever, so at that point it's off to support to find out a day later that ah, yes, you can't use an assumed role just there, it's got to be an actual role, and no, that's not written down anywhere, you've just got to know it, so you need to redesign how you're using the roles completely, and right about this point is when I usually want to buy a farm, raise goats, and get way too into oil painting, instead of whatever this insane waste of life is.
Can no longer login to my AWS account, because I never set up MFA.
Want to set up MFA ... login required to request device.
Yes, I know, they warned us far ahead of time. But not being able to request one of their MFA devices without a login is ... sucky.
Talk to support: https://support.aws.amazon.com/#/contacts/one-support?formId...
I did. That's just an AI, which says this:
> I understand your situation is a bit unique, where you are unable to log in to your AWS account without an MFA device, but you also can't order an MFA device without being able to log in. This is a scenario that is not directly covered in our standard operating procedures.
The best course of action would be for you to contact AWS Support directly. They will be able to review your specific case and provide guidance on how to obtain an MFA device to regain access to your account. The support team may have alternative options or processes they can walk you through to resolve this issue.
Please submit a support request, and one of our agents will be happy to assist you further. You can access the support request form here: https://console.aws.amazon.com/support/home
That last URL? You need to login to use it ...
Looks like something that you could solve easily through their support, no?
Support don't talk to you unless you pay for support
Easy, just log into your account and pay them for support.
Yes, they do.
https://support.aws.amazon.com/#/contacts/one-support?formId...
You know what's still stupid? That if you have an S3 bucket in the same region as your VPC that you will get billed on your NAT Gateway to send data out to the public internet and right back in to the same datacenter. There is simply no reason to not default that behavior to opt out vs opt in (via a VPC endpoint) beyond AWS profiting off of people's lack of knowledge in this realm. The amount of people who would want the current opt-in behavior is... if not zero, infinitesimally small.
It's a design that is secure by default. If you have no NAT gateway and no VPC Gateway Endpoint for S3 (and no other means of Internet egress) then workloads cannot access S3. Networking should be closed by default, and it is. If the user sets up things they don't understand (like NAT gateways), that's on them. Managed NAT gateways are not the only option for Internet egress and users are responsible for the networks they build on top of AWS's primitives (and yes, it is indeed important to remember that they are primitives, this is an IaaS, not a PaaS).
Fine for when you have no NAT gateway and have a subnet with truly no egress allowed. But if you're adding a NAT gateway, it's crazy that you need to setup the gateway endpoint for S3/DDB separately. And even crazier that you have to pay for private links per AWS service endpoint.
There's very real differences between NAT gateways and VPC Gateway Endpoints.
NAT gateways are not purely hands-off, you can attach additional IP addresses to NAT gateways to help them scale to supporting more instances behind the NAT gateway, which is a fundamental part of how NAT gateways work in network architectures, because of the limit on the number of ports that can be opened through a single IP address. When you use a VPC Gateway Endpoint then it doesn't use up ports or IP addresses attached to a NAT gateway at all. And what about metering? If you pay per GB for traffic passing through the NAT gateway, but I guess not for traffic to an implicit built-in S3 gateway, so do you expect AWS to show you different meters for billed and not-billed traffic, but performance still depends on the sum total of the traffic (S3 and Internet egress) passing through it? How is that not confusing?
It's also besides the point that not all NAT gateways are used for Internet egress, indeed there are many enterprise networks where there are nested layers of private networks where NAT gateways help deal with overlapping private IP CIDR ranges. In such cases, having some kind of implicit built-in S3 gateway violates assumptions about how network traffic is controlled and routed, since the assumption is for the traffic to be completely private. So even if it was supported, it would need to be disabled by default (for secure defaults), and you're right back at the equivalent situation you have today, where the VPC Gateway Endpoint is a separate resource to be configured.
Not to mention that VPC Gateway Endpoints allow you to define policy on the gateway describing what may pass through, e.g. permitting read-only traffic through the endpoint but not writes. Not sure how you expect that to work with NAT gateways. This is something that AWS and Azure have very similar implementatoons for that work really well, whereas GCP only permits configuring such controls at the Organization level (!)
They are just completely different networking tools for completely different purposes. I expect closed-by-default secure defaults. I expect AWS to expose the power of different networking implements to me because these are low-level building blocks. Because they are low-level building blocks, I expect for there to be footguns and for the user to be held responsible for correct configuration.
My objections here are in terms of how this manifests in billing. Especially when you consider the highway robbery rates for internet egress.
Again, you are dealing with low-level primitives. You can provision an EC2 VM with multiple GPUs at high cost and use it to host nginx. That is not a correct configuration. There are much cheaper ways available to you. It's ridiculous to imply that AWS shouldn't send you a higher bill because you didn't use the GPUs or that AWS shouldn't offer instances with GPUs because they are more expensive. You, the user, are responsible for building a correct configuration with the low-level primitives that have been made available to you! If it's too much then feel free to move up the stack and host your workloads on a PaaS instead.
It being low level is not an excuse for systems that lead people down the wrong path.
And the traffic never even reaches the public internet. There's a mismatch between what the billing is supposedly for and what it's actually applied to.
> do you expect AWS to show you different meters for billed and not-billed traffic, but performance still depends on the sum total of the traffic (S3 and Internet egress) passing through it?
Yes.
> How is that not confusing?
That's how network ports work. They only go so fast, and you can be charged based on destination. I don't see the issue.
> It's also besides the point that not all NAT gateways are used for Internet egress
Okay, if two NAT gateways talk to each other it also should not have egress fees.
> some kind of implicit built-in S3 gateway violates assumptions
So don't do that. Checking if the traffic will leave the datacenter doesn't need such a thing.
This is the intended use case for S3 VPC Gateway Endpoints, which are free of charge.
https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpo...
(Disclaimer: I work for AWS, opinions are my own.)
I think they know it. They are complaining it's not enabled by default (and so do I).
AWS VPCs are secure by default, which means no traffic traverses their boundaries unless you intentionally enable it.
There are many IaC libraries, including the standard CloudFormation VPC template and CDK VPC class, that can create them automatically if you so choose. I suspect the same is also true of commonly-used Terraform templates.
As others have pointed out, this is by design. If VPCs have access to AWS resources (such as S3, DynamoDB, etc), an otherwise locked down VPC can still have data leaks to those services, including to other AWS accounts.
It's a convenience VS security argument, though the documentation could be better (including via AWS recommended settings if it sees you using S3).
I've been testing our PrivateLink connectivity at work in the past few weeks. This means I've been creating and destroying a bunch of VPCs to test the functionality. The flow in the AWS console when you select the "VPC and more" wizard does have an S3 Gateway enabled by default
The problem is that the default behavior for this is opt-in, rather than opt-out. No one prefers opt-in. So why is it opt-in?
If it were opt-out someone would accidentally leave it on and eventually realize that entire systems had been accidentally "backed up" and exfiltrated to S3.
What? The same is possible whether it's opt-in or opt-out. It's just that if you have the gateway as opt-out you wouldn't also have this problem AND a massive AWS bill. You would just have this problem.
The bad situation is if you created a VPC with no internet access but the hypothetical automatic VPC endpoint still let instances access S3. Then a compromised instance has a vector for data exfiltration.
No, with opt-in the VPC subnet is secure by default. Someone has to explicitly allow access to S3 (or anything else.)
AWS VPCs are secure by default, which means no traffic traverses their boundaries unless you intentionally enable it.
"The door is locked, so instead of suggesting to the end user that they should unlock the door with this key that we know how to give the end user deterministically, we instead tell them to drive across town and back on our toll roads and collect money from it"
This has been a common gotcha for over a decade now: https://www.lastweekinaws.com/blog/the-aws-managed-nat-gatew...
Speaking solely on my own behalf: I don't know a single person at AWS (and I know a lot of them) who wants to mislead customers into spending more money than they need to. I remember a time before Gateway Endpoints existed, and customers (including me at the time) were spending tons of money passing traffic through pricey NAT Gateways to S3. S3 Gateway Endpoints saved them money.
Clearly you guys are aware of the problem though. I mean, every time this thing happens there's probably a ticket. I've personally filed one myself years ago when it happened to me. So why has the behavior not changed? You don't have to give up security to remove this footgun, it's possible to remove it and still make it an opt-in action for security purposes.
Your job depends upon you misunderstanding the problem.
Having experienced the joy of setting up VPC, subnets and PrivateLink endpoints the whole thing just seems absurd.
They spent the effort of branding private VPC endpoints "PrivateLink". Maybe it took some engineering effort on their part, but it should be the default out of the box, and an entirely unremarkable feature.
In fact, I think if you have private subnets, the only way to use S3 etc is Private Link (correct me if I'm wrong).
It's just baffling.
You can provision gateway endpoints for S3 and DynamoDB. They are free and considered best practice. They are opt-in though, but easy to enable.
And ECR, which I would guess impacts more folks than DynamoDB https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-e...
And, as as added benefit, they distinguish between "just pull" and "pull and push" which is nice
True, I forgot that. But depending on services you still have to have some Gateway and some Interface endpoints.
VPC endpoints in general should be free and enabled by default. That you need to pay extra to reach AWS' own API endpoints from your VPC feels egregious.
Gateway endpoints are free. Network endpoints (which are basically AWS-managed ENIs that can tunnel through VPC boundaries) are not free.
S3 can use either, and we recommend establishing VPC Gateway endpoints by default whenever you need S3 access.
(Disclaimer: I work for AWS, opinions are my own.)
Why don't you have gateway endpoints for all your APIs?
The original private endpoints implementation required meaningful work from the service teams (ec2 networking, s3, & ddb). It also changed how the "front end" API servers handled requests and how their infrastructure was deployed (at the time?). The newer LB/ENI style privatelink abstracts away _most_ of that "per service" implementation effort at the cost of more per-request/connection work fromthe virtual network. Hence why theres more support from other services, and it includes a cost.
The service teams don’t talk to each other…
I think that is by design https://konghq.com/blog/enterprise/api-mandate
That is price segmentation. People who are price insensitive will not invest the time to fix it
People who are probably shouldn't be on aws - but they usually have to for unrelated reasons, and they will work to reduce their bill.
> People who are price insensitive will not invest the time to fix it
This just sounds like a polite way of saying "we're taking peoples' money in exchange for nothing of value, and we can get away with it because they don't know any better".
It's more like: we made loads of stuff super cheap but here's where we make some money because it scales with use.
Price segmentation happens all the time in pretty much every industry.
There’s an entire Pandora’s box of shitty things that happen in pretty much every industry. I don’t think you want to use that defense.
Must be un-criticizable then?
>People who are price insensitive will not invest the time to fix it
Hideous.
The problem is that VPC endpoints aren't free.
They should be, of course, at least when the destination is an AWS service in the same region.
[edit: I'm speaking about interface endpoints, but S3 and DynamoDB can use gateway endpoints, which are free to the same region]
Gateway endpoints are free. Network endpoints (which are basically AWS-managed ENIs that can tunnel through VPC boundaries) are not free.
S3 can use either, and we recommend establishing VPC Gateway endpoints by default whenever you need S3 access.
(Disclaimer: I work for AWS, opinions are my own.)
That's fascinating! I hadn't found that in the documentation; everything seems to steer people towards PrivateLink, not gateway endpoints.
Would you recommend using VPC Gateway even on a public VPC that has an Internet gateway (note: not a NAT gateway)? Or only on a private VPC or one with a NAT gateway?
I recommend S3 Gateways for all VPCs that need to access S3, even those that already have routes to the Internet. Plus they eliminate the need for NAT Gateway traversal for requests that originate from private subnets.
> I recommend S3 Gateways for all VPCs that need to access S3, even those that already have routes to the Internet.
Fascinating. What's the advantage of doing that?
It's a much more direct/efficient connection from the EC2 instance to the S3 storage servers through the virtual network layer. It reduces the network path/length through the AWS network _and_ removes the number of virtual network functions/servers (ala "LB") that your connections will traverse.
That's helpful to know, thank you! I'll take a look at that and see if it improves S3 performance.
> everything seems to steer people towards PrivateLink, not gateway endpoints
Gateway endpoints only work for some things.
Privatelink endpoints can be of type gateway or interface. Only gateway is free and only S3 and dynamodb supports it.
Fair point, and valid for S3 (the topic at hand) and DynamoDB.
Other AWS services, though, don't support gateway endpoints.
https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-e...
~~I get the impression there are several others, too, but that one is of especial interest to me~~ Wowzers, they really are much better now:
If you're saying "other services should offer VPC Endpoints," I am 100% on-board. One should never have to traverse the Internet to contact any AWS control planeThose are VPC endpoints, not gateway endpoints.
Both interface endpoints and gateway endpoints are also called VPC endpoints. The former get distinct IP addresses in your VPC subnets while the latter get distinct entries in your VPC routing tables. They are even created with the same API call: https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_C...
Well yeah that's the point....why route through the public internet.
I doubt the traffic ever actually leaves AWS. Assuming it does make it all the way out to their edge routers, the destination ASN will still be one of their own. Not that the pricing will reflect this, of course.
The other problem with (interface) VPC endpoints is that they eat up IP addresses. Every service/region permutation needs a separate IP address drawn from your subnets. Immaterial if you're using IPv6, but can be quite limiting if you're using IPv4.
Sounds like a good reason to use IPv6.
There were still a couple of services/features that choked on IPv6 last time I looked (1.5-2 years ago) but it works with most things and they do seem to be making progress on the others.
I don’t know any software that doesn’t work with IPv6 as of 2021.
Just some internet services that haven’t upgraded. (But fixed by NAT.)
There's a matrix provided by Amazon themselves listing the level of IPv6 support in their various services: https://docs.aws.amazon.com/vpc/latest/userguide/aws-ipv6-su...
If you had an ALB inside the VPC that routed the requests to something that lives inside the VPC, which called the AWS PutObject api on the bucket, would that still be the case?
A company making revenue is not stupid.
Some good stuff here. I wish AWS would just focus on these boring, but ultimately important, things that they’re good at instead of all the current distractions trying to play catch up on “AI.” AWS leadership missed the boat there big time, but that’s OK.
Ultimately AWS doesn’t have the right leadership or talent to be good at GenAI, but they do (or at least used to) have decent core engineers. I’d like to see them get back to basics and focus there. Right now leadership seems panicked about GenAI and is just throwing random stuff at the wall desperately trying to get something to stick. Thats really annoying to customers.
They continue to have large teams working on core stuff. It’s just that they’re working at such a low level (like high perf virtualized networking on their custom network cards) that most people don’t hear about it or care that much.
Leadership is looking to provide infrastructure for anyone to just pick a model and get on doing things without the hassle of setting things up.
> You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots.
From my understanding, I don't think this is completely accurate. But, to be fair, AWS doesn't really document this very well.
From my (informal) conversations with AWS engineers a few months ago, it works approximately like this (modulo some details I'm sure the engineers didn't really want to share):
S3 requests scale based on something called a 'partition'. Partitions form automatically based on the smallest common prefixes among objects in your bucket, and how many requests objects with that prefix receive. And the bucket starts out with a single partition.
So as an example, if you have a bucket with objects "2025-08-20/foo.txt" and "2025-08-19/foo.txt", the smallest common prefix is "2" (or maybe it considers the root as the generator partition, I don't actually know). (As a reminder, a / in an object key has no special significance in S3 -- it's just another character. There are no "sub-directories"). Therefore a partition forms based on that prefix. You start with a single partition.
Now if the object "2025-08-20/foo.txt" suddenly receives a ton of requests, what you'll see happen is S3 throttle those requests for approximately 30-60 minutes. That's the amount of time it takes for a new partition to form. In this case, the smallest common prefix for "2025-08-20/foo.txt" is "2025-08-2". So a 2nd partition forms for that prefix. (Again, the details here may not be fully accurate, but this is the example conveyed to me). Once the partition forms, you're good to go.
But the key issue here with the above situation is you have to wait for that warm up time. So if you have some workload generating or reading a ton of small objects, that workload may get throttled for a non-trivial amount of time until partitions can form. If the workload is sensitive to multi-minute latency, then that's basically an outage condition.
The way around this is that you can submit an AWS support ticket and have them pre-generate partitions for you before your workload actually goes live. Or you could simulate load to generate the partitions. But obviously, neither of these is ideal. Ideally, you should just really not try and store billions of tiny objects and expect unlimited scalability and no latency. For example, you could use some kind of caching layer in front of S3.
Yep, this is still a thing. In the past year I’ve been throttled due to hot partitions. They’ve improved the partitioning so you hit it less, but if you scale too fast you will get limited.
Hit it when building an iceberg Lakehouse using pre existing data. Using object prefixes fixed the issue.
This is my understanding too, and this is particularly problematic for workloads that are read/write heavy on very recent data. When partitioning by a date or by an auto-incrementing id, you still run into the same issue.
Ex: your prefix is /id=12345. S3, under the hood, generates partitions named `/id=` and `/id=1`. Now, your id rolls over to `/id=20000`. All read/write activity on `/id=2xxxx` falls back to the original partition. Now, on rollover, you end up with read contention.
For any high-throughput workloads with unevenly distributed reads, you are best off using some element of randomness, or some evenly distributed partition key, at the root of your path.
> Availability Zones used to be randomized between accounts (my us-east-1a was your us-east-1c)
WTH?
They did this to stop people from overloading us-east-1a.
It was fine, until there started to be ways of wiring up networks between accounts (eg PrivateLink endpoint services) and you had to figure out which AZ was which so you could be sure you were mapping to the the same AZs in each account.
I built a whole methodology for mapping this out across dozens of AWS accounts, and built lookup tables for our internal infrastructure… and then AWS added the zone ID to AZ metadata so that we could just look it up directly instead.
It was for spreading load out. If someone was managing resources in a bunch of accounts and always defaulted to, say, 1b, AWS randomized what AZs corresponded to what datacenter segments to avoid hot spots.
The canonical AZ naming was provided because, I bet, they realized that the users who needed canonical AZ identifiers were rarely the same users that were causing hot spots via always picking the same AZ.
Almost everyone went with 1a, every time. It causes significant issues for all sorts of reasons, especially considering the latency target for network connections between data centres in an AD
Presumably it would help ensure that everyone selecting us-east-1a in their base configs didn't actually all land in the same AZ.
Yeah this one drove me crazy
>VPC peering used to be annoying; now there are better options like Transit Gateway, VPC sharing between accounts, resource sharing between accounts, and Cloud WAN.
TGW is... twice as expensive as vpc peering?
VPC sharing is the sleeper here. You can do cross account networking all in the same VPC and skip all the expensive stuff.
as long as your VPCs aren't too big, yea.
Shared vpcs can get pretty big. Even if you approach the NAU limit you can use privatelink or TGW to have more large shared vpcs.
If you are exhausting an entire VPC I’d be pretty impressed!
Used to work at nflx, biiiiig headaches
And vpc sharing is free. Cost and architecture are tied.
More than twice as same AZ is free with peering. But if you're big enough you can get better deals on cost.
But unlike peering TGW traffic flows through an additional compute layer so it has additional cost.
> You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots.
As of when? According to internal support, this is still required as of 1.5 years ago.
I think there is some nuance needed here. If you ask support to partition your bucket then they will be a bit annoying if you ask for specific partition points and the first part of the prefix is not randomised. They tried to push me to refactor the bucket first to randomise the beginning of the prefix, but eventually they did it.
The auto partitioning is different. It can isolate hot prefixes on its own and can intelligently pick the partition points. Problem is the process is slow and you can be throttled for more than a day before it kicks in.
> but eventually they did it
They can do this with manual partitioning indeed. I've done it before, but it's not ideal because the auto partitioner will scale beyond almost anything AWS will give you with manual partitioning unless you have 24/7 workloads.
> you can be throttled for more than a day before it kicks in
I expect that this would depends on your use case. If you are dropping content you need to scale out to tons of readers, that is absolutely the case. If you are dropping tons of content with well distributed reads, then the auto partitioner is The Way.
He's not talking about the prefix, just the beginning of the object key.
The prefix is not separate from the object key. It's part of it. There's no randomization that needs to be done on either anymore.
and indeed the bucket is not separate from the object key. the API separates it logically "for humans" but it's all one big string
It would've been nice if each of those claims in the article also linked to either the relevant announcement or to the documentation. If I'm interested in any of these headline items, I'd like to learn more.
I don't believe AWS offers permalinks, so it would only help until they rolled over the next documentation release :-(
They actually used to have the upstream docs in GitHub, and that was super nice for giving permalinks but also building the docs locally in a non-pdf-single-file setup. Pour one out, I guess
Strictly off topic:
Everything you know is wrong.
Weird Al. https://www.youtube.com/watch?v=W8tRDv9fZ_c
Firesign Theatre. https://www.youtube.com/watch?v=dAcHfymgh4Y
Paid AWS support got a lot less capable on average during these two decades . :/
My recent interactions with them would probably have been better if they were an LLM.
They probably are an LLM and if they aren't, their higher management is pushing for them to be LLMs by 2027 at the latest.
I still get strong Hyderabad vibes from the copy/pasting plus the occasional original sentence.
Perhaps they trained the LLM using that data though.
(Small customer though: yearly AWS spend around 80k. Support is 10% of that.)
TAMs are super hit and miss. We’ve had great ones (hi Nick!) and not so great ones. ($7-10M/mo customer AWS spend, support is a complicated sliding scale % of that, gogo ES!). Non-ES at smaller customers has been universally useless, except at quota increases.
If you're pushing 10 mil per month through AWS, I pray you have some bitcoin miners hidden in that spend. The mind reels
Multiple large scale SAP S4/HANA workloads wasting electrons…
At this scale it may make sense to run your own mini data centers?
There’s lots of those too.
Every AWS update can potentially affect your SOC 2 or HIPAA compliance posture. I've seen companies fail audits because they assumed their security configurations were still current.
The cloud moves fast. Compliance processes need to keep up. Manual annual reviews aren't enough when your infrastructure is changing constantly.
This is also why we built automated compliance monitoring - because what worked last quarter might not work today.
Also S3 related: the bucket owner can now be configured as the object owner no matter where the object originated. In the past this was exceedingly painful if you wanted to allow one account contribute objects to a bucket in another account. You could do the initial contribution, but the contributor always owned the object, and you couldn't delegate access to a third account.
This article was a relief. I’m always a tiny bit worried Amazon will change some thing drastically and I’ll have to migrate. I’ve had an ec2 instance running since 2013. It requires effectively zero maintenance. So I am glad there were no surprises in this article. Thanks OP.
That. Was a decent investment of my time as a devops engineer. Right to the point. I learned things.
A "Catch me up" on AWS (and for that matter other large platforms) would be very useful for many folks.
Ideally it should be a stream of important updates that can be interactively filtered by time-range. For example, if I have not been actively consuming AWS updates firehose for last 18 months, I should be able to "summarize" that length of updates.
Why this is not already a feature of "What's New" section of AWS and other platforms -- I dont know. Waiting to be built -- either by OEM or by the Community.
I played a lot of DOTA2 in the past and I've often thought that big tech could learn something from Valve's patch notes. Especially in the context of process changes, stuff you should know, etc. Expecting folk to read a series of lengthy emails/blog posts to stay up to date is unrealistic.
I've had two people tell me in the last week that SQS doesn't support FIFO queues.
Would love an AWS equivalent to Cloud Run but the lambda changes are welcome nonetheless.
Isn't Fargate the AWS equivalent of Cloud Run?
It seems Cloud Run comes "batteries included" for a lot more things that one must provision separate AWS resources to achieve https://cloud.google.com/architecture/migrate-aws-lambda-to-...
I haven’t used AWS in the last 5 years. Is IPv6 still somewhat of an issue? I remember some services not supporting it at all and making it impossible to manage as a IPv6-only network.
Yeah, it’s still limited, and a few things still require at least a dual stack setup.
gotta milk those ipv4 investment
You'll be glad(sic) to know they have specialized hostnames for IPv6 control plane endpoints, because AAAA records scare the olds https://github.com/aws/aws-cli/blob/2.28.14/awscli/botocore/...
We enabled ipv6 for our APIs at work. Nothing broke immediately, but we've had a steady stream of unreachable host errors related to ipv6 since then.
Turns out there're many incorrect implementations of Happy Eyeballs that cancel the ipv4 connection attempts after the timeout, and then switch to trying the AAAA records and subsequently throwing a "Cannot reach host" error. For context, in Happy Eyeballs you're supposed to continue trying both network families in parallel.
This only impacts our customers who live far away from the region they're accessing, however, and there's usually a workaround - in Node you can force the network family to be v4 for instance
> because AAAA records scare the olds
No. They break existing customer expectations.
There are heaps of dualstack API endpoints https://docs.aws.amazon.com/general/latest/gr/rande.html#dua... if that's what the client wants.
The amazonaws.com domain endpoints did not introduce ipv6/AAAA directly is (mostly) due to access control. For better or worse there are a lot of "v4 centric" IAM statements, like aws:SourceIp, in identity/resource/bucket policies. Introducing a new v6 value is going to break all of those existing policies with either unexpected DENYs or, worse, ALLOWs. Thats a pretty poor customer experience to unexpectedly break your existing infrastructure or compromise your access control intentions.
AWS _could_ have audited every potential IAM policy and run a MASSIVE outreach campaign, but something as simple as increasing (opaque!) instance ID length was a multi year effort. And introducing backwards compatibility on a _per policy_ basis is its own infinite security & UX yak shaving exercise as well.
So thats why you have opt-in usage of v6/dualstack in the client/SDK/endpoint name.
This is super helpful. I would read a yearly summary like this.
I have a preempt-able workload for which I could use Spot instances or Savings Plans.
Does anyone have experience running Spot in 2025? If you were to start over, would you keep using Spot?
In terms of cost, from cheapest to most expensive:
1. Spot with autoscaling to adjust to demand and a savings plan that covers the ~75th percentile scale
2. On-demand with RIs (RIs will definitely die some day)
3. On-demand with savings-plans (More flexible but more expensive than RIs)
3. Spot
4. On-demand
I definitely recommend spot instances. If you're greenfielding a new service and you're not tied to AWS, some other providers have hilariously cheap spot markets - see http://spot.rackspace.com/. If you're using AWS, definitely auto-scaling spot with savings plans are the way to go. If you're using Kubernetes, the AWS Karpenter project (https://karpenter.sh/) has mechanisms for determining the cheapest spot price among a set of requirements.
Overall tho, in my experience, ec2 is always pretty far down the list of AWS costs. S3, RDS, Redshift, etc wind up being a bigger bill in almost all past-early-stage startups.
To "me, too" this, it's not like that AWS spot instance just go "poof," they do actually warn you (my recollection is 60s in advance of the TerminateInstance call), and so a resiliency plane on top of the workloads (such as the cited Kubernetes) can make that a decided "non-event". Shout out to the reverse uptime crew, a subset of Chaos Engineering
The spot termination warnings arrive 120 seconds before termination https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-ins...
I just saw Weird Al in concert, and one of my favorite songs of his is "Everything You Know is Wrong." This is the AWS version of that song! Nice work Corey!
I also saw that concert.
Oh yeah, we were in the same row!
Weird AL or Weird A.I.?
> Glacier restores are also no longer painfully slow.
Wouldn't this always depend on the length of the queue to access the robotic tape library? Once your tape is loaded it should move really quickly:
https://www.ibm.com/docs/en/ts4500-tape-library?topic=perfor...
> Once upon a time Glacier was its own service that had nothing to do with S3. If you look closely (hi, billing data!) you can see vestiges of how this used to be, before the S3 team absorbed it as a series of storage classes.
Your assumption holds if they still use tape. But this paragraph hints at it not being tape anymore. The eternal battle between tape versus drive backup takes another turn.
I am also assuming that Amazon intends for the Deep Archive tier to be a profitable offering. At $0.00099/gb-month, I don't see how it could be anything other than tape.
I wonder if it's where old S3 hard drives go to die? Presumably AWS have the world's single largest collection of used storage devices - if you RAID them up you can probably get reliable performance out of them for Glacier?
Not quite. Hardware with customer data or corp IP (eg any sort of storage or nvram) doesnt leave the "red zone"[1] without being destroyed. And reusing EOL hardware is a nightmare of failure rates and consistency issues. Its usually more cost effective to scrap the entire rack once its depreciated, or potentially at the 4-5 year mark at most. More revenue is generated by replacing the entire rack with new hardware that will make better use of the monthly recurring cost (MRC) of that rack position/power whips/etc.
[1] https://www.aboutamazon.com/news/aws/aws-data-center-inside
I still don't know if it's possible to make it profitable with old drives in this kind of arrangement, especially if we intend to hit their crazy durability figures. The cost of keeping drives spinning is low, but is double-digit margin % in this context. You can't leave drives unpowered in a warehouse for years on end and say you have 11+ nines of durability.
Unpowered in a warehouse is a huge latency problem.
For storage especially we now build enough redundancy into systems that we don't have to jump on every fault. That reduces the chance of human error when trying to address it, and pushing the hardware harder during recovery (resilvering, catching up in a distributed concensus system, etc).
When the entire box gets taken out of the rack due to hitting max faults, then you can piece out the machine and recycle parts that are still good.
You could in theory ship them all off to the backend of nowhere, but it seems that Glacier is all the places where AWS data centers are, so it's not that. But Glacier being durable storage, with a low expectation of data out versus data in, they could and probably are cutting the aggregate bandwidth to the bone.
How good do your power backups have to be to power a pure Glacier server room? Can you use much cheaper in-rack switches? Can you use old in-rack switches from the m5i era?
Also most of the use cases they mention involve linear reads, which has its own recipe book for optimization. Including caching just enough of each file on fast media to hide the slow lookup time for the rest of the stream.
Little's Law would absolutely kill you in any other context but we are linear write, orders of magnitude fewer reads here. You have hardware sitting around waiting for a request. "Orders of magnitude" is the space where interesting solutions can live.
You don’t raid old drives as it creates cascading failures because recovering from a failed drive adds major wear to other drives
Only if you have low redundancy. RAIDZ is better about this isn’t it? And Backblaze goes a lot farther. They just decommission the rack when it hits the limit for failed disks, and the files on the cluster are stored on m of n racks, so adding a rack and “resilvering” doesn’t even require scanning the entire cluster, just m/n of it.
This is less of a concern with RAID 6, and especially in Glacier's use case where reading any piece of data happens seldom, I'd expect it to be fine.
My understanding is some AWS products (e.g. RDS) need very fast disks with lots of IOPS. To get the IOPS, though, you have to buy +++X TB sized SSDs, far more storage space than RDS actually needs. This doesn't fully utilize the underlying hardware, you are left with lots of remaining storage space but no IOPS. It's perfect for Glacier.
The disks for Glacier cost $0 because you already have them.
Since ~2014 or so the constraint on all HDD based storage has been IOPs/throughput/queue time. Shortly after that we started seeing "minimum" device sizes that were so large as to be challenging to productively use their total capacity. Glacier type retrieval is also nice in that you have much more room for "best effort" scheduling and queuing compared to "real time" request like S3:PutObject.
Last I was aware flash/nvme storage didnt have quite the same problem, due to orers of magnitude improved access times and parallelism. But you can combine the two in a kind of distributed reimplementation of access tiering (behind a single consistent API or block interface).
There’s a really old trick with HDDs where you buy a big disc and then allocate less than half of it. There’s more throughput on the first half of the disk, more tracks per cylinder so fewer seeks, and never having to read half the disk reduces the worst case seek time. All increase IOPs.
But then what do you do with the other half of the disk? If you access it when the machine isn’t dormant you lose most of these benefits.
For deep storage you have two problems. Time to access the files, and resources to locate the files. In a distributed file store there’s the potential for chatty access or large memory footprints for directory structures. You might need an elaborate system to locate file 54325 if you’re doing some consistent hashing thing, but the customer has no clue what 54325 is. They want the birthday party video. So they still need a directory structure even if you can avoid it.
http://www.patentbuddy.com/Patent/20140047261
Is tape even cost competitive anymore? The market would be tiny.
One way to know is to see if new tape products exist, indicating ongoing development. As of May, 2025, LTO-10 is available, offering 30TB/75TB (raw/compressed) storage per cartridge. Street price is a bit under $300 each. Two manufacturers are extant: Fujifilm and IBM.
It's gone in cycles for as long as I recall and older devs around 2010 said it had been going on for as long as they could recall.
> You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots.
Not strictly true.
I should have been more clear. You still need to partition, but randomizing the prefixes hasn't been needed since 2018: https://web.archive.org/web/20240227073321/https://aws.amazo...
Generally speaking this isn't something Amazon S3 customers need to worry about - as others have said, S3 will automatically scale index performance over time based on load. The challenge primarily comes when customers need large bursts of requests within a namespace that hasn't had a chance to scale - that's when balancing your workload over randomized prefixes is helpful.
Please see the documentation: https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimi...
This 2024 re:Invent session "Optimizing storage performance with Amazon S3 (STG328)" which goes very deep on the subject: https://www.youtube.com/watch?v=2DSVjJTRsz8
And this blog that discusses Iceberg's new base-2 hash file layout which helps optimize request scaling performance of large-scale Iceberg workloads running on S3: https://aws.amazon.com/blogs/storage/how-amazon-ads-uses-ice...
This 2024 re:Invent session says exactly the opposite:
"If you want to partition your data even better, you can introduce some randomness in your key names": https://youtu.be/2DSVjJTRsz8?t=2206
FWIW The optimal way we were told was to partition our data was to do this: 010111/some/file.jpg.
Where `010111/` is a random binary string which will please both the automatic partitioning (503s => partition) and manual partitioning you could ask AWS. Please as in the cardinality of partitions grows slower at each characters vs prefixes like `az9trm/`.
We were told that the later version makes manual partitioning a challenge because as soon as you reach two characters you've already created 36x36 partitions (1,296).
The issue with that: your keys are no more meaningful if you're relying on S3 to have "folders" by tenants for example (customer1/..).
Elaborate.
The whole auto-balancing thing isn't instant. If you have a burst of writes with the same key prefix, you'll get throttled.
Not the OP but I’ve had AWS-staff recommend different prefixes even as recently as last year.
If key prefixes don’t matter much any more, then it’s a very recent change that I’ve missed.
Might just be that the AWS staff wasn't up to date on this
I have had the same experience within the last 18 months. The storage team came back to me and asked me to spread my ultra high throughput write workload across 52 (A-Za-z) prefixes and then they pre-partitioned the bucket for me.
S3 will automatically do this over time now, but I think there are/were edge cases still. I definitely hit one and experienced throttling at peak load until we made the change.
That’s sounds like the problem we were having. Lots of writes to a prefix over a short period of time and then low activity to it after about 2 weeks.
By the way, that happens quite frequently. I regularly ask them about new AWS technologies or recent changes, and most of the time they are not aware. They usually say they will call back later after doing some research.
That’s possible but they did consult with the storage team prior to our consultation.
But I don’t know what conversations did or did not happen behind the scenes.
That's right, same for me as of only a few months ago.
API gateway timeout increase has been nice.
It was always there but it required much more activity to get it done (document your use case & traffic levels and then work with your TAM to get the limit changed).
I don't see that in this post.
I just started working with a vendor who has a service behind API Gateway. It is a bit slow(!) and times out at 30 seconds. I've since modified my requests to chunk subsets of the whole dataset, to keep things under the timeout.
Has this changed? Is 30 secs the new or the old timeout?
https://aws.amazon.com/about-aws/whats-new/2024/06/amazon-ap...
> As of very recently, you can also force EC2 instances to stop or terminate without waiting for a clean shutdown or a ridiculous timeout
Not true for GPU instances, they're stuck 5 minutes in a stopping state because they run some GPU health checks.
For our VM solution, we get around this by hot staging VM's. As soon as one customer stops theirs, we reset everything and start it up again. To the end user, our compute seems to be instantly available. Unless of course, we run out.
CloudFront also has 1TB of free data transfer a month under the forever-free perks.
> DynamoDB You can now have empty fields
When this was changed? I think this is still an issue, I've had some such errors quite recently.
18th May 2020
> DynamoDB now supports empty values for non-key String and Binary attributes in DynamoDB tables. Empty value support gives you greater flexibility to use attributes for a broader set of use cases without having to transform such attributes before sending them to DynamoDB. List, Map, and Set data types also support empty String and Binary values.
https://docs.aws.amazon.com/amazondynamodb/latest/developerg...
Lots of this seems to boil down to: AWS shipped something that was barely usable, but then iterated.
That's a reasonable approach, but the fact this post exists shows that this practice is a reputational risk. By all means do this if you think it's the right thing to do, but be aware that first impressions matter and will stick for a long time.
Self-hosting is the future. If you have small-to-modetate traffic it's extremely easy to do and very cheap.
Then you can self-cloud. Several startips are in this space. It gets you the best of both worlds: scaling, freedom, cost-control.
And no marketing jargon that you need to learn, and then unlearn!