> There are two types of people: those who have suffered a catastrophic loss of data, and those who will.
This is an empirical claim, but is it grounded in reality?
99% of people don't backup, and that's probably the right choice because the risk is low, they can't meaningfully improve their restoration rate themselves and they don't care enough about their data to classify its loss as catastrophic.
When a person hears someone talk about the importance of backups, but also haven't heard friends/family suffer this date, they will rightfully ignore this warning.
Or do we all follow the best practices as it relates to backups, exercise, sleep, nutrition, accounting, house maintenance, ... Etc.
I do not. Backups aren't near the top of that list.
This stuff just isn't very important to most people and that's okay.
There are four times in my life I have suffered regrettable data loss incidents.
The first was when the telephone pole outside our house was struck directly by lightning. Not only was it the loudest thing I have ever heard, the current surged through the telephone line, into the internal fax modem, and fries everything within its vicinity. I was 10. I did have backuos, but only only floppy and they didn't cover everything.
The second was storing data in OneDrive - a change to their terms surrounding "lifetime" unlikely noted storage, combined with a client that was unusably slow to download and a deadline for data retrieval meant that I lost most of my files.
The third was SD card failure in digital camera on holiday, the controller chip died catastrophically, leaving the card completely unrecognised. It was a brand new Sony 128GB card, manufactured by Toshiba, and it seemed to be a common issue. I now shoot to two cards simultaneously.
And the fourth time was ... Performing a backup. An errant script deleted the source content, but I'd also deleted the existing backup to free up space for the new backup. I've been weary of using rewritable media for some time now as a consequence, but I think backups themselves are high risk activities.
>I did have backuos, but only only floppy and they didn't cover everything.
This made me remember the time during the 90s when I lost all my BASIC programs due to a hard disk crash, and seeing me sitting sad, my mother handed me a bunch of floppies and asked me to see if they contain something useful.
And turned out that it had all the programs! I took the backups and forgot about that. Apparently that is one of the hard things with backups. You need to track them...
> a change to their terms surrounding "lifetime" unlikely noted storage
I had a similar experience with a Chinese cloud storage provider. They didn't even give a convenient way to export the data. And the client throttled download to like 100KB/s. I luckily was able to parallel download by running the client on several VMs...
> An errant script deleted the source content, but I'd also deleted the existing backup to free up space for the new backup.
Sounds like the backup workflow is fundamentally flawed... But I also have the fear that the backup program with root privileges can go off the rails... So I keep my backup job as simple as just running restic with some systemd restrictions.
I have a weird setup lol.
I use a shared photos library with my partner (so that we have all family photos together) and have a MacBook Air that has Photos being download in raw quality and backups then via Time Machine on a Synology NAS.
Plus, Synology photo sync on my iPhone (backing up on my NAS separately from Time Machine) AND Google Photos on my partner’s (syncing on Google Photos).
Yes it’s horrible, just never had time to clean I up properly but I prefer to have them in multiple cloud locations and locally for now than optimise for having a clean setup.
>I've been weary of using rewritable media for some time now as a consequence
I briefly considered using bluray disks as a backup for my photos and other critical docs. But getting a decent bluray burner seems not so easy these days with most production winding down. Next best thing looks like the "object lock" feature on object store services that prevents deleting objects for a certain time.
I pretty much instantly wrote off physical media because I know I won't reliably move it to a separate location. If I'm going to have backups, I want ones that are not within fire/flood range of my computer.
In terms of preventing "oops" moments, I'm mainly relying on software (restic) for that, where I trust that (A) backups always append data rather than replacing and (B) it's logic works for marking which data to purge based on rules is accurate. [0]
I kinda overdid it with a tape library (40 tapes capacity, LTO-9), but I'm using it for my company's AI model backups. _That_ setup was about $12k.
I had a smaller setup before, with a simple external LTO-5 drive. Used drives are now are about $300, and you can probably find them cheaper. And LTO-5 is the minimum realistic version, it's the first one that supports LTFS and it has reasonable tape capacity.
There's plenty of backup services and cloud solutions. The article mentions Borg and Restic, others like Backblaze are more consumer friendly. Which needs do you have that aren't met by any of these tools?
I also have the same fuji. Digital cameras are still in the stone ages. I think Tony Northrup covered it well. Nothing has changed since digital cameras came to market, the workflow is still the same, copy files from a card to PC. They aren't connected to the internet, they don't have automatic syncing to cloud for backups or sharing, fuji doesn't even have a gps chip built in to store the location (only works if connected to a smartphone), they can't be locked via fingerprint scanner. It's all tech from 25 years ago.
In some senses it's good. That feature list just says "use a smartphone then". No thank you. It's everything that a camera should not be.
GPS is useful, but not wifi. I don't need my camera becoming another IOT always online telemetry device. They're one of the few electronic devices built to last. Imagine needing fingerprint unlock on a camera, and for what? Prevent others from viewing your photos? Changing your settings? Also copy from card to pc is the best possible workflow. Imagine cameras like smartphone with no expandable storage.
Stop giving these people ideas! I am now very afraid they will listen to this guy since he actually has a follower base.
You would often not have wifi available, and you wouldn't want to have to type in wifi passwords to your camera all the time. But your phone will almost always be nearby and persistent bluetooth is easy.
I really like ZFS snapshots with offsite pull-mode sync using
Jim Salter's sanoid/syncoid [1]. ZFS is the base for all OS/filesystems on top of it. If you have a good system for organizing ZFS datasets, and separating ephemeral from persistent data (e.g. [2]), then this is 90% of the backup requirements already fullfilled.
A friend of mine used to work at Veritas[0] making enterprise data retention solutions. When I spoke about their product as being "making backups", he corrected me by saying:
We are not in the backup business. We are in the restoration
business.
No. But, well, yes: if you use BE - you will need a restore pretty soon.
I'm the person who actually experienced BE walking around the file server and deleting files. Helpful suuport person had done the needful and said what this is not an issue and it wouldn't be reported as a bug.
I’m setting up 3-2-1-ish backups for my infra of 3 hosts, and definitely leaning towards Restic + Backrest.
All my hosts run the same CoreOS setup (https://github.com/ebrahim37/infra-template), where container volumes are placed in one central volumes/ folder and that is the only thing I have to backup.
I plan to implement it like this:
vps1:
- restic container with custom sh entrypoint that will backup volumes/ to homelab every 24 hours
homelab:
- backrest container, to back up volumes/, do prune/check, replicate repo to offsite
- rest-server container, will store backups from vps1, homelab, offsite
offsite:
- restic container, backs up volumes/ to homelab every 24 hours
- rest-server container, store copy of backups from homelab
Only caveat is backing up databases, will either have to do: stop container, backup volume/database-data, start container; or use pg dump etc.
The deduplication is nice, you can have a snapshot for each week of the past year without crazy storage cost
If your system has a way to take consistent snapshots in the filesystem (btrfs, zfs) or volume manager (lvm2, perhaps with a bit of filesystem support to obtain fs consistency), that can be used to avoid database downtime, if desired.
The issue is companies love to fire QA people who should be the ones testing these.
Microsoft says we don’t need no SDETs, our products can ship full of issues.
What are the hostages, I mean customers going to do about it?
I have no sympathy for any major corp who suffers data loss. Now individuals who lost grandmas wedding photos, that sucks.
But my recovery strategy for that is simple. After scanning I emailed it out to one cousin who emailed them out to a few more. Plus I have a cloud backup ( I have no idea where the the original files are).
My setup: rsync to Synology NAS, snapshotting there, sync to remote Synology NAS over VPN.
(The other NAS is at my parent's and they have similar setup syncing to me)
The best part: it is just files, not some proprietary archive. I have high confidence in this. I can restore with the most rudimentary tools.
This setup has worked for me well on linux, windows and now mac. I migrated to new computer&os by restoring the backup mostly.
"encrypted, chunk-level deduplicated, GFS-rotated, point-in-time archived, cloud, 3-2-1 backup solution" is now my newest password, no commas. (Don't tell anyone!)
On my home PC I use restic, and (shockingly) I run it manually (via a script). I have a terminal banner that nags me when backup hasn't run in 24 hours, or displays spanish verb tables when it has (so I will notice if the banner doesn't show). I've been burned in the past losing a backup cron, and dead man's snitch seemed overkill for my needs here. And I've also forgotten what backup system was in place on a prior install, and it took me way longer than I liked figuring out how it was setup when I needed to do a restore.
Obviously for production systems automation & monitoring is the way. I kind of like this setup on my own machine though.
> have this happen every morning at 5AM by creating a temporary text file containing this line:
0 5 * * * rsync -vax --delete --ignore-errors / /Volumes/Backup/
This leaves you needlessly exposed to this failure mode: Commit some "braino" that wipes out some recent work. Go to bed without noticing the loss. 5am rsync run wipes your backup as well.
The fix is to back up to not just one but to a rotation of images. Use --link-dest to reduce storage overhead to the size of the directory hierarchy only.
If you don't control your data size, it will end up controlling you and your backup choices, which will eventually lead to many avoidable, disastrous outcomes. Somehow, this gets missed in data storage and backup planning.
A bit of a tangent, but the linked rsnapshot tool is about 7k lines of perl code. The COPYING file is 20 years old and it's still maintained, the last change was 6 months ago. I'm feeling really old.
For my Oracle Linux systems, I rely on the vendor ReaR RPM package for the baseline backup. This is similar to the "Ignite" utility that I used in my HP-UX days.
This is two-fold useful, first as it enables boot from backup media for total recovery, second in that it creates a backup.tar.gz that holds everything that was not explitictly ignored in the /etc/rear/local.conf file.
I'm running Oracle databases on these systems (and the local.conf is configured to ignore the datafile directories). I have standby databases (not Dataguard in that I don't have online redolog replication) that allow me to recover them to the primary in a disaster, with some loss of committed transactions.
I finally have rsync configured for some scratch temporary files.
I have dallied with btrfs snapshot replication for home directories (in a loopback mount). My vendor support (via a CSI number) has been on and off, so I don't use btrfs in areas that we need it most.
My replacements do not like the complexity (and I am retiring).
One of our customers decided to go for off site backups with us - they had been a customer for 15 odd years for other services. Backups are not our core business, its one of the services and I got around to setting them up sooner rather than later.
Two weeks later they had a fire in one of their huge printing machines which caused quite a lot of direct damage due to heat and rendered a lot of equipment broken due to smoke/fumes. It was also, as it turns out, a week before the first COVID 19 lockdown in the UK.
Their backups landed on a XFS file system with reflinks enabled. I cloned their repo and fired up NFS with their VMs running on our gear. It took me another hour to spin up another OpenVPN server (CA etc) for them to use and sort out a few other details (coffee doesn't brew itself).
They ran via VPN out of our data centre for most of the pandemic.
That was an utter triumph but I also have some rather less triumphant stories about backups and lack thereof. Let's skip over those 8)
I'm polishing my backup system right now, and found that `tar | zstd | gpg` plus a small vibe-coded wrapper is pretty much all I need.
GNU tar has its own incremental index via `--listed-incremental=FILE`. Unlike Borg or Restic, which have their own more complicated repository formats, this leaves me with just one additional file (the `.snar`) alongside a dumb, portable full-disk tar archive.
The nice part is that, unlike repositories that require both read/write access patterns, tar can compute deltas using only the small `.snar` file, while the main `full-disk.tar` can remain buried in write-only Glacier storage. This makes it a much better fit for Glacier's write-once model and 180-day minimum retention.
My current plan is to upload a full-disk tar to an external HDD + Glacier on a weekly cadence, and the `.snar`-based deltas daily. That gives me a pretty simple cloud backup solution for a few bucks a month (after burning through $100+ of free credits).
For me, the ability to mount a backup and look at a particular file proved to be important. Also, I restored from my backups four times; two of them was moving between machines, pretty quickly.
You can exercise it against regular, non-Glacier S3. Same flow; the only difference is that you skip one attribute on upload and one request/wait step on download.
Something is better than nothing. I accidentally spilled liquid on my macbook just the other day. It's still in the shop. Luckily, I periodically clone my drive with CCC onto a single external SSD. Don't wait to set up the perfect backup solution, just have a copy of your data in more than one place, that's a good start imho.
I have a cron script that rsync my entire home directory every day with a remote server. Since rsync doesn't remove remote files, I re-create a backup from scratch the first day of each month and store a copy of the previous one.
Using old standardized open source tools means it survives system updates without having to fix anything for years. I got my laptop stolen/lost 3 times over the past 15 years, and I have always been able to restore everything the next day on a new laptop, in the time it takes to transfer the files over the network.
I haven't given much thought, I'm sure there is a realistic scenario where this strategy would fails but I haven't found it yet.
ZFS snapshots + restic backups to backblaze for my homeserver. My secret sauce is a healtchecks.io instance that blows up my phone if ZFS scrubs throw any errors, when local snapshots fail, or when restic checks or backups fail.
>Motivated by our success in developing this solution, we try to use it to backup the homelab with its 10 Docker containers. But later we find out from logs on the individual machines that backups are failing. The reason being that many Docker containers like to create root-owned files, and if you’re not careful you can create a cronjob running as the default user.
Yeah it feels it can be risky to "make the backup also dump the databases, and give it full filesystem permissions on our Docker volumes", depending on what binaries get the root privileges.
What I do is having a minimal systemd timer under root that only calls restic for backing up files (with some additional systemd seucurity restrictions). App-dependent backup logics, like dumping databases, are done by user/container-level cronjobs separately.
On the topic of backups, 7zip recently changed its handling of the FILE_ATTRIBUTE_ARCHIVE bit on unzipped files which caused an issue with some old software I was using. This is from the old DOS days with the BACKUP/RESTORE commands - the bit was set if any backup was made of the file. It's comically naive because it only assumes a single backup and this bit was forgotten to the sands of time until it was disturbed by some meddling kids!
As an average-techie person, the only piece of data that i truly practice 3-2-1 is photos and videos. 1 on my phone, 1 on my small server running Immich, and 1 is on Google Photo(sadly, the quality is loss, but i can live with that).
Most of other data i only have 2 backup, usually 1 at my local machine(code) and 1 online(github). Yeah it's not ideal, but another question to ask yourself, is it truly worth it. I trust the engineers at MS doing much better job at backing up their data than me.
I tried to set up something similar, photos and videos on local NAS + cloud storage. Also with data loss, eg. I don't sync any large RAW camera files to cloud. I clone my local machine with CCC onto an external drive, that way it should be easy to just restore the last state onto a new machine. So I think it's archival storage for media vs snapshots of your working setup so you can restore quickly.
Try running a business on that basis - where do the restores come from? 8)
We charge our customers for backup storage by volume and that includes an annual demonstration of recovery. We fire up some or all of their systems on our gear in isolation and show that they are reasonably functional.
Your teacher's comment is riffing on the well trod lines of: "heights don't kill people, it's depths that kill people", which is all about perspective. This is is not related to "guns don't kill people ..." which is about agency.
It is true that very few people proactively care about backups. It’s like flossing. People know they’re supposed to do it, but in the moment it feels like a waste of time and energy, when there are more important things to do.
When data is lost, or decay takes over, the value can skyrocket. Depending on what is lost, someone may be willing to pay 100x when they would have spent on backup in order to get their data back, this is why data recovery services can basically charge whatever they want.
My assumption is the Venn diagram between people who pass the marshmallow test, and those who proactively backup their systems, has a lot of overlap.
Running a business on that basis would look like the white(ish)-hat version of ransomware. "Oh, your data is gone? How much is it really worth to you?"
It is a special sensation when you go to your hand rolled backup "solution" to make some simple recoveries and start finding problems. It is a complex mix of not dread but disgust, sick to your stomach, self-anger, and loss. I can only image what that's like with like all of your children's baby pictures being the lost material. Borg and restic are fantastic, it still seems like some sort of dummy proof mode might be desirable, all the features in the world and they have some some reasonably sane policy or something that includes verifications.
On the flip side of that though, really and truly, I've restored a laptop from a timemachine backup and it came back damn nearly identical and that was magical.
I needed to temporarily move off 20TB from my NAS and bought a Hetzner storage box.
I wanted to encrypt the data before sending it so i went with the borg ssh mount. Long story short, borg failed a couple of times due to https://github.com/borgbackup/borg/issues/7672, i resumed it per documentation and got 2gb of silently corrupted data. There was no way to do data integrity checks due to broken pipes and there was no option to resume the job from the previous point either. I couldn't afford to go with the s3, backblaze wasn't an option due to placing trust into their proprietary client doing the encryption and i got majorly screwed in the end.
concerning backups, when it comes to disaster recovery you need a strategy for restoring as well, sometimes it's not only the data you would need but the systems as well
When it comes to software, it's almost never simple.
What people think of as simple is rarely so.
For example:
Requirement:
"Just copy a file from folder A to folder B."
Minimal translation:
"Implement the capability to copy any file, in any format and of any size from SharePoint, located at a configurable path, with appropriate authentication and access control checks, then stream it in chunks to a different, configurable path inside an S3 bucket, also with appropriate authentication and access controls in place. Ensure that any disruption in either service which may occur while the file is streaming can be recovered from at the point it failed instead of having to restart from the beginning. Ensure that the retry mechanism is built-in and that the retry window is configurable; if the file cannot be copied within the specific time window, then an error should be sent via email to a configured email address. Ensure that the entire transfer is encrypted in transit... If the file happens to be a folder, then you must copy across all of its contents recursively up to a certain configured MAX_DEPTH to avoid DoS and ensuring that the system does not get caught in an infinite loop due to symlinks pointing to a parent folder... In this case, send an email to the configured address... Etc... Etc..."
And the thing is; if you tell AI "Just copy a file from folder A to folder B." - It will not meet your 'basic' needs because even if it does a great job at filling the gaps in your requirements, it will still take shortcuts. In order for an AI to avoid taking shortcuts, it would have to make you fill out a questionnaire and make you sign up for and configure services; it would not be a pleasant user experience. The user experience cannot be pleasant, because the AI cannot read your mind and it cannot know your intent.
Here's a thing: my intent and requirements actually are:
"Just copy a file from folder A to folder B."
All that access control and authentication and symlink nonsense? That's a self-inflicted problem that exists only in enterprise, and shouldn't be assumed - much less created - for regular users.
> I don't want backups. I want synced duplicated data in geographically separated by shared credentials.
If you accidentally delete something, how long does it take for the delete to propagate to all of your geographically separated datacenters? You need some sort of point-in-time snapshots to be able to recover from accidents and silent corruption.
that's not really the same thing, what you are describing is redundancy so failure in one (or more) place doesn't bring the system down. Backups are about preventing data loss. Data loss can happen in many different ways, not just a system crashing and burning. It can be deliberately/accidently done such that it would be sync'd everywhere.
Accepts stuff from pipe too so you can just pipe mysqldump or pg_dumpall without intermediate files
Very decent options for checking repo integrity, personally (well, at work too, we use it on few hundred user machines and servers) I also added "roll a dice for backup and try to restore it" test script to make sure it is working
Decent deduplication too so "store year's worth of weekly snapshot" is very sensible strategy once you exclude the "constantly changing and useless" (caches etc.) files out of it
can mount FUSE directory with all backups on sensible OSes.
https://kopia.io/ does the same +GUI but a bit worse on CLI front (it insists on keeping local config which makes scripting a bit more involved, but not much)
IMHO, the easiest way to perform a restore test is to use your production backups to refresh a lesser environment on a weekly cadence. Naturally this has to be appropriate to the type of data you're restoring, e.g., our E2E (end to end) testing environment has been built to the same risk tolerances of production so it's suitable for production data and the PII (Personally Identifiable Information) it hosts. If this isn't in your own risk tolerances, you can still perform the same test but ensure the data from prod is anonymised or alternatively destroyed, and perform a second restore after production restore with the dataset that preceeded it that has no PII data or similar.
With this the E2E tests confirm the backups are expected and you can tick a box that when shit does eventually hit the fan, data corruption isn't a problem. Normal caveats apply, you must make sure your backups are immutable so they cannot be modified or changed _after_ the tests have been ran.
I was using Restic + Backrest as well for my personal Windows machines and it was great. Then I found out that Backrest stores the vault password (among other things) in a simple plaintext file on your local hard drive. Not sure if this applies to all platforms but for Windows users that is a huge security hole in today's environment.
importantly, does your die have a side that corresponds to a multi-continental cataclysmic natural disaster of magnitude only known unknown to pre-recorded-history
> So our first principle is to have a backup, i.e. a copy of your files someplace else. So far so good.
So who are we? A household with a localhost administrator?[0]
Then you are surely doesn't need encrypted, chunk-level corporate level bla-bla-bla.
You need a Syncthing copy to some other device not at home - and an additional backup procedure to maintain the history and protect against PEBKAC errors - which can run on your local Syncthing copy or/and the other one.
And no, nobody needs your 555GB of RAWs of sunsets/flowers/precious_family_moments you shot - not even you.[1]
[0] well considering the tone of TFA and most of the comments here
[1] that's Instagram/Google Photos/iCloud for nowadays, totally with "Remember this day N years ago?"
> There are two types of people: those who have suffered a catastrophic loss of data, and those who will.
This is an empirical claim, but is it grounded in reality?
99% of people don't backup, and that's probably the right choice because the risk is low, they can't meaningfully improve their restoration rate themselves and they don't care enough about their data to classify its loss as catastrophic.
When a person hears someone talk about the importance of backups, but also haven't heard friends/family suffer this date, they will rightfully ignore this warning.
Or do we all follow the best practices as it relates to backups, exercise, sleep, nutrition, accounting, house maintenance, ... Etc.
I do not. Backups aren't near the top of that list.
This stuff just isn't very important to most people and that's okay.
There are four times in my life I have suffered regrettable data loss incidents.
The first was when the telephone pole outside our house was struck directly by lightning. Not only was it the loudest thing I have ever heard, the current surged through the telephone line, into the internal fax modem, and fries everything within its vicinity. I was 10. I did have backuos, but only only floppy and they didn't cover everything.
The second was storing data in OneDrive - a change to their terms surrounding "lifetime" unlikely noted storage, combined with a client that was unusably slow to download and a deadline for data retrieval meant that I lost most of my files.
The third was SD card failure in digital camera on holiday, the controller chip died catastrophically, leaving the card completely unrecognised. It was a brand new Sony 128GB card, manufactured by Toshiba, and it seemed to be a common issue. I now shoot to two cards simultaneously.
And the fourth time was ... Performing a backup. An errant script deleted the source content, but I'd also deleted the existing backup to free up space for the new backup. I've been weary of using rewritable media for some time now as a consequence, but I think backups themselves are high risk activities.
>I did have backuos, but only only floppy and they didn't cover everything.
This made me remember the time during the 90s when I lost all my BASIC programs due to a hard disk crash, and seeing me sitting sad, my mother handed me a bunch of floppies and asked me to see if they contain something useful.
And turned out that it had all the programs! I took the backups and forgot about that. Apparently that is one of the hard things with backups. You need to track them...
> a change to their terms surrounding "lifetime" unlikely noted storage
I had a similar experience with a Chinese cloud storage provider. They didn't even give a convenient way to export the data. And the client throttled download to like 100KB/s. I luckily was able to parallel download by running the client on several VMs...
> An errant script deleted the source content, but I'd also deleted the existing backup to free up space for the new backup.
Sounds like the backup workflow is fundamentally flawed... But I also have the fear that the backup program with root privileges can go off the rails... So I keep my backup job as simple as just running restic with some systemd restrictions.
OneDrive...
So much fail in one single product.
It is made by Microsoft, the experts in this domain.
Since we’re talking about it: anyone backing up their iCloud photos? If so, how?
From iPhone into local machine through LocalSend or AirDrop, then your usual favorite backup pipeline.
I have a weird setup lol. I use a shared photos library with my partner (so that we have all family photos together) and have a MacBook Air that has Photos being download in raw quality and backups then via Time Machine on a Synology NAS. Plus, Synology photo sync on my iPhone (backing up on my NAS separately from Time Machine) AND Google Photos on my partner’s (syncing on Google Photos). Yes it’s horrible, just never had time to clean I up properly but I prefer to have them in multiple cloud locations and locally for now than optimise for having a clean setup.
Arq.
https://www.arqbackup.com/
>I've been weary of using rewritable media for some time now as a consequence
I briefly considered using bluray disks as a backup for my photos and other critical docs. But getting a decent bluray burner seems not so easy these days with most production winding down. Next best thing looks like the "object lock" feature on object store services that prevents deleting objects for a certain time.
I pretty much instantly wrote off physical media because I know I won't reliably move it to a separate location. If I'm going to have backups, I want ones that are not within fire/flood range of my computer.
In terms of preventing "oops" moments, I'm mainly relying on software (restic) for that, where I trust that (A) backups always append data rather than replacing and (B) it's logic works for marking which data to purge based on rules is accurate. [0]
[0] https://restic.readthedocs.io/en/stable/060_forget.html#remo...
I bought a tape library for my backups, but that's probably an overkill.
On the other hand, LTO-5 drives are now pretty affordable. And each tape cartridge holds around 2Tb of data for about $20.
Wait really ? That's a huge change from last time I looked. How much did this setup cost you?
I kinda overdid it with a tape library (40 tapes capacity, LTO-9), but I'm using it for my company's AI model backups. _That_ setup was about $12k.
I had a smaller setup before, with a simple external LTO-5 drive. Used drives are now are about $300, and you can probably find them cheaper. And LTO-5 is the minimum realistic version, it's the first one that supports LTFS and it has reasonable tape capacity.
> Next best thing looks like the "object lock" feature on object store services that prevents deleting objects for a certain time.
Lol, no. "As of today we are closed. Goodbye."
Especially considering what giving even a two weeks now considered "generous".
For digital media we really need a good stable archival storage solution. Nothing on the market right now for ordinary people.
There's plenty of backup services and cloud solutions. The article mentions Borg and Restic, others like Backblaze are more consumer friendly. Which needs do you have that aren't met by any of these tools?
>I now shoot 2 cards simultaneously
A very lovely change over the last 8 years or so lol. I came up in film during the DSLR revolution. 5D2’s/7D’s/Rebels (i series) years.
8bit 420 nasty aliasing recording on single cards and praying baby. Magic lantern booted on those same SD’s!
Mainstream DSLRs with dual card slots came out 22 years ago.
Sadly after 22 years, it's still considered a "high-end" feature. Even on $1600 Fujifilm x100vi, there is still single SD slot.
I also have the same fuji. Digital cameras are still in the stone ages. I think Tony Northrup covered it well. Nothing has changed since digital cameras came to market, the workflow is still the same, copy files from a card to PC. They aren't connected to the internet, they don't have automatic syncing to cloud for backups or sharing, fuji doesn't even have a gps chip built in to store the location (only works if connected to a smartphone), they can't be locked via fingerprint scanner. It's all tech from 25 years ago.
In some senses it's good. That feature list just says "use a smartphone then". No thank you. It's everything that a camera should not be.
GPS is useful, but not wifi. I don't need my camera becoming another IOT always online telemetry device. They're one of the few electronic devices built to last. Imagine needing fingerprint unlock on a camera, and for what? Prevent others from viewing your photos? Changing your settings? Also copy from card to pc is the best possible workflow. Imagine cameras like smartphone with no expandable storage.
Stop giving these people ideas! I am now very afraid they will listen to this guy since he actually has a follower base.
Windows OS is also stone age. There is no simple way to list all user data, or user edited system data for simple backup in readable format.
Btw X100F has Wi-Fi and automated file transfer to PC. But it is faster to transfer it via SD card.
I would hope they have some feature for automatic backup to your phone over bluetooth or such today.
They do transfer, but it's not automatic.
My Nikon ZF transfers to my phone through Nikon's SnapBridge app automatically.
Oh, that's cool, kudos to Nikon, could be a good reason to switch over from Fuji.
WiFi is a little faster I guess? In the primitive days people used the EyeFi.
You would often not have wifi available, and you wouldn't want to have to type in wifi passwords to your camera all the time. But your phone will almost always be nearby and persistent bluetooth is easy.
Wifi between your camera and the phone. Or your camera and a laptop in your bag or whatever...
I really like ZFS snapshots with offsite pull-mode sync using Jim Salter's sanoid/syncoid [1]. ZFS is the base for all OS/filesystems on top of it. If you have a good system for organizing ZFS datasets, and separating ephemeral from persistent data (e.g. [2]), then this is 90% of the backup requirements already fullfilled.
A friend of mine used to work at Veritas[0] making enterprise data retention solutions. When I spoke about their product as being "making backups", he corrected me by saying:
0 - https://en.wikipedia.org/wiki/Backup_ExecBackups are boring. Restores, on the other hand, are often exciting.
Backups can be automated. Restores are a manual, heightened-attention, deep-focus process.
Correct. In data protection, backup is not the feature. Restore is the feature.
> https://en.wikipedia.org/wiki/Backup_Exec
No. But, well, yes: if you use BE - you will need a restore pretty soon.
I'm the person who actually experienced BE walking around the file server and deleting files. Helpful suuport person had done the needful and said what this is not an issue and it wouldn't be reported as a bug.
"This is the business we've chosen."
I’m setting up 3-2-1-ish backups for my infra of 3 hosts, and definitely leaning towards Restic + Backrest.
All my hosts run the same CoreOS setup (https://github.com/ebrahim37/infra-template), where container volumes are placed in one central volumes/ folder and that is the only thing I have to backup.
I plan to implement it like this:
Only caveat is backing up databases, will either have to do: stop container, backup volume/database-data, start container; or use pg dump etc.The deduplication is nice, you can have a snapshot for each week of the past year without crazy storage cost
If your system has a way to take consistent snapshots in the filesystem (btrfs, zfs) or volume manager (lvm2, perhaps with a bit of filesystem support to obtain fs consistency), that can be used to avoid database downtime, if desired.
https://www.postgresql.org/docs/current/backup-file.html
Huge fan of borg + vorta for backups and time Maschine like „Jump back in time“. Wrote about it years ago: https://simon-frey.com/blog/borgvorta-is-finally-a-usable-ba...
They are simple.
The issue is companies love to fire QA people who should be the ones testing these.
Microsoft says we don’t need no SDETs, our products can ship full of issues.
What are the hostages, I mean customers going to do about it?
I have no sympathy for any major corp who suffers data loss. Now individuals who lost grandmas wedding photos, that sucks.
But my recovery strategy for that is simple. After scanning I emailed it out to one cousin who emailed them out to a few more. Plus I have a cloud backup ( I have no idea where the the original files are).
Data is meant to be shared after all
My setup: rsync to Synology NAS, snapshotting there, sync to remote Synology NAS over VPN. (The other NAS is at my parent's and they have similar setup syncing to me)
The best part: it is just files, not some proprietary archive. I have high confidence in this. I can restore with the most rudimentary tools.
This setup has worked for me well on linux, windows and now mac. I migrated to new computer&os by restoring the backup mostly.
"encrypted, chunk-level deduplicated, GFS-rotated, point-in-time archived, cloud, 3-2-1 backup solution" is now my newest password, no commas. (Don't tell anyone!)
Change the o in rotated to a 0 and you might have a secure password.
On my home PC I use restic, and (shockingly) I run it manually (via a script). I have a terminal banner that nags me when backup hasn't run in 24 hours, or displays spanish verb tables when it has (so I will notice if the banner doesn't show). I've been burned in the past losing a backup cron, and dead man's snitch seemed overkill for my needs here. And I've also forgotten what backup system was in place on a prior install, and it took me way longer than I liked figuring out how it was setup when I needed to do a restore.
Obviously for production systems automation & monitoring is the way. I kind of like this setup on my own machine though.
I've given out jwz's link to several people:
https://www.jwz.org/doc/backups.html
(maybe copy/paste this into browser instead of following link, since referrer from hn apparently does something)
> have this happen every morning at 5AM by creating a temporary text file containing this line: 0 5 * * * rsync -vax --delete --ignore-errors / /Volumes/Backup/
This leaves you needlessly exposed to this failure mode: Commit some "braino" that wipes out some recent work. Go to bed without noticing the loss. 5am rsync run wipes your backup as well.
The fix is to back up to not just one but to a rotation of images. Use --link-dest to reduce storage overhead to the size of the directory hierarchy only.
Oh that's fantastic (HN redirect)
If you don't control your data size, it will end up controlling you and your backup choices, which will eventually lead to many avoidable, disastrous outcomes. Somehow, this gets missed in data storage and backup planning.
A bit of a tangent, but the linked rsnapshot tool is about 7k lines of perl code. The COPYING file is 20 years old and it's still maintained, the last change was 6 months ago. I'm feeling really old.
[1]: https://github.com/rsnapshot/rsnapshot/blob/master/rsnapshot...
For my Oracle Linux systems, I rely on the vendor ReaR RPM package for the baseline backup. This is similar to the "Ignite" utility that I used in my HP-UX days.
https://relax-and-recover.org/
This is two-fold useful, first as it enables boot from backup media for total recovery, second in that it creates a backup.tar.gz that holds everything that was not explitictly ignored in the /etc/rear/local.conf file.
I'm running Oracle databases on these systems (and the local.conf is configured to ignore the datafile directories). I have standby databases (not Dataguard in that I don't have online redolog replication) that allow me to recover them to the primary in a disaster, with some loss of committed transactions.
I finally have rsync configured for some scratch temporary files.
I have dallied with btrfs snapshot replication for home directories (in a loopback mount). My vendor support (via a CSI number) has been on and off, so I don't use btrfs in areas that we need it most.
My replacements do not like the complexity (and I am retiring).
One of our customers decided to go for off site backups with us - they had been a customer for 15 odd years for other services. Backups are not our core business, its one of the services and I got around to setting them up sooner rather than later.
Two weeks later they had a fire in one of their huge printing machines which caused quite a lot of direct damage due to heat and rendered a lot of equipment broken due to smoke/fumes. It was also, as it turns out, a week before the first COVID 19 lockdown in the UK.
Their backups landed on a XFS file system with reflinks enabled. I cloned their repo and fired up NFS with their VMs running on our gear. It took me another hour to spin up another OpenVPN server (CA etc) for them to use and sort out a few other details (coffee doesn't brew itself).
They ran via VPN out of our data centre for most of the pandemic.
That was an utter triumph but I also have some rather less triumphant stories about backups and lack thereof. Let's skip over those 8)
I'm polishing my backup system right now, and found that `tar | zstd | gpg` plus a small vibe-coded wrapper is pretty much all I need.
GNU tar has its own incremental index via `--listed-incremental=FILE`. Unlike Borg or Restic, which have their own more complicated repository formats, this leaves me with just one additional file (the `.snar`) alongside a dumb, portable full-disk tar archive.
The nice part is that, unlike repositories that require both read/write access patterns, tar can compute deltas using only the small `.snar` file, while the main `full-disk.tar` can remain buried in write-only Glacier storage. This makes it a much better fit for Glacier's write-once model and 180-day minimum retention.
My current plan is to upload a full-disk tar to an external HDD + Glacier on a weekly cadence, and the `.snar`-based deltas daily. That gives me a pretty simple cloud backup solution for a few bucks a month (after burning through $100+ of free credits).
How well did it work for restoring?
For me, the ability to mount a backup and look at a particular file proved to be important. Also, I restored from my backups four times; two of them was moving between machines, pretty quickly.
The problem with using glacier for backups is it's hard to run restore exercises, and without restore exercises, a backup is pretty dangerous
You can exercise it against regular, non-Glacier S3. Same flow; the only difference is that you skip one attribute on upload and one request/wait step on download.
Something is better than nothing. I accidentally spilled liquid on my macbook just the other day. It's still in the shop. Luckily, I periodically clone my drive with CCC onto a single external SSD. Don't wait to set up the perfect backup solution, just have a copy of your data in more than one place, that's a good start imho.
I have a cron script that rsync my entire home directory every day with a remote server. Since rsync doesn't remove remote files, I re-create a backup from scratch the first day of each month and store a copy of the previous one.
Using old standardized open source tools means it survives system updates without having to fix anything for years. I got my laptop stolen/lost 3 times over the past 15 years, and I have always been able to restore everything the next day on a new laptop, in the time it takes to transfer the files over the network.
I haven't given much thought, I'm sure there is a realistic scenario where this strategy would fails but I haven't found it yet.
rsync can delete remote files with --delete or --delete-after
What is missing is recovering accidentally deleted or corrupted data.
+1 for restic.
ZFS snapshots + restic backups to backblaze for my homeserver. My secret sauce is a healtchecks.io instance that blows up my phone if ZFS scrubs throw any errors, when local snapshots fail, or when restic checks or backups fail.
I use ZFS and do off site backups using rsync.net.
>Motivated by our success in developing this solution, we try to use it to backup the homelab with its 10 Docker containers. But later we find out from logs on the individual machines that backups are failing. The reason being that many Docker containers like to create root-owned files, and if you’re not careful you can create a cronjob running as the default user.
and a privesc also.
Yeah it feels it can be risky to "make the backup also dump the databases, and give it full filesystem permissions on our Docker volumes", depending on what binaries get the root privileges.
What I do is having a minimal systemd timer under root that only calls restic for backing up files (with some additional systemd seucurity restrictions). App-dependent backup logics, like dumping databases, are done by user/container-level cronjobs separately.
On the topic of backups, 7zip recently changed its handling of the FILE_ATTRIBUTE_ARCHIVE bit on unzipped files which caused an issue with some old software I was using. This is from the old DOS days with the BACKUP/RESTORE commands - the bit was set if any backup was made of the file. It's comically naive because it only assumes a single backup and this bit was forgotten to the sands of time until it was disturbed by some meddling kids!
As an average-techie person, the only piece of data that i truly practice 3-2-1 is photos and videos. 1 on my phone, 1 on my small server running Immich, and 1 is on Google Photo(sadly, the quality is loss, but i can live with that).
Most of other data i only have 2 backup, usually 1 at my local machine(code) and 1 online(github). Yeah it's not ideal, but another question to ask yourself, is it truly worth it. I trust the engineers at MS doing much better job at backing up their data than me.
I tried to set up something similar, photos and videos on local NAS + cloud storage. Also with data loss, eg. I don't sync any large RAW camera files to cloud. I clone my local machine with CCC onto an external drive, that way it should be easy to just restore the last state onto a new machine. So I think it's archival storage for media vs snapshots of your working setup so you can restore quickly.
Time Machine (to NAS) + iCloud (w/ ADP) + local copy is the lowest friction path I have found.
"Obviously, none of this is worth anything if you don’t actually test restores."
One of my teachers claimed that contrary to common behavior, no one actually valued backups; it was restores that were worth paying for.
Try running a business on that basis - where do the restores come from? 8)
We charge our customers for backup storage by volume and that includes an annual demonstration of recovery. We fire up some or all of their systems on our gear in isolation and show that they are reasonably functional.
Your teacher's comment is riffing on the well trod lines of: "heights don't kill people, it's depths that kill people", which is all about perspective. This is is not related to "guns don't kill people ..." which is about agency.
It is true that very few people proactively care about backups. It’s like flossing. People know they’re supposed to do it, but in the moment it feels like a waste of time and energy, when there are more important things to do.
When data is lost, or decay takes over, the value can skyrocket. Depending on what is lost, someone may be willing to pay 100x when they would have spent on backup in order to get their data back, this is why data recovery services can basically charge whatever they want.
My assumption is the Venn diagram between people who pass the marshmallow test, and those who proactively backup their systems, has a lot of overlap.
Running a business on that basis would look like the white(ish)-hat version of ransomware. "Oh, your data is gone? How much is it really worth to you?"
"It's not the fall that kills you, it's the sudden stop at the end."
It is a special sensation when you go to your hand rolled backup "solution" to make some simple recoveries and start finding problems. It is a complex mix of not dread but disgust, sick to your stomach, self-anger, and loss. I can only image what that's like with like all of your children's baby pictures being the lost material. Borg and restic are fantastic, it still seems like some sort of dummy proof mode might be desirable, all the features in the world and they have some some reasonably sane policy or something that includes verifications.
On the flip side of that though, really and truly, I've restored a laptop from a timemachine backup and it came back damn nearly identical and that was magical.
>Borg ... fantastic
I needed to temporarily move off 20TB from my NAS and bought a Hetzner storage box.
I wanted to encrypt the data before sending it so i went with the borg ssh mount. Long story short, borg failed a couple of times due to https://github.com/borgbackup/borg/issues/7672, i resumed it per documentation and got 2gb of silently corrupted data. There was no way to do data integrity checks due to broken pipes and there was no option to resume the job from the previous point either. I couldn't afford to go with the s3, backblaze wasn't an option due to placing trust into their proprietary client doing the encryption and i got majorly screwed in the end.
Yep, backup is a verb, restore is a noun. Testing restores is where the real work begins.
concerning backups, when it comes to disaster recovery you need a strategy for restoring as well, sometimes it's not only the data you would need but the systems as well
Truenas, Crashplan gives me a 3-2-1 with rolling, incremental backups at a pretty low price
Amazon are finding this out atm :-P
https://archive.is/Ay7RJ
When it comes to software, it's almost never simple. What people think of as simple is rarely so.
For example:
Requirement:
"Just copy a file from folder A to folder B."
Minimal translation:
"Implement the capability to copy any file, in any format and of any size from SharePoint, located at a configurable path, with appropriate authentication and access control checks, then stream it in chunks to a different, configurable path inside an S3 bucket, also with appropriate authentication and access controls in place. Ensure that any disruption in either service which may occur while the file is streaming can be recovered from at the point it failed instead of having to restart from the beginning. Ensure that the retry mechanism is built-in and that the retry window is configurable; if the file cannot be copied within the specific time window, then an error should be sent via email to a configured email address. Ensure that the entire transfer is encrypted in transit... If the file happens to be a folder, then you must copy across all of its contents recursively up to a certain configured MAX_DEPTH to avoid DoS and ensuring that the system does not get caught in an infinite loop due to symlinks pointing to a parent folder... In this case, send an email to the configured address... Etc... Etc..."
And the thing is; if you tell AI "Just copy a file from folder A to folder B." - It will not meet your 'basic' needs because even if it does a great job at filling the gaps in your requirements, it will still take shortcuts. In order for an AI to avoid taking shortcuts, it would have to make you fill out a questionnaire and make you sign up for and configure services; it would not be a pleasant user experience. The user experience cannot be pleasant, because the AI cannot read your mind and it cannot know your intent.
Here's a thing: my intent and requirements actually are:
"Just copy a file from folder A to folder B."
All that access control and authentication and symlink nonsense? That's a self-inflicted problem that exists only in enterprise, and shouldn't be assumed - much less created - for regular users.
I don't want backups. I want synced duplicated data in geographically separated by shared credentials.
> I don't want backups. I want synced duplicated data in geographically separated by shared credentials.
If you accidentally delete something, how long does it take for the delete to propagate to all of your geographically separated datacenters? You need some sort of point-in-time snapshots to be able to recover from accidents and silent corruption.
> You need some sort of point-in-time snapshots to be able to recover from accidents and silent corruption.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/versio...
Sounds like you are the second type of people mentioned in the article.
that's not really the same thing, what you are describing is redundancy so failure in one (or more) place doesn't bring the system down. Backups are about preventing data loss. Data loss can happen in many different ways, not just a system crashing and burning. It can be deliberately/accidently done such that it would be sync'd everywhere.
So is apparently making readable site.
Software like https://restic.net/ does a good job. Few choices of backend
Accepts stuff from pipe too so you can just pipe mysqldump or pg_dumpall without intermediate files
Very decent options for checking repo integrity, personally (well, at work too, we use it on few hundred user machines and servers) I also added "roll a dice for backup and try to restore it" test script to make sure it is working
Decent deduplication too so "store year's worth of weekly snapshot" is very sensible strategy once you exclude the "constantly changing and useless" (caches etc.) files out of it
can mount FUSE directory with all backups on sensible OSes.
https://kopia.io/ does the same +GUI but a bit worse on CLI front (it insists on keeping local config which makes scripting a bit more involved, but not much)
I'm using Restic & Backrest (https://garethgeorge.github.io/backrest/) everywhere I can, they're both exceptional OSS.
IMHO, the easiest way to perform a restore test is to use your production backups to refresh a lesser environment on a weekly cadence. Naturally this has to be appropriate to the type of data you're restoring, e.g., our E2E (end to end) testing environment has been built to the same risk tolerances of production so it's suitable for production data and the PII (Personally Identifiable Information) it hosts. If this isn't in your own risk tolerances, you can still perform the same test but ensure the data from prod is anonymised or alternatively destroyed, and perform a second restore after production restore with the dataset that preceeded it that has no PII data or similar.
With this the E2E tests confirm the backups are expected and you can tick a box that when shit does eventually hit the fan, data corruption isn't a problem. Normal caveats apply, you must make sure your backups are immutable so they cannot be modified or changed _after_ the tests have been ran.
I was using Restic + Backrest as well for my personal Windows machines and it was great. Then I found out that Backrest stores the vault password (among other things) in a simple plaintext file on your local hard drive. Not sure if this applies to all platforms but for Windows users that is a huge security hole in today's environment.
https://github.com/garethgeorge/backrest/issues/1047
importantly, does your die have a side that corresponds to a multi-continental cataclysmic natural disaster of magnitude only known unknown to pre-recorded-history
> So our first principle is to have a backup, i.e. a copy of your files someplace else. So far so good.
So who are we? A household with a localhost administrator?[0]
Then you are surely doesn't need encrypted, chunk-level corporate level bla-bla-bla.
You need a Syncthing copy to some other device not at home - and an additional backup procedure to maintain the history and protect against PEBKAC errors - which can run on your local Syncthing copy or/and the other one.
And no, nobody needs your 555GB of RAWs of sunsets/flowers/precious_family_moments you shot - not even you.[1]
[0] well considering the tone of TFA and most of the comments here
[1] that's Instagram/Google Photos/iCloud for nowadays, totally with "Remember this day N years ago?"