Having built a nas recently (luckily before ram/diskpocalypse) there seems to be a huge difference between making a small nas (say 4 disks) and a big one. With a small one you are essentially building a pc, and the differentiating factor is the software. Mobo, memory, case, PSU… buy consumer (cheap) stuff and you’re ok. Above that you have to start buying very expensive business oriented stuff,disk enclosure, better psu… the costs increase more that linearly , especially if you don’t live in the US and can easily get used stuff from eBay.
And that stuff is often not just more expensive, but uglier and noisier. I ended up making my own “enclosure”
Not a great time price-wise to be building a NAS, but I have been doing so for the last two weeks. Inside a Jonsbo N6 case, which is pretty nice with an 8x SATA backplane and drive bays (unlike the earlier Jonsbo variants).
I ended up on shucking 4x the 14 TB WD Elements Desktop. They contain helium drives, the WD140EDGZ in my case, and are about a third cheaper than 4x the 12 TB WD Red Plus drives (which are air-filled). The shucking was easier than I expected too, and the performance seems very comparable. The warranty is a definite downside (European, so no Magnuson-Moss), but I think I can even get them back in their enclosure should they fail during the 2-year warranty period.
I've put some second hand 256 GB M.2 SSDs in there as boot drives. It was a bit of a struggle to get it to work in a way that failure of one of the drives doesn't hold up booting, combined with LUKS, TPM keys and ZFS on root. Learned a lot about systemd-boot which I have never used before, but feels a lot saner to me than grub ever was. So now I have a large script which debootstraps a Debian based NAS into being.
I noticed that there are a lot of ZFS myths and cargo culting. For example TFA mentions ECC RAM, which in some circles is a must-have because ZFS would wreck your pool during a scrub otherwise, which is a myth. It's also very expensive, especially this year. You also don't need much RAM for ZFS, L2ARC doesn't use much RAM at all, to name a few others.
Still doubting about setting `dnodesize=auto` (which is the default), because there are some horror stories about that [1]. And it seems impossible to find a cloud storage provider with reasonable prices that supports `zfs send`. Rsync.net upped their minimum order to 10 TiB recently, which is far too much for my use case.
> Not a great time price-wise to be building a NAS
That under-states the matter. It is a terrible time, price-wise, to build a NAS.
I'd almost rather have no AI whatsoever and have storage 1/10 the price of pre-AI times.
(If there were a magical choice between having AI and significantly more expensive storage, and having no AI and some program to dump that investment money into getting and somehow leveraging significantly more available storage, that is.)
> I'd almost rather have no AI whatsoever and have storage 1/10 the price of pre-AI times.
Almost? ALMOST!?!?!
If you handed me a button that would make it like LLMs had never existed I'd be slamming that button so hard Sam Altman's clothes would spin around.
Return memory and storage prices to normal, undo the sloppification of ~everything, remove all these annoying "features" that are so useful they have to force them upon people, and make scammers actually have to put in a slight bit of effort, all at the "cost" of real human developers, artists, writers, etc. getting paid for their work....
If this is a hard decision for you, you are the problem.
Same. Every time I think about this, I come back to the fact that I was excellent at my job before AI, and even enjoyed aspects of it, so I would be fine if tomorrow it all blew up.
Same, I'd smash it like it was offering me a million dollars. I cannot think of a single positive thing that has come out of the AI boom. It has been a net negative for all of society.
Oh come on! I'll take a few years of expensive RAM in order for humanity to get wide access to something near as makes no difference to the Star Trek Ship computer.
Which is 100% what this has felt like to work with, all spring and summer.
So we're in a slop phase, it'll pass. The first few years of youtube gave no hints that we'd get stuff like Veritasium.
Ignore the slop and use the tools to create something you never thought you could do. That's what it's for!
Hey everyone, stop progress! Wolrah has decided that November 29, 2022 is as far as things should go. Because everyone knows before that day, there were no scams, every artist lived lavishly from commissions, and software was flawless and without any extraneous features.
I had a drive go bad in my 4 year old home-built NAS a few weeks ago, and it cost 2x what it originally did to buy the same capacity drive, and that was going to a grey-market importer on eBay, it was more on "reputable" sites.
I obtained a used Dell PowerEdge and just stuffed a bunch of drives in there. It's old enough that the RAM is not overly expensive (DDR3), so it has 192GB of RAM to effectively cache the data, and then 32TB of raw storage. (Of course I regret not spending the extra money to take it to 384GB now.)
The nice thing about the array being big is that I can just RAID 1 it instead of worrying about tinkering with RAID 5, and leave a drive as a hot spare.
PCIe 2.5G NIC for the uplink, and then it can serve over SMB or iSCSI. The main use case for this thing, incidentally, is it just is a caching proxy that holds Docker images, models off of Huggingface, and so on.
The 36 bay was all I could find on fb marketplace.
I fell in love with the silly idea of squeezing 10G NIC + 10G writes (to a 2x12 RAID 10 HDD array) on the thunderbolt bus of my 12 year old macbook pro.
Then I got distracted with the server mb and truenas and ZFS.
I would kill (figuratively, anyway) to have no AI whatsoever. No slop machine threatening to replace my job, or turn my job into babysitting its stupidity, and hardware would be reasonably priced? That would be awesome. AI has brought me nothing but downside.
AI has freed most people I know from the tedious job of writing actual emails and birthday greetings - leaving then more time to e.g. wash dishes or clean the floor.
Seriously, if arts and creativity is what sets humans apart from other animals, then AI has almost completely displaced our capacity to even consider doing these activities ourselves. People reach for AI when they should be composing a birthday greeting themselves.
someone at work unironically said they use ai to compose "thoughtful responses" to people, and my immediate thought was that what they are doing is the literal opposite of thoughtful
Was it a genuine birthday greeting when you bought a card? Or used an online card? I don’t really see the difference. The point is you thought of them, not the actual words
Great business opportunity there. Make a free service that sends AI generated birthday notes to people. All the user has to provide is the other persons birthday.
People cannot be relied upon to provide accurate birthdays but nobody would suffer the social faux pas of an incorrect birthday congrats note. Nor would they send it to a spam catching email, but rather are guaranteed to send it to a regularly checked address. They know the persons birthday after all.
You can then sell high quality birthday information correlated to contact information to ad agencies.
I used to write birthday emails to my friends years ago. I quit when calendar systems started automatically promoting me when birthdays happened. Remembering someone's birthday no longer seemed like a significant effort on my part and so it wasn't worth the bother since it didn't say as much.
For 90% of birthday greetings, are they ever genuine?
Like seriously, outside of some close friends and family, are you sitting there deliberating over a message for your coworker Steve in the Slack chat or a cousin you barely talk to outside of a birthday and Christmas card?
Are you being sarcastic? There's no way you believe people couldn't wash dishes or clean floors because of all the emails and birthday greetings they had to write, or that people have almost completely lost the ability to consider making art.
It's a paraphrasing of the popular 2024 Joanna Maciejewska quote: "I want AI to do my laundry and dishes so that I can do art and writing, not for AI to do my art and writing so that I can do my laundry and dishes."
I believe it was rhetorical. They were trying to illustrate that we've automated tasks that people might find to be fulfilling (drawing, writing) but are still a long way off from affordably automating drudgery.
> or that people have almost completely lost the ability to consider making art
They haven't lost it, no, but there's a lot of financial incentive to stop paying people to do it.
I immigrated wanting to become a millionaire. Never found time working my ass off. Now that I get pension and tokens from Musk and Zuck, I can finally try!
I’m running an 8-drive ZFS RAIDZ2 pool. I’m wondering if you know — are the free space recommendations around ZFS cargo or real?
Like I’m already giving up two full drives for redundancy (which saved my ass - I recently had two drives fail on me in quick succession — both SSDs from what looks like an identical batch) but then the advice is kinda saying I need to keep at least another drive worth of space free for the pool to perform well and not crap itself. That hurts with current prices for sure.
> I’m running an 8-drive ZFS RAIDZ2 pool. I’m wondering if you know — are the free space recommendations around ZFS cargo or real?
I'm not entirely sure, but it seems to me that free space (and the 20% reservation) is mostly a proxy for fragmentation, and you can therefore better look at fragmentation directly. That would mean that if you mostly store large files, there shouldn't be a lot of fragmentation even at high utilization. The whole "ZFS changes allocation algorithm from 80% usage on" is something of 10+ years distant past, and lots of things around the allocator have been improved. It's also something that probably isn't too different from the performance of other filesystems at high utilization, so it shouldn't be exaggerated.
ZFS will auto-degrade performance if there isn't enough headroom. In addition, if you use SSD, you also want the headroom, because otherwise you end up writing and rewriting on the same small empty space which kills SSD (unless you have enterprise SSD, which have built-in headroom).
Not sure why I'm being downvoted for being factually correct (and also, why I can't edit my above comment).
ZFS will definitely degrade write performance gradually from 90% utilization, and will hit a stronger cliff at 95%+. Same with SSD, using a consumer SSD above 90%+ utilization would rapidly degrade its lifetime. The effect will be smaller for very large pools and very large files, but the effect is stil there.
I'm repeating this so that people who set up these drives know what is actually going to happen.
I appreciate the reply, but my warning consumer SSD is not incorrect. Perhaps I should have clarified that While they typically have a small overprovisioning of 7%, enterprise SSDs have close to 4 times as much, exactly to allow more writes (which is very relevant if running a ZFS raid on the drive). Keeping 10-20% headroom will prolong your SSD's life significantly, and increase performance of ZFS.
For home usage, if you have backups raidz1 is fine (just do an incremental backup at the first sign of trouble). If you don’t have backups, then you probably shouldn’t be running a NAS in the first place.
The risk of z1 is that if you get a read error during resilvering, that data is permanently corrupted. The odds of this happening go way up the larger your individual drives are. This is why I chose RAIDZ2 for my NAS. I've had to resilver 2-3 times over the past 10 years, and never lost a byte of data.
In the Linux ISO community the math is different because none of the data is essential, however there is a lot of it.
Backing it all up even one time to an off-site location would be a 2x expense for a hobby. So 3-2-1 is hard to justify.
Instead the math can be different e.g. it's ok to lose some data but not all. Therefore you might prefer the unraid approach over zfs where losing more than parity doesn't kill the whole pool.
I'm actually in this boat right now. I have 2x26TB drives in a 4 base nas (two empty, plus 3x M2 empty) and I'm trying to decide the best way to set them up. I have around 3TB of backups that I care about as backups (but they don't need to be online, just archive backups at this point), and around 5TB of media that can be replaced.
I don't want to lose 26TB just for a mirror, and I have a spare 8TB USB HDD. I'm torn between unraid and just Debian, and I'm torn between just two separate devices and one RAID 1 partition one RAID 0
For most people snapshots are enough backup. It is still useful to have an offsite backup, but realistically fire is rare enough that you can risk it, and that is about the only risk most people have (you can have your NAS in a location likely far from where a fire might break out to reduce the risk farther).
Fire, malware, accidental deletions, capricious RAID controller (pre-ZFS). And that’s only the stuff that happened to me. Add power surge, theft, correlation in SSD failures (eg power on counter overflow firmware bug), damaging the array while moving, etc.
Malware, accidental deletion, RAID controller issues - all things ZFS with snapshots are immune to. You can decide if the other worry you or not - the risk is not zero, but it may be acceptable. Your backups are all subject to similar issues.
Snapshots don’t protect you from malware. I bet 99% of home users use the same credentials on all their machines, once a malware compromised one, the others are compromised within seconds.
[edit] also snapshots aren’t really workable for large files. Remux a movie file and now it occupies twice the space.
The middle ground I go with is to structure the pool using datasets and zvols, and then periodically backup the critical ones using zfs send. For example, I can live without my movie collection, but I cannot afford to lose personal photos.
This isn’t black and white. You might have huge amounts of non-essential data that aren’t worth the cost of off-site backups, but worth the cost of an extra disk of redundancy to lessen the risk. Even when you do have backups, it will reduce the risk of extended downtime (and possibly egress costs) caused by having to restore large amounts of data from backups.
I run my cheap hosting box with a cleartext boot setup that uses ssh to automatically grab the key for the real root from my home server (or an alternate at my MILs house). Using FreeBSD, but similar concepts.
A previous hoster once gave me someone else's drives without wiping them. I don't want random customers snooping around on my data if a similar mistake happens with my disks.
For home use, I run without disk encryption. If I ever need to do data recovery, it's not going to be possible with encrypted disks and one point of a centralized NAS is to have stable long term storage.
If you are worried about someone breaking in to your house and replacing the bootloader while leaving your drives in place I probably wouldn't use the TPM auto unlock even if in theory secure boot should be able to handle this.
But in reality that will never happen and the only actual attack you need to be worried about is junkies breaking in and flogging the drives on facebook marketplace. For which, this level of security is fine.
You have SATA or SAS to pick from. The CPU requirements for a storage server are not high. On a typical ATX board you have motherboard SATA and can put SAS controllers in the spare PCIe slots.
My first "NAS" was two 22TB hard drives in a ZFS pool on my motherboard SATA
no. just sata & sas. There is no nvme spinning drive and flash loses bits just from sitting, it's not archival. flash is good for working not for storing.
It was €329 per hard drive (through a reputable store), and I chose the ASRock B650M-HDV/M.2 mATX motherboard combined with a Ryzen 5 8500G. Stock CPU cooler, replaced the Jonsbo case fans with Arctic P12 Pro PST LN. I only slightly regret the PSU (MSI MAG A650GL), which could be quieter. Not that it's very noisy, and it's a great PSU otherwise, but I should've chosen one that just shuts the fan down at low power usage.
I recommend staying away from SATA drives (huge consumer rush) and look for SAS drives on eBay, particularly HC520 Helium drives from HGST/WD. Need a SAS3008 PCIe adapter ($20) and a SFF-8643 splitter ($30). No backplane is required. Huge lots of decommissioned drives frequently appear. I bought Qty 10* of HC520 (12TB) SAS drives for $1000 about 3 weeks ago, avg age is about 2.5 years, still well within its rated lifetime.
Yea may be some of the stuff is fear mongering and cargo culting. I was told ECC is necessary for ZFS. When the article was written, it was cheap af (2024) to buy ECC RAM so not much consideration was given to it.
--
(*) I have no idea what to do with it. Anyone has any good idea for using 120TB space? I have about 40TB unused bandwidth in the datacenter, may be host a Debian mirror? Donate storage/bandwidth to Internet Archive? Please contact me, appreciate it.
> I bought Qty 10* of HC520 (12TB) SAS drives for $1000 about 3 weeks ago, avg age is about 2.5 years, still well within its rated lifetime.
That's cheap indeed. Enough headroom for some failing disks too. How is the noise and power usage? I didn't look at SAS drives at all, because my impression was that they're very noisy. I can place my NAS in a closed off room, but it's not too far away and I was afraid SAS drives would be audible through the wall. At the same time, the shucked drive I'm using presents itself as an WD Ultrastar, which comes very close to a SAS drive, and isn't very noisy.
This reply is exactly right about using enterprise drives (hdd), and ideally SAS. Note the power usage of SAS vs sata for an identical drive make/model is very small sometimes 0.5 W to 1.5 W at idle (idle power draw is what matters for spinning HDD’s as underload it doesn’t increase much and idle is what you’ll be at 98% of the time).
Also enterprise SSDs are much better than consumer SSD‘s for ZFS, and be aware that when you get above the 4 TB size SSD drives frequently use more power than HDD drives (i’m mainly referring to enterprise non-consumer level drives as that’s all I run for 100s of disks across many zfs systems over past 15 yrs).
Also something people forget for home nas or figuring out cost for a nas, the power draw of the entire system times your electricity price you then have to multiply this times 2x to 4x times if your climate requires cooling air conditioning of any kind
I'm also planning a new home server build now and the prices are definitely relatively ass. So far I've spent 820€ on two 22 TB WD Elements HDDs, 375€ on 2x16 GB DDR5 kit, and 520€ on two 2 TB M.2 SSDs (cache). About 1700€ and I still have no server to show for it. Doesn't help that I've been procrastinating on picking the CPU and motherboard.
You assert that ECC RAM being necessary for ZFS is just a myth but provide no justification for why that is untrue.
Is it not the case that if you don't have ECC memory, ZFS could end up writing a checksum that does not match the data if you get a bitflip in just the right (wrong) spot?
Yes, indeed. ECC RAM is better than non-ECC RAM, also for ZFS.
The myth, popularized by a notorious thread on the TrueNAS forums [1], is specifically that ZFS requires ECC RAM, and will do worse than other filesystems without it, because scrubbing will multiply a single bitflip into a failed pool.
A ZFS core developer says that that isn't the case [2]. Here's some more reasoning [3], also about many other myths.
Usually if you’re using a NAS you don’t want to lose data, ZFS is not significantly more sensitive than everything else.
But everything is actually quite sensitive.
We’ve accepted lack of ECC because Intel decided it would be a product line differentiator, and serious customers who didn’t want random crashes or to lose data would buy chips with ECC.
It’s actually less of an issue these days because DDR5 has (by spec) some in-line ECC; won’t help with multi-bit errors but its an improvement on what came before.
> We’ve accepted lack of ECC because Intel decided it would be a product line differentiator, and serious customers who didn’t want random crashes or to lose data would buy chips with ECC.
AMD has been allowing ECC on lots of regular hardware for a long time.
People don't tend to buy ECC for desktop use because it costs significantly more (used server ram is/was often cheap... but it often doesn't work in desktop boards), and the performance specs are poor.
My home servers are mostly retired desktops, so they get my old desktop ram and I don't want to pay premium prices for jedec speed ecc ram on my desktops, thanks.
Since DDR5 doesn't include reporting on bit errors (afaik), it likely means much fewer single bit errors, but most experienced errors will be multi-bit. Although, I dunno what proportion of bit errors is on the ram chips and what's on the bus... there's no protection from bus errors.
If there were reporting, you could replace chips with high error rates, but without reporting you'll keep running them until they fail enough to notice.
DDR5 on-die ECC is to achieve acceptable yields in the face of denser process nodes that decrease the reliability of RAM cells. It’s not clear how much of an improvement that is to what we had before, other than allowing for higher RAM speeds. It doesn’t replace side-band ECC.
I know that LPDDR5 has ECC, and not just single bit AFAIK, but if you enable it you lose some memory capacity (the 128 bit bus minus 16 bit for ECC error correction, making it effectively 112 bits)
The only hangup with the myth "debunking" is that the point is that the corruption doesn't happen to a per-disk buffer, but to the in-flight data before it's persisted to your stripe.
Which means all copies of the data will be corrupted. This can be anything from an irrecoverable file to complete filesystem corruption.
But generally, yeah, not any more dangerous than any other filesystem, and ECC used to be cheap so it was a no-brainer, you should have backups anyway TBH if we're being honest about storage resiliency.
What has been debunked is the "scrub of death" issue, on a scrub a bad bit flip would cause an error, which would be copied over with good data -- well it was technically good before. It would be statistically difficult to have a fault on a read, then a clean read, then a second bit flip destroying the data.
Had it happen to my PBS backups running on ZFS without ECC just the other day. Turned out to be heat related, memory was fine but during long backups heat was causing bit flips.
Install avahi-daemon. Samba will automatically register with it to advertise SMB/CIFS to macOS and Linux clients over DNS-SD.
Install wsdd2 so that your server will be auto-discovered by Explorer on Windows 10+ clients with SMB 1.0 disabled, too.
Your Linux hostname is probably lower-case, but by default, Samba publishes a capitalized rendering of the hostname to NetBIOS and Avahi. If this bothers you, set “host-name=something” in the [server] section of /etc/avahi/avahi-daemon.conf, and set “mdns name = mdns” in the [global] section of /etc/samba/smb.conf.
If you have macOS clients, you should enable vfs_fruit in your Samba configuration: https://www.samba.org/samba/docs/current/man-html/vfs_fruit..... There are some compatibility reasons to do this, but mostly it means you can set the “fruit:model” so that your server has a fun icon in the Finder sidebar.
Although macOS deprecated AFP in favour of SMB years ago (and are slated to remove AFP client support altogether in the upcoming macOS 27), SMB client support in macOS is still pretty miserable. The upcoming macOS 27 is set to drop AFP support, but until then I will continue to run Netatalk side-by-side with Samba. Netatalk also registers itself with Avahi, and macOS will (tellingly) use AFP preferentially to SMB, so clients will talk to the right daemon automatically.
Yeah, the default [homes] share is really not necessary on a home network where you really just want a common share. On my home storage server, I just have a “files” user, and a corresponding, anonymously readable/writable share:
[public]
vfs objects = catia fruit streams_xattr
fruit:encoding = native
comment = Public share
path = /home/files
force user = files
force group = files
guest ok = yes
browseable = yes
read only = no
create mask = 0644
directory mask = 0755
readdir_attr:aapl_max_access = no
The default [homes] share is still there in case I need it for anything, but it's “browseable = no” so it doesn't confuse visitors.
I think it's appropriate to link to my blog post about building such a NAS but with very different technologies: dm-integrity, mdadm and XFS. The reason being that as a C/C++ dev following OpenZFS development closely, I found the focus on ginormous features over general stability and the constant problems due to the SPL/separate page cache too worrying.
Sure, I traded the convenience of kitchen sink (or Swiss army knife for the more charitable souls) ZFS for some initial pain, but I'm very happy with my choice today.
That's what I did after my 15 year old Synology DS1010+ built in USB DOM failed. Put a network booted debian on it via netboot.xyz with zfs and now i get to reuse those 15 year old 2TB still chugging along disks. Fully open source, pretty nice way to keep old hardware going. It's my tertiary backup that wakes up once a week pulls in some open weights LLM models that I hoard just in case and goes back to sleep.
Where can I have a discussion with other developers about storage?
Is here okay?
I want to have physical storage over here, and logical storage over there, and I want to control the mapping from one to the other. I want to talk about encryption, replication, latency. Make "time machine" available on this logical storage. And then I buy some new physical storage, and it joins the story. This physical storage is over at my friend's house for backup. This physical storage is slow and archival. This is a device for writing archival media, and there's a brand new media in it, go ahead and write to it. Show me the health report on all of the physical media, and show me what you've done to protect the logical storage. Graph my usage and make suggestions about when to add more physical storage.
Buying physical storage with power and wifi, and configuring it with a QR code that's on an e-ink display - seems like it should be the most obvious thing in the world, that we should all be really used to doing by now.
bcachefs is slowly working up to what you're describing - describing what you want and letting the filesystem sort it out. We're basically there for local storage, and other people have been building some nice reporting on top.
Next year (post Rust, because networking code is so much nicer in Rust) will be send/recv, and I think we should be able to make some nice improvements over the state of the art there.
This is what I'm hoping to do. I haven't done much research yet, so I look forward to reading about your learnings. (Also to the person who responded to you with their experience and link.)
Thanks for saving me probably some hours...
EDIT: I'm also coming from an old Synology NAS from the ~2015 era.
Same, using NixOS as a NAS, though it kept growing and now I'm trying to share it with other people. I use BTRFS on MDADM though. It's ended up being an all-in-one home server and router now because I was tired of the noisy power hungry rack I had in my closet.
I'm sure people have reasons for taking these things as far as they do with ZFS, and everything else that goes into what is commonly considered a "NAS." But I've found great convenience in a tight NFSv3 config running from a single high capacity HDD with ext4 on a Linux machine (primary system that's always on anyway while I'm around/awake) making things available to my other devices for the electricity and overhead cost of one machine instead of multiple.
I try to keep my network configuration restrictive by default, so I'm not too concerned about possible security arguments running it from my main machine. I've probably committed some great sin here, but is plain NFSv3 and secondarily Samba (for compatibility) really not enough?
First: Do you have backups of your single high-capacity HDD? That's my biggest worry. What's your plan for when that HDD fails one day, as it will?
The main reasons people go for ZFS and a "NAS" is checksums and data integrity protection, as well as maybe not wanting to keep their primary machine awake all the time (I personally don't).
Then there are useful features like snapshots, which means I don't have to worry as much about accidentally deleting, or over-writing a file and losing data.
I don't see anything wrong with using a main machine that's up 7x24 as a NAS, don't buy things for the sake of something, but I'm worried about your reliability and bitrot protection. (Yes, it happens, I've seen it first-hand thanks to ZFS).
I have external parity for everything with rsync. I don't have any storage that isn't matched with external backups. Fortunately I've never seen any bitrot or had a drive fail on me since I tend to replace them before they do. And yeah, I could add in some extra HDDs for RAID, but this is more a personal choice because of how much I hate SATA.
Filesystems like ext4 are because I value boring and rock solid stability over semi-experimental status of modern filesystems with all the features.
If you were to replace your current drive with a new larger one and your RAM or your SATA controller silently corrupted the files how would things play out? Would these corrupted files propegate to the backups?
Ext4 is battle tested so I understand your reasoning. I think you just need to figure out a way to detect silent corruption and a way to snapshot your files in case they do get corrupted.
From experience I can tell you that it is an absolute pain having to manually sort through a bunch of files trying to detect which ones are corrupt and which ones are good.
I suppose I couldn't guarantee I haven't, but I keep a total backup, and other backups based on file lists. I keep a record of every file of importance that I've ever written/modified/interacted with, and those lists themselves are also git managed.
I compulsively look at the rsync backup running to see what's going to transfer, and typically do a dry-run first. That's no perfect solution, but it works for me. If I see something odd I don't quite remember, I check the hashes between both drives, and every time it turns out it was just a path change since I try to keep my data as organized as possible.
rsync arguments are just the plain `--archive --acls --xattrs --verbose` and depending on which backup I'm doing, `--recursive`, `--delete`, `--files-from=`. Nothing other than vanilla at all.
The cables and how they have to be managed/positioned in a case. I've had panics over a drive failing when it was just the cable after I'd already replaced the drive, restored data, and ran tests on the old drive that turned out to be fine each time.
rsync does not protect you from bitrot unless you have some script that specifically collects the rsync log and tracks when existing files change, and you somehow know every file that should not have changed and get alerted that a file changed when it shouldn't have.
Even with that, that is still just 2 equally authoritative copies, without any way to know which one suffered the bitrot.
Ok the tie-breaker data could come from history. Yesterdays log and the day before gives you the extra data points to say which side changed. As long as the log itself is somehow above reproach.
Except this script is a myth anyway. You would still need to have something where you tag files as "this file shall never change again, so if it ever appears to, tell me so I can ok or reject the change." and you would have to actually do that tagging and reviewing.
It's more the single drive storage layout that is an issue in your case. No resilience to a single disk failing and you are risking all your data if your hardware starts silently corrupting data. This is assuming all your backups are done using the data on the single hard drive.
If you want to keep your setup simple I would consider the following:
- Keep a list of file hashes and check your files against this list periodically.
- Use some type of backup program that supports snapshots. Restic is a good choice.
- Get a hard drive the same size as the one you have and use SnapRAID to manage file integrity or set up a two disk mirror using btrfs.
I still pay for snooty, and the reason for that is that when a disk goes bad (not if; when) I pop its tray out, replace the disk, pop the tray with the disk back in, click a couple of widgets, and that’s it. I know it will be rebuilt properly.
(And I know I have to do that, because when the disk fails it beeps and lights a led near the bad disk)
It’s easy to build a NAS such as the one described in this article, but in the long run, data loss is significantly more likely.
Also, any guide like this that doesn’t guide you through “disk 3 failed, this is how you safely replace it” is imho incomplete, even if it doesn’t go through telling you how you know a disk has failed.
That is kind of exactly how zfs works though. The guide isn't complete, sure, but "rebuilding" the array is just replace the disk and run a single zfs command.
That’s comforting to know; that wasn’t true in the past for ext4 over lvm (is it true now?).
But what is that command? And how do you know which disk has gone bad?
I am sure I can get an answer from Google / Claude / ChatGPT, but a guide is incomplete without it - and the failure report should be active like a beep or flashing hardware light - I typically log into my NAS only a few times a year. A motd or other banner isn’t sufficient.
I used to look after storage arrays for VFX places.
everytime I saw lvm I inwardly sighed. The docs were terrible, almost as bad as MDADM. snapshots were for a long time unrecoverable. You'd then have to work out what pattern of LV you had, was it a suprise raid0? or a misaligned raid1?
zfs is a night and day compared to LVM/mdadm, two tools, rich help, the man pages are reasonably good, and once you understand zfs vs zpool, you're usually good to go.
man zfs, man zpool, cron, your mta of choice (I like dma) :)
schedule a zpool scrub every month, send an email if it finds errors. zpool status will show the errored drive. zpool replace will initiate the drive replacement.
ZFS handles drive failures more robustly than anything else. There's a reason synology uses mdadm under BTRFS instead of the built-in BTRFS RAID features, and mdadm operates at the device level. That means that to replace a drive, mdadm has to rebuild the entire drive while zfs will only rebuild what's actually in use.
> Also, any guide like this that doesn’t guide you through “disk 3 failed, this is how you safely replace it” is imho incomplete, even if it doesn’t go through telling you how you know a disk has failed.
To be fair, the "beeps and lights a led" part is very important in case of a dead disk in an array. The only time, I disrupted a service in production for longer than a few hours was when I didn't get the email of a degraded raid 1 array and the second disk died a few weeks later.
I run conceptually the same thing, but on FreeBSD instead of Debian. Different set of trade offs; not per se better or worse, just different, and in the end works just as well.
I found cockpit to be incredibly useful for managing NAS and VPS. Even though any configuration is not an issue with LLMs, having a dashboard with toggles, formatted logs and such is helpful, and I can even run shell commands on phone without ssh. In the NAS case, you can easily see the results of backup/maintenance cronjobs, manage samba (which is a huge pain in the ass to get to work properly on iPhone), and monitor disk usage
> and I can even run shell commands on phone without ssh.
How do you do the authentication to cockpit, basic HTTP auth in front or something? I've always ended up making my phone SSH-capable rather than dropping down to anything else but public/private keys for authentication to my servers, guess I have a bit of conspiracism when it comes to that.
I'd love reading some blogs about what's the best usage for actually minimal computers, any suggestions?
It always gets me how the world of self-hosting is usually introduced from claiming that you can start giving a second life to a Raspberry Pi or a forgotten laptop, and suddenly the next blog you read calls "minimal" a beast machine meant for racks and semi- or professional environments.
Bought a ThinkCentre M910q with an internal SSD and 16 GB of RAM for €200 a couple years ago... Right now I got it chugging along with TrueNAS + 2 USB disks in ZFS Mirror (sitting in a closed cupboard so no chance of cable disconnections).
For me, "minimal home server" means a small computer that fits on a cabinet on the living room, is practically silent, and has a very small power consumption profile (less than a decent Hetzner otherwise the cost wouldn't be worth it). I have a mini-PC in mind, but people think of Dell PowerEdges. Even if given for free, I would never install at my home a PowerEdge for a home server.
I guess it must be the difference between living on a flat vs a 2 story house :-)
I love these machines. I have 3 of them and an M720Q. Needs changed so only one of each is actually in use, but they are so low profile and quiet that I'm considering deploying the other two at home just in case I ever get the urge to play with K8s or something. The only thing I'm missing is a splitter so I can use one plug socket instead of three - not sure if something like that exists.
One nice little fact about the M910Q is that they are rated to maximum 32GB but I have two of them with 2x32GB no problem. Recent RAM prices and forever regret is the only reason the third one doesn't have the same. The M720Q only takes 32GB though.
So it has the potential to even look and work like an actual NAS! That is so cool. Sad state of affairs that 64GB seem to be around €400 nowadays, that's actually double of what the whole machine cost. I'll wait for the AIpocalypse and come out later to buy some sticks.
Building a NAS from scratch is really fun! A small hinderance, but definitely part of the fun as well, is the lack of a "complete resource" on the topic covering how to do every single thing you need to do. Part of the point of my blog post is actually to bring to the internet yet another opinionated NAS setup "guide" (eventhough I would hesitate to call it a guide, but if I ever had to do the same thing again I would definitely read my own post first).
> I am creating a RAIDZ1 (RAID 5) zpool. That means 1 drive redundancy in-case of failure
A friend once told me that RAID5 has a high latency cost, because every Write requires a Read to update the stripes across all drives, and while this made sense when drives were expensive, nowadays you might as well do a RAID10 instead, and trade space for latency.
It's complicated, but https://jro.io/capacity/ actually walks you though how it works, towards the middle of the page.
```
Unlike traditional RAID5 and RAID6 implementations, ZFS supports partial-stripe writes. This has a number of important advantages but also presents some implications for space calculation that we'll need to consider. Supporting partial stripe writes means that in our 7wZ2 vdev example, we can support a write of 12 total sectors even though 12 is not an even multiple of our stripe width (7). 12 is evenly divisible by +1 (3 in this case), so we don't even need any padding. We would have a single full stripe of 7 sectors (2 parity sectors plus 5 data sectors) followed by a partial stripe with 2 parity sectors and 3 data sectors. This will be important because even though we can support partial stripe writes, every stripe (including those partial stripes) need a full set of p parity sectors.
```
RAIDZ1 will add more stress to the consumer ssd's as to enterprise one due to surprinsingly different hardware. Consumer ssd's life times are calculated on a 8 hour shift (the tbw) and enterprise are calculated on 24/7.
Please - dont - use - consumer ssd's - with zfs raidz1
This is pretty much what I did, except that since I use Proxmox I used that to manage the volumes and then just did a pass-through mount to an LXC that runs Samba - that way I get a decent, simple GUI to monitor and manage volumes and can do user management inside the LXC.
I really, do not want to have a big server rack lying around at my house. I have a couple of SSDs that are not being used and 3 USB HDD drives. I also have a PN50 that I am not using as well.
I am thinking of buying a USB bay with 5 SSD slots in it and then 3 HDD drive.
My use case is very, very cold. It is mostly just readonly data with some rclone sync every week.
Does anyone have a suggestion? I am pretty much relying on ZFS to do all the redundancy for me.
10 months ago I've built a Raspberry Pi 5 mini NAS with 3 SSDs connected through RADXA Penta Hat. It runs OpenMediaVault, system is on the SDcard, other data are on the SSD ZFS pool. I have autosnapshots enabled, and I tune compression methods on specific datasets (lz4 for almost everything, zstd for file backups).
We use it for storing (backups, media files), playing (PaperMC), and watching (Jellyfin). I can only complain about the lack of hardware decoding in Raspberry Pi 5. Jellyfin loads CPU much if I enable transcoding so it's always disabled. If I knew this, I'd consider a cheaper and faster, but less popular, RADXA machine. Storage is fast enough for me, rsync and Samba speeds are usually limited by my network. PaperMC also runs without a hitch, thanks for asking!
As I didn't have high requirements for the machine, I also considered USB Bays at first but they wouldn't go well with ZFS.
USB/SATA adapters are incredibly flaky. Some don't support TRIM, some have major firmware bugs, many overheat during long data transfers and disconnect from the host. I'd strongly advise against using them for any important data.
I'd probably just buy a cheap prebuilt PC with a big case, or surplus enterprise tower. You can get cheap LSI PCI cards and do miniSAS to SATA if needed.
The biggest concern I'd have with USB is power delivery to the hard drives, but I haven't even done the napkin math so maybe it's fine. The SSDs seem like they might be a waste of money. USB hard drives have a poor reputation, but I don't have a ton of experience to say how much of that is deserved. On a practical level, I'd also be concerned about knocking the cable out.
"It is important to regularly monitor the snapshot’s storage usage. If a snapshot reaches 100% of its allocated space, it will become invalid." [1]
ZFS snapshots use Copy-on-Write differently and have no such allocated space limit. Thus, we can do interesting things like snapshotting a file system after an OS is installed, and then roll back to that snapshot upon an OS upgrade failure, or even clone a new file system from that snapshot to have a different copy of the OS.
Essentially, the nature of the snapshots are different in both.
Just yesterday I was brainstorming with ChatGPT about this. I have an ancient QNAP plus a slightly less ancient NUC running PiHole, Wireguard and other services. Both need replaced, so why not combine them?
I don't know much about ZFS, but it sounds like I need to learn. Docker may have conquered the world, but I plan to stay with LXD for services.
The one thing I take issue with: an appliance like this runs 24/7. It should be low power and fanless. A processor like the N100 seems like the obvious choice.
I run zfs as the storage pool for my incus (next lxd) services. It is the ideal fit. Here is a list:
- Instant, zero-copy container cloning from images via Copy-on-Write. If you boot a new image like the existing ones it's seconds.
- Atomic, millisecond-level instance snapshots regardless of storage size
- Block-level container migration using native 'zfs send' and 'zfs receive', very short command lines and seems to work perfectly.
- Granular dataset nesting (every instance, image, and custom volume gets its own ZFS dataset). You can see every filesystem even on the host.
- Transparent, inline data compression (LZ4/ZSTD) enabled automatically per dataset. For services that don't change much, you might as well use a compressed image to make them even smaller.
- Mirroring / Raid
- Sub-volume sharing and direct management via native ZFS administration tools. If my home directory has a build area and a million files, I can just save time and put my home, pre cooked into a new machine and not copy or even rebuild on my new machine.
- Dedup keeps blocks with the same data as a reference. This costs a lot of memory and has not saved much for me as a lot of my images are similar and already shared I think, but it's cool.
The N100 supports "In-band ECC" (IBECC), which uses regular non-ECC RAM at the cost of less available memory and a 10-20% performance drop. It’s unclear how well it works, and almost nobody uses it.
Can you explain a bit more about this. Would this help in a single server to facilitate backups? Or does this need the the NAS and the Proxmox Backup Server on a different system?
My homelab "NAS" is unfortunately just a few USB based external disk enclosures, externally powered. Is there no way of making these reliable ? Every so often (maybe once a month) I get timeout errors or some kind of issues with at least one of the disks.
I've been thinking about setting up something like this for a while. I have a Broadwell dual socket Xeon workstation that I'm going to upgrade to Proxmox. Would it be reasonable to run something like this as an LXC or VM or would you put it in the base kernel?
I made 2 Raspberry Pi5 NAS's. one with 4x 6TB and one with 4x 12TB.
Used mergerfs and snapraid, and a simple NFS share. Absolutely perfect for Proxmox backups, our pictures, media, etc, etc. No fuss, easy replacement of drives without needing to keep drives of the same size around.
I'm good with ZFS, have been for years now managing storage for $COMPANY. And I still freaking love using zfs send/receive with proxmox ;-)
But for most at home stuff: mergerfs and snapraid are just more logical.
if you are after quietness and "power" then an old workstation is a great bet. They normally have space for at least 4 fullsized HDDs, the more modern ones have lots of PCI space for nvme-ssd cards (for space) and some have lots of lanes for speed
They also tend to come with SAS/whatever remote nvme is called/SATA expnasion options
The down side is they are not as space efficient, they also tend to have 60-120watt CPUs, so expensive to run
I'm sure there's better options now but the HP ProLiant MicroServers (used).
They support ECC ram, 4 caddies, one extra PCIe slot, and to my knowledge you're not cpu limited for a zfs file server usecase.
Keep in mind though, all you need is linux* support, iDRAC, ECC if you're a snob, and drive bays ... and that's basically any free server.
In my extremely opinionated opinion I would only get used enterprise server gear, because a zfs file server will just work unless hardware fails. And a UPS.
*ZFS will be a more natural choice on FreeBSD. It's better documented, and will meet Linux 1:1 in hardware compatibility for this.
Agreed. I have used a HP N40L ProLiant MicroServer since 2013 as home NAS and Time Machine backups via samba. Rock solid hardware, incredibly expandable, and today runs FreeBSD 15.1 with ZFS. Additional hardware modifications include; CD-ROM replaced by two 3.5" HDDs on mounts (now six HDDs of 10TB each in ZRAID1), a SAS card to add two mirrored bootable SSDs underneath CD-ROM drive space, a 2x 2.5G NIC (limited to 4GB/s slot) for dedicated NFS link, while additional internal SATA and external SATA ports unused. Next: replace PSU fan with quieter Noctua fan.
I'm still running an old Gen 8 MicroServer. Modern drives can actually saturate the SATA controller, and because it only has a single PCIe slot I can't add both a 10Gb NIC and a storage controller - I went with the 10Gb NIC.
It works well enough though and has lasted me over a decade at this point. 16GB DDR3 ECC, an old 4 core/8 thread Xeon, 4x14TB drives and the Mellanox NIC.
Not strictly a recommendation, but Terramaster is a good brand to look at if you want Synology-shaped hardware which can run TrueNAS or Proxmox or any flavour of Linux you want.
Along with various other devices (including a large Synology which I wouldn’t buy today), I run Proxmox on a small two bay+two nvme Terramaster. I have a bare bones Ubuntu LXC running Samba configured for Apple Time Machine, an VM running Scrypyed, and PBS for Proxmox backups. Nothing on it is critical so I don’t bother with any storage redundancy.
I have the F8 Plus, great little unit. It did need a BIOS update when I first got it to enable Proxmox/other OSes to work properly.
Recently replaced the internal USB boot drive with a small NVMe USB enclosure; using a 90-degree USB connector and using a dremel to sand away an opening for the cable to come out so I could mount the enclosure externally.
It's a horrible idea likely, but I have an ancient old Dell PowerEdge R510. Probably sucks way too much energy, but it does what it does and the price of SSDs have skyrocketed so I'm not touching it.
A friendly reminder since it's probably relevant: when's the last time you tested disaster recovery for your own setup? If you haven't verified recovering it, you don't have an offsite backup.
I had a cloud backup (Backblaze B2, using rclone and encryption) for my home NAS. Some unlucky drive failures later, I was getting ready to recover from my Backblaze backup... and then I couldn't find/remember where I saved the rclone encryption backup.
I lost all data in that NAS, including irreplaceable personal/family photos, due to that mistake. My lesson to share: please verify you can indeed recover from backups.
One alternative for those who don't want any of the major NAS vendors, just use RHEL10. It's free up to 16 licenses, it's ultra stable, cockpit is a very mature gui for a lot of maintenance tasks.
It's a set and forget OS that will run for years without requiring your attention. But these days it has decent container support for hosting services on.
The problem with RHEL is that the free version exists at the sufferance of IBM, a company not well known for being motivated by keeping tech enthusiasts happy. I would use Debian, personally. Not as long of a support cycle as RHEL, but still quite stable and no possibility of corporate rug-pulls.
its probably dumb but what i do is use proxmox with a debian VM with docker installed and each docker container gets a virtual disk i make on the proxmox host. obviously more involved than running unraid or truenas but i think the complexity underneath is simpler.
Thats perfectly reasonable. Double isolation but that isn’t too costly.
I’m talking more about people doing the same as you except they’re linking the VM storage to another vm (truenas) over network despite it all being on same host. Think it’s mostly because people don’t want to deal with zfs via command line
To this day I still use a ZFS array for my critical backup storage. I've migrated over to using snap raid + mergerfs for my larger Linux iso storage array. Simple enough and I can pull a drive on it's own without any other stuff.
I'm hardly a sysadmin and I think Unraid is great. My usecase is some Docker containers and media backups. The UI gets the job done well, whereas Synology tries to dumb things down and make things pretty.
this is really cool. I've been dealing with an aging Synology nas and this is something I can pick up, evaluate performance and how safe it is to serve as home for my data.
I killed 2x 4tb Nvme drives on Pciexpress with ZFS raid1 under proxmox, got them replaced, tried again on both pci-express and usb adapter kept getting io problems there too with zfs, drive gets disconnected during scrub or high load, tried already kernel or power management options for pciexpress and usb without sucess, managed to reproduce with high load.
or proxmox ( especially community edition) is fucked or zfs is still unstable, surprisingly with different drives on different interfaces I obtained same results, didnt swap proxmox for clean os, that might being some changes since modules and options would be different i guess. ( drives firmware etc are fine and performing well when inspected with proprietary windows tools)
just search zfs nvme pcie or usb problema or disconnect and you see similar stability problem for different users in different cases (os, drives etc), but also unraid and others, maybe somewhere is rithe right combo of options/glitches but didnt find it yet
Yeah no. When it comes to backups and data storage, I would rather use a proven reliable system that’s been used and tested by millions of other users, keep these hacky stuff for your hyperland set up.
Not really, zfs is part of it, but not the whole picture, you as a user you need more than just a file system.
Truenas for example add replication, external cloud sync, gui (yes it’s important not to wipe your data with wrong command), HA, custom caching, containers to extend ot with other apps, raid-z expansion with openzfs, and other for monitoring, user management and a lot more.
Other solutions also add more than just a zfs with ssh access.
Run ZFS backed filestore on FreeBSD, have migrated it to/from Debian. At work and home, not petabyte scale but certainly multi hundred terrabyte. Over 15 years, on over 50 hosts/NAS/SAN instances, different hardware.
Run ZFS on Raspberry Pi, on home builds, on Intel, on AMD, on other ARM chipsets.
I think you're over-stating things. Debian is fine for this. I do think FreeBSD is a better platform for myself.
The code bases adhere (modulo ZFS version numbers) to a spec and you can safely migrate the pools between OS. I've done it multiple times both directions.
You can not do this with BTRFS and other Linux things, I consider this feature of (Open)ZFS a killer-context for me: It's OS portable. I wish Mac OSX hadn't walked out of the room when Oracle went legal.
There is actually a btrfs driver for Windows [0]. I've used it a few times before, and it works surprisingly well. You probably wouldn't want to use it for any serious work, but that's not because it's technically flawed, but more because it isn't extensively tested or commercially-supported.
Yet everyone is (again) lost in the details and missing the big picture, which is Linux is doing its best to rat fuck OpenZFS at every opportunity, the last of which was the elimination of write_cache_pages in 6.18 behind the GPL iron curtain a mere few months ago.
I don't know about you but I don't want to build my file storage atop hacks on top of more hacks. The kernel has made it clear non-GPL code is not welcome. Struggles will continue in perpetuity. There are better options.
I understand and I agree from an experience point of view it felt very unstable on debian and proxmox that is debian based.
I wanted to share this experience too as a warning to users investing time and money and possibly hitting instabilities that can cause raid problems and data loss. Don't know why my comment got downvoted, if I knew about this I would have handled things differently.
I usually recovered the pool thanks to other disks being fine, but beside zfs being super cool in terms of features and flexibility at the beginning it actually felt unreliable and I would not suggest it neither!
as mentioned it is probably more stable on other families but I didnt experience that yet.
would freebsd be the most reliable? or which one would have the most reliable zfs module state?
do other solutions like unraid or truenas or similar use zfs the background?
> Running ZFS on anything but Solaris/Illumos/FreeBSD is asinine.
> ZFS is a permanent second-class citizen on Linux
Linux is the primary target of OpenZFS [0] [1], and has been since 2020 [2]. It may not be supported by the Linux kernel developers, but it's supported by the ZFS developers, and that's all that really matters.
> I don't want to trust my data to some half-assed out-of-tree solution that may or may not break in a week.
Sure, it's an out-of-tree module, but that doesn't mean that it will randomly break all the time; it just means that you may occasionally need to wait for a new OpenZFS release before upgrading your kernel.
> FreeBSD ZFS support has matured and is outstanding.
Agreed, but Linux and FreeBSD both use the same ZFS [3], so I don't really see how the ZFS in FreeBSD can be better than the one in Linux. The tooling and install procedure is certainly better on FreeBSD, but the actual filesystem code is the same (and is probably slightly more robust on Linux since that's going to be where most of the testing occurs).
If you run ZFS with an LTS kernel you're pretty much fine. Yes new Linux releases will break existing ZFS releases - but the LTS tree is in support for long enough that this is never an issue.
I am a BSD user yet I think this is largely irrelevant in the scope of a NAS which is the conversation we have here. Even in the cases bleeding edge hardware support is relevant, the latest Linux LTS kernel has probably wider hardware support than the freebsd-current at any given point in time anyway.
FreeBSD is a great choice, but there is no need to invent silly reasons to justify using it.
Having built a nas recently (luckily before ram/diskpocalypse) there seems to be a huge difference between making a small nas (say 4 disks) and a big one. With a small one you are essentially building a pc, and the differentiating factor is the software. Mobo, memory, case, PSU… buy consumer (cheap) stuff and you’re ok. Above that you have to start buying very expensive business oriented stuff,disk enclosure, better psu… the costs increase more that linearly , especially if you don’t live in the US and can easily get used stuff from eBay.
And that stuff is often not just more expensive, but uglier and noisier. I ended up making my own “enclosure”
Not a great time price-wise to be building a NAS, but I have been doing so for the last two weeks. Inside a Jonsbo N6 case, which is pretty nice with an 8x SATA backplane and drive bays (unlike the earlier Jonsbo variants).
I ended up on shucking 4x the 14 TB WD Elements Desktop. They contain helium drives, the WD140EDGZ in my case, and are about a third cheaper than 4x the 12 TB WD Red Plus drives (which are air-filled). The shucking was easier than I expected too, and the performance seems very comparable. The warranty is a definite downside (European, so no Magnuson-Moss), but I think I can even get them back in their enclosure should they fail during the 2-year warranty period.
I've put some second hand 256 GB M.2 SSDs in there as boot drives. It was a bit of a struggle to get it to work in a way that failure of one of the drives doesn't hold up booting, combined with LUKS, TPM keys and ZFS on root. Learned a lot about systemd-boot which I have never used before, but feels a lot saner to me than grub ever was. So now I have a large script which debootstraps a Debian based NAS into being.
I noticed that there are a lot of ZFS myths and cargo culting. For example TFA mentions ECC RAM, which in some circles is a must-have because ZFS would wreck your pool during a scrub otherwise, which is a myth. It's also very expensive, especially this year. You also don't need much RAM for ZFS, L2ARC doesn't use much RAM at all, to name a few others.
Still doubting about setting `dnodesize=auto` (which is the default), because there are some horror stories about that [1]. And it seems impossible to find a cloud storage provider with reasonable prices that supports `zfs send`. Rsync.net upped their minimum order to 10 TiB recently, which is far too much for my use case.
[1] https://github.com/openzfs/zfs/issues/11353
[2] https://www.rsync.net/products/zfsintro.html
> Not a great time price-wise to be building a NAS
That under-states the matter. It is a terrible time, price-wise, to build a NAS.
I'd almost rather have no AI whatsoever and have storage 1/10 the price of pre-AI times.
(If there were a magical choice between having AI and significantly more expensive storage, and having no AI and some program to dump that investment money into getting and somehow leveraging significantly more available storage, that is.)
> I'd almost rather have no AI whatsoever and have storage 1/10 the price of pre-AI times.
Almost? ALMOST!?!?!
If you handed me a button that would make it like LLMs had never existed I'd be slamming that button so hard Sam Altman's clothes would spin around.
Return memory and storage prices to normal, undo the sloppification of ~everything, remove all these annoying "features" that are so useful they have to force them upon people, and make scammers actually have to put in a slight bit of effort, all at the "cost" of real human developers, artists, writers, etc. getting paid for their work....
If this is a hard decision for you, you are the problem.
Same. Every time I think about this, I come back to the fact that I was excellent at my job before AI, and even enjoyed aspects of it, so I would be fine if tomorrow it all blew up.
Same, I'd smash it like it was offering me a million dollars. I cannot think of a single positive thing that has come out of the AI boom. It has been a net negative for all of society.
Oh come on! I'll take a few years of expensive RAM in order for humanity to get wide access to something near as makes no difference to the Star Trek Ship computer.
Which is 100% what this has felt like to work with, all spring and summer.
So we're in a slop phase, it'll pass. The first few years of youtube gave no hints that we'd get stuff like Veritasium.
Ignore the slop and use the tools to create something you never thought you could do. That's what it's for!
Same here, the usefulness of the slop machines is quite limited compared to the downsides.
Hey everyone, stop progress! Wolrah has decided that November 29, 2022 is as far as things should go. Because everyone knows before that day, there were no scams, every artist lived lavishly from commissions, and software was flawless and without any extraneous features.
That's quite uncharitable. What are you trying to achieve here?
They (probably) don't want to stop progress (especially unqualified like this) in general. They'd like a world where LLMs didn't come to exist.
And whether LLMs are progress at all remains to be proven.
There's a difference between standing in the rain and being hit by a tsunami, even if it's both just water.
You're telling me.
I had a drive go bad in my 4 year old home-built NAS a few weeks ago, and it cost 2x what it originally did to buy the same capacity drive, and that was going to a grey-market importer on eBay, it was more on "reputable" sites.
I'm doing it anyway and I've found some secondhand deals. Maybe COVID homelabbers are offloading gear?
I built a 24 TB HDD NAS in a 36 bay chassis for around $1000 all-in: mb, ram, rails, chassis, rack, disks, hba, nic
Awful Watts/TB but the plan is to run it and the GPU rig on solar.
36 bays and only 24T of storage?
Can't you get that in a single disk that costs less than a kilodollar?
I obtained a used Dell PowerEdge and just stuffed a bunch of drives in there. It's old enough that the RAM is not overly expensive (DDR3), so it has 192GB of RAM to effectively cache the data, and then 32TB of raw storage. (Of course I regret not spending the extra money to take it to 384GB now.)
The nice thing about the array being big is that I can just RAID 1 it instead of worrying about tinkering with RAID 5, and leave a drive as a hot spare.
PCIe 2.5G NIC for the uplink, and then it can serve over SMB or iSCSI. The main use case for this thing, incidentally, is it just is a caching proxy that holds Docker images, models off of Huggingface, and so on.
36 bay? Why so many?
The 36 bay was all I could find on fb marketplace.
I fell in love with the silly idea of squeezing 10G NIC + 10G writes (to a 2x12 RAID 10 HDD array) on the thunderbolt bus of my 12 year old macbook pro.
Then I got distracted with the server mb and truenas and ZFS.
I would kill (figuratively, anyway) to have no AI whatsoever. No slop machine threatening to replace my job, or turn my job into babysitting its stupidity, and hardware would be reasonably priced? That would be awesome. AI has brought me nothing but downside.
AI has freed most people I know from the tedious job of writing actual emails and birthday greetings - leaving then more time to e.g. wash dishes or clean the floor.
Seriously, if arts and creativity is what sets humans apart from other animals, then AI has almost completely displaced our capacity to even consider doing these activities ourselves. People reach for AI when they should be composing a birthday greeting themselves.
someone at work unironically said they use ai to compose "thoughtful responses" to people, and my immediate thought was that what they are doing is the literal opposite of thoughtful
People were Googling these things. Those who were good with words, didn’t do that before AI and won’t do it today. They don’t have to.
Is it a genuine birthday greeting if you just let AI do the job?
Was it a genuine birthday greeting when you bought a card? Or used an online card? I don’t really see the difference. The point is you thought of them, not the actual words
With AI you don't have to think about them.
Great business opportunity there. Make a free service that sends AI generated birthday notes to people. All the user has to provide is the other persons birthday.
People cannot be relied upon to provide accurate birthdays but nobody would suffer the social faux pas of an incorrect birthday congrats note. Nor would they send it to a spam catching email, but rather are guaranteed to send it to a regularly checked address. They know the persons birthday after all.
You can then sell high quality birthday information correlated to contact information to ad agencies.
Fuck this current internet. So dystopian.
Then the receiver AI can thank the sending AI before deleting.
I used to write birthday emails to my friends years ago. I quit when calendar systems started automatically promoting me when birthdays happened. Remembering someone's birthday no longer seemed like a significant effort on my part and so it wasn't worth the bother since it didn't say as much.
For 90% of birthday greetings, are they ever genuine?
Like seriously, outside of some close friends and family, are you sitting there deliberating over a message for your coworker Steve in the Slack chat or a cousin you barely talk to outside of a birthday and Christmas card?
that was their point
Are you being sarcastic? There's no way you believe people couldn't wash dishes or clean floors because of all the emails and birthday greetings they had to write, or that people have almost completely lost the ability to consider making art.
It's a paraphrasing of the popular 2024 Joanna Maciejewska quote: "I want AI to do my laundry and dishes so that I can do art and writing, not for AI to do my art and writing so that I can do my laundry and dishes."
I believe it was rhetorical. They were trying to illustrate that we've automated tasks that people might find to be fulfilling (drawing, writing) but are still a long way off from affordably automating drudgery.
> or that people have almost completely lost the ability to consider making art
They haven't lost it, no, but there's a lot of financial incentive to stop paying people to do it.
But we now have trillionaires, aren't you happy about that? It means you might be the one some day too /s
I immigrated wanting to become a millionaire. Never found time working my ass off. Now that I get pension and tokens from Musk and Zuck, I can finally try!
I’m running an 8-drive ZFS RAIDZ2 pool. I’m wondering if you know — are the free space recommendations around ZFS cargo or real?
Like I’m already giving up two full drives for redundancy (which saved my ass - I recently had two drives fail on me in quick succession — both SSDs from what looks like an identical batch) but then the advice is kinda saying I need to keep at least another drive worth of space free for the pool to perform well and not crap itself. That hurts with current prices for sure.
> I’m running an 8-drive ZFS RAIDZ2 pool. I’m wondering if you know — are the free space recommendations around ZFS cargo or real?
I'm not entirely sure, but it seems to me that free space (and the 20% reservation) is mostly a proxy for fragmentation, and you can therefore better look at fragmentation directly. That would mean that if you mostly store large files, there shouldn't be a lot of fragmentation even at high utilization. The whole "ZFS changes allocation algorithm from 80% usage on" is something of 10+ years distant past, and lots of things around the allocator have been improved. It's also something that probably isn't too different from the performance of other filesystems at high utilization, so it shouldn't be exaggerated.
> It's also something that probably isn't too different from the performance of other filesystems at high utilization, so it shouldn't be exaggerated.
Well, other filesystems can defragment.
ZFS will auto-degrade performance if there isn't enough headroom. In addition, if you use SSD, you also want the headroom, because otherwise you end up writing and rewriting on the same small empty space which kills SSD (unless you have enterprise SSD, which have built-in headroom).
Not sure why I'm being downvoted for being factually correct (and also, why I can't edit my above comment).
ZFS will definitely degrade write performance gradually from 90% utilization, and will hit a stronger cliff at 95%+. Same with SSD, using a consumer SSD above 90%+ utilization would rapidly degrade its lifetime. The effect will be smaller for very large pools and very large files, but the effect is stil there.
I'm repeating this so that people who set up these drives know what is actually going to happen.
> Not sure why I'm being downvoted for being factually correct
Maybe because your statement about SSDs is incorrect. Consumer SSDs have spare blocks too, and wear leveling prevents the scenario you're describing.
I appreciate the reply, but my warning consumer SSD is not incorrect. Perhaps I should have clarified that While they typically have a small overprovisioning of 7%, enterprise SSDs have close to 4 times as much, exactly to allow more writes (which is very relevant if running a ZFS raid on the drive). Keeping 10-20% headroom will prolong your SSD's life significantly, and increase performance of ZFS.
Go ahead and let them try consumer SSD's. It seems to be a lesson everyone has to learn with ZFS once.
For home usage, if you have backups raidz1 is fine (just do an incremental backup at the first sign of trouble). If you don’t have backups, then you probably shouldn’t be running a NAS in the first place.
The risk of z1 is that if you get a read error during resilvering, that data is permanently corrupted. The odds of this happening go way up the larger your individual drives are. This is why I chose RAIDZ2 for my NAS. I've had to resilver 2-3 times over the past 10 years, and never lost a byte of data.
In the Linux ISO community the math is different because none of the data is essential, however there is a lot of it. Backing it all up even one time to an off-site location would be a 2x expense for a hobby. So 3-2-1 is hard to justify.
Instead the math can be different e.g. it's ok to lose some data but not all. Therefore you might prefer the unraid approach over zfs where losing more than parity doesn't kill the whole pool.
I'm actually in this boat right now. I have 2x26TB drives in a 4 base nas (two empty, plus 3x M2 empty) and I'm trying to decide the best way to set them up. I have around 3TB of backups that I care about as backups (but they don't need to be online, just archive backups at this point), and around 5TB of media that can be replaced.
I don't want to lose 26TB just for a mirror, and I have a spare 8TB USB HDD. I'm torn between unraid and just Debian, and I'm torn between just two separate devices and one RAID 1 partition one RAID 0
For most people snapshots are enough backup. It is still useful to have an offsite backup, but realistically fire is rare enough that you can risk it, and that is about the only risk most people have (you can have your NAS in a location likely far from where a fire might break out to reduce the risk farther).
Fire, malware, accidental deletions, capricious RAID controller (pre-ZFS). And that’s only the stuff that happened to me. Add power surge, theft, correlation in SSD failures (eg power on counter overflow firmware bug), damaging the array while moving, etc.
Malware, accidental deletion, RAID controller issues - all things ZFS with snapshots are immune to. You can decide if the other worry you or not - the risk is not zero, but it may be acceptable. Your backups are all subject to similar issues.
Snapshots don’t protect you from malware. I bet 99% of home users use the same credentials on all their machines, once a malware compromised one, the others are compromised within seconds.
[edit] also snapshots aren’t really workable for large files. Remux a movie file and now it occupies twice the space.
The middle ground I go with is to structure the pool using datasets and zvols, and then periodically backup the critical ones using zfs send. For example, I can live without my movie collection, but I cannot afford to lose personal photos.
This isn’t black and white. You might have huge amounts of non-essential data that aren’t worth the cost of off-site backups, but worth the cost of an extra disk of redundancy to lessen the risk. Even when you do have backups, it will reduce the risk of extended downtime (and possibly egress costs) caused by having to restore large amounts of data from backups.
I don't think they are. I've already filled a ZFS pool almost to the brim with no adverse effects.
My experience has been escalating fragmentation past 90%, along with lower performance owing to the fragmentation.
> combined with LUKS, TPM keys
Does it work? Server can reboot and use TPM to unlock rootfs? What about /boot - encrypted and tamper proof? Resists evil maid attack?
Rabbit hole, I know, but so fascinating if you solved it all :)
I run my cheap hosting box with a cleartext boot setup that uses ssh to automatically grab the key for the real root from my home server (or an alternate at my MILs house). Using FreeBSD, but similar concepts.
A previous hoster once gave me someone else's drives without wiping them. I don't want random customers snooping around on my data if a similar mistake happens with my disks.
For home use, I run without disk encryption. If I ever need to do data recovery, it's not going to be possible with encrypted disks and one point of a centralized NAS is to have stable long term storage.
If you are worried about someone breaking in to your house and replacing the bootloader while leaving your drives in place I probably wouldn't use the TPM auto unlock even if in theory secure boot should be able to handle this.
But in reality that will never happen and the only actual attack you need to be worried about is junkies breaking in and flogging the drives on facebook marketplace. For which, this level of security is fine.
> junkies breaking in and flogging the drives
For this you dont need TPM. Just a LUKS key in rootfs /etc. He said TPM :)
I've been thinking about building my own NAS as well. Mind sharing how much did you pay for those hard drives, and what motherboard did you choose?
https://diskprices.com/ is great for this
You have SATA or SAS to pick from. The CPU requirements for a storage server are not high. On a typical ATX board you have motherboard SATA and can put SAS controllers in the spare PCIe slots.
My first "NAS" was two 22TB hard drives in a ZFS pool on my motherboard SATA
There should be more sites like diskprices. So refreshing to see no styling, no cookie banners, just pure information.
ATA (SATA), SCSI (SAS), and NVM (NVMe).
no. just sata & sas. There is no nvme spinning drive and flash loses bits just from sitting, it's not archival. flash is good for working not for storing.
It was €329 per hard drive (through a reputable store), and I chose the ASRock B650M-HDV/M.2 mATX motherboard combined with a Ryzen 5 8500G. Stock CPU cooler, replaced the Jonsbo case fans with Arctic P12 Pro PST LN. I only slightly regret the PSU (MSI MAG A650GL), which could be quieter. Not that it's very noisy, and it's a great PSU otherwise, but I should've chosen one that just shuts the fan down at low power usage.
> 4x the 14 TB WD Elements Desktop.
I recommend staying away from SATA drives (huge consumer rush) and look for SAS drives on eBay, particularly HC520 Helium drives from HGST/WD. Need a SAS3008 PCIe adapter ($20) and a SFF-8643 splitter ($30). No backplane is required. Huge lots of decommissioned drives frequently appear. I bought Qty 10* of HC520 (12TB) SAS drives for $1000 about 3 weeks ago, avg age is about 2.5 years, still well within its rated lifetime.
Yea may be some of the stuff is fear mongering and cargo culting. I was told ECC is necessary for ZFS. When the article was written, it was cheap af (2024) to buy ECC RAM so not much consideration was given to it.
-- (*) I have no idea what to do with it. Anyone has any good idea for using 120TB space? I have about 40TB unused bandwidth in the datacenter, may be host a Debian mirror? Donate storage/bandwidth to Internet Archive? Please contact me, appreciate it.
> I bought Qty 10* of HC520 (12TB) SAS drives for $1000 about 3 weeks ago, avg age is about 2.5 years, still well within its rated lifetime.
That's cheap indeed. Enough headroom for some failing disks too. How is the noise and power usage? I didn't look at SAS drives at all, because my impression was that they're very noisy. I can place my NAS in a closed off room, but it's not too far away and I was afraid SAS drives would be audible through the wall. At the same time, the shucked drive I'm using presents itself as an WD Ultrastar, which comes very close to a SAS drive, and isn't very noisy.
> How is the noise and power usage?
I think they're same as SATA drives, just different interface. AFAIK they have the same physical dimensions and same internals.
This reply is exactly right about using enterprise drives (hdd), and ideally SAS. Note the power usage of SAS vs sata for an identical drive make/model is very small sometimes 0.5 W to 1.5 W at idle (idle power draw is what matters for spinning HDD’s as underload it doesn’t increase much and idle is what you’ll be at 98% of the time). Also enterprise SSDs are much better than consumer SSD‘s for ZFS, and be aware that when you get above the 4 TB size SSD drives frequently use more power than HDD drives (i’m mainly referring to enterprise non-consumer level drives as that’s all I run for 100s of disks across many zfs systems over past 15 yrs).
Also something people forget for home nas or figuring out cost for a nas, the power draw of the entire system times your electricity price you then have to multiply this times 2x to 4x times if your climate requires cooling air conditioning of any kind
I'm also planning a new home server build now and the prices are definitely relatively ass. So far I've spent 820€ on two 22 TB WD Elements HDDs, 375€ on 2x16 GB DDR5 kit, and 520€ on two 2 TB M.2 SSDs (cache). About 1700€ and I still have no server to show for it. Doesn't help that I've been procrastinating on picking the CPU and motherboard.
You assert that ECC RAM being necessary for ZFS is just a myth but provide no justification for why that is untrue.
Is it not the case that if you don't have ECC memory, ZFS could end up writing a checksum that does not match the data if you get a bitflip in just the right (wrong) spot?
Yes, indeed. ECC RAM is better than non-ECC RAM, also for ZFS.
The myth, popularized by a notorious thread on the TrueNAS forums [1], is specifically that ZFS requires ECC RAM, and will do worse than other filesystems without it, because scrubbing will multiply a single bitflip into a failed pool.
A ZFS core developer says that that isn't the case [2]. Here's some more reasoning [3], also about many other myths.
[1] https://www.truenas.com/community/threads/ecc-vs-non-ecc-ram...
[2] https://news.ycombinator.com/item?id=18480016
[3] https://kldload.com/zfs-wiki/myths
Usually if you’re using a NAS you don’t want to lose data, ZFS is not significantly more sensitive than everything else.
But everything is actually quite sensitive.
We’ve accepted lack of ECC because Intel decided it would be a product line differentiator, and serious customers who didn’t want random crashes or to lose data would buy chips with ECC.
It’s actually less of an issue these days because DDR5 has (by spec) some in-line ECC; won’t help with multi-bit errors but its an improvement on what came before.
> We’ve accepted lack of ECC because Intel decided it would be a product line differentiator, and serious customers who didn’t want random crashes or to lose data would buy chips with ECC.
AMD has been allowing ECC on lots of regular hardware for a long time.
People don't tend to buy ECC for desktop use because it costs significantly more (used server ram is/was often cheap... but it often doesn't work in desktop boards), and the performance specs are poor.
My home servers are mostly retired desktops, so they get my old desktop ram and I don't want to pay premium prices for jedec speed ecc ram on my desktops, thanks.
Since DDR5 doesn't include reporting on bit errors (afaik), it likely means much fewer single bit errors, but most experienced errors will be multi-bit. Although, I dunno what proportion of bit errors is on the ram chips and what's on the bus... there's no protection from bus errors.
If there were reporting, you could replace chips with high error rates, but without reporting you'll keep running them until they fail enough to notice.
DDR5 on-die ECC is to achieve acceptable yields in the face of denser process nodes that decrease the reliability of RAM cells. It’s not clear how much of an improvement that is to what we had before, other than allowing for higher RAM speeds. It doesn’t replace side-band ECC.
I know that LPDDR5 has ECC, and not just single bit AFAIK, but if you enable it you lose some memory capacity (the 128 bit bus minus 16 bit for ECC error correction, making it effectively 112 bits)
The only hangup with the myth "debunking" is that the point is that the corruption doesn't happen to a per-disk buffer, but to the in-flight data before it's persisted to your stripe.
Which means all copies of the data will be corrupted. This can be anything from an irrecoverable file to complete filesystem corruption.
But generally, yeah, not any more dangerous than any other filesystem, and ECC used to be cheap so it was a no-brainer, you should have backups anyway TBH if we're being honest about storage resiliency.
What has been debunked is the "scrub of death" issue, on a scrub a bad bit flip would cause an error, which would be copied over with good data -- well it was technically good before. It would be statistically difficult to have a fault on a read, then a clean read, then a second bit flip destroying the data.
Updated link to the post from Matt Ahrens here[1], which is one of the ZFS creators[2] not "just" a core developer.
[1]: https://arstechnica.com/civis/threads/ars-walkthrough-using-...
[2]: https://en.wikipedia.org/wiki/ZFS#2004%E2%80%932010:_Develop...
Had it happen to my PBS backups running on ZFS without ECC just the other day. Turned out to be heat related, memory was fine but during long backups heat was causing bit flips.
> You assert that ECC RAM being necessary for ZFS is just a myth but provide no justification for why that is untrue.
ZFS without ECC is no more risky than any other file system / software RAID without ECC.
As no one owes you an explanation, it would take you five seconds to Google this and discover:
1. It's been disproven, with one of the original ZFS developers chiming in.
2. The original source of the rumor was a forum post that somehow became canon.
> five seconds to Google this and discover
...(after much longer) that it's a rabbit hole with nuance. https://news.ycombinator.com/item?id=18481910
What an incredibly rude comment
Some other things I find useful on NASes:
Install avahi-daemon. Samba will automatically register with it to advertise SMB/CIFS to macOS and Linux clients over DNS-SD.
Install wsdd2 so that your server will be auto-discovered by Explorer on Windows 10+ clients with SMB 1.0 disabled, too.
Your Linux hostname is probably lower-case, but by default, Samba publishes a capitalized rendering of the hostname to NetBIOS and Avahi. If this bothers you, set “host-name=something” in the [server] section of /etc/avahi/avahi-daemon.conf, and set “mdns name = mdns” in the [global] section of /etc/samba/smb.conf.
If you have macOS clients, you should enable vfs_fruit in your Samba configuration: https://www.samba.org/samba/docs/current/man-html/vfs_fruit..... There are some compatibility reasons to do this, but mostly it means you can set the “fruit:model” so that your server has a fun icon in the Finder sidebar.
To avoid the creation .DS_Store files, you can disallow them: https://ryanoberto.github.io/blog/2015/04/01/disabling-the-c.... I think you can also set “fruit:resource = xattr” to store Finder preferences in xattrs, but I haven't tried it.
Although macOS deprecated AFP in favour of SMB years ago (and are slated to remove AFP client support altogether in the upcoming macOS 27), SMB client support in macOS is still pretty miserable. The upcoming macOS 27 is set to drop AFP support, but until then I will continue to run Netatalk side-by-side with Samba. Netatalk also registers itself with Avahi, and macOS will (tellingly) use AFP preferentially to SMB, so clients will talk to the right daemon automatically.
For Samba, add a separate entry for the user home directory because the homes directory default triggers some sort of OCD.
Yeah, the default [homes] share is really not necessary on a home network where you really just want a common share. On my home storage server, I just have a “files” user, and a corresponding, anonymously readable/writable share:
The default [homes] share is still there in case I need it for anything, but it's “browseable = no” so it doesn't confuse visitors.I think it's appropriate to link to my blog post about building such a NAS but with very different technologies: dm-integrity, mdadm and XFS. The reason being that as a C/C++ dev following OpenZFS development closely, I found the focus on ginormous features over general stability and the constant problems due to the SPL/separate page cache too worrying.
Sure, I traded the convenience of kitchen sink (or Swiss army knife for the more charitable souls) ZFS for some initial pain, but I'm very happy with my choice today.
https://world-playground-deceit.net/blog/2025/06/nas-setup-l...
That's what I did after my 15 year old Synology DS1010+ built in USB DOM failed. Put a network booted debian on it via netboot.xyz with zfs and now i get to reuse those 15 year old 2TB still chugging along disks. Fully open source, pretty nice way to keep old hardware going. It's my tertiary backup that wakes up once a week pulls in some open weights LLM models that I hoard just in case and goes back to sleep.
Where can I have a discussion with other developers about storage?
Is here okay?
I want to have physical storage over here, and logical storage over there, and I want to control the mapping from one to the other. I want to talk about encryption, replication, latency. Make "time machine" available on this logical storage. And then I buy some new physical storage, and it joins the story. This physical storage is over at my friend's house for backup. This physical storage is slow and archival. This is a device for writing archival media, and there's a brand new media in it, go ahead and write to it. Show me the health report on all of the physical media, and show me what you've done to protect the logical storage. Graph my usage and make suggestions about when to add more physical storage.
Buying physical storage with power and wifi, and configuring it with a QR code that's on an e-ink display - seems like it should be the most obvious thing in the world, that we should all be really used to doing by now.
What am I missing?
bcachefs is slowly working up to what you're describing - describing what you want and letting the filesystem sort it out. We're basically there for local storage, and other people have been building some nice reporting on top.
Next year (post Rust, because networking code is so much nicer in Rust) will be send/recv, and I think we should be able to make some nice improvements over the state of the art there.
I came to the same conclusion when I built my NAS. Just Nix for the system, zfs for raid and docker compose for any service I might want to run. https://www.splitbrain.org/blog/2025-08/03-diy_nas_on_nixos
This is what I'm hoping to do. I haven't done much research yet, so I look forward to reading about your learnings. (Also to the person who responded to you with their experience and link.)
Thanks for saving me probably some hours...
EDIT: I'm also coming from an old Synology NAS from the ~2015 era.
I'm also using NixOS for my NAS, you should look into quadlets (docker containers managed by systemd) to manage containers with nix.
Same, using NixOS as a NAS, though it kept growing and now I'm trying to share it with other people. I use BTRFS on MDADM though. It's ended up being an all-in-one home server and router now because I was tired of the noisy power hungry rack I had in my closet.
https://HomeFree.host
I'm sure people have reasons for taking these things as far as they do with ZFS, and everything else that goes into what is commonly considered a "NAS." But I've found great convenience in a tight NFSv3 config running from a single high capacity HDD with ext4 on a Linux machine (primary system that's always on anyway while I'm around/awake) making things available to my other devices for the electricity and overhead cost of one machine instead of multiple.
I try to keep my network configuration restrictive by default, so I'm not too concerned about possible security arguments running it from my main machine. I've probably committed some great sin here, but is plain NFSv3 and secondarily Samba (for compatibility) really not enough?
First: Do you have backups of your single high-capacity HDD? That's my biggest worry. What's your plan for when that HDD fails one day, as it will?
The main reasons people go for ZFS and a "NAS" is checksums and data integrity protection, as well as maybe not wanting to keep their primary machine awake all the time (I personally don't).
Then there are useful features like snapshots, which means I don't have to worry as much about accidentally deleting, or over-writing a file and losing data.
I don't see anything wrong with using a main machine that's up 7x24 as a NAS, don't buy things for the sake of something, but I'm worried about your reliability and bitrot protection. (Yes, it happens, I've seen it first-hand thanks to ZFS).
I have external parity for everything with rsync. I don't have any storage that isn't matched with external backups. Fortunately I've never seen any bitrot or had a drive fail on me since I tend to replace them before they do. And yeah, I could add in some extra HDDs for RAID, but this is more a personal choice because of how much I hate SATA.
Filesystems like ext4 are because I value boring and rock solid stability over semi-experimental status of modern filesystems with all the features.
What do you use for external parity? Par2?
If you were to replace your current drive with a new larger one and your RAM or your SATA controller silently corrupted the files how would things play out? Would these corrupted files propegate to the backups?
Ext4 is battle tested so I understand your reasoning. I think you just need to figure out a way to detect silent corruption and a way to snapshot your files in case they do get corrupted.
From experience I can tell you that it is an absolute pain having to manually sort through a bunch of files trying to detect which ones are corrupt and which ones are good.
Misspoke. I didn't mean that type of parity, but just hashes, and a matching drive (in make and capacity) externally for an exact copy.
How do you know you haven't seen bitrot? What rsync arguments do you use to error out when an old, untouched file suddenly changes?
I suppose I couldn't guarantee I haven't, but I keep a total backup, and other backups based on file lists. I keep a record of every file of importance that I've ever written/modified/interacted with, and those lists themselves are also git managed.
I compulsively look at the rsync backup running to see what's going to transfer, and typically do a dry-run first. That's no perfect solution, but it works for me. If I see something odd I don't quite remember, I check the hashes between both drives, and every time it turns out it was just a path change since I try to keep my data as organized as possible.
rsync arguments are just the plain `--archive --acls --xattrs --verbose` and depending on which backup I'm doing, `--recursive`, `--delete`, `--files-from=`. Nothing other than vanilla at all.
What do you dislike about SATA?
The cables and how they have to be managed/positioned in a case. I've had panics over a drive failing when it was just the cable after I'd already replaced the drive, restored data, and ran tests on the old drive that turned out to be fine each time.
rsync does not protect you from bitrot unless you have some script that specifically collects the rsync log and tracks when existing files change, and you somehow know every file that should not have changed and get alerted that a file changed when it shouldn't have.
Even with that, that is still just 2 equally authoritative copies, without any way to know which one suffered the bitrot.
Ok the tie-breaker data could come from history. Yesterdays log and the day before gives you the extra data points to say which side changed. As long as the log itself is somehow above reproach.
Except this script is a myth anyway. You would still need to have something where you tag files as "this file shall never change again, so if it ever appears to, tell me so I can ok or reject the change." and you would have to actually do that tagging and reviewing.
It's more the single drive storage layout that is an issue in your case. No resilience to a single disk failing and you are risking all your data if your hardware starts silently corrupting data. This is assuming all your backups are done using the data on the single hard drive.
If you want to keep your setup simple I would consider the following:
- Keep a list of file hashes and check your files against this list periodically.
- Use some type of backup program that supports snapshots. Restic is a good choice.
- Get a hard drive the same size as the one you have and use SnapRAID to manage file integrity or set up a two disk mirror using btrfs.
I still pay for snooty, and the reason for that is that when a disk goes bad (not if; when) I pop its tray out, replace the disk, pop the tray with the disk back in, click a couple of widgets, and that’s it. I know it will be rebuilt properly.
(And I know I have to do that, because when the disk fails it beeps and lights a led near the bad disk)
It’s easy to build a NAS such as the one described in this article, but in the long run, data loss is significantly more likely.
Also, any guide like this that doesn’t guide you through “disk 3 failed, this is how you safely replace it” is imho incomplete, even if it doesn’t go through telling you how you know a disk has failed.
Is snooty a autocorrect for Synology or some other product?
Indeed. Didn’t notice and it’s too late to edit now.
That is kind of exactly how zfs works though. The guide isn't complete, sure, but "rebuilding" the array is just replace the disk and run a single zfs command.
That’s comforting to know; that wasn’t true in the past for ext4 over lvm (is it true now?).
But what is that command? And how do you know which disk has gone bad?
I am sure I can get an answer from Google / Claude / ChatGPT, but a guide is incomplete without it - and the failure report should be active like a beep or flashing hardware light - I typically log into my NAS only a few times a year. A motd or other banner isn’t sufficient.
> in the past for ext4 over lvm
I used to look after storage arrays for VFX places.
everytime I saw lvm I inwardly sighed. The docs were terrible, almost as bad as MDADM. snapshots were for a long time unrecoverable. You'd then have to work out what pattern of LV you had, was it a suprise raid0? or a misaligned raid1?
zfs is a night and day compared to LVM/mdadm, two tools, rich help, the man pages are reasonably good, and once you understand zfs vs zpool, you're usually good to go.
zpool-replace(8)
https://openzfs.github.io/openzfs-docs/man/master/8/zpool-re...
ZFS got this right. MDADM on ext4 somehow did not.
EDIT: ext4 on MDADM.
How ZFS gets this right: zfs(8) on zpool(8). So, basically, ext4 and mdadm can talk to each other.
man zfs, man zpool, cron, your mta of choice (I like dma) :)
schedule a zpool scrub every month, send an email if it finds errors. zpool status will show the errored drive. zpool replace will initiate the drive replacement.
ZFS handles drive failures more robustly than anything else. There's a reason synology uses mdadm under BTRFS instead of the built-in BTRFS RAID features, and mdadm operates at the device level. That means that to replace a drive, mdadm has to rebuild the entire drive while zfs will only rebuild what's actually in use.
> Also, any guide like this that doesn’t guide you through “disk 3 failed, this is how you safely replace it” is imho incomplete, even if it doesn’t go through telling you how you know a disk has failed.
`zpool replace my_pool disk3 newdisk`
The worst part of either approach is: how do I find disk number 39 out of 48.
ZFS makes this completely trivial except for the "beeps and lights a led" part
To be fair, the "beeps and lights a led" part is very important in case of a dead disk in an array. The only time, I disrupted a service in production for longer than a few hours was when I didn't get the email of a degraded raid 1 array and the second disk died a few weeks later.
smartmontools makes this trivial and can email you which might be better than beeping and lights if you’re not always near your server.
I use that plus smartctl_exporter so I can do metrics and alerting off that.
That’s awesome. But a guide that does not include a howto for alerting (such as the one linked) is incomplete.
smartmontools misses data corruption at the drive layer, as it trusts the drives to report faithfully.
"In hashes I trust"
Does Synology with its beeps and bops catch that?
I run conceptually the same thing, but on FreeBSD instead of Debian. Different set of trade offs; not per se better or worse, just different, and in the end works just as well.
I found cockpit to be incredibly useful for managing NAS and VPS. Even though any configuration is not an issue with LLMs, having a dashboard with toggles, formatted logs and such is helpful, and I can even run shell commands on phone without ssh. In the NAS case, you can easily see the results of backup/maintenance cronjobs, manage samba (which is a huge pain in the ass to get to work properly on iPhone), and monitor disk usage
> and I can even run shell commands on phone without ssh.
How do you do the authentication to cockpit, basic HTTP auth in front or something? I've always ended up making my phone SSH-capable rather than dropping down to anything else but public/private keys for authentication to my servers, guess I have a bit of conspiracism when it comes to that.
I'd love reading some blogs about what's the best usage for actually minimal computers, any suggestions?
It always gets me how the world of self-hosting is usually introduced from claiming that you can start giving a second life to a Raspberry Pi or a forgotten laptop, and suddenly the next blog you read calls "minimal" a beast machine meant for racks and semi- or professional environments.
Bought a ThinkCentre M910q with an internal SSD and 16 GB of RAM for €200 a couple years ago... Right now I got it chugging along with TrueNAS + 2 USB disks in ZFS Mirror (sitting in a closed cupboard so no chance of cable disconnections).
For me, "minimal home server" means a small computer that fits on a cabinet on the living room, is practically silent, and has a very small power consumption profile (less than a decent Hetzner otherwise the cost wouldn't be worth it). I have a mini-PC in mind, but people think of Dell PowerEdges. Even if given for free, I would never install at my home a PowerEdge for a home server.
I guess it must be the difference between living on a flat vs a 2 story house :-)
> ThinkCentre M910q
I love these machines. I have 3 of them and an M720Q. Needs changed so only one of each is actually in use, but they are so low profile and quiet that I'm considering deploying the other two at home just in case I ever get the urge to play with K8s or something. The only thing I'm missing is a splitter so I can use one plug socket instead of three - not sure if something like that exists.
One nice little fact about the M910Q is that they are rated to maximum 32GB but I have two of them with 2x32GB no problem. Recent RAM prices and forever regret is the only reason the third one doesn't have the same. The M720Q only takes 32GB though.
You can also run 2 SATA off of the M2, and someone created a 3D printable enclosure for them: https://makerworld.com/en/models/1280680-thinknas-2x-hdd-enc...
So it has the potential to even look and work like an actual NAS! That is so cool. Sad state of affairs that 64GB seem to be around €400 nowadays, that's actually double of what the whole machine cost. I'll wait for the AIpocalypse and come out later to buy some sticks.
I used this blog post as one learning resource while building my own NAS a while back: https://xyny.art/blog/2026-building-nas/
Building a NAS from scratch is really fun! A small hinderance, but definitely part of the fun as well, is the lack of a "complete resource" on the topic covering how to do every single thing you need to do. Part of the point of my blog post is actually to bring to the internet yet another opinionated NAS setup "guide" (eventhough I would hesitate to call it a guide, but if I ever had to do the same thing again I would definitely read my own post first).
Tangential, but about this:
> I am creating a RAIDZ1 (RAID 5) zpool. That means 1 drive redundancy in-case of failure
A friend once told me that RAID5 has a high latency cost, because every Write requires a Read to update the stripes across all drives, and while this made sense when drives were expensive, nowadays you might as well do a RAID10 instead, and trade space for latency.
Is this still true with ZFS RAIDZ1?
It's complicated, but https://jro.io/capacity/ actually walks you though how it works, towards the middle of the page.
``` Unlike traditional RAID5 and RAID6 implementations, ZFS supports partial-stripe writes. This has a number of important advantages but also presents some implications for space calculation that we'll need to consider. Supporting partial stripe writes means that in our 7wZ2 vdev example, we can support a write of 12 total sectors even though 12 is not an even multiple of our stripe width (7). 12 is evenly divisible by +1 (3 in this case), so we don't even need any padding. We would have a single full stripe of 7 sectors (2 parity sectors plus 5 data sectors) followed by a partial stripe with 2 parity sectors and 3 data sectors. This will be important because even though we can support partial stripe writes, every stripe (including those partial stripes) need a full set of p parity sectors. ```
> and while this made sense when drives were expensive
I don't have the answer to the latency question, but HDDs have shot up in $/TB over the last couple years too. They are once again kind of expensive.
The RAID5 write hole is not present in RAIDZ1 I believe.
RAIDZ1 will add more stress to the consumer ssd's as to enterprise one due to surprinsingly different hardware. Consumer ssd's life times are calculated on a 8 hour shift (the tbw) and enterprise are calculated on 24/7.
Please - dont - use - consumer ssd's - with zfs raidz1
This is pretty much what I did, except that since I use Proxmox I used that to manage the volumes and then just did a pass-through mount to an LXC that runs Samba - that way I get a decent, simple GUI to monitor and manage volumes and can do user management inside the LXC.
I really, do not want to have a big server rack lying around at my house. I have a couple of SSDs that are not being used and 3 USB HDD drives. I also have a PN50 that I am not using as well.
I am thinking of buying a USB bay with 5 SSD slots in it and then 3 HDD drive. My use case is very, very cold. It is mostly just readonly data with some rclone sync every week.
Does anyone have a suggestion? I am pretty much relying on ZFS to do all the redundancy for me.
10 months ago I've built a Raspberry Pi 5 mini NAS with 3 SSDs connected through RADXA Penta Hat. It runs OpenMediaVault, system is on the SDcard, other data are on the SSD ZFS pool. I have autosnapshots enabled, and I tune compression methods on specific datasets (lz4 for almost everything, zstd for file backups).
We use it for storing (backups, media files), playing (PaperMC), and watching (Jellyfin). I can only complain about the lack of hardware decoding in Raspberry Pi 5. Jellyfin loads CPU much if I enable transcoding so it's always disabled. If I knew this, I'd consider a cheaper and faster, but less popular, RADXA machine. Storage is fast enough for me, rsync and Samba speeds are usually limited by my network. PaperMC also runs without a hitch, thanks for asking!
As I didn't have high requirements for the machine, I also considered USB Bays at first but they wouldn't go well with ZFS.
USB/SATA adapters are incredibly flaky. Some don't support TRIM, some have major firmware bugs, many overheat during long data transfers and disconnect from the host. I'd strongly advise against using them for any important data.
I'd probably just buy a cheap prebuilt PC with a big case, or surplus enterprise tower. You can get cheap LSI PCI cards and do miniSAS to SATA if needed.
The biggest concern I'd have with USB is power delivery to the hard drives, but I haven't even done the napkin math so maybe it's fine. The SSDs seem like they might be a waste of money. USB hard drives have a poor reputation, but I don't have a ton of experience to say how much of that is deserved. On a practical level, I'd also be concerned about knocking the cable out.
The big USB racks all have separate power.
If you use LVM you can have a similar pool of storage with 1GB RAM or even less, instead 8GB (ZFS requirement).
Caveat: LVM does not have built-in snapshots like ZFS has.
LVM does have snapshots.
"It is important to regularly monitor the snapshot’s storage usage. If a snapshot reaches 100% of its allocated space, it will become invalid." [1]
ZFS snapshots use Copy-on-Write differently and have no such allocated space limit. Thus, we can do interesting things like snapshotting a file system after an OS is installed, and then roll back to that snapshot upon an OS upgrade failure, or even clone a new file system from that snapshot to have a different copy of the OS.
Essentially, the nature of the snapshots are different in both.
[1] https://docs.redhat.com/en/documentation/red_hat_enterprise_...
You can do those things with LVM snapshots too, that's what they are used for.
Given the current "crisis", I wonder if older off-the-shelf NAS from QNAP, Synology, Asus, UGreen etc. could be re-flashed using FOSS.
A lot of them can, yes.
Just yesterday I was brainstorming with ChatGPT about this. I have an ancient QNAP plus a slightly less ancient NUC running PiHole, Wireguard and other services. Both need replaced, so why not combine them?
I don't know much about ZFS, but it sounds like I need to learn. Docker may have conquered the world, but I plan to stay with LXD for services.
The one thing I take issue with: an appliance like this runs 24/7. It should be low power and fanless. A processor like the N100 seems like the obvious choice.
I run zfs as the storage pool for my incus (next lxd) services. It is the ideal fit. Here is a list:
- Instant, zero-copy container cloning from images via Copy-on-Write. If you boot a new image like the existing ones it's seconds.
- Atomic, millisecond-level instance snapshots regardless of storage size
- Block-level container migration using native 'zfs send' and 'zfs receive', very short command lines and seems to work perfectly.
- Granular dataset nesting (every instance, image, and custom volume gets its own ZFS dataset). You can see every filesystem even on the host.
- Transparent, inline data compression (LZ4/ZSTD) enabled automatically per dataset. For services that don't change much, you might as well use a compressed image to make them even smaller.
- Mirroring / Raid
- Sub-volume sharing and direct management via native ZFS administration tools. If my home directory has a build area and a million files, I can just save time and put my home, pre cooked into a new machine and not copy or even rebuild on my new machine.
- Dedup keeps blocks with the same data as a reference. This costs a lot of memory and has not saved much for me as a lot of my images are similar and already shared I think, but it's cool.
It also surprised me that the author said "4 Cores, Xeon Server CPU can be had for cheap".
But the specs also said ECC RAM and I don't think the N100 supports that.
The N100 supports "In-band ECC" (IBECC), which uses regular non-ECC RAM at the cost of less available memory and a 10-20% performance drop. It’s unclear how well it works, and almost nobody uses it.
> But the specs also said ECC RAM and I don't think the N100 supports that.
If I remember correctly it can in theory but in practice I have never seen a N100 with ECC.
This post mentions https://github.com/wyager/zfs-backup for (offsite) backups. Why would they recommend this over something more established like https://github.com/jimsalterjrs/sanoid? Has anybody tried it?
It's pretty minimal and straightforward written in Haskell. Probably can write one from scratch with Claude now.
What's next!? Let Claude analyse clinic visits!? Oh well...
Rather than 'plain' debian you can put it on Proxmox, and enjoy the bliss of Proxmox Backup Server.
Can you explain a bit more about this. Would this help in a single server to facilitate backups? Or does this need the the NAS and the Proxmox Backup Server on a different system?
My homelab "NAS" is unfortunately just a few USB based external disk enclosures, externally powered. Is there no way of making these reliable ? Every so often (maybe once a month) I get timeout errors or some kind of issues with at least one of the disks.
I've been thinking about setting up something like this for a while. I have a Broadwell dual socket Xeon workstation that I'm going to upgrade to Proxmox. Would it be reasonable to run something like this as an LXC or VM or would you put it in the base kernel?
I made 2 Raspberry Pi5 NAS's. one with 4x 6TB and one with 4x 12TB.
Used mergerfs and snapraid, and a simple NFS share. Absolutely perfect for Proxmox backups, our pictures, media, etc, etc. No fuss, easy replacement of drives without needing to keep drives of the same size around.
I'm good with ZFS, have been for years now managing storage for $COMPANY. And I still freaking love using zfs send/receive with proxmox ;-)
But for most at home stuff: mergerfs and snapraid are just more logical.
Can anyone recommend a good server for a homelab to use for a storage purpose like this?
Depends on what you are after.
if you are after quietness and "power" then an old workstation is a great bet. They normally have space for at least 4 fullsized HDDs, the more modern ones have lots of PCI space for nvme-ssd cards (for space) and some have lots of lanes for speed
They also tend to come with SAS/whatever remote nvme is called/SATA expnasion options
The down side is they are not as space efficient, they also tend to have 60-120watt CPUs, so expensive to run
I'm sure there's better options now but the HP ProLiant MicroServers (used).
They support ECC ram, 4 caddies, one extra PCIe slot, and to my knowledge you're not cpu limited for a zfs file server usecase.
Keep in mind though, all you need is linux* support, iDRAC, ECC if you're a snob, and drive bays ... and that's basically any free server.
In my extremely opinionated opinion I would only get used enterprise server gear, because a zfs file server will just work unless hardware fails. And a UPS.
*ZFS will be a more natural choice on FreeBSD. It's better documented, and will meet Linux 1:1 in hardware compatibility for this.
Agreed. I have used a HP N40L ProLiant MicroServer since 2013 as home NAS and Time Machine backups via samba. Rock solid hardware, incredibly expandable, and today runs FreeBSD 15.1 with ZFS. Additional hardware modifications include; CD-ROM replaced by two 3.5" HDDs on mounts (now six HDDs of 10TB each in ZRAID1), a SAS card to add two mirrored bootable SSDs underneath CD-ROM drive space, a 2x 2.5G NIC (limited to 4GB/s slot) for dedicated NFS link, while additional internal SATA and external SATA ports unused. Next: replace PSU fan with quieter Noctua fan.
I'm still running an old Gen 8 MicroServer. Modern drives can actually saturate the SATA controller, and because it only has a single PCIe slot I can't add both a 10Gb NIC and a storage controller - I went with the 10Gb NIC.
It works well enough though and has lasted me over a decade at this point. 16GB DDR3 ECC, an old 4 core/8 thread Xeon, 4x14TB drives and the Mellanox NIC.
I got a gen10 plus microserver and liked it so much that I got a second one.
Throw FreeBSD on it and add a couple lines to /etc/exports and rc.conf and it's a NAS right out of the box
Not strictly a recommendation, but Terramaster is a good brand to look at if you want Synology-shaped hardware which can run TrueNAS or Proxmox or any flavour of Linux you want.
Along with various other devices (including a large Synology which I wouldn’t buy today), I run Proxmox on a small two bay+two nvme Terramaster. I have a bare bones Ubuntu LXC running Samba configured for Apple Time Machine, an VM running Scrypyed, and PBS for Proxmox backups. Nothing on it is critical so I don’t bother with any storage redundancy.
I have the F8 Plus, great little unit. It did need a BIOS update when I first got it to enable Proxmox/other OSes to work properly.
Recently replaced the internal USB boot drive with a small NVMe USB enclosure; using a 90-degree USB connector and using a dremel to sand away an opening for the cable to come out so I could mount the enclosure externally.
It's a horrible idea likely, but I have an ancient old Dell PowerEdge R510. Probably sucks way too much energy, but it does what it does and the price of SSDs have skyrocketed so I'm not touching it.
To be totally honest, I don't think I can live without ZFS anymore. Deduplication and snaphots in 2026 are a must.
and checksums!
A friendly reminder since it's probably relevant: when's the last time you tested disaster recovery for your own setup? If you haven't verified recovering it, you don't have an offsite backup.
I had a cloud backup (Backblaze B2, using rclone and encryption) for my home NAS. Some unlucky drive failures later, I was getting ready to recover from my Backblaze backup... and then I couldn't find/remember where I saved the rclone encryption backup.
I lost all data in that NAS, including irreplaceable personal/family photos, due to that mistake. My lesson to share: please verify you can indeed recover from backups.
One alternative for those who don't want any of the major NAS vendors, just use RHEL10. It's free up to 16 licenses, it's ultra stable, cockpit is a very mature gui for a lot of maintenance tasks.
It's a set and forget OS that will run for years without requiring your attention. But these days it has decent container support for hosting services on.
Or rocky, which is a 1:1 clone of RHEL that doesn't require licenses. https://rockylinux.org/
The problem with RHEL is that the free version exists at the sufferance of IBM, a company not well known for being motivated by keeping tech enthusiasts happy. I would use Debian, personally. Not as long of a support cycle as RHEL, but still quite stable and no possibility of corporate rug-pulls.
Needs a 2024 in the title maybe.
Also, are there neal.* besides .fun and .computer?
I'll check all the other TLDs real quick...
Also you can use zfs on a proxmox host and use that both for NAS duty and for the VMs.
For some reason people insist on doing truenas on top of proxmox and then introducing a networking layer between everything they do. Noooo…
its probably dumb but what i do is use proxmox with a debian VM with docker installed and each docker container gets a virtual disk i make on the proxmox host. obviously more involved than running unraid or truenas but i think the complexity underneath is simpler.
Thats perfectly reasonable. Double isolation but that isn’t too costly.
I’m talking more about people doing the same as you except they’re linking the VM storage to another vm (truenas) over network despite it all being on same host. Think it’s mostly because people don’t want to deal with zfs via command line
To this day I still use a ZFS array for my critical backup storage. I've migrated over to using snap raid + mergerfs for my larger Linux iso storage array. Simple enough and I can pull a drive on it's own without any other stuff.
Has anybody used unraid for his Home Server? Is it worth it?
unraid.net
I'm hardly a sysadmin and I think Unraid is great. My usecase is some Docker containers and media backups. The UI gets the job done well, whereas Synology tries to dumb things down and make things pretty.
Android is to Unraid as iPhone is to Synology.
personally, after buying X380, using my old T430 as a mini NAS..(also immich) with a ZFS mirror. works wonderfully :3
this is really cool. I've been dealing with an aging Synology nas and this is something I can pick up, evaluate performance and how safe it is to serve as home for my data.
I killed 2x 4tb Nvme drives on Pciexpress with ZFS raid1 under proxmox, got them replaced, tried again on both pci-express and usb adapter kept getting io problems there too with zfs, drive gets disconnected during scrub or high load, tried already kernel or power management options for pciexpress and usb without sucess, managed to reproduce with high load.
or proxmox ( especially community edition) is fucked or zfs is still unstable, surprisingly with different drives on different interfaces I obtained same results, didnt swap proxmox for clean os, that might being some changes since modules and options would be different i guess. ( drives firmware etc are fine and performing well when inspected with proprietary windows tools)
just search zfs nvme pcie or usb problema or disconnect and you see similar stability problem for different users in different cases (os, drives etc), but also unraid and others, maybe somewhere is rithe right combo of options/glitches but didnt find it yet
> usb adapter kept getting io problems there too with zfs, drive gets disconnected during scrub or high load
That's usually caused by the adapter chipset overheating. Don't use USB adapters for important data.
Very cool
Yeah no. When it comes to backups and data storage, I would rather use a proven reliable system that’s been used and tested by millions of other users, keep these hacky stuff for your hyperland set up.
I've lost more data to "proven reliable systems" than to my homegrown hacked-up stuff (which may be because I trust the latter much less).
This. My synology nas got pwned so many times i cant remember.
> I would rather use a proven reliable system that’s been used and tested by millions of other users
You're describing ZFS.
> You're describing ZFS.
..running on illumos.
Not really, zfs is part of it, but not the whole picture, you as a user you need more than just a file system.
Truenas for example add replication, external cloud sync, gui (yes it’s important not to wipe your data with wrong command), HA, custom caching, containers to extend ot with other apps, raid-z expansion with openzfs, and other for monitoring, user management and a lot more.
Other solutions also add more than just a zfs with ssh access.
Stopped reading at "Debian".
Running ZFS on anything but Solaris/Illumos/FreeBSD is asinine.
ZFS is a permanent second-class citizen on Linux (due to usual open-source politics). This will never resolve.
I don't want to trust my data to some half-assed out-of-tree solution that may or may not break in a week.
FreeBSD ZFS support has matured and is outstanding. Quality-wise it has reached parity with Illumos.
If you can afford Solaris then you're probably not building your own NAS from parts of lesser computers.
Run ZFS backed filestore on FreeBSD, have migrated it to/from Debian. At work and home, not petabyte scale but certainly multi hundred terrabyte. Over 15 years, on over 50 hosts/NAS/SAN instances, different hardware.
Run ZFS on Raspberry Pi, on home builds, on Intel, on AMD, on other ARM chipsets.
I think you're over-stating things. Debian is fine for this. I do think FreeBSD is a better platform for myself.
The code bases adhere (modulo ZFS version numbers) to a spec and you can safely migrate the pools between OS. I've done it multiple times both directions.
You can not do this with BTRFS and other Linux things, I consider this feature of (Open)ZFS a killer-context for me: It's OS portable. I wish Mac OSX hadn't walked out of the room when Oracle went legal.
I guess you can try Windows next - https://github.com/openzfsonwindows/openzfs
> You can not do this with BTRFS
There is actually a btrfs driver for Windows [0]. I've used it a few times before, and it works surprisingly well. You probably wouldn't want to use it for any serious work, but that's not because it's technically flawed, but more because it isn't extensively tested or commercially-supported.
[0]: https://github.com/maharmstone/btrfs
There is Fuse support in BSD. I don't consider that a good choice for this role.
Sure you can migrate pools.
Yet everyone is (again) lost in the details and missing the big picture, which is Linux is doing its best to rat fuck OpenZFS at every opportunity, the last of which was the elimination of write_cache_pages in 6.18 behind the GPL iron curtain a mere few months ago.
I don't know about you but I don't want to build my file storage atop hacks on top of more hacks. The kernel has made it clear non-GPL code is not welcome. Struggles will continue in perpetuity. There are better options.
I understand and I agree from an experience point of view it felt very unstable on debian and proxmox that is debian based.
I wanted to share this experience too as a warning to users investing time and money and possibly hitting instabilities that can cause raid problems and data loss. Don't know why my comment got downvoted, if I knew about this I would have handled things differently.
I usually recovered the pool thanks to other disks being fine, but beside zfs being super cool in terms of features and flexibility at the beginning it actually felt unreliable and I would not suggest it neither!
as mentioned it is probably more stable on other families but I didnt experience that yet.
would freebsd be the most reliable? or which one would have the most reliable zfs module state?
do other solutions like unraid or truenas or similar use zfs the background?
> or which one would have the most reliable zfs module state?
Can't get more reliable ZFS than on illumos. OmniOS on napp-it if you want GUI[0].
[0] https://www.napp-it.org/index_en.html
> Running ZFS on anything but Solaris/Illumos/FreeBSD is asinine.
> ZFS is a permanent second-class citizen on Linux
Linux is the primary target of OpenZFS [0] [1], and has been since 2020 [2]. It may not be supported by the Linux kernel developers, but it's supported by the ZFS developers, and that's all that really matters.
> I don't want to trust my data to some half-assed out-of-tree solution that may or may not break in a week.
Sure, it's an out-of-tree module, but that doesn't mean that it will randomly break all the time; it just means that you may occasionally need to wait for a new OpenZFS release before upgrading your kernel.
> FreeBSD ZFS support has matured and is outstanding.
Agreed, but Linux and FreeBSD both use the same ZFS [3], so I don't really see how the ZFS in FreeBSD can be better than the one in Linux. The tooling and install procedure is certainly better on FreeBSD, but the actual filesystem code is the same (and is probably slightly more robust on Linux since that's going to be where most of the testing occurs).
[0]: https://github.com/openzfs/zfs#supported-kernels-and-distrib...
[1]: https://github.com/openzfs/zfs/pull/8987
[2]: https://github.com/openzfs/zfs/releases/tag/zfs-2.0.0
[3]: https://docs.freebsd.org/en/books/handbook/zfs/
> Linux is the primary target of OpenZFS
Which is worrying as there's a high risk of linux EEE breaking portability.
If you run ZFS with an LTS kernel you're pretty much fine. Yes new Linux releases will break existing ZFS releases - but the LTS tree is in support for long enough that this is never an issue.
Unless you need driver support only found in newer kernels. Then you are screwed.
Do I want my hardware to work or do I want to be able to read my files?
I am a BSD user yet I think this is largely irrelevant in the scope of a NAS which is the conversation we have here. Even in the cases bleeding edge hardware support is relevant, the latest Linux LTS kernel has probably wider hardware support than the freebsd-current at any given point in time anyway.
FreeBSD is a great choice, but there is no need to invent silly reasons to justify using it.
I run ZFS on Linux and FreeBSD and FreeBSD is less of a faff. If you don’t need docker on your NAS, I would go FreeBSD as well.
And yet TrueNAS is moving away from BSD towards … Debian!
At a $company we were planning deploying TrueNAS as our new main NAS system. Then this happened and now everything is running OmniOS.
AFAIK this is because TrueNAS is trying to be all things to everyone, and they like Docker more than BSD jails.
You can use past tense there.