I work as a junior software engineer at the company developing this software (Tuxera Inc.), and I have done some release testing work on this. I might have influenced the "Arch Linux" support that is mentioned on the page under System Requirements, because I also tested it on my machine :)
I didn't see that the first time around so this re-post was interesting, thanks! There's a bit of discussion in the other thread about the "protected area" - anyone got good links to the minutiae of that? How big is it, what tools exist to access it etc. ?
The “protected area” was probably reserved for DRM keys for video distribution, an envisioned use case that never took off for SD cards, and hasn’t been relevant for at least a decade.
Yeah, I'm vaguely aware of this - jsut because of the "S" in SD standing for "Secure" as a euphemism for DRM stuff. From your link it seems likely that most cards probably don't even support it nowadays, but I'm still kind of curious about how exactly it was intended to be accessed - for example what extended commands might have been available. Presumably there's a spec somewhere for it, and presumably one can at the bare minimum ask a card "do you have a protected area available?"
I'd be interested in the specific commands/spec too, however as with all DRM techs I wouldn't be surprised if it's hidden behind NDAs.
However you can see from the parent comment:
> Success of mutual authentication enables host to access card protected area.
So it's something the memory control must make available after a handshake, and then I presume either some specific memory address or set of retrieval commands would follow.
As you mention, it also states this spec is now optional.
Pro tip: Use a read only rootfs and use overlayfs to write any changes to ram instead of the SD. Then you just gotta put your state on a USB drive, network etc.
Likely worn out sectors. Sadly such products are easy in that they don't have a read only file system, but for the majority of users that would help. Maybe there should be a simple script called "finalize" that turns it read only
Didn't there use to be a physical read lock on SD cards?
I know "modern" cards have no space, but if it's important to you, you can still use the full size adapters which should (all?) have the physical lock. Super glue that in place...
There's a flag you can set electronically, to make the card read only, but if the OS isn't set up to support running from a read-only filesystem, setting the card to read only will make the OS crash. If the OS is set up for a read-only filesystem, it won't try to write to it, regardless of the flag.
Setting up Linux to run from a read only filesystem only takes a handful of commands, but having a tool to automate it would be nice.
I love this tool. I use it not just for SD cards but for all sorts of portable storage.
It seems to fix things where Windows File Explorer's formatter and other tools fail. A simple tool that does a really good job. It has even fixed some partition weirdness I've needed to deal with.
While this tool saved my beef a couple of times, I found out that zeroing the device (partially or fully) with dd and partitioning it with gparted (the CLI tool) did also wonders.
I had a wonky HIKSemi USB flash drive, which turned out to be partitioned in a very unaligned way, and neither the flash nor the controller liked that much. Doing it manually and making sure it's aligned (which gparted does automatically) converted it to a very dependable drive.
The only thing I use SD cards for are my digital cameras. I have always read that the cards should be formatted within the camera rather than the computer. I've never seen any issues but I the only thing writing to the card is the camera so I would rather format the card in the camera in case there are any incompatibilities.
afaik, thats so the device can format the card in its preferred filesystem. Instead of pestering the user (who may only use their computer as a "Facebook machine") to make sure they format the card to X specification, the device can just do that for them. Outside of "that", the device isn't doing anything special during format (unless its using the "secure" bits of a SD card, which pretty much no one does).
However, as at least some of the devices users will be Windows users, it does tend to limit the FS choices to FAT, exFAT or NTFS if the user expects to treat the card as removable storage to transfer files, like in a digital camera, so the issue is pretty much moot. Unless MS are still charging royalties on FAT and the device manufacturer wants to avoid those.
These days with people mainly using their phones, and the transfer of files being done over the air, allows device manufacturers more freedom with their SD card FS choice.
Most professional cameras still use ExFAT and AFAIK Microsoft doesn't charge inclusion of ExFAT drivers on these devices anymore.
However, sometimes devices format these cards in slightly specific ways they like (sector sizes, partition offsets and like) so the cards work well with the devices.
My Sony A7-III has an intelligent way of testing cards without reading/writing extensive data and reporting whether the card can handle particular video bitrates. I think SD cards have some tricks we still don't know as consumers much.
Sorry, yeah I prob dumbed it down too much by just saying the file system. But you are right that some devices will prefer a certain sector size, partition layout, etc, and while these can be done manually by the user outside of the device, its just "easier" for the vast majority of people if the device just does that for them.
Which IMO is where the whole "Its better to let the device format the card" came from. Because techs just got sick of trying to explain to less tech savvy users that "yes its possible to format the card in your computer, but just use the devices in built formatter handle it for you", because I know I told users that all the time back in the day, lol.
It's so odd they don't publish the source. You'd think an organization like this which at least claims to have strong opinions about how sd card formatting should work would want to actually communicate what those opinions are.
It's also an organization created by Panasonic, SanDisk, and Toshiba. The entire FOSS scene are technically irrelevant to those companies - you might think software like Linux Kernel would be pretty important, but that tech is available to everybody, which makes it just a constant.
Also, this tool exists to fix card errors at field and reduce support costs, not to advocate their superior jutsu of formatting. The latter is not the point.
There was a period of time when linux only supported MMCs (the forerunner standards), because none of the partner members provided updated SD card drivers. This limitation meant you were limited to using older media that capped out at 128MB. This was a significant problem with using Linux on PDAs.
On the one hand, I get that and I get that you might not want to take criticism etc on a small utility that you create...
However, on the other hand, if this such a pervasive issue as to merit an official "correct" implementation, then I would think you would want more implementations working correctly, so that your brand behaves more consistently, correctly, across impls.
I wonder how much money have these three companies made from FOSS in the last quarter alone. I guess they don't know either, it's easy to take all of that work for granted.
They also tend to buy Linux from someone, often MontaVista last I did some Googling long time ago...
The fundamental problem, I think, is that FOSS didn't really take off in Japan the way it did in US/EU, for some reason. There are tons of code-literate engineers but way less who would be sympathetic with developer community building, code sharing, licensing discussions, etc., that are more common in communities from US/EU.
That might explain why their software never took off to the same extent that the US's did, despite having been leaders in the hardware space.
You can of course "just program" the same way you can "just build" a car out of plywood and some electric motors, however the presence of an engineering mindset (and tools/resources) is what separates a fire hazard from a Ferrari.
> There are tons of code-literate engineers but way less who would be sympathetic with developer community building, code sharing, licensing discussions, etc., that are more common in communities from US/EU
I find these cultural differences very interesting. I wonder why it’s that way. I noticed some lower willingness to share work (and any form of bad news) in more competitive societies, but I’m unsure this would be the case.
I think one of the reasons for the proprietary licensing is that this software is built with Tuxera IP. As is typical for a mostly B2B software company, Tuxera wants to keep potentially advantageous features and optimizations proprietary.
How can you license the correct formatting process and create a sweet recurring value stream for the shareholders of the SD Card Association if you open source this?
Are you a communist? /s
Jokes aside. I'd love to see this standardized and widely available with a document why SD cards needs to be formatted this way, but I think everyone is afraid that someone's patent or secret or something will be revealed, and its library form is probably, really a revenue stream for them.
What if you're formatting your own SD cards for Raspberry Pi, and need control over the partition layout, filesystem, etc. Surely there are some best practices that this tool embodies, which one can follow to achieve the same level of robustness and performance?
It can discard with the proper SD commands (use `--discard`), if your SD adapter supports that. So not with USB card readers, but with many built-in SD card slots. The main reason to use this formatter instead of the OS built-in formatter (or mkfs) is that this creates the correct type of file system (FAT or exFAT depending on card size) and uses correct allocation sizes etc. for devices expecting a factory-formatted card.
The main difference seems to be stuff about block size and alignment.
>Technically speaking, the SD Memory Card Formatter optimizes the layout of data structures on a SD memory card in accordance with flash parameters defined by the SDA. This includes placing the partition at the correct offset for the internal flash layout of the card, properly aligning the FAT and the cluster heap (an area containing the file and directory data) to internal flash boundaries, and minimizing wear-leveling, while maximizing read/write performance.
>Manufacturers of small embedded devices most often expect a file system to be formatted to the above stated parameters, so they optimize their relatively simple system implementations based on the assumption that the file system is already optimized for the internal flash layout.
I don't know enough about flash memory to understand why SSDs don't have this issue. Presumably they use more advanced controllers that just hide all this
Can this tool burn ISO images directly? I currently use the Raspberry Pi Imager (on macOS) for burning my ISOs to SD Cards and USB drives and it works great.
I work as a junior software engineer at the company developing this software (Tuxera Inc.), and I have done some release testing work on this. I might have influenced the "Arch Linux" support that is mentioned on the page under System Requirements, because I also tested it on my machine :)
Outside of Arch BTW memes, yours is the first I've seen an somone unprompted describe themeselves as an Arch user. I gues it does really happen.
Related discussion from 2023: https://news.ycombinator.com/item?id=35610243
I didn't see that the first time around so this re-post was interesting, thanks! There's a bit of discussion in the other thread about the "protected area" - anyone got good links to the minutiae of that? How big is it, what tools exist to access it etc. ?
The “protected area” was probably reserved for DRM keys for video distribution, an envisioned use case that never took off for SD cards, and hasn’t been relevant for at least a decade.
https://www.sdcard.org/developers/sd-standard-overview/conte...
Yeah, I'm vaguely aware of this - jsut because of the "S" in SD standing for "Secure" as a euphemism for DRM stuff. From your link it seems likely that most cards probably don't even support it nowadays, but I'm still kind of curious about how exactly it was intended to be accessed - for example what extended commands might have been available. Presumably there's a spec somewhere for it, and presumably one can at the bare minimum ask a card "do you have a protected area available?"
I'd be interested in the specific commands/spec too, however as with all DRM techs I wouldn't be surprised if it's hidden behind NDAs.
However you can see from the parent comment:
> Success of mutual authentication enables host to access card protected area.
So it's something the memory control must make available after a handshake, and then I presume either some specific memory address or set of retrieval commands would follow.
As you mention, it also states this spec is now optional.
This seems to be the spec in question: https://picture.iczhiku.com/resource/eetop/wYKESjuFHlDHkbcM....
I have legitimately had this fix weird errors with SD cards in certain devices when nothing else worked.
Sadly it doesn’t seem to be able to fix the several SD cards killed by the raspberry pis I have.
Pro tip: Use a read only rootfs and use overlayfs to write any changes to ram instead of the SD. Then you just gotta put your state on a USB drive, network etc.
You can also mount /var/log as a tmpfs. Worked for the Debian based OS but not so well with Fedora (on an eMMC volume).
Likely worn out sectors. Sadly such products are easy in that they don't have a read only file system, but for the majority of users that would help. Maybe there should be a simple script called "finalize" that turns it read only
Didn't there use to be a physical read lock on SD cards?
I know "modern" cards have no space, but if it's important to you, you can still use the full size adapters which should (all?) have the physical lock. Super glue that in place...
There's a flag you can set electronically, to make the card read only, but if the OS isn't set up to support running from a read-only filesystem, setting the card to read only will make the OS crash. If the OS is set up for a read-only filesystem, it won't try to write to it, regardless of the flag.
Setting up Linux to run from a read only filesystem only takes a handful of commands, but having a tool to automate it would be nice.
Fun fact: SD write lock is just a switch wired to host controller, it doesn't actually prevent writing to the card.
The latest Pi OS writes all logs to RAM, which might change your experience.
SD cards do not make good SSDs. For Plan 9 I only put the kernel on the SD card and pull root from another source e.g. a file server via tcp/tls.
I love this tool. I use it not just for SD cards but for all sorts of portable storage.
It seems to fix things where Windows File Explorer's formatter and other tools fail. A simple tool that does a really good job. It has even fixed some partition weirdness I've needed to deal with.
While this tool saved my beef a couple of times, I found out that zeroing the device (partially or fully) with dd and partitioning it with gparted (the CLI tool) did also wonders.
I had a wonky HIKSemi USB flash drive, which turned out to be partitioned in a very unaligned way, and neither the flash nor the controller liked that much. Doing it manually and making sure it's aligned (which gparted does automatically) converted it to a very dependable drive.
It works on really old versions of Windows too, even XP I think.
The only thing I use SD cards for are my digital cameras. I have always read that the cards should be formatted within the camera rather than the computer. I've never seen any issues but I the only thing writing to the card is the camera so I would rather format the card in the camera in case there are any incompatibilities.
afaik, thats so the device can format the card in its preferred filesystem. Instead of pestering the user (who may only use their computer as a "Facebook machine") to make sure they format the card to X specification, the device can just do that for them. Outside of "that", the device isn't doing anything special during format (unless its using the "secure" bits of a SD card, which pretty much no one does).
However, as at least some of the devices users will be Windows users, it does tend to limit the FS choices to FAT, exFAT or NTFS if the user expects to treat the card as removable storage to transfer files, like in a digital camera, so the issue is pretty much moot. Unless MS are still charging royalties on FAT and the device manufacturer wants to avoid those.
These days with people mainly using their phones, and the transfer of files being done over the air, allows device manufacturers more freedom with their SD card FS choice.
Most professional cameras still use ExFAT and AFAIK Microsoft doesn't charge inclusion of ExFAT drivers on these devices anymore.
However, sometimes devices format these cards in slightly specific ways they like (sector sizes, partition offsets and like) so the cards work well with the devices.
My Sony A7-III has an intelligent way of testing cards without reading/writing extensive data and reporting whether the card can handle particular video bitrates. I think SD cards have some tricks we still don't know as consumers much.
Sorry, yeah I prob dumbed it down too much by just saying the file system. But you are right that some devices will prefer a certain sector size, partition layout, etc, and while these can be done manually by the user outside of the device, its just "easier" for the vast majority of people if the device just does that for them.
Which IMO is where the whole "Its better to let the device format the card" came from. Because techs just got sick of trying to explain to less tech savvy users that "yes its possible to format the card in your computer, but just use the devices in built formatter handle it for you", because I know I told users that all the time back in the day, lol.
I hope this is a short term solution, pending the various OS vendors modifying their native tools to handle the formatting correctly.
It's so odd they don't publish the source. You'd think an organization like this which at least claims to have strong opinions about how sd card formatting should work would want to actually communicate what those opinions are.
Someone actually reverse engineered the formatting and made a open source tool that does the equivalent on Linux: https://github.com/profi200/sdFormatLinux
It's also an organization created by Panasonic, SanDisk, and Toshiba. The entire FOSS scene are technically irrelevant to those companies - you might think software like Linux Kernel would be pretty important, but that tech is available to everybody, which makes it just a constant.
Also, this tool exists to fix card errors at field and reduce support costs, not to advocate their superior jutsu of formatting. The latter is not the point.
There was a period of time when linux only supported MMCs (the forerunner standards), because none of the partner members provided updated SD card drivers. This limitation meant you were limited to using older media that capped out at 128MB. This was a significant problem with using Linux on PDAs.
> The latter is not the point
On the one hand, I get that and I get that you might not want to take criticism etc on a small utility that you create...
However, on the other hand, if this such a pervasive issue as to merit an official "correct" implementation, then I would think you would want more implementations working correctly, so that your brand behaves more consistently, correctly, across impls.
I wonder how much money have these three companies made from FOSS in the last quarter alone. I guess they don't know either, it's easy to take all of that work for granted.
They also tend to buy Linux from someone, often MontaVista last I did some Googling long time ago...
The fundamental problem, I think, is that FOSS didn't really take off in Japan the way it did in US/EU, for some reason. There are tons of code-literate engineers but way less who would be sympathetic with developer community building, code sharing, licensing discussions, etc., that are more common in communities from US/EU.
Japan is Japan, most of their history can be summarised as isolationism/NIH.
Software developers are also not seen as "real" engineers there. They're just typing words into a computer, right?
That might explain why their software never took off to the same extent that the US's did, despite having been leaders in the hardware space.
You can of course "just program" the same way you can "just build" a car out of plywood and some electric motors, however the presence of an engineering mindset (and tools/resources) is what separates a fire hazard from a Ferrari.
What exactly separates a fire hazard from a Ferrari? Those vehicles are notoriously unreliable
> There are tons of code-literate engineers but way less who would be sympathetic with developer community building, code sharing, licensing discussions, etc., that are more common in communities from US/EU
I find these cultural differences very interesting. I wonder why it’s that way. I noticed some lower willingness to share work (and any form of bad news) in more competitive societies, but I’m unsure this would be the case.
Culture eats strategy for breakfast
I think one of the reasons for the proprietary licensing is that this software is built with Tuxera IP. As is typical for a mostly B2B software company, Tuxera wants to keep potentially advantageous features and optimizations proprietary.
A surprising name for a company so decided to keep IP secret.
Doubly idiotic since just running it under strace and feeding that to a LLM would get you an open source version in no time..
You now need an LLM to go thru write calls?
I used an AI to help reverse engineer a weird JS login flow so I could automate it.
I could've done it without AI (I've done it before) but it would've taken me 20x longer.
How can you license the correct formatting process and create a sweet recurring value stream for the shareholders of the SD Card Association if you open source this?
Are you a communist? /s
Jokes aside. I'd love to see this standardized and widely available with a document why SD cards needs to be formatted this way, but I think everyone is afraid that someone's patent or secret or something will be revealed, and its library form is probably, really a revenue stream for them.
Shortsighted, I may say.
Not open source.
What if you're formatting your own SD cards for Raspberry Pi, and need control over the partition layout, filesystem, etc. Surely there are some best practices that this tool embodies, which one can follow to achieve the same level of robustness and performance?
I wonder if this does anything more than fill with zeros or 0xFFs and then run the normal file system formatting… Maybe trim/discard operation too.
Seems weird it can be applied to bitlocker to go volumes.
It can discard with the proper SD commands (use `--discard`), if your SD adapter supports that. So not with USB card readers, but with many built-in SD card slots. The main reason to use this formatter instead of the OS built-in formatter (or mkfs) is that this creates the correct type of file system (FAT or exFAT depending on card size) and uses correct allocation sizes etc. for devices expecting a factory-formatted card.
https://www.sdcard.org/press/thoughtleadership/the-sd-memory... has more details, and a comparison to disk utility format. There is also this open source implementation https://github.com/profi200/sdFormatLinux
The main difference seems to be stuff about block size and alignment.
>Technically speaking, the SD Memory Card Formatter optimizes the layout of data structures on a SD memory card in accordance with flash parameters defined by the SDA. This includes placing the partition at the correct offset for the internal flash layout of the card, properly aligning the FAT and the cluster heap (an area containing the file and directory data) to internal flash boundaries, and minimizing wear-leveling, while maximizing read/write performance.
>Manufacturers of small embedded devices most often expect a file system to be formatted to the above stated parameters, so they optimize their relatively simple system implementations based on the assumption that the file system is already optimized for the internal flash layout.
I don't know enough about flash memory to understand why SSDs don't have this issue. Presumably they use more advanced controllers that just hide all this
brew install --cask sdformatter
So, not OSX?
The Windows/Mac page can be found at https://www.sdcard.org/downloads/formatter/ . Caveat:
Note: If you have a Mac with Apple silicon, e.g. M1, you might be asked to install Rosetta in order to open the SD Card Formatter.
It's there. The website just has poor navigation & OP linked specifically to the Linux version for whatever reason.
You can either link the linux version or the Win/OSX version. This is a petty complaint
What did I complain about? You're getting upset over absolutely nothing. That's not healthy.
"for whatever reason"
I just meant that I don't know why you chose to link the Linux version. Sorry, I didn't intend to be rude.
Can this tool burn ISO images directly? I currently use the Raspberry Pi Imager (on macOS) for burning my ISOs to SD Cards and USB drives and it works great.