This has been on the back of my mind for a while. And, as other commenters have noted, it would be great for the file to contain the (self-modifiable) Lisp image, a builtin virtual file system, and whatever the application want to use as (runtime modifiable) extra tables.
I find SQLite dynamic linking being basically compatible with ELF dynamic linking to be very impressive, I can imagine that if well done, it cloud replace most uses of AppImages with a much more efficient format, like the author suggests.
How about an option for compressing section contents within the SQLite blobs, since the author mentioned you can't mmap directly the text pages and have to copy anyway ?
There are two extensions that I was thinking would make a SQLite executable truly unique.
First, a linking extension that would allow patching in functions and hooks more directly to allow for a very powerful plug-in system. Imagine the plug-in SQLite defining a BEFORE/AFTER/REPLACE hook for some symbol the host SQLite defines as extensible.
Second, re-linking at runtime. This would require application author cooperation because you won't be able to do that from anywhere, but imagine changing a dependency or loading a plugin at runtime through editing the db, and the interpreter just maps that on demand/automatically in the background, now next time your web server accept(), it calls the new version of the handling function.
The copied vs. mapped memory situation is the only deal breaker in this experiment. Otherwise, file format unification would be a big step forward. The PE/COFF executable file format used by Windows (and some older Unix systems) is a relational database as well. The same goes to .NET assembly format - it's a relational database too. The wheel gets reinvented over and over again.
I don't think this goes far enough! Make the actually app store be the same file itself. so it's a living application and the file is constantly updated to how you use it. Copy it around, and you carry your data wiht you as well.
Let's go deeper. it's a webserver app + the server code + application code + db, so pocketbase++ where it's also the deployment target.
Then combine it with APE liek system, and the same file loads and stores things on every platform. evil laugh
I didn't have this idea exactly but I did play around with sqlite as an embedded database for packing ruby apps into a single file a while ago. It was less direct than this: you basically compiled all the dependencies (including .so's) into a fake filesystem that an overridden `require` loaded from, which was unpacked to a `:memory:` database at runtime from a `.data` section that got bolted onto the interpreter. It died when 1.8->1
9 changed how the build system works and I never got round to updating.
A version based on this which carries around an overlay filesystem would be comparatively straightforward, the hard work is already done.
Yeah... Original and clever, and a great read! But it seems mostly useful to the handful of people dealing with ELF internals, than to the vast majority of people executing ELF blissfully ignorant of its internals... Maybe if the latency and size trade-offs were the other way around it would be more appealing to the masses.
I wonder how flexible is the SQLite binary format to allow to design a tool that would take a SELF binary/db and rewrite it to make BLOB values more mmap-able (OS page aligned) in order to allow shortcuts in the self-exec loader.
(Even with ELF, a kernel can have a relatively simple parser and loader and leave more challenging work to user space... like relocations, dynamic libraries, etc.)
Great read. I always find these turn a feature of a computer into a database an interesting read/idea. It certainly has upside, and downsides. For example having an entire filesystem as a sqlite database etc, or in this case an executable. It would be nice to have a structured way of talking to our utilities, rather than interpreting various commands as json through a few layers of tools to then get the output in a common format. That said I'd probably rather have a split up tool-set, than everything having to be in a single database for it to interoperate.
I was thinking that someone managed to put an ELF header in an SQLite file while keeping it compatible with SQLite... but no, "just" a new binary format. Very interesting nevertheless!
So where does this lead to? Is it leading us towards SQLite as some kind of alternative container image format? Because the benefit seems to only occur when it incorporates dependencies
I’m wondering if the anti-clickbait filter could benefit from some lightweight LLM integration. E.g. when the submission hits the front page, if the title was “de-clickbaited” and wasn’t changed manually yet, run it through a cheap model and see if that was the right call.
Yeah, I have been amazed for an entire life how many tools the IT industry invented during last 50 years to just disguise a database.
OK, in '70s, '80s or '90s when compute and storage resources were limited and every bit counted, specialized formats did make sense.
But nowadays we'd save enormous efforts by just packaging stuff in SQLite databases. Microsoft's proprietary file formats (Office, Power BI etc.), OpenOffice/LibreOffice OpenDocument format, or almost everything else would fit perfectly.
Documents, files, are all data (called "data files", aren't they?), including executables, which as can be seen in this article are also databases.
Yes !
This has been on the back of my mind for a while. And, as other commenters have noted, it would be great for the file to contain the (self-modifiable) Lisp image, a builtin virtual file system, and whatever the application want to use as (runtime modifiable) extra tables.
I find SQLite dynamic linking being basically compatible with ELF dynamic linking to be very impressive, I can imagine that if well done, it cloud replace most uses of AppImages with a much more efficient format, like the author suggests.
How about an option for compressing section contents within the SQLite blobs, since the author mentioned you can't mmap directly the text pages and have to copy anyway ?
There are two extensions that I was thinking would make a SQLite executable truly unique.
First, a linking extension that would allow patching in functions and hooks more directly to allow for a very powerful plug-in system. Imagine the plug-in SQLite defining a BEFORE/AFTER/REPLACE hook for some symbol the host SQLite defines as extensible.
Second, re-linking at runtime. This would require application author cooperation because you won't be able to do that from anywhere, but imagine changing a dependency or loading a plugin at runtime through editing the db, and the interpreter just maps that on demand/automatically in the background, now next time your web server accept(), it calls the new version of the handling function.
The copied vs. mapped memory situation is the only deal breaker in this experiment. Otherwise, file format unification would be a big step forward. The PE/COFF executable file format used by Windows (and some older Unix systems) is a relational database as well. The same goes to .NET assembly format - it's a relational database too. The wheel gets reinvented over and over again.
I don't think this goes far enough! Make the actually app store be the same file itself. so it's a living application and the file is constantly updated to how you use it. Copy it around, and you carry your data wiht you as well.
Let's go deeper. it's a webserver app + the server code + application code + db, so pocketbase++ where it's also the deployment target.
Then combine it with APE liek system, and the same file loads and stores things on every platform. evil laugh
Very cool hacking! My hats off to the author.
I didn't have this idea exactly but I did play around with sqlite as an embedded database for packing ruby apps into a single file a while ago. It was less direct than this: you basically compiled all the dependencies (including .so's) into a fake filesystem that an overridden `require` loaded from, which was unpacked to a `:memory:` database at runtime from a `.data` section that got bolted onto the interpreter. It died when 1.8->1 9 changed how the build system works and I never got round to updating.
A version based on this which carries around an overlay filesystem would be comparatively straightforward, the hard work is already done.
The application writing its state to the executable is basically Smalltalk (or Lisp). It's an extremely powerful way of writing software, IMO.
<_< >_> https://github.com/blue-monads/potatoverse
A lot of kernel filesystem features should be inside a database as well.
I appreciate the inventiveness of this idea. But I don’t find myself thinking I must have this.
Yeah... Original and clever, and a great read! But it seems mostly useful to the handful of people dealing with ELF internals, than to the vast majority of people executing ELF blissfully ignorant of its internals... Maybe if the latency and size trade-offs were the other way around it would be more appealing to the masses.
This was super fun. Dude’s been on an absolute tear recently.
I wonder how flexible is the SQLite binary format to allow to design a tool that would take a SELF binary/db and rewrite it to make BLOB values more mmap-able (OS page aligned) in order to allow shortcuts in the self-exec loader.
What if, we put the sqlite driver in kernel space? So this is your only filesystem like a unikernel
It's a lot of machinery to put into a kernel.
(Even with ELF, a kernel can have a relatively simple parser and loader and leave more challenging work to user space... like relocations, dynamic libraries, etc.)
Great read. I always find these turn a feature of a computer into a database an interesting read/idea. It certainly has upside, and downsides. For example having an entire filesystem as a sqlite database etc, or in this case an executable. It would be nice to have a structured way of talking to our utilities, rather than interpreting various commands as json through a few layers of tools to then get the output in a common format. That said I'd probably rather have a split up tool-set, than everything having to be in a single database for it to interoperate.
I was thinking that someone managed to put an ELF header in an SQLite file while keeping it compatible with SQLite... but no, "just" a new binary format. Very interesting nevertheless!
So where does this lead to? Is it leading us towards SQLite as some kind of alternative container image format? Because the benefit seems to only occur when it incorporates dependencies
The handling of dynamic libraries and closing over them into single dbs made this a fun read.
The word "Your" was dropped from the start of the title for some reason.
Well, for the usual reason :-)
I’m wondering if the anti-clickbait filter could benefit from some lightweight LLM integration. E.g. when the submission hits the front page, if the title was “de-clickbaited” and wasn’t changed manually yet, run it through a cheap model and see if that was the right call.
Maybe "your" at the beginning of a title should be replaced by "my" instead of getting stripped.
That would preserve the non-clickbait meaning of all such titles I can spontaneously think of.
“SQLite database as an executable format”? A bit farther away from the original but captures the meaning and sounds more neutral.
Yeah, I have been amazed for an entire life how many tools the IT industry invented during last 50 years to just disguise a database.
OK, in '70s, '80s or '90s when compute and storage resources were limited and every bit counted, specialized formats did make sense.
But nowadays we'd save enormous efforts by just packaging stuff in SQLite databases. Microsoft's proprietary file formats (Office, Power BI etc.), OpenOffice/LibreOffice OpenDocument format, or almost everything else would fit perfectly.
Documents, files, are all data (called "data files", aren't they?), including executables, which as can be seen in this article are also databases.
https://sqlite.org/aff_short.html
https://sqlite.org/appfileformat.html