It is not a schizophrenic zip file to have inline headers that are not referenced in the TOC. the TOC is the only source of truth in a zip file. It was designed this way in purpose so that you can add new versions of files on your 20 disk zip without having to re-write all 20 disks. Pkzip would read the TOC from disk 20, append your new file to disk 20 (or 21 if there was not enough space) and then write a new TOC at the end that does not reference the old file still in the zip. That is by design. Reading anything other than the files in the TOC is an invalid zip reader
While that is true (even if some people really want to stream read zips and will thus trigger tar-style conflicts) that is not what the article is about. The article is really more about software using the "size of central directory" field to find the central directory (a sin Python's standard library apparently commits) versus the "offset of central directory".
A commenter on an other post of that article noted that technically there's no reason for the central directory to be right before the EOCD so seeking backwards from the EOCD by the size of CD is just incorrect. In fact zip was designed such that the central directory could be split across multiple disks (and later files), so it was not possible to guarantee a simple backwards jump from the EOCD to the start of CD.
> technically there's no reason for the central directory to be right before the EOCD so seeking backwards from the EOCD by the size of CD is just incorrect
Yes, and in fact APPNOTE.TXT expects implementations to deal with discontiguous CD/EOCD records; it actually mandates that some records appear between them. (It's where ZIP64-specific records go, for example.)
The whole thing seems to stem from a belief by the author of this post that Info-ZIP is the canonical implementation. (This is of course wrong. That would be PKZIP.)
If Info-ZIP is exhibiting the behavior described, then the Info-ZIP implementation is flat out wrong.
(This has happened before. See <https://news.ycombinator.com/item?id=27925393> where the author of that post provides pushback to Mark Adler, receives pushback from Adler in response, and then just stops pushing and defers to Adler's reading. Adler is an expert in his domain, and he's made valuable contributions to free software, but his expertise is in compression, and he's not the authority on the spec or the format; Adler didn't create ZIP or write APPNOTE.TXT—Phil Katz did.)
> a belief by the author of this post that Info-ZIP is the canonical implementation
↑ Where did you get this from?
I don't think nowadays there's such a thing as a canonical implementation — ZIP implementation world is too fragmented and implementations are too widespread.
One more note is that the article isn't about who's right or wrong in terms of interpreting APPNOTE.TXT — this is besides the point. The point there is that a parser discrepancy can cause security issues, and the article documents just another discrepancy found in the real world implementations (and there are A LOT of these with regards to the ZIP format).
> this makes sense—technically both the offset of the Central Directory and the size of the Central Directory are redundant when you consider that the End of Central Directory Record should be, well, at the end of the Central Directory. In a proper ZIP file the following equation should be true: (position_of_EoCDR - size_of_CD == offset_of_CD)
This is a pretty clear statement on normativity.
You also left out an important part of my comment when you quoted it. (The operative word at the front of the sentence is "seems".)
> This is a pretty clear statement on normativity.
I disagree – it's a technical remark on redundancy of fields (this is related to my hypothesis in the article that redundancy in formats is the primary cause of parser discrepancy). It doesn't acknowledge InfoZIP being canonical in any way.
> You also left out an important part of my comment when you quoted it. (The operative word at the front of the sentence is "seems".)
It's talking about how the EOCD contains both the size of the central directory and the offset of the start of the central directory, which is redundant. So we end up with some tools honoring the offset, while some subtract the size from the EOCD.
It's true that the other comment is unrelated to the content of the article, but it's not true that the offset and size in the EOCD are redundant (for the reasons given in the earlier sibling comment).
I can think they would test which parser you have using some social engineering and create the zip appropriately, but they would also need to know the accountings parser. So I think they would just do the first step and leave it up to chance.
PDFs in .zip files is in itself a huge smell to begin with.
I don't think I've actually ever received a zipped up .pdf, so it's clearly not legitimately necessary for anyone to do so and should you ever see it you should treat it highly suspiciously.
I get admin@ emails for our company domain and there is a somewhat steady stream of run of the mill fraud attempts.
A trick I've seen happen quite a lot recently is emails with .svg attachments, which have some lightly obfuscated JavaScript in them and which ultimately redirects you to some dodgy looking URL (which I never visited).
I simply made a rule to outright reject all .svg files from external sources and I get a report any time it's attempted. In about the last 12 months this has been running, it's probably blocked about 20 incoming emails and only one of those was a false positive and even the false positive was a weird case as we were sending a .svg file to a creative company who for some reason had our .svg attachment in their reply back to us.
Well, there is a case for PDFs in a .zip, but it's when you're e.g. downloading some set of documents from your e-government site, and it's like 4 PDFs, all cryptographically signed: so you get a .zip with 4 PDFs in it and 4 .sig files with matching names.
But in emails? Just attach however many PDFs you need and send, they don't really compress anyway; and I think most web-mail fronts actually allow you to download all the attachments as a single .zip — but obviously those .zip-files are not maliciously crafted (I hope, at least).
Also, now that I think of it, forwarding the PDF you've extracted and visually reviewed instead of the original .zip-file would defeat this attack (unless, of course, it's the PDF file that's schizophrenic).
This is often a useful practice, because it prevents providers' badly-written/-configured attempts at antimalware from blocking files that the recipient knows they want to accept.
> I don't think I've actually ever received a zipped up .pdf, so it's clearly not legitimately necessary for anyone to do so and should you ever see it you should treat it highly suspiciously.
Not sure... S.O. is an accountant and they're constantly emailing zips of Excel files and PDFs. Most of the monthly financials are a handful of different reports and they all get zipped together and emailed around.
I'm sure there's other common reporting use cases as well
It's an interesting hole that the test cases don't cover any of Microsoft Office, Windows Explorer, PowerShell's various cmdlets, or the several major .NET ZIP archive libraries. It would seem that the author just does not use Microsoft Windows.
There's a whole extra level of archive file format tooling gotchas that one misses out on when one assumes "UNIX" for everything, and does not account for "FAT", "NTFS", "HPFS", and even "OpenVMS".
The trick depends upon different implementations doing different things. Not likely for Word (though I suppose it is -possible- across different versions or different OSes).
To respond to Grandfather comment, modern Office files are really just ZIPs with different extensions, they even have the magic string "PK" at the very beginning of the file.
I do wonder, since a lot of tools outside of the MS ecosystem can read Office files (e.g. LibreOffice and Google Docs as well as plenty of other online tools), if indeed the hack as described by the article is possible. One would just need to figure out the ZIP stacks used by said tools.
You can even just rename a docx file to use the zip extension and then manually unzip it for those curious. If I remember correctly, the contents are XML files with structure encoding the formatting around the content.
docx is supposed to be a portable format. There are many tools other than Word that open them. Just LibreOffice and Google Docs for example. They might well differ from Word or from each other in how they handle this case. Definitely worth testing!
The described exploit seems theoretical. In order to create the schizophrenic ZIP, the attacker would have to figure out what ZIP stacks are being used and ensure they act differently - if the 2 departments use the same stack, then the exploit can't work, can it?
A more realistic attack would be something like, slipping a malicious payload past a scanner by emailing a zip file that appears innocent when unpacked with the scanner’s zip implementation but produces malware when unpacked with the email client’s implementation. There’s a decent chance they’ll be different, and it wouldn’t be too hard to guess which ones a target might be using.
There was a time when passing ZIP files around was a very popular method of software distribution, and things like this were gotchas that had to be watched for. It was widely known, at least amongst sysops, that the varied toolsets that handled ZIP archives were functionally different. And there were scanners and sanity checkers, and bugfixes to PKUNZIP, that dealt in this stuff for uploaded files and FREQ responses.
Did people exploit the differences? Yes. Although it was mainly on the level of creating prank ZIP files on non-Microsoft operating systems with 8.3 filenames such as "PRN" or "CLOCK$".
However, the truly terrible idea of self-extracting archives was popular, which meant that archives with "interesting" arrangements of the archive within the overall file were widespread. ZIP comments were also liberally applied and altered by pretty much every BBS that passed an archive along. And the Unix people wanted to be able to use pipes, something that the MS-DOS original never had to cater for.
Also, there were people who exploited the fact that different tools took different things as gospel. Even within the past decade one can find people still being caught out by the fact that there's a header field that instructs what the pathname separator character(s) used are; and that ZIP tools that expect non-seekable streams operate differently to ZIP tools that expect seekable regular files.
In theory there isn't, and everyone is supposed to use forward slash. But in practice there is, because not everyone has stuck to using forward slashes over the decades. Some tools are more forgiving of the differences amongst the various ZIP archive creators than others.
These people having fun with "Unix" versus "FAT" over the past decade are seeing the tip of the iceberg, given that there was a PKZIP for OS/400, there is a PKZIP for z/OS (and a competitor that claims to be cheaper), there are tools of varying degrees of Unixiness for systems like the Atari ST and OS/2, and a whole bunch of things have accrued over the years such as an outright extra header giving alternative filenames specifically for MacOS.
It's not unlikely that devs and such will be on Mac/Linux and accounting will be on Windows. I agree it's still somewhat theoretical but interesting nonetheless.
The author explains in the article that they previously gave a presentation outlining various techniques to achieve a "schizophrenic" zip file. The blog post discusses an additional technique that was not present in their previous presentation.
Are you saying that parser discrepancy is a well known problem OR that this exact technical issue described (CD offset vs CD size used to derive CD location) is a well known problem?
If the prior, you are of course right — parser discrepancy and the existence of these kinds of "multiple personality" files isn't a new thing. I'm pretty sure this can be traced back to at least the '80 :)
However, if you mean the latter, I would appreciate a link if you can recall where you've seen it before / read about it — I can then add it to the article.
I don't know much about the subject but I do know that PDF is an executable format, and I suppose it could use fingerprinting (based on e.g. screen size or resolution) to become schizophrenic (?)
It is not a schizophrenic zip file to have inline headers that are not referenced in the TOC. the TOC is the only source of truth in a zip file. It was designed this way in purpose so that you can add new versions of files on your 20 disk zip without having to re-write all 20 disks. Pkzip would read the TOC from disk 20, append your new file to disk 20 (or 21 if there was not enough space) and then write a new TOC at the end that does not reference the old file still in the zip. That is by design. Reading anything other than the files in the TOC is an invalid zip reader
While that is true (even if some people really want to stream read zips and will thus trigger tar-style conflicts) that is not what the article is about. The article is really more about software using the "size of central directory" field to find the central directory (a sin Python's standard library apparently commits) versus the "offset of central directory".
A commenter on an other post of that article noted that technically there's no reason for the central directory to be right before the EOCD so seeking backwards from the EOCD by the size of CD is just incorrect. In fact zip was designed such that the central directory could be split across multiple disks (and later files), so it was not possible to guarantee a simple backwards jump from the EOCD to the start of CD.
> technically there's no reason for the central directory to be right before the EOCD so seeking backwards from the EOCD by the size of CD is just incorrect
Yes, and in fact APPNOTE.TXT expects implementations to deal with discontiguous CD/EOCD records; it actually mandates that some records appear between them. (It's where ZIP64-specific records go, for example.)
The whole thing seems to stem from a belief by the author of this post that Info-ZIP is the canonical implementation. (This is of course wrong. That would be PKZIP.)
If Info-ZIP is exhibiting the behavior described, then the Info-ZIP implementation is flat out wrong.
(This has happened before. See <https://news.ycombinator.com/item?id=27925393> where the author of that post provides pushback to Mark Adler, receives pushback from Adler in response, and then just stops pushing and defers to Adler's reading. Adler is an expert in his domain, and he's made valuable contributions to free software, but his expertise is in compression, and he's not the authority on the spec or the format; Adler didn't create ZIP or write APPNOTE.TXT—Phil Katz did.)
Author here :)
> a belief by the author of this post that Info-ZIP is the canonical implementation
↑ Where did you get this from?
I don't think nowadays there's such a thing as a canonical implementation — ZIP implementation world is too fragmented and implementations are too widespread.
One more note is that the article isn't about who's right or wrong in terms of interpreting APPNOTE.TXT — this is besides the point. The point there is that a parser discrepancy can cause security issues, and the article documents just another discrepancy found in the real world implementations (and there are A LOT of these with regards to the ZIP format).
> this makes sense—technically both the offset of the Central Directory and the size of the Central Directory are redundant when you consider that the End of Central Directory Record should be, well, at the end of the Central Directory. In a proper ZIP file the following equation should be true: (position_of_EoCDR - size_of_CD == offset_of_CD)
This is a pretty clear statement on normativity.
You also left out an important part of my comment when you quoted it. (The operative word at the front of the sentence is "seems".)
> This is a pretty clear statement on normativity.
I disagree – it's a technical remark on redundancy of fields (this is related to my hypothesis in the article that redundancy in formats is the primary cause of parser discrepancy). It doesn't acknowledge InfoZIP being canonical in any way.
> You also left out an important part of my comment when you quoted it. (The operative word at the front of the sentence is "seems".)
Fair, though my point still stands.
This is not what the article is talking about.
It's talking about how the EOCD contains both the size of the central directory and the offset of the start of the central directory, which is redundant. So we end up with some tools honoring the offset, while some subtract the size from the EOCD.
It's true that the other comment is unrelated to the content of the article, but it's not true that the offset and size in the EOCD are redundant (for the reasons given in the earlier sibling comment).
And how bad actors could take advantage of that.
I can think they would test which parser you have using some social engineering and create the zip appropriately, but they would also need to know the accountings parser. So I think they would just do the first step and leave it up to chance.
PDFs in .zip files is in itself a huge smell to begin with.
I don't think I've actually ever received a zipped up .pdf, so it's clearly not legitimately necessary for anyone to do so and should you ever see it you should treat it highly suspiciously.
I get admin@ emails for our company domain and there is a somewhat steady stream of run of the mill fraud attempts.
A trick I've seen happen quite a lot recently is emails with .svg attachments, which have some lightly obfuscated JavaScript in them and which ultimately redirects you to some dodgy looking URL (which I never visited).
I simply made a rule to outright reject all .svg files from external sources and I get a report any time it's attempted. In about the last 12 months this has been running, it's probably blocked about 20 incoming emails and only one of those was a false positive and even the false positive was a weird case as we were sending a .svg file to a creative company who for some reason had our .svg attachment in their reply back to us.
Well, there is a case for PDFs in a .zip, but it's when you're e.g. downloading some set of documents from your e-government site, and it's like 4 PDFs, all cryptographically signed: so you get a .zip with 4 PDFs in it and 4 .sig files with matching names.
But in emails? Just attach however many PDFs you need and send, they don't really compress anyway; and I think most web-mail fronts actually allow you to download all the attachments as a single .zip — but obviously those .zip-files are not maliciously crafted (I hope, at least).
Also, now that I think of it, forwarding the PDF you've extracted and visually reviewed instead of the original .zip-file would defeat this attack (unless, of course, it's the PDF file that's schizophrenic).
The other reason is to use zip's incredibly mega super secure password protection, to keep super secret docs safe.
(I see it all the time. Along with the password in the very same email)
This is often a useful practice, because it prevents providers' badly-written/-configured attempts at antimalware from blocking files that the recipient knows they want to accept.
I’m not saying this is good security hygiene, but not necessarily everyone with access to the file will also have access to the password in the email.
e.g. someone downloaded the password protected zip on a public computer, logged out of their email, but forgot to delete the file.
> I don't think I've actually ever received a zipped up .pdf, so it's clearly not legitimately necessary for anyone to do so and should you ever see it you should treat it highly suspiciously.
What kind of crazy logic is that?
Not sure... S.O. is an accountant and they're constantly emailing zips of Excel files and PDFs. Most of the monthly financials are a handful of different reports and they all get zipped together and emailed around.
I'm sure there's other common reporting use cases as well
Since docx files are similar to a zip file with the extension changed, could this trick fake out Microsoft Word?
It's an interesting hole that the test cases don't cover any of Microsoft Office, Windows Explorer, PowerShell's various cmdlets, or the several major .NET ZIP archive libraries. It would seem that the author just does not use Microsoft Windows.
There's a whole extra level of archive file format tooling gotchas that one misses out on when one assumes "UNIX" for everything, and does not account for "FAT", "NTFS", "HPFS", and even "OpenVMS".
Or ZIP64. (-:
* https://github.com/dotnet/runtime/blob/main/src/libraries/Sy...
* https://github.com/mihula/ProDotNetZip/blob/main/src/Zip/Zip...
The trick depends upon different implementations doing different things. Not likely for Word (though I suppose it is -possible- across different versions or different OSes).
It’s very common for organisations to only give expensive MS Office licenses to a subset of employees while the rest rely on O365 or Google Docs.
Then you have people on Linux or macOS who might also use LibreOffice, Apples Office suite, or something else entirely.
And given MS Office is the de facto standard, you’ll often see people open OOXML documents within non-MS office suites.
After all, OOXML is an open standard (sarcasm).
ODF (the document formats favoured by most other office suites) is also ZIP-based XML. So they too could be vulnerable.
To respond to Grandfather comment, modern Office files are really just ZIPs with different extensions, they even have the magic string "PK" at the very beginning of the file.
I do wonder, since a lot of tools outside of the MS ecosystem can read Office files (e.g. LibreOffice and Google Docs as well as plenty of other online tools), if indeed the hack as described by the article is possible. One would just need to figure out the ZIP stacks used by said tools.
You can even just rename a docx file to use the zip extension and then manually unzip it for those curious. If I remember correctly, the contents are XML files with structure encoding the formatting around the content.
The Office365 online and desktop implementations of zip could be different.
docx is supposed to be a portable format. There are many tools other than Word that open them. Just LibreOffice and Google Docs for example. They might well differ from Word or from each other in how they handle this case. Definitely worth testing!
> As everything looks good, you forward the ZIP with the invoice to the payment team
Nope, because a typical accounting asset wouldn't make it and you know it so you forward the PDF to them.
Obviously it sucks in the real world but I do always appreciate the cleverness of exploits like these.
The described exploit seems theoretical. In order to create the schizophrenic ZIP, the attacker would have to figure out what ZIP stacks are being used and ensure they act differently - if the 2 departments use the same stack, then the exploit can't work, can it?
A more realistic attack would be something like, slipping a malicious payload past a scanner by emailing a zip file that appears innocent when unpacked with the scanner’s zip implementation but produces malware when unpacked with the email client’s implementation. There’s a decent chance they’ll be different, and it wouldn’t be too hard to guess which ones a target might be using.
Often you don't have to guess, just use how the software responds as an oracle.
None of this stuff is theoretical. It's just old.
There was a time when passing ZIP files around was a very popular method of software distribution, and things like this were gotchas that had to be watched for. It was widely known, at least amongst sysops, that the varied toolsets that handled ZIP archives were functionally different. And there were scanners and sanity checkers, and bugfixes to PKUNZIP, that dealt in this stuff for uploaded files and FREQ responses.
Did people exploit the differences? Yes. Although it was mainly on the level of creating prank ZIP files on non-Microsoft operating systems with 8.3 filenames such as "PRN" or "CLOCK$".
* https://groups.google.com/g/alt.comp.virus/c/zLV-Y2a71gs/m/U...
However, the truly terrible idea of self-extracting archives was popular, which meant that archives with "interesting" arrangements of the archive within the overall file were widespread. ZIP comments were also liberally applied and altered by pretty much every BBS that passed an archive along. And the Unix people wanted to be able to use pipes, something that the MS-DOS original never had to cater for.
Also, there were people who exploited the fact that different tools took different things as gospel. Even within the past decade one can find people still being caught out by the fact that there's a header field that instructs what the pathname separator character(s) used are; and that ZIP tools that expect non-seekable streams operate differently to ZIP tools that expect seekable regular files.
Btw, in ZIP archives there is *no* header field that instructs what the pathname separator character(s) used are.
In theory there isn't, and everyone is supposed to use forward slash. But in practice there is, because not everyone has stuck to using forward slashes over the decades. Some tools are more forgiving of the differences amongst the various ZIP archive creators than others.
These people having fun with "Unix" versus "FAT" over the past decade are seeing the tip of the iceberg, given that there was a PKZIP for OS/400, there is a PKZIP for z/OS (and a competitor that claims to be cheaper), there are tools of varying degrees of Unixiness for systems like the Atari ST and OS/2, and a whole bunch of things have accrued over the years such as an outright extra header giving alternative filenames specifically for MacOS.
* https://michaelrommel.com/create/2022-12-28-malformed-zip-fi...
* https://unix.stackexchange.com/q/166159/5132
* https://github.com/filebrowser/filebrowser/issues/1768
It's not unlikely that devs and such will be on Mac/Linux and accounting will be on Windows. I agree it's still somewhat theoretical but interesting nonetheless.
Like spam, the exploit would still be profitable if only a small fraction worked.
Check out CVE-2017-13156 which is a real exploit that leveraged differences in zip parsing to bypass a signature scheme.
I don't see anything "another" about this; this problem is well known by $((CURRENTYEAR-10)) or so.
The author explains in the article that they previously gave a presentation outlining various techniques to achieve a "schizophrenic" zip file. The blog post discusses an additional technique that was not present in their previous presentation.
Author here :)
Are you saying that parser discrepancy is a well known problem OR that this exact technical issue described (CD offset vs CD size used to derive CD location) is a well known problem?
If the prior, you are of course right — parser discrepancy and the existence of these kinds of "multiple personality" files isn't a new thing. I'm pretty sure this can be traced back to at least the '80 :)
However, if you mean the latter, I would appreciate a link if you can recall where you've seen it before / read about it — I can then add it to the article.
Ok, so how do you handle them? Do you take screenshots of your PDFs before you forward them? Other ideas?
You can forward the PDF itself, not the zip file.
I don't know much about the subject but I do know that PDF is an executable format, and I suppose it could use fingerprinting (based on e.g. screen size or resolution) to become schizophrenic (?)