Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Similarly, for bzip2 there is pbzip2 (http://compression.great-site.net/pbzip2/?i=1).

zstd & xz support the "-T" argument for setting thread count. If you pass "-T 0" it will attempt to detect and use a thread per physical core.



AFAIK (not 100% sure), multithreading support is different - parallel versions split the file in multiple segments, and compress each independently, which multithreaded functionalities apply to the same stream (no hard splitting). For this reason, there's for example pzstd, in addition to zstd.


Yes, pbzip divides up the file into blocks per core. Though I think some versions (older?) of bzip are unable to handle pbzip archives.

I used pbzip2 on an old octo core xeon server with a decent sas raid and was able to compress at well over 200MB/sec, closer to 300MB in some cases.


bzip2 is already block based though, so there is no compatibility issue in that specific case (vs pbzip2) though I think pbzip2 supports larger blocks than the original bzip2.


tangential question: compressed files look like hashes in terms of if something changes in the beginning, all the other parts are different, right?


Usually but it depends on the compression scheme. There’s usually a “window” of how far back they look, so they can resync after a while, but it’s unlikely and the offsets will likely have changed so you need to handle that.

You can force this property by introducing synchronisation points though, gzip has an `—-rsyncable` which makes that a lot more likely, at a small compression cost.

Edit: apparently zstd has also had —-rsyncable for the last 5 years.


For compression efficiency, it makes sense to have one large common dictionary. For compression speed, it is easier to have a dictionary per chunk. I still hope they use the common dictionary; if so, any change in the beginning likely affects further parts if it affects the dictionary and thus the way how the later parts are compressed. Same for farther parts affecting the way earlier parts are compressed.


I have seen at least one case where pbzip2 created files which could not be opened by some .NET implementation of the decoder, but the same decoder could open files created by lbzip just fine. No idea why.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: