Facebook/Zstandard – Fast real-time compression algorithm

  • ZSTD seems to be relatively revolutionary in the field of compression, at least in most of the test cases I've tried. Binaries typically compress as well as LZMA2, averaging about a 50% compression ratio, with nearly an order of magnitude faster decompression.

    ZSTD beats DEFLATE and Gzip seemingly in every metric. It decompresses faster than almost any algorithm out there, save maybe LZO or LZ4, and has a compression ratio comparable to much slower algorithms.

    Arch Linux switched to ZSTD for compressing their package database at the beginning of this year, and only increased the overall average compressed size by 0.8%, while speeding up decompression several orders of magnitude compared to LZMA2.

  • As it happens, [I was adding Zstandard support](https://github.com/kafkajs/zstd) to [KafkaJS](https://kafka.js.org) yesterday. Sadly, the closest thing there is to official bindings for NodeJS haven't been updated for 4 years and don't build against newer versions of Node, so I ended up having to rely on some random fork that I have no reason to believe is going to be maintained.

    Unfortunately, the overlap of people that know C++ well enough to write high quality bindings and also have an interest in NodeJS is not that large. I tried to get into it myself, but there's a pretty steep learning curve and once you go beyond tiny toy examples it gets pretty hairy.

    If someone has the skills and interest in taking on a medium sized open source project that would be immediately useful to a lot of people in the Node community, Zstandard would be a good choice.

  • > Zstandard is dual-licensed under BSD and GPLv2.

    Both of which don't offer a patent license.

  • How well does it perform in WebAssembly?

  • I just would like for wireshark or tshark to handle zstd- or lz4-compressed pcaps. Then all would be good.

  • so hooli wasn't google after all

  • Yeah but what's its Weissman score?