From 0c2f76b32eff1defe2b6767ecf09f3ecb77f5298 Mon Sep 17 00:00:00 2001 From: Jonathan Dieter Date: Mon, 19 Apr 2021 20:44:57 +0100 Subject: [PATCH] Add documentation for generating zdicts Signed-off-by: Jonathan Dieter --- README.md | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/README.md b/README.md index 379ebe0..9244aa4 100644 --- a/README.md +++ b/README.md @@ -50,6 +50,37 @@ To read a zchunk header, run: zck_read_header ``` + +## Zchunk dictionaries + +If you're creating a zchunk file that has any repetitive data, you may be +able to reduce the file size by using a [zstd dictionary](https://facebook.github.io/zstd/#small-data). +The dictionary is stored as extra data in chunk 0, but is used as a starting +point for compressing each chunk, which can give a significant overall savings. + +It is important that all further revisions of the zchunk file use the same +dictionary. If the dictionary changes, none of the chunks will match from the +old file, and the full new file will be downloaded. + +Zchunk can use any zstd dictionary, but also includes a utility to generate the +ideal zstd dictionary for a zchunk file. + +To create an ideal dictionary for a zchunk file, run: +``` +zck_gen_zdict +``` + +The dictionary will be saved as ``. + +You will then need to recompress the file with the dictionary: +``` +zck -D +``` + +Note that `zck_gen_zdict` does require that the `zstd` binary be installed on +your system. + + ## Documentation - [Format definition](zchunk_format.txt) - [Initial announcement](https://www.jdieter.net/posts/2018/04/30/introducing-zchunk) -- 2.30.2