321 lines
14 KiB
Markdown
321 lines
14 KiB
Markdown
# rust-brotli
|
|
|
|
[](https://crates.io/crates/brotli)
|
|
[](https://travis-ci.org/dropbox/rust-brotli)
|
|
|
|
# What's new in 8.0.4
|
|
Fix: adjust versions of rust-decompressor and rust-alloc-no-stdlib and
|
|
alloc-stdlib so the Allocator<> trait is identical for all associated crates.
|
|
Return BrotliFileNotCraftedForConcatenation when a new stream header advertises more whole source bytes than have been buffered. This prevents the unsigned subtraction in shift_and_check_new_stream_header from underflowing on truncated metadata headers.
|
|
Return NULL from BrotliEncoderCreateInstance and BrotliEncoderCreateWorkPool when a caller-provided allocator returns NULL, rather than writing state through a NULL pointer.
|
|
Wrap the mutable Broccoli FFI entry points in a local catch_unwind helper, matching the encoder FFI convention so Rust panics do not unwind across extern C when std panic catching is available.
|
|
Return BrotliFileNotCraftedForConcatenation on caught panics and keep the existing pass-through behavior for no-std or pass-through-ffi-panics builds. Add regression coverage for a crafted stream input that previously panicked through BroccoliConcatStream.
|
|
Reject serialized BroCatli buffers with out-of-range live state fields before constructing the state. This keeps deserialize_from_buffer on its existing Result<BroCatli, ()> API while returning Err(()) for corrupt buffers that would otherwise panic on later use
|
|
|
|
## What's new in 8.0.3
|
|
Fix: avoid panic across Broccoli FFI boundary with BroCatLi
|
|
Fix: CompressMulti worker joins on errors
|
|
|
|
|
|
## What's new in 8.0.2
|
|
Fix for memory leak in ffi API
|
|
|
|
## What's new in 8.0.1
|
|
Compatibility for ffi builds
|
|
|
|
## What's new in 8.0.0
|
|
* Fixed LZ77 to comply with the specification
|
|
* No longer seed the context by the end of the lz77 dictionary. Instead use
|
|
zero for the seed as would happen without a dictionary. This matches the
|
|
behavior of brotli with a custom dictionary as specified in
|
|
* https://datatracker.ietf.org/doc/draft-vandevenne-shared-brotli-format/
|
|
|
|
## What's new in 7.0.0
|
|
* Fix error with short writes
|
|
* allow quality=10 for certain APIs and make it default to 9.5
|
|
|
|
## What's new in 6.0.0
|
|
* Remove unused SIMD use statements
|
|
* hide a few warnings - these are TODOs, and should be fixed in separate PRs
|
|
* do NOT build SIMD as part of MSRV -- doesn't make any sense to combine nightly with MSRV
|
|
|
|
## What's new in 5.0.0
|
|
* The FFI is no longer active by default to avoid ODR issues if multiple versions of brotli are included in several dependent crates.
|
|
|
|
## What's new in 4.0.0
|
|
Pinned to a rust-brotli-decompressor that can disable the ffi with the ffi-api
|
|
flag.
|
|
This can help avoid symbol conflicts with other brotli libs.
|
|
|
|
## What's new in 3.5
|
|
Updated SIMD support.
|
|
Better CI integration.
|
|
Cleaned up some of the clippy warnings.
|
|
|
|
## What's new in 3.4
|
|
Brotli decompressor's reader and writer has better behavior when operating upon brotli streams with extra bits at the end.
|
|
Optional features like stdsimd are now tested or disabled for now.
|
|
|
|
## What's new in 3.2
|
|
* into_inner conversions for both Reader and Writer classes
|
|
|
|
## What's new in 3.0
|
|
* A fully compatible FFI for drop-in compatibiltiy with the https://github.com/google/brotli binaries
|
|
* custom allocators fully supported
|
|
* Multithreaded compression so multiple threads can operate in unison on a single file
|
|
* Concatenatability mode to add the feature requested in https://github.com/google/brotli/issues/628
|
|
* binary tool catbrotli can accomplish this if the first file was specified with -apendable and the second with -catable
|
|
* validation mode where a file is double-checked to be able to be decompressed with the same settings; useful for benchmarking or fuzzing
|
|
* Magic Number: where the brotli file can have a useful header with a few magic bytes, concatability info and a final output size for pre-allocating memory
|
|
|
|
## What's new in 2.5
|
|
* In 2.5 The callback also passes down an allocator to make new StaticCommands and PDFs and 256 bit floating point vectors.
|
|
* In 2.4 The callback with the compression intermediate representation now passes a full metablock at a time. Also these items are mutable
|
|
in case futher optimization is desired
|
|
|
|
## What's new in 2.3
|
|
|
|
* Flush now produces output instead of calling finish on the stream. This allows you to use the writer abstraction to
|
|
get immediate output without having to resort to the CompressStream internal abstraction
|
|
|
|
## Project Requirements
|
|
|
|
Direct no-stdlib port of the C brotli compressor to Rust
|
|
|
|
no dependency on the Rust stdlib: this library would be ideal for decompressing within a rust kernel among other things.
|
|
|
|
This is useful to see how C and Rust compare in an apples-to-apples
|
|
comparison where the same algorithms and data structures and
|
|
optimizations are employed.
|
|
|
|
## Compression Usage
|
|
|
|
Rust brotli currently supports compression levels 0 - 11
|
|
They should be bitwise identical to the brotli C compression engine at compression levels 0-9
|
|
Recommended lg_window_size is between 20 and 22
|
|
|
|
### With the io::Read abstraction
|
|
```rust
|
|
let mut input = brotli::CompressorReader::new(&mut io::stdin(), 4096 /* buffer size */,
|
|
quality as u32, lg_window_size as u32);
|
|
```
|
|
then you can simply read input as you would any other io::Read class
|
|
|
|
### With the io::Write abstraction
|
|
|
|
```rust
|
|
let mut writer = brotli::Compressor::new(&mut io::stdout(), 4096 /* buffer size */,
|
|
quality as u32, lg_window_size as u32);
|
|
```
|
|
|
|
There are also methods to build Compressor Readers or Writers using the with_params static function
|
|
|
|
eg:
|
|
```rust
|
|
let params = BrotliEncoderParams::default();
|
|
// modify params to fit the application needs
|
|
let mut writer = brotli::Compressor::with_params(&mut io::stdout(), 4096 /* buffer size */,
|
|
params);
|
|
```
|
|
or for the reader
|
|
```rust
|
|
let params = BrotliEncoderParams::default();
|
|
// modify params to fit the application needs
|
|
let mut writer = brotli::CompressorReader::with_params(&mut io::stdin(), 4096 /* buffer size */,
|
|
params);
|
|
```
|
|
|
|
|
|
### With the Stream Copy abstraction
|
|
|
|
```rust
|
|
match brotli::BrotliCompress(&mut io::stdin(), &mut io::stdout(), &brotli_encoder_params) {
|
|
Ok(_) => {},
|
|
Err(e) => panic!("Error {:?}", e),
|
|
}
|
|
```
|
|
|
|
## Decompression Usage
|
|
|
|
### With the io::Read abstraction
|
|
|
|
```rust
|
|
let mut input = brotli::Decompressor::new(&mut io::stdin(), 4096 /* buffer size */);
|
|
```
|
|
then you can simply read input as you would any other io::Read class
|
|
|
|
### With the io::Write abstraction
|
|
|
|
```rust
|
|
let mut writer = brotli::DecompressorWriter::new(&mut io::stdout(), 4096 /* buffer size */);
|
|
```
|
|
|
|
### With the Stream Copy abstraction
|
|
|
|
```rust
|
|
match brotli::BrotliDecompress(&mut io::stdin(), &mut io::stdout()) {
|
|
Ok(_) => {},
|
|
Err(e) => panic!("Error {:?}", e),
|
|
}
|
|
```
|
|
|
|
### With manual memory management
|
|
|
|
There are 3 steps to using brotli without stdlib
|
|
|
|
1. setup the memory manager
|
|
2. setup the BrotliState
|
|
3. in a loop, call BrotliDecompressStream
|
|
|
|
in Detail
|
|
|
|
```rust
|
|
// at global scope declare a MemPool type -- in this case we'll choose the heap to
|
|
// avoid unsafe code, and avoid restrictions of the stack size
|
|
|
|
declare_stack_allocator_struct!(MemPool, heap);
|
|
|
|
// at local scope, make a heap allocated buffers to hold uint8's uint32's and huffman codes
|
|
let mut u8_buffer = define_allocator_memory_pool!(4096, u8, [0; 32 * 1024 * 1024], heap);
|
|
let mut u32_buffer = define_allocator_memory_pool!(4096, u32, [0; 1024 * 1024], heap);
|
|
let mut hc_buffer = define_allocator_memory_pool!(4096, HuffmanCode, [0; 4 * 1024 * 1024], heap);
|
|
let heap_u8_allocator = HeapPrealloc::<u8>::new_allocator(4096, &mut u8_buffer, bzero);
|
|
let heap_u32_allocator = HeapPrealloc::<u32>::new_allocator(4096, &mut u32_buffer, bzero);
|
|
let heap_hc_allocator = HeapPrealloc::<HuffmanCode>::new_allocator(4096, &mut hc_buffer, bzero);
|
|
|
|
// At this point no more syscalls are going to be needed since everything can come from the allocators.
|
|
|
|
// Feel free to activate SECCOMP jailing or other mechanisms to secure your application if you wish.
|
|
|
|
// Now it's possible to setup the decompressor state
|
|
let mut brotli_state = BrotliState::new(heap_u8_allocator, heap_u32_allocator, heap_hc_allocator);
|
|
|
|
// at this point the decompressor simply needs an input and output buffer and the ability to track
|
|
// the available data left in each buffer
|
|
loop {
|
|
result = BrotliDecompressStream(&mut available_in, &mut input_offset, &input.slice(),
|
|
&mut available_out, &mut output_offset, &mut output.slice_mut(),
|
|
&mut written, &mut brotli_state);
|
|
|
|
// just end the decompression if result is BrotliResult::ResultSuccess or BrotliResult::ResultFailure
|
|
}
|
|
```
|
|
|
|
This interface is the same interface that the C brotli decompressor uses
|
|
|
|
Also feel free to use custom allocators that invoke Box directly.
|
|
This example illustrates a mechanism to avoid subsequent syscalls after the initial allocation
|
|
|
|
## Using the C interface
|
|
|
|
rust-brotli is a drop-in replacement for the official https://github.com/google/brotli C
|
|
implementation. That means you can use it from any place that supports that library.
|
|
To build rust-brotli in this manner enter the c subdirectory and run make there
|
|
|
|
cd c && make
|
|
|
|
this should build c/target/release/libbrotli.so and should build the vanilla
|
|
command line tool in C for compressing and decompressing any brotli file.
|
|
|
|
the libbrotli.so in c/target/release should be able to replace any other libbrotli.so
|
|
file, but with all the advantages of using safe rust (except in the FFI bindings)
|
|
|
|
The code also allows a wider range of options, including forcing the prediction mode
|
|
(eg UTF8 vs signed vs MSB vs LSB) and changing the weight of the literal cost from 540
|
|
to other values.
|
|
|
|
## Stream Concatenation
|
|
|
|
Brotli supports creating streams that can be concatenated together, useful for streaming
|
|
scenarios where you want to compress chunks independently but decompress as a single stream.
|
|
|
|
### Simple Concatenation (Fast)
|
|
|
|
Use `-bare -appendable` for the first file and `-bare -catable` for subsequent files.
|
|
These can be combined using plain byte concatenation without special tools, with a
|
|
finalization byte (`0x03`) added at the end:
|
|
|
|
```bash
|
|
# Create the base file with header but no trailer (must specify window size)
|
|
brotli -c -bare -appendable -w22 input1.txt > base.br
|
|
|
|
# Create bare-catable streams (no header, no trailer, same window size!)
|
|
brotli -c -bare -catable -w22 input2.txt > part2.br
|
|
brotli -c -bare -catable -w22 input3.txt > part3.br
|
|
|
|
# Simple concatenation with finalization byte
|
|
# Note: printf '\x03' adds the required final byte
|
|
(cat base.br part2.br part3.br; printf '\x03') > combined.br
|
|
|
|
# Decompress normally
|
|
brotli -d combined.br -o output.txt
|
|
```
|
|
|
|
**Advantages:**
|
|
- Instant concatenation (no processing)
|
|
- No special tools required
|
|
- Bare streams can be appended in any order
|
|
|
|
**Requirements:**
|
|
- All files must use the same window size (`-w22` recommended)
|
|
- First file: `-bare -appendable` (has header, no trailer)
|
|
- Subsequent files: `-bare -catable` (no header, no trailer, no dictionary refs)
|
|
- A final `0x03` byte must be appended to complete the stream
|
|
|
|
### Efficient Concatenation (Size-optimized)
|
|
|
|
Use the `catbrotli` tool with `-catable` and `-appendable` flags for better compression
|
|
at the cost of processing time:
|
|
|
|
```bash
|
|
# Create files for catbrotli tool
|
|
brotli -c -appendable input1.txt > appendable.br
|
|
brotli -c -catable input2.txt > catable1.br
|
|
brotli -c -catable input3.txt > catable2.br
|
|
|
|
# Concatenate using catbrotli tool
|
|
catbrotli appendable.br catable1.br catable2.br > combined.br
|
|
```
|
|
|
|
**Tradeoff:** `catbrotli` produces smaller output but requires CPU time to process the
|
|
streams intelligently. Use this when size matters more than concatenation speed.
|
|
|
|
### Technical Reference: Stream Parameter Interactions
|
|
|
|
**Stream Types and Their Parameters:**
|
|
|
|
| Stream Type | bare_stream | byte_align | appendable | catable | use_dictionary | Description |
|
|
|-------------|-------------|------------|------------|---------|----------------|-------------|
|
|
| Standard | false | false | false | false | true | Normal brotli stream with header and trailer |
|
|
| First (simple concat) | true | true | true | false | true | Has header, no trailer - for simple `cat` concatenation |
|
|
| Subsequent (simple concat) | true | true | true | true | false | No header, no trailer, no dict refs - append to first |
|
|
| Appendable (catbrotli) | false | varies | true | false | true | For use with `catbrotli` tool |
|
|
| Catable (catbrotli) | false | varies | true | true | false | For use with `catbrotli` tool |
|
|
|
|
**Important Notes:**
|
|
- **Parameter dependencies are applied automatically** by the library in both CLI and API usage
|
|
- The library's `SanitizeParams` function ensures:
|
|
- `catable = true` → automatically sets `appendable = true` and `use_dictionary = false`
|
|
- `bare_stream = true` → automatically sets `byte_align = true`
|
|
- `!appendable` → automatically sets `byte_align = false`
|
|
- When using `set_parameter()`, dependencies are applied immediately
|
|
- When setting fields directly (e.g., `params.catable = true`), dependencies are applied during compression initialization
|
|
- **No manual fixups needed** - the library handles all parameter dependencies
|
|
- The `use_dictionary = false` for catable streams prevents references to bytes before the chunk boundary
|
|
- Simple concatenation requires a final `0x03` byte to complete the stream
|
|
- All concatenated streams must use the same window size
|
|
|
|
**Example API Usage:**
|
|
```rust
|
|
// First file: -bare -appendable equivalent
|
|
params.bare_stream = true; // Sets bare_stream=true, byte_align=true (automatic)
|
|
params.appendable = true; // Sets appendable=true
|
|
|
|
// Subsequent files: -bare -catable equivalent
|
|
params.bare_stream = true; // Sets bare_stream=true, byte_align=true (automatic)
|
|
params.catable = true; // Sets catable=true, appendable=true, use_dictionary=false (automatic)
|
|
|
|
// All parameter dependencies are handled automatically by the library.
|
|
// No manual fixups required - just set the primary flags you want.
|
|
```
|