ZFP wavelet compression
OIV Xtras - ZFP Wavelet Compression
Open Inventor volume rendering (VolumeViz) provides a native file format called LDM. The LDM format was designed over ten years ago and introduced the concept of storing volume data as multi-resolution tiles. This innovation allows random access to all data, very fast display of initial (approximately correct) images, and “best possible” rendering on any size hardware.
VolumeViz supports the concept of data compression in LDM using a plug-in architecture to allow enhancement and customization by our customers. We currently support compression using the gzip, JPEG, and JPEG3D algorithms. Gzip is currently the best option, but gzip only supports lossless compression and is not optimized for numeric data. Modern wavelet compression algorithms, designed for numeric data, can provide much higher compression ratios with “near-lossless” and “visually lossless” results. JPEG is not very useful because it is lossy only, introduces artifacts, and only supports 8-bit data. JPEG3D is not very useful because it does not support floating-point data, and decompression is very slow.
Introducing: Data compression for LDM using the “zfp” library
Zfp is an open-source data compression library developed at LLNL for HPC applications. Zfp is designed to maximize both compression performance and compression ratio for numeric values. In addition to lossless compression, zfp provides lossy but error-bounded compression. Zfp lossy compression substantially reduces data size without introducing visible artifacts, as you see with JPEG algorithms. Zfp is the compression algorithm used in the ZGY file format via the OpenZGY API. There is an in-depth discussion here. Zfp is also used in Equinor’s seismic-zfp format. So we know that zfp is a good solution for seismic data.
Overall results:
- Zfp is typically 4X faster than gzip for float data.
- Zfp lossless compression ratios are similar to gzip.
- Zfp visually lossless compression ratios are 2X to 3X better than gzip.
- Exception: Zfp lossless compression is less good for 8-bit data.
Some specific results:
Notes: gzip compression level 3 is typical, and Zfp compression level 40 is a visually lossless PSNR. All sizes in bytes.
| Original | Gzip level 3 | Zfp level 40 | |
|---|---|---|---|
| 8-bit data Engine | 9,011,712 | 3,720,192 58.7% 0.5 sec | 3,016,192 66.5% 0.5 sec |
| 16-bit data DicomSample | 398,461,440 | 177,954,304 55.3% 11.2 sec | 12,130,304 97.0% 6.6 sec |
| Float data F3_entire (seismic) | 1,853,888,000 | 854,443,008 53.9% 71 sec | 380,078,080 79.5% 15 sec |
| Float data Kahu (seismic) | 8,346,689,024 | 6,657,977,344 20.2% 290.5 sec | 1,417,525,248 83.0% 79.8 sec |
| 16-bit data CGG (seismic) | 28,852,782,592 | 20,715,173,888 27% very long… | 4,546,709,504 84.2% 780 sec |
Usage:
Adding zfp compression support in a C++ application is very easy: Just include “ZfpDataCompressor.h” and call the method ZfpDataCompressor::initClass().
- The zfp compressor is now registered with VolumeViz.
- Compression
- You can now use the LDMConverter (after rebuilding it) to create an LDM file with zfp compression using the command line argument “-c zfp”.
- You can now use SoConverter to create an LDM file by calling the method SoConverterParameters::setCompressionName(“zfp”)
- Decompression
- VolumeViz will automatically select the zfp decompressor if the LDM file header specifies “zfp” as compression.
The zfp data compressor header and source files are part of the OIV Xtras project. To get them please contact your Product Application Specialist (PAS) or account manager.
