[PLUG] Compressed data files

Matt Alexander m at netpro.to
Tue Nov 12 05:36:31 UTC 2002


If you really want to reduce the file sizes for your programs, check out
lzip:
http://lzip.sourceforge.net/
It provides better compression than even bzip2, which is pretty
impressive.
~M


On Mon, 11 Nov 2002, Bear Giles wrote:

> Ed Sawicki wrote:
> > I'm wondering how some programs deal with data files that
> > are sometimes compressed using gzip. I haven't yet written
> > code for this.
>
> The zlib (and bz2lib) libraries are easy to use.  You can either use the
> stdio-ish routines, e.g., zfopen(), zread(), zprintf(), or call the
> inflater and deflater routines directly.  That takes a bit more work,
> but it gives you *much* more flexibility in doing things like inserting
> a Z_SYNC_FLUSH after every meaningful chunk of data (e.g., every file in
> an archive) so that you can recover the maximum amount of data even if
> the file is subsequently corrupted.
>
> As for gzip in particular, it's a well-documented format that provides a
> wrapper around the zlib output.  Things like original filename,
> timestamp, and at the end of the file the uncompressed size and CRC
> checksum.
>
> Put it all together and you get a very nice function that I'm surprised
> has never been put into a library - maybe it's just too obvious - ...
>
> 1) open the file
> 2) lock it
> 3) mmap it
> 4) check the first few bytes - if it's a GZIP file mmap a buffer large
> enough to hold the uncompressed data and blitz your original file into
> the second mmap file.  Unlock and close the original file.
> 5) return the structure containing the information needed when you close
> the file - file pointer, locks, mmap ptr, etc.
>
> This is the type of code that can be a pain to get right the first time,
> but once you have it working it can be reused everywhere and you wonder
> how you ever got by without it.
>
>
> _______________________________________________
> PLUG mailing list
> PLUG at lists.pdxlinux.org
> http://lists.pdxlinux.org/mailman/listinfo/plug
>





More information about the PLUG mailing list