Ubuntu QA:
BlogBrainstormPackage status
Log in
Ubuntu QA
The Ubuntu community has contributed 17459 ideas, 107690 comments, 2263278 votes
Idea sandbox Idea sandbox
Popular ideas Popular ideas
Ideas in development Ideas in development
Implemented ideas Implemented ideas
Idea #21158: allow parallel (de-)compression of archives

Written by timblech the 23 Aug 09 at 10:24. Category: System. Related project: Nothing/Others. Status: New
Rationale
the commonly used compression tools gzip and bzip2 are single-threaded, however, parallel implementations of these algorithms exist, pigz as parallel implementation of gzip, pbzip2 for bzip2.

it would be nice, if ubuntu would provide a facility, so that these parallel implementations could be used as drop-in replacement for the classic tools in order to make use of the processing power of current multiprocessor machines.
Tags: (none)

363
votes
up equal down
Solution #1: debian alternatives
Written by timblech the 23 Aug 09 at 10:24.
this could probably be introduced with the help of debian alternatives, providing a wrapper to the corresponding binaries.
1
votes
up equal down
Solution #2: use LZMA2
Written by kwinz the 30 Sep 09 at 21:26.
LZMA2 supports multi-threaded encoding using 2,4,8 or more threads while giving superior compression over gzip and bzip2.

Propose your solution

Attachments
No attachments.


Duplicates


Comments
cos wrote on the 25 Aug 09 at 16:58
Out of curiosity, do those make much of a difference over the single-threaded versions? I'd think the real bottleneck there is your harddisk, not your CPU.

psquared89 wrote on the 27 Aug 09 at 17:21
Depends on the size of the archive and the quality of the compression (as well as how much RAM you have). At the end of the day, there is no harm in using the multithreaded varieties, but there are benefits.

Shnatsel wrote on the 27 Aug 09 at 18:25
I've got a 4-core processor and 2 gb of ram, and often deal with 3-5 gb gzip and bzip2 archives. Multithreading is exactly what I need.

slashdotaccount wrote on the 3 Sep 09 at 09:12
Think about big archives - for example backups that contain GBs of data. I think this idea is really useful!

twright wrote on the 4 Sep 09 at 22:49
Well if it provides a big performance boost it should be done as the default so that everyday users can benefit from it instead of just the geeks.

OpenNingia wrote on the 7 Sep 09 at 08:59
When compressing bottleneck is CPU ( especially if set to an high compression rate )

In a most general way, any encoding processing ( video/music/crypto ) use a lot of CPU processing while decoding ( or decompressing ) is easier on CPU.

So here is my +1 for multi threading :)

MighMoS wrote on the 20 Sep 09 at 05:02
I've used pbzip2 before, and I can tell you that it is MUCH faster for decompressing archives, which is what is important. The HDD is not the bottleneck. Please try creating a largish (200-500MB archive) with bzip and pbzip and you can tell the difference.


Post your comment