Idea #9609: Store dpkg and apt files in an SQLite database

Written by flammon the 7 Jun 08 at 01:34. Category: System.
Related to: Nothing/Others. Status: New

Description

Store the dpkg available and status files in an SQLite database to improve performance, reduce file size, increase integrity, have better flexibility when searching and simplify updates.

The proof of concept has already started.

http://people.debian.org/~seanius/dpkg-sqlite/

sean finney wrote:

I have checked it out, very nice, besides missing a -ldl flag, everything went
according to plan. The speed is simply amazing, here is a small benchmark I did
on my machine:

omega3:/home/joe/build/dpkg-sqlite# echo 1 > /proc/sys/vm/drop_caches
omega3:/home/joe/build/dpkg-sqlite# time dpkg -S /usr/bin/gcc
gcc: /usr/bin/gcc

real 0m4.778s
user 0m0.252s
sys 0m0.152s
omega3:/home/joe/build/dpkg-sqlite# echo 1 > /proc/sys/vm/drop_caches
omega3:/home/joe/build/dpkg-sqlite# time ./poc /usr/bin/gcc
querying as if listing package /usr/bin/gcc:
querying as if for owner of file /usr/bin/gcc:
gcc: /usr/bin/gcc

real 0m1.306s
user 0m0.240s
sys 0m0.096s
omega3:/home/joe/build/dpkg-sqlite#

The difference is drastic. One of the main concerns now is reliability. Does
anyone here know of any SQLite horror stories they might like to share ?

Tags: (none)

Attachments

No attachments.

Duplicates

[Report a duplicate and merge its votes to this idea]

Comments

flammon wrote on the 7 Jun 08 at 02:12

I forgot to mention, SQLite also supports triggers which could be more robust than the current /var/lib/dpkg/triggers method and it would allow parallelization of tasks.

Triggers would be fired based on package dependencies. Packages with no dependacies would be installed first in parallel and they would fire triggers to continue the installation of their dependents.

Auzy wrote on the 7 Jun 08 at 06:03

This is a great idea, and it doesn't introduce any backward compatibility issues.

And it makes perfect sense :D

SQL reduces the need to reinvent the wheel. Because they can handle all the optimisation of the database and the code. We just worry about using it.

And, SQL has more features. We get the new features free.

Awesome idea. +1

Sector wrote on the 7 Jun 08 at 10:21

I completely agree. I use SQLite in programs I make all the time, its simple, fast and just works. Although I don't really get why the firefox people took so long to make a bookmarking system in SQLite :).

I think application managing systems in a distro, which are responsible for updates/new programs, should be at the pinnacle of their speed capacity, because it's a thing that many people often come in contact with

+1

sebsauvage wrote on the 9 Jun 08 at 11:14

+1

SQLite is very fast, space-efficient and ensures data integrity.

But woudln't the fsync() problem (as in Firefox 3) cause any performance problem ?
(Shouldn't since dpkg is not the kind of program you run all the time.)

Auzy wrote on the 9 Jun 08 at 15:12

The fsync problem I think they said was due to windows not properly syncing with the disk. I might be wrong though.

Post your comment