Ubuntu QA:
BlogBrainstormPackage status
Log in
Ubuntu QA
The Ubuntu community has contributed 22700 ideas, 138270 comments, 2629576 votes
Idea sandbox Idea sandbox
Popular ideas Popular ideas
Ideas in development Ideas in development
Implemented ideas Implemented ideas
Idea #9737: Logging to database

Written by Eldmannen the 10 Jun 08 at 18:30. Category: System. Related project: Nothing/Others. Status: New
Rationale
== Problem ==
Current system logs events to a plain-text file. Parsing a large text file is slow and inflexible, while advanced data selection, filtering and sorting operations are impossible.

Currently all events in the log are indistinguishable. There are no semantic difference between information, warnings and errors. Which makes log analysis difficult, tedious, boring and time consuming due to the signal-to-noise ratio.

== Solution ==
Using a database (such as SQLite), would be very fast and flexible, and allow many advanced operations that just are not possible with what we have today.

It would allow chronological and reverse-chronological sorting of events. Grouping. It would allow to filter out or display events by time, category, severity, application, or tag, etc.

With a database it would be possible to sort or filter events by severity (e.g. only errors) or by tags (such as disk, local, network, security, etc).

Example, you could choose to only show the last 50 network-related events in the last 4 days sorted by severity in chronological order.
It would make system administration much more effective.

6
votes
up equal down
Solution #1: Auto-generated solution of idea #9737
Written by Eldmannen the 10 Jun 08 at 18:30.
Ubuntu Brainstorm was updated in January 2009. Since the idea #9737 was submitted before this update, its rationale and solution are not separated. Please vote accordingly, and if you have the necessary rights, please separate the rationale from the solution. Thanks!

Propose your solution

Attachments
No attachments.


Duplicates


Comments
deadowl wrote on the 11 Jun 08 at 05:44
You forgot to mention, though it would definitely speed up operations pertaining to queries, how much this would affect system performance in other aspects. I honestly don't care enough about the system log for it to slow other system operations for something that isn't system critical. If you had worded it such that this would be optional, I'd be all for it. I've got to -1 though, sorry.

Eldmannen wrote on the 11 Jun 08 at 15:28
I don't think this would negatively affect system performance at all. I don't think it would make any noticeable effect.

bgfeldm wrote on the 27 Jul 08 at 22:57
All databases add extra IO dependencies which slow down read write operations over reading and writing straight to a flat file.

Databases offer fast flexible searching across data.

Also log files are best when kept in the simplest format possible to ensure easy readability, portability and archive-ability.

Matthias Müller-Reineke wrote on the 26 Aug 10 at 10:34
One further advantage of such solution is that
Log messages would be stored without redundance. Now
logger -p local0.warn 'This becomes 2 messages!'
creates two entries (Ubuntu 10.04.1 with default log configuration). A database would be able to store log data as it was obtained with the C function syslog (see man 3 syslog) without data loss while the default logging configuration does much data processing (route messages to different files, replace utc with local time, strip the level ...) and data processing means data loss!

By the way using a database would ease solving http://brainstorm.ubuntu.com/idea/5082

Configuring the system log daemon to log into different plain files (as it is done in the default configuration) requires knowing the needs of log analysis in advance. This is inflexible.

It's clear that logging to database requires changes in packages like logcheck. One solution would be to create a command line client for the database which can write the results of the database query to stdout in traditional logfile format.

It my be that Sqlite isn't most suitable for this task as it introduces unnecessary overhead.

ulerch wrote on the 2 Feb 11 at 15:34
Why not keep writing the logs in a file as a default, and adding an extra database-solution for those who need and/or want it?

I could imagine synchronizing the database synchron or asynchron by configuration. So the argument of the performance loss gets less important, since you could choose between actuality and system performance.

I'm contributing to the project Apache ALOIS. I believe it could be a good starting point for such an idea. Moreover, we're currently working on getting the software fit for usage on a laptop and doing exactly what I briefly described above. Since we all use Ubuntu and expect that there's still a lot of work to optimize the system, we would warmly welcome any new contributor.

See: http://incubator.apache.org/alois/


Post your comment