Ubuntu QA:
BlogBrainstormPackage status
Log in
Ubuntu QA
The Ubuntu community has contributed 17459 ideas, 107690 comments, 2263278 votes
Idea sandbox Idea sandbox
Popular ideas Popular ideas
Ideas in development Ideas in development
Implemented ideas Implemented ideas
Idea #21124: To give people computers that do thing much more effectively

Written by bigsam72 the 20 Aug 09 at 09:15. Category: Usability. Related project: Nothing/Others. Status: New
Rationale
Nowdays the users have thousands files on their filesystem's. Exists toools like find/grep/locate/slocate & exists thing/api's like inotify/dnotify. Example, i remember that i want file which holds text "Perpetum" but i don't rememeber where is stored (in worst case on which of my 5 computers). I could try "find / -exec grep "Perpetum" \; and wait for several minutes/hours. I think that we have "stupid approach" to files, like slocate/locate. Indexing services which kill's computers because the try to index files via "full scan" of filesystem and kills the system via I/O.
My idea is simple.
Modify Linux kerenel ( i am not kernel developer ) on VFS level to store 2 new numbers,word (FILE_GROUP_TYPE_xxxx & FILE_TYPE_xxxx ) for each file and create new call "set/get_file_type". For exmple when gimps create's jpeg sets set_file_type(int handle, FILE_GROUP_TYPE_IMAGES, FILE_TYPE_JPEG); Second step is to modify inotify/dnotify or create new mechanism to populate "changes/events" from kernel space to userspace. This API allows me and others to create "thousands" of plugins/daemons/scripts like indexers, mail convertors, mail backupers.media convertors and whatever you want to do or better utilize existing tools.

Exapmple one.
Somebody creates indexd (userspace daemon) which "catch" from kernel via "dnotify/inotify/new_api" event new file created. and store this info to memory ( inode,name,group_type,file_type). The next step is to modify "find" or create new "tool" to find files ...

find --file_type=FILE_TYPE_JPEG ... which respons in miliseconds instead of minutes.

Example two:
Via kernel_api i register "new script" on folder "convert_to_flash". I wrote my "script" which convers .avi -> .flv So when user do cp /media/disk/file0001.avi /home/user/convert_to_flash systems automaticaly do conversion on the fly to .flv even better user for example in nautilus do drag and drop to this folder.

Example three:
Unix admin want to know how many shared objects files hold his file system ? find --file_type FILE_TYPE_SO | wc -l

Example four:
Imagine i have mounted xx TB filesystem which holds thousands of files and i want to find some file. During mount ( my system catch event "NEW_MOUNT" populate this for example via DBUS ? and because i have my new "indexd" deamon, this daemon try to contacts "indexd" on remote machine after successfull mount. So when i want to find file i have chance to find this file across all mounted filesystems :-).

Example five:
In unix is everithing file .. so dev/sda4 has FILE_GROUP_TYPE_FILESYSTEM, FILE_TYPE_EXT4 ..



Benefits ?
No stupid background indexers like slocate/locate and others which kills your notebook/machine in worst time of day. Every user loves situation when he need to do something, but his notebook do huge amount I/O operations because of "

( user space daemon which index "FILE_GROUP_DOCUMENTS" files on the fly)

Incredible fast searching via "new find".

When i want to search document which hol'd text "

Indexers which will try index YOUR DOCUMENTS' via FILE_TYPE_ID not 400GB of yours MP3's. And so on.

I know that this idea should be posted to kerenel developers but i am not kernel developer and from my point of view i don't know how to contact them.


6
votes
up equal down
Solution #1: Better Filesystems
Written by bigsam72 the 20 Aug 09 at 09:15.
Lot of work ...
13
votes
up equal down
Solution #2: help / improve / push fanotify
Written by dino the 22 Aug 09 at 18:00.
fanotify is like an advance versoin of "inotify". One of its goals is to overcome inotify's scalability problems.

fanotify would help Tracker with several (scalability / performance) problems.
6
votes
up equal down
Solution #3: Mimetypes
Written by slashdotaccount the 3 Sep 09 at 09:27.
using mimetypes instead of the proposed type-constants is more flexible and based on what we have already today

Propose your solution

Attachments
No attachments.


Duplicates


Comments
Lachu wrote on the 20 Aug 09 at 10:56
Very good idea.

Also, look at inotify-tools or http://slawek.lach.art.pl/FlashMovieMp3Conv.sh .

The main reason do not do that, is that are already implemented.

Look also on Nepuk. Nepomuk is file indexer with tags. It works in very similar way, but in userspace.

dino wrote on the 22 Aug 09 at 18:08
So there are two "new" things to the kernel in your propsal:

1. storing mimetypes in filesystem
2. change notification for all files regardles where they are

Hope I got it correct?

1. is possible right now. Not for all filesystems though. Requires to change userspace programs to provide them.

2. a recursive inotify would perfectly solve the issue of indexers - fanotify will probably solve this too.

I dont get the point why would the indexer just "fly"? It still needs to crawl through all your documents and extract the words out if it causing lots of I/O.

bigsam72 wrote on the 27 Aug 09 at 08:34
"on the fly" i means continually. When new file is created, index them. When file is modified index him.

Lachu wrote on the 28 Aug 09 at 06:10
fsnotify is bad, because it can block many operation. Inotify is pretty good - fsnotify is designed for Anti-Malware userspace programs, but on Linux we don't have viruses.


Post your comment