Ubuntu QA:
BlogBrainstormPackage status
Log in
Ubuntu QA
The Ubuntu community has contributed 13767 ideas, 65602 comments, 1277055 votes

Idea #8083: Avoid data loss with unclean removal of USB Flash drives



up
150
down
Written by mp3phish the 5 May 08 at 02:16. Category: Hardware support.
Related to: Nothing/Others. Status: New
Description
This idea ties together ideas #8033 and #7989 with some improvements.

Problems with USB Flash Unmounting: 1) When you pull the plug on a drive, the system does not detect it and take care of the forced unmount for you. User has to go to command line. 2) When you unplug a drive, even when the light isn't blinking, unflushed buffers can cause significant data loss for no apparent reason.

Solution:

1) the system should detect when the drive is unplugged and if it is still mounted, a forced unmount should occur. This will prevent the user from having to use the command line for an every day occurrence. Many newbies reboot the system to get rid of this error state.

2) There should be a timeout set that after X seconds of inactivity (I propose 5 seconds) the write buffer should be flushed.

3) A new .fslog (or similar) file should be installed in all usb flash drives upon connection. If its a journaled filesystem, then don't, but for fat32 and others, it should do this. It will have the following function:
a. for every flush of the write buffer due to the timeout in part (2), write an entry into the log at the end indicating all writes are complete. This can be called the "clean" state
b. when writing to the disk with an initially empty write buffer, check to see if the log indicates a clean or dirty state. If clean, change the state to dirty.
c. when mounting a flash drive the system should read the log file, detect whether it is dirty or clean, and do a fsck if dirty, warning the user to not unplug the drive when it is in use.
d. Due to (a) and (b), at maximum, a log write could only occur once after 5 seconds of inactivity if the buffer has data in it. So it would not reduce lifespan of the flash. And even if your worried about flushing after 5 seconds, it would only correspond to writes already on their way to the disk, negating any problem with limited writes and lifespan of the drive.

The log file will be of negligible size and only grow with one entry for file transfers that happen greater than 5 seconds apart. Lifespan of the drive will not be hurt because these log entries will be VERY small and will only happen when writing activity is already going to happen to the disk anyway. The log can be cleaned up upon mounting of the drive. Since there is already a daemon running in the background to auto mount a USB flash drive, this functionality should be added to said daemon.



Tags: (none)

Attachments
No attachments.


Duplicates


Comments
mp3phish wrote on the 5 May 08 at 15:17
What are your thoughts on this idea, does anyone have any feedback on technical reasons this idea will or won't work? or other thoughts from a user perspective? One idea I heard is that maybe when the buffer is flushed the icon can change slightly to indicate its not in use.

Thanks.

cdurst wrote on the 5 May 08 at 18:05
How about just making mounted USB disks with FAT (or VFAT) filesystems mount with the "sync" option by default. That would force all I/O to be done synchronously as Windows does. Then it would be safe to pull it out as soon as the copy completes.

I think that, at least, users ought to be able to select "sync" as the default for hot-plugged storage devices from somewhere in the systemsettings GUI(s). At the moment, I don't think it's even possible without editing some magic hot-plug script somewhere.

The obvious problem is that if you use a USB interface for disks you don't expect to remove often (e.g. a Drobo), it would slow down all processes doing I/O to them.

Redrazor39 wrote on the 6 May 08 at 00:27
YES

Ssdg wrote on the 6 May 08 at 13:56
NO NO NO NOOOOO.
I'll vote this down like the others:"please kill my flash memory as fast as posible ideas".

because according to me disk buffers should stay for 2-5 minutes and it's too long to prevent "surprise" umounts.

And I'd rather learn to properly umount drives than loose drives frequently (with their files of course)

Mojo wrote on the 7 May 08 at 13:53
Please read my comments on related item http://brainstorm.ubuntu.com/idea/7989/ as I did not see this one first. Certainly thought should go into what proper time-outs or sync vs async modes. But regardless, some attempt at sane behavior and dialogs/alerts should be made. This is GOING to happen.

ALSO... Is it known which devices have lower (re)write lifetimes? Perhaps behavior in the background with regard to flush timeouts or use of sync/write-thru (whatever) can be decided intelligently by examining the device IDs.

Mojo

mp3phish wrote on the 8 May 08 at 02:24
Ssdg. You are just plain FLAT OUT WRONG. Please quit trolling these posts about the flash going out if you flush the write buffer. Everything in the write buffer has to be written anyway, so you aren't writing more than you normally would, just at different chosen times.

The whole purpose of the write buffer is so applications don't have to wait on disk I/O to work properly. So they get instant write access to RAM and the GUI is snappy for the user.

Are you telling me that you would sit around modifying the same file multiple times, and saving it every so often, and you would prefer that it NOT be written to the disk?

Your perception of reality is just plain backwards.

Auzy wrote on the 8 May 08 at 03:42
What we really need is a way, to mark devices as unique devices, and be able to select the level of protection they have.


1) Be able to mark them as fully synced, or not synced FROM A GUI!!!
2) Allow users to be able to change if they are synced or unsynced. Apple does this with the ipods.
3) Also track all transactions in ram, and if the drive is pulled out whilst there are still some, tell the user to plug it back in or they will lose data. Then they plug it in, it finishes the transactions, and then automatically eject it and tell the user when done.
4) I dont agree flushing every 5 seconds, I find it better to perform the same as it is now, but when idle, flush. In heavy I/O situations, you don't want to also be flushing your USB drive.

Moderator saivann (Moderator) wrote on the 8 May 08 at 07:57
I changed the title to speak only about the main part of this idea. This post should not contains more than 1 idea at the time. Automatic unmounting and avoiding data loss are two different ideas so idea 7989 is actually not a duplicate of this one.

mp3phish wrote on the 8 May 08 at 23:06
Auzy, I there there is confusion about the 5 second proposal, and I'll edit it to reflect. When I said 5 seconds, it means a 5 second timeout of an idle drive. You would not get a flush every 5 seconds unless you wrote to the disk, went idle for 5 seconds, and then wrote again, and repeat... So it would not flush when the disk is idle after the initial flush, and it will not flush when the disk has activity.

Auzy wrote on the 9 May 08 at 09:56
Ok, i guess that makes more sense..

seventynine wrote on the 15 May 08 at 03:10
+1. I hope that some smart person implements it.

frogitts wrote on the 8 Jun 08 at 08:09
What at least needs to happen is a way for the system to deal with a drive that has been unproperly removed, whether or not there are files waiting to be written. As far as I know, once the device is removed, that data is lost. Let's not have it's slot in /dev be taken up as well, adding insult to injury! The first item under solutions definitely needs implemented - I'm not an idiot, but I'm one of those newbies who thought he (since everyone's a he on brainstorm!) had to restart his computer every time Skype randomly restarted his usb devices for no reason :)

That is, before he made his own custom script to guide him through the process of remounting the drive at the proper place.


Post your comment