Right now, the boot menu has regular Ubuntu, recovery mode, and memtest. With just a simple Down arrow, someone can accidentally boot into recovery mode.
Perhaps we should make recovery mode a little less obvious. Is there some way to make a special key combination or something to access recovery mode? Mac OS X has you hold down Cmd-S during bootup to boot into single-user mode. I'm not sure if Grub can do this, but it'd be cool if it could.
The reasoning behind this is more user-friendliness. Users who aren't comfortable at the command prompt might accidentally end up there and not know what to do.
The added benefit is that those who need the illusion of better security would also stop complaining that the existence of recovery mode is a security problem.