|
Description
This is an attempt at a unification of:
http://brainstorm.ubuntu.com/idea/7081/
http://brainstorm.ubuntu.com/idea/7390/
http://brainstorm.ubuntu.com/idea/7649/
http://brainstorm.ubuntu.com/idea/7725/
I can't think, nor have I heard, of any showstopper reason for why BitTorrent shouldn't be used as the primary download method of Ubuntu respository packages. Although the specifics of the implementation of this idea will be different for ISOs and repositories, I feel they should be unified in the brainstorm because the goal is to allow the rapid, efficient, reliable, and available download of Ubuntu software.
Implementation Benefits
1) Speed. All Ubuntu downloads (ISO downloads, dist upgrades, regular system updates, and new application installs) will as a whole be faster. Generally torrent download speeds benefit from higher numbers of downloaders that seed, which Ubuntu users have demonstrated they are prone to do. BitTorrent is better able to absorb (and eventually use as an asset) large numbers of users attempting to download data at the same time, such as with the recent mad rush of Hardy downloaders/upgraders.
2) Efficiency. The BitTorrent protocol has proven to be one of the most efficient methods of distributing data amongst a large number of clients. It will harness the collective upstream of tens of thousands of Ubuntu users, from DSL and cable connections to the fastest of corporate connections.
3) Reliability. Checksums guarantee the integrity of BitTorrent downloads, so data corruption is much less likely to occur. Only the pieces that fail checksum are redownloaded, contributing to points 1 and 2.
4) Availability. Each seed's data and each distributed copy of data from the peer collective combine to provide a level of redundancy that far exceeds even the most exhaustive list of HTTP and FTP mirrors. The ISOs and repositories would never be offline, even if Canonical's servers went down for weeks (many torrents retain availability after the trackers have gone offline).
Implementation Ideas and Challenges
1) Package/ISO Tracker Allocation. Due to the huge number of packages in the various Ubuntu repositories, special attention would have to be given to how the packages were allocated to Canonical's BitTorrent trackers. A logical relationship would need to be established between the existence of a tracker and the torrents and torrent data it was serving. I suggest that an algorithm be developed generate both trackers and torrent files programmatically, based on the current method of tracking packages and their dependencies. Synaptic, etc., will synchronize with this list.
2) Because a single torrent can contain multiple files, any of which can be downloaded by themselves or in groups, it would be logical to group multiple related packages in a single torrent. For instance, a package with 50 MB in 10 required dependencies could include these dependencies in the torrent. The torrent could point to a single copy of each package on Canonical's servers to eliminate storage redundancy. Synaptic, etc., would only download the packages in the torrent that weren't already installed, rather than downloading the entire torrent. This would also eliminate the need for a single app install to download hundreds of separate torrents.
3) Much like the popularity contest opt-in, Ubuntu users could choose to seed their installed packages as a way to contribute to Ubuntu. Users could schedule seeding and throttle the allocated bandwidth, and it could be performed as a silent background process. Even more, Canonical could award such seeders with Ubuntu store credit for each gigabyte seeded. This would result in more people with Ubuntu T-shirts, hats, etc., and a stronger community and public presence, while saving Canonical untold dollars in bandwidth expense.
4) Some smaller downloads would be slower to install, particularly if for whatever reason both multiple torrents and trackers needed to be connected to. Torrent downloads tend to start slow and speed up as more connections are gained. Generally there is a significant lag before any data is downloaded, proportionally greatly increasing the download time of a small download. This could be eliminated by allowing the traditional method of download for smaller packages/updates. The Update Manager or Synaptic could make this judgement.
5) Downloads of multiple applications from Add/Remove or Synaptic could be executed simultaneously rather than sequentially, greatly decreasing the total time spend downloading.
6) System updates and other packages could automatically be seeded by the system after download until a user-defined share ratio is attained.
7) Heavy use of trackers by Canonical's servers would migrate the expense from bandwidth to processing power, possibly requiring a retooling of their server hardware infrastructure.
That's all I got...
Attachments
Duplicates
Comments
|
Adys wrote on the 28 Apr 08 at 21:00
| |
While the idea does sound awesome, once you get at it you'll see a fair few ISPs throttle the entire torrent protocol, and slow connections will have the hardest time ever with this.
|
|
XSP wrote on the 28 Apr 08 at 21:05
|
^^^^
Not to mention that "unlimited" access is indeed limited. Should you exceed their acceptable use limit, they will do one of two things:
1. Charge you the extra bandwidth consumption.
2. Cancel your service.
Not all companies do this, but Comcast has made it clear there is indeed a limit to their "unlimited" accounts.
|
|
zephyrcat wrote on the 28 Apr 08 at 22:31
| |
I still think this is a good idea, but Comcast at least used to just stop BitTorrent downloads, not just slow them down. As long as after a few failed attempts it reverted to regular downloads, it would be great.
|
|
mp3phish wrote on the 29 Apr 08 at 03:35
|
To prevent problems for people who can't torrent, this should be stated like...
8) Torrent should be default but OPTIONAL. There should still be the traditional method of mirroring and downloading directly from the servers. Torrent could be default, but there should be a checkbox or something in add/remove programs to enable/disable peer to peer updating.
9) By default, bittorrent should only offer a 1:1 ratio of uploading. This means once the download is complete and one full upload amount of bandwidth is used up, it should shut off automatically. This will prevent unsuspecting users from having their upload taxed 24/7 unwittingly. You shouldn't be silently punished for using Ubuntu. Likewise, there should be a setting for how large the download to upload ratio can get prior to shutting off, and a bandwidth cap and peer cap for uploads (default to 128kbps up and 1 upload to a peer at a time).
This would be a sane default for most users on cablemodem/DSL on the consumer end, an allow enthusiasts to "crank" their p2p ability up so to speak, and get more prizes from canonical. The primary idea here is that the users who "leech" (ie, don't offer up more bandwidth than default) will not be a net drain on the system, but those that want to contribute more should have that option in an advanced panel.
|
|
Auzy wrote on the 29 Apr 08 at 07:35
| |
Heavy encryption can also help get around filtering. So if it ran at the highest encryption level, its possible, filtering wont really be that plausible any longer
|
|
zooounds wrote on the 29 Apr 08 at 08:03
| |
I get 2 Mbyte/s ordinary download of packages.
|
|
holizz wrote on the 29 Apr 08 at 10:21
| |
-1 because torrents are slow.
|
|
Endolith wrote on the 29 Apr 08 at 14:01
| |
+1 because torrents are super fast.
|
|
U53R wrote on the 29 Apr 08 at 15:35
| |
-1 My router RAPES torrenting /p2p'ing. So any body who has a piece of *$&@ router, or a poorly configured one, under this system, would get ubuntu updates at TERRIBLE speeds, this is a horrible idea. The current update speeds are fine with me. The majority of people dont need 700kbps (for example) if it means a good fraction of people wont get updates at all, or at speeds under 20kbps.
|
|
maniacmusician wrote on the 29 Apr 08 at 17:24
| |
As others have said, ISP's are often just merciless when it comes to P@P handling, as are a lot of college networks. Really not a viable option for everyone until something is done about the throttling and blocking of traffic.
|
|
pturing wrote on the 29 Apr 08 at 17:52
|
+1
But it's important that
1. You can turn it off
2. It automatically fails over to normal http download (like the World of Warcraft updates do)
|
|
kazagistar wrote on the 30 Apr 08 at 00:33
|
-1
My Internet has very small upload speed, and it throttles download speed, so downloading is usually faster then torrenting. Also, in my experience, whenever there is the choice between downloading and torrenting, the provider provides very poor service and speed for the downloaders.
|
|
qaaq wrote on the 30 Apr 08 at 01:31
|
This idea would rule on a large private network, as a 'try this first before hitting the Ubuntu archives' kind of a thing. Swarming downloads may be iffy for some users on the public Internet, but an internal, no-administration, automatically-set-up bittorrent tracker on our internal lan and some autodiscovery magic would be awesome.
|
|
bigfatball2 wrote on the 30 Apr 08 at 10:18
|
excellent idea all the other linux distros do the same
keep a few mirrors up for non p2p users and limited bandwith users
in france no limit upload 24/7 for opensuse but definetely would prefer doing this ubuntu
nice idea about the credits gained for the shop if you upload a lot and therefore help ubuntu community
by the way http and ftp servers can be used to boost the slow torrents and there could be a dedicated page for willing uploaders to reference the torrents which need to have an upload boost
+1
|
|
keros wrote on the 30 Apr 08 at 13:46
|
Once I started using BT, I thought what a great way to distribute Linux. However, I'd like to see the following:
1. BT should be recommended, but not default. Add a nice description and this will keep most users from being punished by their ISPs.
2. Allow max bandwidth option. This should included both the option to cap the up/down rate as well as a maximum total upload size.
3. Allow the option of only getting torrented packages through the local subnet. If not available, then download from public repository. This would be a beautiful thing for corporations with Internet connections that are near capacity. (or see #2)
|
|
markush wrote on the 1 May 08 at 22:00
|
I'm living in a students-residence and torrent ports are blocked ->
-1
|
|
Wouter.de.Groot wrote on the 2 May 08 at 10:38
|
Number 3: popularity contest submits a list of packages, it does not consume lots and lots of bandwidth. This system effectively turns Ubuntu in a pay-for release: I have to fork over bandwidth.
Also, giving users store credit is strange: we save money by giving people stuff! Please, give us some numbers before you make claims like that.
Furthermore: giving users the choice to regulate bandwidth, times and whatnot for seeding adds quite a bit to setup complexity, and most users will see this as something they need to do to restrain their OS from doing bad things. We do not want that image.
Remember folks, this is supposed to be a mainstream OS. All this stuff appeals to some hardcore users and will likely scare away the rest ("It will use MY internet connection so that THEY can save money?!")
Sidenote: I am all for offering Ubuntu ISO downloads as BitTorrent. I know Canonical already does, I get mine that way, but they're buried deep in download pages most people never see. Put up links to the torrents front and center.
|
|
tgape wrote on the 2 May 08 at 11:42
|
I'd be more interested in this idea, except I paid attention to what was going on during my install.
There are many packages on the install CDs which are currently being downloaded from ubuntu.com even if the CD version is the latest current.
Shameless plug: if you want to fix the download problem, the highest improvement for the required effort fix right now is probably http://brainstorm.ubuntu.com/idea/7959/.
Of course, if you have a 4x CDROM drive, and a 12Mb download speed, the current solution may work faster. I doubt most people have that kind of a situation, however.
|
|
Baggers wrote on the 2 May 08 at 15:48
|
Good idea but should not be default.
Many universities and other institutes block torrenting. We need ubuntu to work out the box.
|
|
bigfatball2 wrote on the 4 May 08 at 00:25
|
hey guies if you listened a bit it is said torrenting the iso would become the default source but wouldn't delete all the mirrors which means people with small downloads would be able to use it.
opensuse does it just do the same!:
http://software.opensuse.org/
it's obvious that credit idea isn't going to work cause their's no way you're going to make economies like that
anyway leaving ubuntu cause it's 10 time worse quality than any windows distros so changed for opensuse now. bye
|
|
Ape wrote on the 7 May 08 at 15:49
| |
Is this really good idea for apt-get? The updates are quite small files and I can currently download them with the speed of 100Mbps from the repos. Bittorrent wouldn't make this any better.
|
|
Auzy wrote on the 8 May 08 at 01:34
|
In ideal periods, HTTP is ok..
- Except, HTTP sucks at validating files really.
- Some mirrors aren't kept up to date well, or have issues. Bittorrent is more "fuzzy", and doesn't suffer from bad mirrors really.
- Bittorrent also gets faster under more load. The download mirrors die every dist upgrade.. Bittorrent will get around that
|
 saivann (Moderator) wrote on the 8 May 08 at 07:02
| |
I removed ISO from the title since we should not mix multiple ideas in the same post. Idea 7390 is already about the ISOs. Apt-get based on bittorrent is a really unique idea which require the involvement of specific teams and it should not be mixed with other ideas. Keeping one idea per post is the best way to guarantee that people will be able to work on ideas.
|
|
barsanuphe wrote on the 10 May 08 at 13:03
|
I am amazed that many people vote this down because THEY, specifically, will not be using this, even though it has been specified many times that this idea should be an OPTION.
To people mentioning Comcast: all ubuntu users are not American subscribers to Comcast. Please think of the overall quality to worldwide users.
I think it would be the best use of my unused bandwidth yet. Personnally, I would more than gladly seed packages.
Plus, I think it would really help during version releases: many people would get up to date before the release, and therefore be able to seed those who would use update-manager the day of the release, hence considerably lowering the strain on the official http servers.
|
|
leight wrote on the 12 May 08 at 12:34
|
Bittorrent if it is to be used should only be an option.
I for one don't use Bittorrent or any peer to peer programs
ISP's here in Australia are starting to reduce the bandwith available to peer to peer connections.
So I don't see any advantage in using Bittorrent.
As I said make it an option but not the default.
|
|
Mishtal wrote on the 13 May 08 at 02:18
|
I'm a little confused why other commenter's seem so against this suggestion.
Most of you know that P2P is a decentralized download protocol. You download the .Torrent file from a server, your respective Bittorrent program (transmission, uTorrent, BitTorrent, BitComment, Asurous (spelling?), W/e) will connect to the tracker and then download the file from Everyone connected to the tracker. So, instead of everyone downloading the packages directly from the official package repos mirrors (which, after a while, gets expensive in terms of server maintenance, bandwidth, employees to keep stuff working, etcetera etcetera) the mirror only has to send one single copy of the file into the swarm before the torrent can be distributed 100% to all downloaders, so long as there is at least one copy of every chunk of the file in the swarm at any given moment. this cuts the server load for the mirror operators down by a seriously significant amount.
torrent speeds might not be all that great sometimes, and sometimes any peer to peer at all is blocked by your ISP (college, work, w/e). thats why, the package manager would do a check first with the server "what speed can you give me today, mr server? do you know if the torrent swarm would be faster today, mr. server?" and, if you allow synaptic to use bittorrent to download the package, you cut the server load, and potentially get your package faster when using bittorrent.
there is significantly less chance of getting your download preempted with someone elses malicious file, considering that your download isnt coming from One Single place which might just happen to have a man-in-the-middle. if any particular piece of the package comes from a malicious host, BitTorrent would reject it, it wouldnt pass the checksum, and BitTorrent would re-download it, over and over and over until it did. (which would more than likely happen the first time). so out of the box implementing Bittorrent increases security slightly.
if for some reason the download is taking much longer than it really should (or if some some reason there is a failure during the download with Bittorrent, synaptic would default directly back to HTTP. potentially without the user even noticing that that happened.
for those of you who are afraid of being used and abused by your isp (and for those who are tracking the media fiasco with P2P being claimed to be used for nothing but illegal transfers (how can you copyright a number? seriously..) implementing package downloads in ubuntu will help to raise awareness in the general public about other open source projects being fully legal and in some cases better than their closed-source equivalent)
to sum up my points: the downloader could check which was faster (on average) (http, ftp, bittorrent, w/e other option), and download via the fastest route. optionally it could seed the file obtained by http or ftp to increase the swarm speed for other torrent users
there is a possibility of creating a max bandwidth / day setting, for those worried about overstepping caps. but if your downloading via http anyway, there doesnt seem much point to that.
bittorrent would be Optional.
bittorrent will decrease the server load on repos mirror hosts. i dont care HOW you calculate it, if you decrease server load, the cost of operation for the owner/operator for that server goes down. meaning more resources for other things.
|
|
Mishtal wrote on the 13 May 08 at 02:22
|
sorry for double post, just realized this.
consider the situation where you have a household with multiple computers (for purposes of this thought, consider they are all running the same version of ubuntu)
if you want to update ALL of your computers at the same time, you will be downloading a copy of each package once for EACH computer. so if you have 5 computers, and want to download 20 1 megabyte packages for each of them. instead of downloading 20 megabytes, you have to download 100 megabytes.
but thats only for HTTP
potentially (best case, people, best case)
if you use the bittorrent download method for each of these stations, you might end up having the gateway computer (if you have one, go with me here) download the packages via bittorrent, and then instead of each other station downloading from the external wan, you would just bittorrent from your gateway. you just cut your bandwidth down by 1/5th of what it would have been for that update by using bittorrent.
|
|
mp3phish wrote on the 14 May 08 at 03:34
|
@Mistal:
you wouldn't have to have your gateway run a bittorrent client. Each client on your local LAN would automatically find each other in the swarm (because they are close to each other, they would prefer each other) and then most of your download speed would come from your local systems, allowing the seed to spread automatically throughout your household (or business) LAN.
The end result is that you would download the updates and that comp would share it to your other systems automatically, and it would scale very very well on local LANs.
|
|
Endolith wrote on the 14 May 08 at 21:10
|
"I for one don't use Bittorrent or any peer to peer programs
ISP's here in Australia are starting to reduce the bandwith available to peer to peer connections.
So I don't see any advantage in using Bittorrent."
It's insanely faster. I did my Hardy Heron upgrade by torrenting the CD very quickly on the release day when the regular servers weren't even responding. If your ISPs are reducing bandwidth, you should be finding a different ISP or fighting against them if they are a monopoly.
|
|
Mishtal wrote on the 14 May 08 at 22:31
|
@mp3Phish
your absolutely right.
when i posting that i had in my mind a model that as soon as one system (in the example, the gateway system, but in reality any would qualify) downloaded a chunk, it would be the preferred source for that chunk among the rest of the local systems.
but the way you describe it is more accurate.
|
|
azimout wrote on the 20 May 08 at 14:32
|
+1
However, one problem is that currently many corporate networks don't allow p2p (such as bittorrent) traffic. This is the case where I work for example, where I had trouble with the admins for downloading the new Ubuntu .iso via torrent... So, +1, but there should be a way to turn it off...
|
|
mp3phish wrote on the 21 May 08 at 02:50
|
azimout: I think that is what they are proposing. Bittorrent with sane defaults as the default update method, with a checkbox to turn it off. and a panel to allow power users to "share more" than default.
|
|
manishmahabir wrote on the 23 May 08 at 07:09
|
This idea is ultimate....simply too good to resist.
It should be implemented as soon as possible.
Those who want to vote against it,first understand what is being discussed.Just because YOU wont benefit from it does not mean millions of others wont.Please help to promote it for the sake of your brothers!
|
|
Endolith wrote on the 27 May 08 at 15:59
|
"However, one problem is that currently many corporate networks don't allow p2p (such as bittorrent) traffic."
Then the network admins are stupid. This would BENEFIT corporate networks a lot because they would download the updates from each other instead of having to download it separately for each computer. This would be a great thing for corporations that use Ubuntu.
|
|
DPic wrote on the 1 Jun 08 at 04:48
| |
Would this be possible with gnunet?
|
|
Klayy wrote on the 21 Jun 08 at 21:50
| |
Not everyone can use p2p. Simply face it. BUT I'd love to have a choice, since I would use p2p at home (but I can't at the dorm)
|
|
Xlage wrote on the 6 Jul 08 at 09:13
|
+ 1
1. Not to download faster in usual conditions, but to lower the load on servers at a new release, major updates...
2. It should be first download option, and apt should give away if they are no peers, or if download is much too slow. Real widespread cooperation !!
debtorrent and apt-transport-torrent already did a part of the job.
(finally universities, ISPs would not confuse thepiratebay and bittorrent ;) )
|
|
Mishtal wrote on the 7 Jul 08 at 02:56
|
I installed debtorrent, as some other people mentioned. for those of you who may want to do this on your own, and not wait for it to be implemented as part of a release. it was actually very easy.
http://debtorrent.alioth.debian.org/
theres the webpage (google "debtorrent")
i followed the install steps exactly and as a test of it, on an old crap system i have, i ran
for pkg in `dpkg --get-selections | awk 'print $1' | egrep -v '(dpkg|apt|mysql|mythtv)'` ; do apt-get -y --force-yes install --reinstall $pkg ; done
(grabbed from somewhere in the ubuntu forums)
everything works just dandy.
im not sure what issues debtorrent still has, as i didnt spend too much time looking at it, but id like to suggest looking into possibility of integrating debtorrent into APT
(for those who mention that some places dont allow p2p protocols, it should be noted that when debtorrent couldnt find a peer to download from (a few of the repos had no peers at the time) it reverted to http and proceeded as normal. all in all, i think i ended up with a net speed increase, and waiting to fall back to http happened with no noticeable overhead)
-Mishtal
|
|
gabtrat wrote on the 31 Jul 08 at 19:02
| |
I think this would really be in the spirit of open source software.
|
|
flint wrote on the 3 Aug 08 at 18:31
|
+? I live here in Central Vermont USA.
While this is technically not the end of the network world, I can see it clearly from here.
When I use the following Bittorrent configuration:
$ btshowmetainfo ubuntu-8.04.1-desktop-i386.iso.torrent
btshowmetainfo 20021207 - decode BitTorrent metainfo files
metainfo file.: ubuntu-8.04.1-desktop-i386.iso.torrent
info hash.....: cdbfcd61f52525559d08514ba525e2d3189953b3
file name.....: ubuntu-8.04.1-desktop-i386.iso
file size.....: 728221696 (1388 * 524288 + 509952)
announce url..: http://torrent.ubuntu.com:6969/announce
I get really sucky speed:
| file: ubuntu-8.04.1-desktop-i386.iso |
| size: 728,221,696 (694.5 M) |
| dest: /svrland/isos/ubuntu/ubuntu-8.04.1-desktop-i386.iso |
| progress: ##________________________________________________________________ |
| status: connecting to peers (4.0%) |
| speed: 0 B/s down - 0 B/s up |
| totals: 0.0 M down - 0.0 M up |
| error(s):
Note that this is after two hours...
Anybody with any ideas?
Regards,
Flint
|
|
skip wrote on the 6 Aug 08 at 09:05
|
Default option should be pretty light :
Remove as soon as ratio is 1:1
Upload is really slow
It can fall back on http when a problem occured.
it should be optionnal, and not default (or if default, make it very easy to change it, say tright after install, the first time you see the update manager, it's an already checked checkbox, so it's easy to change).
But I promise I'd use it, with fast upload (I can easly give 40k/s for canonical ! ) and high ratio (5:1 at least) so to help the guys who made it possible.
|
|
Endolith wrote on the 8 Aug 08 at 15:30
| |
For security purposes, could we download just hashes from the central server, and then check that the files we get from peers match the hashes? Certainly people have thought of ways to accomplish this already.
|
|
notyetroot wrote on the 10 Aug 08 at 18:58
| |
Only if it isn't default.
|
|
notyetroot wrote on the 10 Aug 08 at 18:59
| |
Or use GNUNet/similar, which won't be throttled in the near future.
|
|
saftaplan wrote on the 19 Aug 08 at 19:16
|
1) Check for UPnP support and try to open the necessary ports
2) Test which protocols (HTTP/Bittorrent/...) are available
3) Download a few packages with all available protocols and measure the speed
4) Remember which protocol was the fastest (if it's a near-tie, prefer P2P over HTTP)
5) If a P2P protocol was the 'winner', and *only in this case*, prompt to seed use this protocol and whether you want to seed downloaded files while not downloading yourself.
This way, people that don't need/can't use a P2P network aren't bugged with the question, and people with a data limit (like in Belgium, 25 GB *sigh*) can choose not to use this option.
|
|
Mishtal wrote on the 20 Aug 08 at 19:15
|
@Endolith,
That is already built into the bittorrent protocol. The .Torrent file has a hash for each "chunk" of the torrent, and then a hash for the entire file. Because we know that the publisher of the .Torrent file will be the server (or someone pretending to be the repo server, but that's an authentication issue), we know that if the chunk doesn't match with what the .Torrent file says it should be, we need to discard it. Most Bittorrent clients automatically stop downloading chunks from peers that continually send bad chunks.
Additionally, since half of a file might come from peer A, and a quarter might come from peer B, while the last quarter might come from peer C, even if peer A and peer B were both trying to seed the same malicious file, and even if they some how got past the .Torrent file's hashing checks, the last quarter (not necessarily a contiguous chunk of the file, i mean 25 % of the total file, potentially randomly distributed) came from peer C, who (since this is a made up situation) we know is seeding the correct, non-malicious, file. unless peer A, and peer B, were REALLY good at writing their malicious program, the chances of the file even (if it was executable, anyway) running without segfaulting and dieing right away are pretty slim.
so, yea, Bittorrent already has security in place. as far as i can see its not a concern.
|
|
Endolith wrote on the 22 Aug 08 at 20:29
| |
Ok, that makes sense.
|
Post your comment
|
Blueprint apt-torrent:
Ubuntuforums.org thread #785206
