Mailing List Archive: NANOG: users

P2P agents for software distribution - saving the WAN from meltdown?!?

Index | Next | Previous | View Threaded

netfortius at gmail

Jun 16, 2008, 6:53 AM

Post #1 of 13 (647 views)
Permalink

P2P agents for software distribution - saving the WAN from meltdown?!?

Has anybody used (and been successful at) a bit-torrent-like agent for fast
distribution of LEGAL software (install programs of large-DVD size), across
multiple sites, all over the globe, with bad WAN connectivity? I have read a
couple of references online (e.g.
http://torrentfreak.com/university-uses-utorrent-080306/) about such, but I
am a little reluctant to do it in a corporate environment, especially in the
light of potential misuse of such ... unless finding a way to install, use
and remove the P2P agent, all in one shot ... catch 22, sort of (distributing
the P2P agent, that is :)) ...

Stefan

P.S. If inappropriate for this mailing list, I apologize - but the "long fat
pipe" thread gave me the idea to ask here, vs. sysadmin-like lists, as the
potential for network impact is my primary concern.

morrowc.lists at gmail

Jun 17, 2008, 11:00 AM

Post #2 of 13 (610 views)
Permalink

Re: P2P agents for software distribution - saving the WAN from meltdown?!? [In reply to]

On Mon, Jun 16, 2008 at 9:53 AM, Netfortius <netfortius [at] gmail> wrote:
> Has anybody used (and been successful at) a bit-torrent-like agent for fast
> distribution of LEGAL software (install programs of large-DVD size), across
> multiple sites, all over the globe, with bad WAN connectivity? I have read a
> couple of references online (e.g.
> http://torrentfreak.com/university-uses-utorrent-080306/) about such, but I
> am a little reluctant to do it in a corporate environment, especially in the
> light of potential misuse of such ... unless finding a way to install, use
> and remove the P2P agent, all in one shot ... catch 22, sort of (distributing
> the P2P agent, that is :)) ...

revision3.com
ubuntu.com
fedoraproject.org
.
.
.

most of the larger free-nix's do BT downloads on release day(s).
Revision3 distributes their content via BT. There were rumors of
Disney and Apple moving to BT models for their content distribution at
one point as well.

If the tracker isn't accessible from untrusted networks, what's the
concern you have?

-Chris

brandon.galbraith at gmail

Jun 17, 2008, 11:14 AM

Post #3 of 13 (616 views)
Permalink

Re: P2P agents for software distribution - saving the WAN from meltdown?!? [In reply to]

On 6/17/08, Christopher Morrow <morrowc.lists [at] gmail> wrote:
>
> On Mon, Jun 16, 2008 at 9:53 AM, Netfortius <netfortius [at] gmail> wrote:
> > Has anybody used (and been successful at) a bit-torrent-like agent for
> fast
> > distribution of LEGAL software (install programs of large-DVD size),
> across
> > multiple sites, all over the globe, with bad WAN connectivity?

<snip>

most of the larger free-nix's do BT downloads on release day(s).
> Revision3 distributes their content via BT. There were rumors of
> Disney and Apple moving to BT models for their content distribution at
> one point as well.
>

<snip>

I believe World of Warcraft uses Bittorrent to push out updates as well
(Steam may, haven't checked, would make sense though). Something we've been
working with for a client is using Amazon's S3 service to host the tracker,
as S3 will natively handle serving it (both the content and the tracker, you
simply need to append ?torrent to the S3 request).

HTH,
-brandon

joelja at bogus

Jun 17, 2008, 11:19 AM

Post #4 of 13 (610 views)
Permalink

Re: P2P agents for software distribution - saving the WAN from meltdown?!? [In reply to]

Netfortius wrote:
> Has anybody used (and been successful at) a bit-torrent-like agent for fast
> distribution of LEGAL software (install programs of large-DVD size), across
> multiple sites, all over the globe, with bad WAN connectivity? I have read a
> couple of references online (e.g.
> http://torrentfreak.com/university-uses-utorrent-080306/) about such, but I
> am a little reluctant to do it in a corporate environment, especially in the
> light of potential misuse of such ... unless finding a way to install, use
> and remove the P2P agent, all in one shot ... catch 22, sort of (distributing
> the P2P agent, that is :)) ...

well if the connectivity universally sucks no amount of p2p is going to
help... We have some experience with large file distribution in this
fashion because of the fedora core DVD iso's and we can say generally at
this point that the mirror infrastructure serves more iso's on release
day than the torrents do...

that said the p2p client does rule out needing to select a mirror that
has free slots during a flash crowd.

> Stefan
>
> P.S. If inappropriate for this mailing list, I apologize - but the "long fat
> pipe" thread gave me the idea to ask here, vs. sysadmin-like lists, as the
> potential for network impact is my primary concern.
>

smb at cs

Jun 17, 2008, 11:43 AM

Post #5 of 13 (629 views)
Permalink

Re: P2P agents for software distribution - saving the WAN from meltdown?!? [In reply to]

On Tue, 17 Jun 2008 11:19:19 -0700
Joel Jaeggli <joelja [at] bogus> wrote:

> that said the p2p client does rule out needing to select a mirror
> that has free slots during a flash crowd.

As Mozilla is learning today:
http://www.techspot.com/news/30486-mozilla-sites-die-shortly-after-download-day-begins.html

--Steve Bellovin, http://www.cs.columbia.edu/~smb

adrian at creative

Jun 18, 2008, 7:42 AM

Post #6 of 13 (615 views)
Permalink

Re: P2P agents for software distribution - saving the WAN from meltdown?!? [In reply to]

On Tue, Jun 17, 2008, Christopher Morrow wrote:

> most of the larger free-nix's do BT downloads on release day(s).
> Revision3 distributes their content via BT. There were rumors of
> Disney and Apple moving to BT models for their content distribution at
> one point as well.

<random type="idea from tonight">
If only there was a way for a SP to run a BitTorrent type service for
their clients, subscribing the BT server(s) to known-good (ie, not warez-y)
torrents pre-seeded from trusted sources and then leaving it the hell
alone and not having to continuously dump specific torrent files into
it.
</random>

Hm!

Adrian

jabley at ca

Jun 18, 2008, 7:52 AM

Post #7 of 13 (606 views)
Permalink

Re: P2P agents for software distribution - saving the WAN from meltdown?!? [In reply to]

On 18 Jun 2008, at 10:42, Adrian Chadd wrote:

> <random type="idea from tonight">
> If only there was a way for a SP to run a BitTorrent type service for
> their clients, subscribing the BT server(s) to known-good (ie, not
> warez-y)
> torrents pre-seeded from trusted sources and then leaving it the hell
> alone and not having to continuously dump specific torrent files into
> it.
> </random>

Automatically leeching and then seeding for long periods is trivial to
set up if you can get an RSS feed with torrent enclosures. It is my
(highly theoretical, naturally) understanding that many BitTorrent
trackers make such feeds available.

However just because you have a fast, on-net seed for particular
torrents doesn't mean that your on-net leechers will necessarily pick
it up. The behaviour I have observed with BitTorrent is that clients
are handed a relatively short list of potential peers by the tracker,
and it's quite common for sensible, close, local peers not to be
included. My assumption has been that the set of potential peers
passed to the client is assembled randomly.

If this behaviour is widespread (i.e. if my observations are valid and
my interpretation of those observations reasonable) then the more
popular the content, the less likely leechers are to see the seed you
want them to see. This relegates your local, on-net, fast seed to be a
way of distributing unpopular content (that which is not being seeded
by many other people).

There has been at least one presentation at NANOG in the past couple
of years which describes the benefit to ISPs of p2p, by virtue of
keeping traffic for popular content on-net. From memory, however, that
presentation was based on a non-deployed p2p protocol which made more
of an effort to help peers find local peers than the clients I
described above.

Joe

warren at kumari

Jun 18, 2008, 8:06 AM

Post #8 of 13 (604 views)
Permalink

Re: P2P agents for software distribution - saving the WAN from meltdown?!? [In reply to]

On Jun 18, 2008, at 10:42 AM, Adrian Chadd wrote:

> On Tue, Jun 17, 2008, Christopher Morrow wrote:
>
>> most of the larger free-nix's do BT downloads on release day(s).
>> Revision3 distributes their content via BT. There were rumors of
>> Disney and Apple moving to BT models for their content distribution
>> at
>> one point as well.
>
> <random type="idea from tonight">
> If only there was a way for a SP to run a BitTorrent type service for
> their clients, subscribing the BT server(s) to known-good (ie, not
> warez-y)
> torrents pre-seeded from trusted sources and then leaving it the hell
> alone and not having to continuously dump specific torrent files into
> it.
> </random>
>

Ah, if only there was a way for my SP to go and look all over the web
and figure out what pages are acceptable for me to browse and block
out all of the other stuff like porn and warez and phishing --- and
other objectionable content like creationism / evolution [delete
whichever is appropriate ], those bastard [insert your least favorite
ethnic / religious group here ] and any mention of [insert political
party]..... Oh, and anything to do with clowns, they freak me out...

Yes, P2P is not the web, but the general principle still applies -- I
don't think that handing over the censorship keys to my ISP is a
reasonable solution...
W

> Hm!
>
>
>
> Adrian
>
>

--
Do not meddle in the affairs of wizards, for they are subtle and quick
to anger.
-- J.R.R. Tolkien

bortzmeyer at nic

Jun 18, 2008, 8:21 AM

Post #9 of 13 (601 views)
Permalink

Re: P2P agents for software distribution - saving the WAN from meltdown?!? [In reply to]

On Wed, Jun 18, 2008 at 10:52:38AM -0400,
Joe Abley <jabley [at] ca> wrote
a message of 41 lines which said:

> The behaviour I have observed with BitTorrent is that clients are
> handed a relatively short list of potential peers by the tracker,
> and it's quite common for sensible, close, local peers not to be
> included. My assumption has been that the set of potential peers
> passed to the client is assembled randomly.

I did not check seriously so I cannot confirm or deny but do note that
there are several proposals to improve "peer selection" behind random
sorting or crude measurements with ping on a few hosts. A summary of
existing work is on the ALTO Web site
<http://alto.tilab.com/resources.html>.

ALTO will have a BoF session at the next IETF in Dublin, so we may see
one day a standard protocol for peer selection.

adrian at creative

Jun 18, 2008, 8:27 AM

Post #10 of 13 (609 views)
Permalink

Re: P2P agents for software distribution - saving the WAN from meltdown?!? [In reply to]

On Wed, Jun 18, 2008, Warren Kumari wrote:

> Yes, P2P is not the web, but the general principle still applies -- I
> don't think that handing over the censorship keys to my ISP is a
> reasonable solution...

I dunno, an RSS type feed of bittorrent files which can be subscribed
to would be useful. You could then just subscribe to certain content,
implictly trusting that they're publishing sensible content (and filtering
the content you seed your torrent with using tags or some such.)

You could then subscribe to various projects' downloads and mirror
appropriately.

Of course, this could already be being done; I haven't any idea. :)

Adrian

nanog at daork

Jun 18, 2008, 8:29 AM

Post #11 of 13 (606 views)
Permalink

Re: P2P agents for software distribution - saving the WAN from meltdown?!? [In reply to]

On 19/06/2008, at 2:52 AM, Joe Abley wrote:
> On 18 Jun 2008, at 10:42, Adrian Chadd wrote:
>> <random type="idea from tonight">
>> If only there was a way for a SP to run a BitTorrent type service for
>> their clients, subscribing the BT server(s) to known-good (ie, not
>> warez-y)
>> torrents pre-seeded from trusted sources and then leaving it the hell
>> alone and not having to continuously dump specific torrent files into
>> it.
>> </random>
>
> Automatically leeching and then seeding for long periods is trivial
> to set up if you can get an RSS feed with torrent enclosures. It is
> my (highly theoretical, naturally) understanding that many
> BitTorrent trackers make such feeds available.
>
> However just because you have a fast, on-net seed for particular
> torrents doesn't mean that your on-net leechers will necessarily
> pick it up. The behaviour I have observed with BitTorrent is that
> clients are handed a relatively short list of potential peers by the
> tracker, and it's quite common for sensible, close, local peers not
> to be included. My assumption has been that the set of potential
> peers passed to the client is assembled randomly.
>
> If this behaviour is widespread (i.e. if my observations are valid
> and my interpretation of those observations reasonable) then the
> more popular the content, the less likely leechers are to see the
> seed you want them to see. This relegates your local, on-net, fast
> seed to be a way of distributing unpopular content (that which is
> not being seeded by many other people).
>
> There has been at least one presentation at NANOG in the past couple
> of years which describes the benefit to ISPs of p2p, by virtue of
> keeping traffic for popular content on-net. From memory, however,
> that presentation was based on a non-deployed p2p protocol which
> made more of an effort to help peers find local peers than the
> clients I described above.

There was a product around that would keep track of torrents and fudge
the tracker responses to direct you to on-net peers where possible.
Not sure what it's called. Inline box thing, much like Sandvine,
Allot, etc. I imagine you could either inject the details of a local
seed you're running, or keep track of on-net users and inject those.

From a tracker software point of view, it would be fairly trivial to
weight peer lists to prefer peers within the same ASN I imagine.
Perhaps that could be turned in to same country, or what not. Better,
combine it with some kind of rough AS adjacency graph and <insert
algorithm here> and viola.
Is there any data available that would let that happen easily?
Obviously routing tables for the ASN/IP mapping, but what about rough
ASN adjacency? It doesn't really need to be updated that often - even
CAIDA's yearly data that they use to make their pretty pictures could
work OK.

Seems like win/win/win - linux distribution vendors can pride
themselves on how much faster their torrents run, end users get better
speeds for their torrents, networks move less traffic off-net.

.. this is the part where someone bustles off and makes it go.

--
Nathan Ward

laird at pando

Jun 18, 2008, 12:43 PM

Post #12 of 13 (599 views)
Permalink

Re: P2P agents for software distribution - saving the WAN from meltdown?!? [In reply to]

To address the original question, there are several p2p companies focusing on optimizing p2p for internal distribution of software and rich media. In particular, Kontiki and Ignite both offer such services, and between the two have many of the Fortune 1000 as customers (Coke, Bank of America, Accenture, McDonalds, Canon, Burger King, etc.). Their systems manage not just the (p2p) physical delivery of the bits, but also the enterprise management aspects (e.g. sending the right versions of the right software to the right desktops, managing data flow in a way that works well on a corporate LAN, security, running the installs/upgrades, etc.).

Addressing the Revision3 comment in the thread, I don't think that the "RIAA and similar organizations" had any problem with Revision3 using the BitTorrent protocol, but with them running an (inadvertently) open Tracker that was hosting 250K pirate torrents. The "attack" was pretty clearly a MediaDefender software bug in their code that monitors pirate torrents, multiplied by the large number of servers that they run, which unfortunately kicked in over a holiday weekend when nobody was around to fix it. Once MediaDefender was notified of the problem, Revision3 said that it was fixed quickly. So while you may not like what MediaDefender does for a living, it doesn't look like they were trying to DDOS Revision3 for using p2p protocols.

- Laird Popkin, CTO, Pando Networks
mobile: 646/465-0570

----- Original Message -----
From: "Blaine Fleming" <groups [at] digital-z>
To: nanog [at] merit
Sent: Wednesday, June 18, 2008 12:20:28 PM (GMT-0500) America/New_York
Subject: Re: P2P agents for software distribution - saving the WAN from meltdown?!?

Christopher Morrow wrote:
> On Mon, Jun 16, 2008 at 9:53 AM, Netfortius <netfortius [at] gmail> wrote:
>
>> Has anybody used (and been successful at) a bit-torrent-like agent for fast
>> distribution of LEGAL software (install programs of large-DVD size), across
>> multiple sites, all over the globe, with bad WAN connectivity? I have read a
>> couple of references online (e.g.
>> http://torrentfreak.com/university-uses-utorrent-080306/) about such, but I
>> am a little reluctant to do it in a corporate environment, especially in the
>> light of potential misuse of such ... unless finding a way to install, use
>> and remove the P2P agent, all in one shot ... catch 22, sort of (distributing
>> the P2P agent, that is :)) ...
>>
>
> revision3.com
>

And we saw how it worked out for Revision3.com. MediaDefender
considered them illegal and launched a Denial of Service attack against
them over Memorial Day weekend. P2P is considered illegal and wrong by
people with lots of money and that makes it hard to use for legitimate
services. Because MediaDefender is backed by the RIAA and similar
organizations they seem to be immune to prosecution. However, if *I*
did the same thing then I know I would be locked up right now.

--Blaine

justin at justinshore

Jun 18, 2008, 3:22 PM

Post #13 of 13 (593 views)
Permalink

Re: P2P agents for software distribution - saving the WAN from meltdown?!? [In reply to]

Nathan Ward wrote:
> There was a product around that would keep track of torrents and fudge
> the tracker responses to direct you to on-net peers where possible. Not
> sure what it's called. Inline box thing, much like Sandvine, Allot, etc.
> I imagine you could either inject the details of a local seed you're
> running, or keep track of on-net users and inject those.

Out of curiosity, how many SPs out there have local Akamai servers on
their network? I inquired about it last Fall and our average bandwidth
to Akamai wasn't enough at the time to warrant placing hardware on our
site, from their perspective anyway. The bandwidth though accounted for
roughly 1/10th of our overall bandwidth. I wonder what it would be
today. Our Internet bandwidth is just over 4x what it was last Fall.

Justin

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads