Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: nsp: ipv6

Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users

 

 

First page Previous page 1 2 3 Next page Last page  View All nsp ipv6 RSS feed   Index | Next | Previous | View Threaded


jeroen at sixxs

Aug 20, 2012, 9:17 AM

Post #1 of 53 (1129 views)
Permalink
Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users

In reference to https://www.sixxs.net/tickets/?msg=tickets-7640234

In other words it makes things "sometimesnot work"....

[code]
grh.sixxs.net> show bgp 2a02:26f0:c:1::5c7a:3289
BGP routing table entry for 2a02:26f0:c::/48
Paths: (100 available, best #20, table Default-IP-Routing-Table)
Not advertised to any peer
25384 3292 3257 20940 20940
2001:15f8:1::1 from 2001:15f8:1::1 (85.89.248.1)
Origin IGP, localpref 100, valid, external
Last update: Mon Aug 20 18:07:40 2012
...
[/code]

[code]
grh.sixxs.net> show bgp 2a02:26f0:5::5f64:f938
BGP routing table entry for 2a02:26f0:5::/48
Paths: (99 available, best #97, table Default-IP-Routing-Table)
Not advertised to any peer
25384 3292 1299 25074 20940 20940
2001:15f8:1::1 from 2001:15f8:1::1 (85.89.248.1)
Origin IGP, localpref 100, valid, external
Last update: Mon Aug 20 17:52:39 2012
...
[/code]

Announcing /48s will get filtered and this randomly breaks stuff.


One would think that Akamai had enough moneyz to pay everybody to accept
those /48s into their or at least then to get a disjunct /32...

Dear Akamai: It is a PA prefix as in AGGREGATED...

and as Facebook has enabled IPv6, this is broken for a LOT of people.


There is not even a special route6, there just is:

inet6num: 2a02:26f0::/32
netname: EU-AKAMAI-20101022
descr: Akamai Technologies
country: EU
org: ORG-AT1-RIPE
admin-c: NARA1-RIPE
tech-c: NARA1-RIPE
status: ALLOCATED-BY-RIR
mnt-by: RIPE-NCC-HM-MNT
mnt-lower: AKAM1-RIPE-MNT
mnt-routes: AKAM1-RIPE-MNT
source: RIPE # Filtered

route6: 2a02:26f0::/32
descr: Akamai Technologies
origin: AS34164
mnt-by: AKAM1-RIPE-MNT
source: RIPE # Filtered


Note that it depends on where I do the DNS query for which returns one
gets as some other locations point elsewhere...

$ host profile.ak.fbcdn.net
profile.ak.fbcdn.net is an alias for profile.ak.facebook.com.edgesuite.net.
profile.ak.facebook.com.edgesuite.net is an alias for a1725.dspl.akamai.net.
a1725.dspl.akamai.net has address 77.109.171.96
a1725.dspl.akamai.net has address 77.109.171.107
a1725.dspl.akamai.net has address 77.109.171.121
a1725.dspl.akamai.net has address 77.109.171.89
a1725.dspl.akamai.net has IPv6 address 2a02:26f0:c:1::5c7a:32a1
a1725.dspl.akamai.net has IPv6 address 2a02:26f0:c:1::5c7a:32aa
a1725.dspl.akamai.net has IPv6 address 2a02:26f0:c:1::5c7a:328b

This of course makes it more fun to debug as it is then "works for me"
in one case while completely different set in another case:

$ host -t any profile.ak.fbcdn.net
profile.ak.fbcdn.net is an alias for profile.ak.facebook.com.edgesuite.net.
profile.ak.facebook.com.edgesuite.net is an alias for a1725.dspl.akamai.net.
a1725.dspl.akamai.net has address 23.62.98.130
a1725.dspl.akamai.net has address 23.62.98.169
a1725.dspl.akamai.net has address 23.62.98.145
a1725.dspl.akamai.net has address 23.62.98.113
a1725.dspl.akamai.net has address 23.62.98.152
a1725.dspl.akamai.net has address 23.62.98.105
a1725.dspl.akamai.net has address 23.62.98.122
a1725.dspl.akamai.net has IPv6 address 2001:668:108:1::4d43:1c38
a1725.dspl.akamai.net has IPv6 address 2001:668:108:1::4d43:1c30
a1725.dspl.akamai.net has IPv6 address 2001:668:108:1::4d43:1c3a


As such it is not that Akamai really needs to announce disjunct /48s as
the are already using PA space from other providers who do not have this
issue.

Greets,
Jeroen


bill at wjw

Aug 20, 2012, 1:59 AM

Post #2 of 53 (1037 views)
Permalink
Re: Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users [In reply to]

Sorry, just had to reply, this section of your email is
brilliant!!! Shame I can't convince my bosses the only way forward is
distributed content....

On Mon, 20 Aug 2012 15:44:46 -0400, Patrick W.
Gilmore wrote:

> One last note, on a slightly more personal nit to
pick. People have been screaming about the "exponential" growth of the
table for well over a decade, and how the world was coming to an end Any
Second Now. I double-checked Nick and he is right, the sky is not
falling. Of course, it is important to not waste slots needlessly, or
otherwise be silly with a limited resource. But The End is not nigh.
>

> I've been doing this for a little while now, and my biggest fear is
not another order of magnitude in the v6 table. Far more likely to
destroy the Internet is the growth of Mbps, Gbps, Tbps - in fact, some
would argue it has already caused us harm.
>
> The only realistic way
to manage the growth of things like VoD is massive distribution of
content. Everyone who has a _LOT_ of traffic is following in Akamai's
footsteps by placing non-network connected caches inside broadband
networks. Assuming ISPs treat content providers alike, you will see this
problem with many content companies.
>
> Without this distribution, I
posit the Internet would have already failed. To put this in
perspective, 1 in 5 bits on most broadband modems worldwide come from an
Akamai server. Google has similar traffic, NetFlix has a lot of traffic
in the US and a non-trivial amount in other countries, etc. It seems
more than obvious to me the real danger is creating obstacles to
distributing traffic more widely, not whether we have a few thousand
more or even a couple orders of magnitude more prefixes in the v6 DFZ.
>

> But then, maybe I'm biased....
>
> --
> TTFN,
> patrick
>
> P.S.
Regarding the Subject line: Jeroen, we have different definitions of
"lots of users".


jeroen at unfix

Aug 20, 2012, 9:20 AM

Post #3 of 53 (1052 views)
Permalink
Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users [In reply to]

In reference to https://www.sixxs.net/tickets/?msg=tickets-7640234

In other words it makes things "sometimesnot work"....

[code]
grh.sixxs.net> show bgp 2a02:26f0:c:1::5c7a:3289
BGP routing table entry for 2a02:26f0:c::/48
Paths: (100 available, best #20, table Default-IP-Routing-Table)
Not advertised to any peer
25384 3292 3257 20940 20940
2001:15f8:1::1 from 2001:15f8:1::1 (85.89.248.1)
Origin IGP, localpref 100, valid, external
Last update: Mon Aug 20 18:07:40 2012
...
[/code]

[code]
grh.sixxs.net> show bgp 2a02:26f0:5::5f64:f938
BGP routing table entry for 2a02:26f0:5::/48
Paths: (99 available, best #97, table Default-IP-Routing-Table)
Not advertised to any peer
25384 3292 1299 25074 20940 20940
2001:15f8:1::1 from 2001:15f8:1::1 (85.89.248.1)
Origin IGP, localpref 100, valid, external
Last update: Mon Aug 20 17:52:39 2012
...
[/code]

Announcing /48s will get filtered and this randomly breaks stuff.


One would think that Akamai had enough moneyz to pay everybody to accept
those /48s into their or at least then to get a disjunct /32...

Dear Akamai: It is a PA prefix as in AGGREGATED...

and as Facebook has enabled IPv6, this is broken for a LOT of people.


There is not even a special route6, there just is:

inet6num: 2a02:26f0::/32
netname: EU-AKAMAI-20101022
descr: Akamai Technologies
country: EU
org: ORG-AT1-RIPE
admin-c: NARA1-RIPE
tech-c: NARA1-RIPE
status: ALLOCATED-BY-RIR
mnt-by: RIPE-NCC-HM-MNT
mnt-lower: AKAM1-RIPE-MNT
mnt-routes: AKAM1-RIPE-MNT
source: RIPE # Filtered

route6: 2a02:26f0::/32
descr: Akamai Technologies
origin: AS34164
mnt-by: AKAM1-RIPE-MNT
source: RIPE # Filtered


Note that it depends on where I do the DNS query for which returns one
gets as some other locations point elsewhere...

$ host profile.ak.fbcdn.net
profile.ak.fbcdn.net is an alias for profile.ak.facebook.com.edgesuite.net.
profile.ak.facebook.com.edgesuite.net is an alias for a1725.dspl.akamai.net.
a1725.dspl.akamai.net has address 77.109.171.96
a1725.dspl.akamai.net has address 77.109.171.107
a1725.dspl.akamai.net has address 77.109.171.121
a1725.dspl.akamai.net has address 77.109.171.89
a1725.dspl.akamai.net has IPv6 address 2a02:26f0:c:1::5c7a:32a1
a1725.dspl.akamai.net has IPv6 address 2a02:26f0:c:1::5c7a:32aa
a1725.dspl.akamai.net has IPv6 address 2a02:26f0:c:1::5c7a:328b

This of course makes it more fun to debug as it is then "works for me"
in one case while completely different set in another case:

$ host -t any profile.ak.fbcdn.net
profile.ak.fbcdn.net is an alias for profile.ak.facebook.com.edgesuite.net.
profile.ak.facebook.com.edgesuite.net is an alias for a1725.dspl.akamai.net.
a1725.dspl.akamai.net has address 23.62.98.130
a1725.dspl.akamai.net has address 23.62.98.169
a1725.dspl.akamai.net has address 23.62.98.145
a1725.dspl.akamai.net has address 23.62.98.113
a1725.dspl.akamai.net has address 23.62.98.152
a1725.dspl.akamai.net has address 23.62.98.105
a1725.dspl.akamai.net has address 23.62.98.122
a1725.dspl.akamai.net has IPv6 address 2001:668:108:1::4d43:1c38
a1725.dspl.akamai.net has IPv6 address 2001:668:108:1::4d43:1c30
a1725.dspl.akamai.net has IPv6 address 2001:668:108:1::4d43:1c3a


As such it is not that Akamai really needs to announce disjunct /48s as
the are already using PA space from other providers who do not have this
issue.

Greets,
Jeroen


nick at foobar

Aug 20, 2012, 9:37 AM

Post #4 of 53 (1041 views)
Permalink
Re: Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users [In reply to]

On 20/08/2012 17:17, Jeroen Massar wrote:
> Announcing /48s will get filtered and this randomly breaks stuff.
>
> One would think that Akamai had enough moneyz to pay everybody to accept
> those /48s into their or at least then to get a disjunct /32...
>
> Dear Akamai: It is a PA prefix as in AGGREGATED...

Jeroen,

I'm counting ~800 prefixes in the DMZ for Akamai's v4 requirements. Can
you suggest an alternative strategy for multihoming 800 ipv6 CDN sites
which is less problematic than announcing 800 /48s out of a single /32?

Nick


phil at philkern

Aug 20, 2012, 9:41 AM

Post #5 of 53 (1043 views)
Permalink
Re: Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users [In reply to]

Jeroen,

am Mon, Aug 20, 2012 at 06:17:24PM +0200 hast du folgendes geschrieben:
> In reference to https://www.sixxs.net/tickets/?msg=tickets-7640234
> In other words it makes things "sometimesnot work"....
> Announcing /48s will get filtered and this randomly breaks stuff.

r-ir-cs-1#show bgp ipv6 unicast 2a02:26f0::/32
BGP routing table entry for 2A02:26F0::/32, version 2801974
Paths: (2 available, best #1, table Default)
Advertised to update-groups:
1
553 3549 4436 34164 34164
2001:7C0:3:702:: (FE80::4255:39FF:FE3A:60C3) from 2001:7C0:3:702:: (193.196.190.67)
Origin IGP, metric 50, localpref 100, valid, external, best
Community: 553:304 3549:4405 3549:30840 4436:42532
553 3549 4436 34164 34164
2001:7C0:3:703:: (FE80::FA66:F2FF:FE11:8E72) from 2001:7C0:3:703:: (193.196.190.68)
Origin IGP, metric 200, localpref 100, valid, external
Community: 553:304 3549:4405 3549:30840 4436:42532

I do see the covering aggregate and in [1] to which you answered an Akamai
employee said that it's supposed to be announced. What's the problem with those
more specific? Aren't they reachable through the covering aggregate, too?
(I cannot test because my peers accept the specifics.)

Also I see a route6 for instance for 2a02:26f0:0005::/48:

route6: 2a02:26f0:0005::/48
descr: Akamai Technologies
origin: AS20940
notify: ip-admin [at] akamai
mnt-by: AKAM1-RIPE-MNT
changed: ck [at] akamai 20101220
source: RIPE

Kind regards
Philipp Kern

[1] <FECAD2E8-7A8E-4A9A-A5C6-98551D30E0C8 [at] ianai>
Attachments: signature.asc (0.19 KB)


jeroen at unfix

Aug 20, 2012, 9:42 AM

Post #6 of 53 (1045 views)
Permalink
Re: Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users [In reply to]

On 2012-08-20 18:37, Nick Hilliard wrote:
> On 20/08/2012 17:17, Jeroen Massar wrote:
>> Announcing /48s will get filtered and this randomly breaks stuff.
>>
>> One would think that Akamai had enough moneyz to pay everybody to accept
>> those /48s into their or at least then to get a disjunct /32...
>>
>> Dear Akamai: It is a PA prefix as in AGGREGATED...
>
> Jeroen,
>
> I'm counting ~800 prefixes in the DMZ for Akamai's v4 requirements.

(s/DMZ/DFZ/ I guess ;)

Most of those are out of PA space that is announced inside the ISP that
that prefix is living in and thus which do not need to be seen in the
rest of the DFZ as they come out of a PA block.

> Can
> you suggest an alternative strategy for multihoming 800 ipv6 CDN sites
> which is less problematic than announcing 800 /48s out of a single /32?

See above about paying folks to get passed those filters.

And they could of course just use address space which is not meant for
de-aggregation...


I guess what the real thing is that the time is RIPE for a RIPE address
plan which is akin to ARIN's Micro Allocations.

Greets,
Jeroen


ryan at u13

Aug 20, 2012, 9:43 AM

Post #7 of 53 (1038 views)
Permalink
Re: Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users [In reply to]

On Aug 20, 2012, at 11:37 AM, Nick Hilliard wrote:

> On 20/08/2012 17:17, Jeroen Massar wrote:
>> Announcing /48s will get filtered and this randomly breaks stuff.
>>
>> One would think that Akamai had enough moneyz to pay everybody to accept
>> those /48s into their or at least then to get a disjunct /32...
>>
>> Dear Akamai: It is a PA prefix as in AGGREGATED...
>
> Jeroen,
>
> I'm counting ~800 prefixes in the DMZ for Akamai's v4 requirements. Can
> you suggest an alternative strategy for multihoming 800 ipv6 CDN sites
> which is less problematic than announcing 800 /48s out of a single /32?
>
> Nick
>


http://www.gossamer-threads.com/lists/nsp/ipv6/35364

It looks like this was resolved once in the past, and I am seeing it announced as a /32 from AS34164 currently:

Announced By
Origin AS Announcement Description
AS20940 2a02:26f0:000c::/48 Akamai Technologies

Less Specific Announcements
Origin AS Announcement Description
AS34164 2a02:26f0::/32 Akamai Technologies

Nick, I believe Jeroen is commenting that he sees only /48s and not the less-specific /32, leading to brokenness in ASNs which filter /48s from PA allocations. However as mentioned above, this does not seem to be the case from my point of view (the pastes above are from bgp.he.net)

Ryan


jeroen at unfix

Aug 20, 2012, 9:46 AM

Post #8 of 53 (1037 views)
Permalink
Re: Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users [In reply to]

On 2012-08-20 18:41, Philipp Kern wrote:
> Jeroen,
>
> am Mon, Aug 20, 2012 at 06:17:24PM +0200 hast du folgendes geschrieben:
>> In reference to https://www.sixxs.net/tickets/?msg=tickets-7640234
>> In other words it makes things "sometimesnot work"....
>> Announcing /48s will get filtered and this randomly breaks stuff.
[..]
> I do see the covering aggregate and in [1] to which you answered an Akamai
> employee said that it's supposed to be announced.

Yes, the covering is there too and likely tries to capture some packets,
but clearly not all of them as for the other route they disappear in
/dev/null.

> What's the problem with those more specific?

That the specific /32 mentioned is a PA prefix and should not have more
specifics at all. Unless the meaning and name of PA is changed to just
be "block of addresses" instead of containing the word "Aggregated".

But primarily at the moment: it breaks things in very very funny ways.

> Aren't they reachable through the covering aggregate, too?
> (I cannot test because my peers accept the specifics.)

I honestly have no idea what goes wrong, as I cannot see the traceroute
from the side of Akamai...

Greets,
Jeroen


gert at space

Aug 20, 2012, 10:44 AM

Post #9 of 53 (1035 views)
Permalink
Re: Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users [In reply to]

Hi,

On Mon, Aug 20, 2012 at 06:42:34PM +0200, Jeroen Massar wrote:
> I guess what the real thing is that the time is RIPE for a RIPE address
> plan which is akin to ARIN's Micro Allocations.

It's called "IPv6 PI" and we have that.

(It doesn't matter for the amount of prefixes in the DFZ - but it does
make a difference if people filter out /48s from /32-range without
checking with their IRRDB whether they might want to make an exception
here - OTOH, if the /48s don't even *have* route6 objects, now that
would be a good reason to bash Akamai)

Gert Doering
-- NetMaster
--
have you enabled IPv6 on something today...?

SpaceNet AG Vorstand: Sebastian v. Bomhard
Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen HRB: 136055 (AG Muenchen)
Tel: +49 (89) 32356-444 USt-IdNr.: DE813185279


nick at foobar

Aug 20, 2012, 10:45 AM

Post #10 of 53 (1039 views)
Permalink
Re: Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users [In reply to]

On 20/08/2012 17:42, Jeroen Massar wrote:
> Most of those are out of PA space that is announced inside the ISP that
> that prefix is living in and thus which do not need to be seen in the
> rest of the DFZ as they come out of a PA block.

2.16.0.0/13: 53 announcements
23.0.0.0/12: 110 announcements
23.32.0.0/11: 170 announcements
23.64.0.0/14: 54 announcements

etc, etc, etc. I got bored at this stage and stopped totting up their
larger allocations (there are lots more), but that's already nearly half
the space they announce. It's PA space, but all allocated to Akamai and
announced as smaller prefixes. I.e. it's not PA space in the traditional
sense of all being announced as a single aggregate from a single ASN. So
why should Akamai's v6 allocation / assignment policies be different?

> See above about paying folks to get passed those filters.

I thought about this for a while and figure that there are two likely
scenarios for this sort of thing:

Scenario #1:

$ISP: oh hai, we are filtering out ur /48s, pls to be givin us moniez

Akamai: sure thing. Here's $money. How much did you want?

$restofworld: oh hai, Akamai, uhhhhhh we've just started filtering out ur
/48s, so pls to be givin us moniez too

Akamai: doh! Okey-dokey, here's money for you too.


Scenario #2:

$ISP: oh hai, we are filtering out ur /48s, pls to be givin us moniez

Akamai: lol no. if you don't want akamai v6 traffic, feel free to have
your helpdesk tell your customers that their interwebs is broken because of
your prefix aggregation demands.

$ISP: how dare you! you're breaking our network! we demand that you stop!

> And they could of course just use address space which is not meant for
> de-aggregation...

You mean, get 800 separate separate PI assignments from the RIRs? What
problem is that going to solve other than annoying the LIRs? Would you be
happier if Akamai announced 800 /32s instead?

> I guess what the real thing is that the time is RIPE for a RIPE address
> plan which is akin to ARIN's Micro Allocations.

The current RIPE recommendations are to filter on /48 for v6. No clue
about Arinland or Apnicistan.

Nick


md at Linux

Aug 20, 2012, 11:25 AM

Post #11 of 53 (1036 views)
Permalink
Re: Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users [In reply to]

On Aug 20, Nick Hilliard <nick [at] foobar> wrote:

> So
> why should Akamai's v6 allocation / assignment policies be different?
Because some people are trying hard to not repeat the same errors which
are causing the tragedy of the commons of the IPv4 DFZ.

> You mean, get 800 separate separate PI assignments from the RIRs? What
> problem is that going to solve other than annoying the LIRs? Would you be
> happier if Akamai announced 800 /32s instead?
In both cases yes, because such cases do not require me to whitelist
Akamai in my border prefix filters (which would not scale).

--
ciao,
Marco


nick at foobar

Aug 20, 2012, 11:48 AM

Post #12 of 53 (1032 views)
Permalink
Re: Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users [In reply to]

On 20/08/2012 19:25, Marco d'Itri wrote:
> Because some people are trying hard to not repeat the same errors which
> are causing the tragedy of the commons of the IPv4 DFZ.

There is no tragedy of the commons here.

The Tragedy of the Commons was a cautionary historical incident where a
small number of people abused a common resource and caused a complete,
permanent collapse of that resource.

In the case of the ipv4 dfz, we have a commonage which is well within the
scaling bounds of equipment which has been on sale for the last 7 years,
and which looks like it will scale for several years to come. At that
stage, normal kit retirement will come into play and the next generation of
kit will scale well beyond what's currently available, which will
accommodate expected DFZ growth for many years to come.

Even if for some reason the v6 table explodes and smashes everyones'
forwarding engines, we still then have the option of targeted filtering
because on a global scale, resource consumption will tend to follow a
Pareto distribution. This means we can cherry pick the greatest abusers
and filter them until they sort out their broken policies (i.e. it will
hurt them more than anyone else).

All of which is to say that in the worse case, it is not feasible at
current usage growth rates that we will sustain a complete collapse of the
Internet due to unconstrained DFZ growth, even in the long term.

The sky is not falling (I checked earlier today).

Even still, let's just be sensible. Let's do our prefix aggregation
carefully because we know that too much is bad.

> You mean, get 800 separate separate PI assignments from the RIRs? What
> problem is that going to solve other than annoying the LIRs? Would you be
> happier if Akamai announced 800 /32s instead?

But if Akamai or some other organisation which has 800 publicly routed
sites, then they're going to need 800 v6 prefixes. It is pointless to tell
them that they need to use /32 for each just to get around peoples'
filters. Insisting on /32 for each site is fixing the wrong problem.

Also, please note ripe-555, section 4.

Nick


patrick at ianai

Aug 20, 2012, 12:44 PM

Post #13 of 53 (1041 views)
Permalink
Re: Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users [In reply to]

Hi everyone. How's it hangin'? Seems like we have once again stirred up a lot of dust unintentionally. I'm going to try to clear up a few things here, so please pardon the length of this response. Feel free to let me know if anything was not clear.

First, some housekeeping: Akamai should have route6 objects for all our announcements. I'll have someone get on that. Mea Culpa, please forgive any inconvenience. (I thought we already cleaned that up after the last thread, so this is especially embarrassing.)

Second, some stats: Akamai has several thousand nodes, but not all use Akamai-owned IP space. We prefer to use a block from the ISP hosting the node, and obviously those do not need a separate announcement. When necessary, i.e. when the hosting ISP cannot give us space, we have to make a decision whether the node is worth using our own IP space. Frequently, but not guaranteed, the answer is "yes". There are also some nodes which are multi-homed, and obviously those use our space, but they are a minor percentage of the total number of nodes.

Either way, since Akamai has no backbone, each node with unique, Akamai-owned IP space must (obviously) announce its block independently. (If anyone suggests something like GRE tunnels, I will ridicule them in public. :)


Now on to the issue at hand. As per an earlier thread here, we have three /32s. We deaggregate and hand out /48s to our individual nodes. We also announce the aggregate, as shown earlier in this thread. If you have trouble reaching a node, please email noc [at] akamai (24/7), or NetSupport-tix [at] akamai (M-F biz-hours++, but likely a better place to get your problem solved).

The interesting thing below is that things "sometimes" work. If a prefix / path is unavailable, it should not work, full stop. And assuming you are using a topologically close recursive name server, our system should see the disconnectivity and not return that AAAA when you resolve v6 hostnames. I'm not sure what is wrong with this particular situation, but we'll be looking into it. Probably some weirdness around SIXXS which I cannot grok in my massively sleep deprived condition, but that's another matter.

As for whether we should deaggregate PA space, I'm afraid that decision is already made. We are not asking for 1000+ /32s from the RIRs, and there really isn't another good solution to this problem AFAIK. We are not trying to cause problems, but we have constraints in which we must work as well.

If this does not fit your view of how the world should work, I am afraid we shall have to agree to disagree - unless you can come up with a better solution than asking for 1000+ /32s.


One last note, on a slightly more personal nit to pick. People have been screaming about the "exponential" growth of the table for well over a decade, and how the world was coming to an end Any Second Now. I double-checked Nick and he is right, the sky is not falling. Of course, it is important to not waste slots needlessly, or otherwise be silly with a limited resource. But The End is not nigh.

I've been doing this for a little while now, and my biggest fear is not another order of magnitude in the v6 table. Far more likely to destroy the Internet is the growth of Mbps, Gbps, Tbps - in fact, some would argue it has already caused us harm.

The only realistic way to manage the growth of things like VoD is massive distribution of content. Everyone who has a _LOT_ of traffic is following in Akamai's footsteps by placing non-network connected caches inside broadband networks. Assuming ISPs treat content providers alike, you will see this problem with many content companies.

Without this distribution, I posit the Internet would have already failed. To put this in perspective, 1 in 5 bits on most broadband modems worldwide come from an Akamai server. Google has similar traffic, NetFlix has a lot of traffic in the US and a non-trivial amount in other countries, etc. It seems more than obvious to me the real danger is creating obstacles to distributing traffic more widely, not whether we have a few thousand more or even a couple orders of magnitude more prefixes in the v6 DFZ.

But then, maybe I'm biased....

--
TTFN,
patrick

P.S. Regarding the Subject line: Jeroen, we have different definitions of "lots of users".


On Aug 20, 2012, at 14:48 , Nick Hilliard <nick [at] foobar> wrote:

> On 20/08/2012 19:25, Marco d'Itri wrote:
>> Because some people are trying hard to not repeat the same errors which
>> are causing the tragedy of the commons of the IPv4 DFZ.
>
> There is no tragedy of the commons here.
>
> The Tragedy of the Commons was a cautionary historical incident where a
> small number of people abused a common resource and caused a complete,
> permanent collapse of that resource.
>
> In the case of the ipv4 dfz, we have a commonage which is well within the
> scaling bounds of equipment which has been on sale for the last 7 years,
> and which looks like it will scale for several years to come. At that
> stage, normal kit retirement will come into play and the next generation of
> kit will scale well beyond what's currently available, which will
> accommodate expected DFZ growth for many years to come.
>
> Even if for some reason the v6 table explodes and smashes everyones'
> forwarding engines, we still then have the option of targeted filtering
> because on a global scale, resource consumption will tend to follow a
> Pareto distribution. This means we can cherry pick the greatest abusers
> and filter them until they sort out their broken policies (i.e. it will
> hurt them more than anyone else).
>
> All of which is to say that in the worse case, it is not feasible at
> current usage growth rates that we will sustain a complete collapse of the
> Internet due to unconstrained DFZ growth, even in the long term.
>
> The sky is not falling (I checked earlier today).
>
> Even still, let's just be sensible. Let's do our prefix aggregation
> carefully because we know that too much is bad.
>
>> You mean, get 800 separate separate PI assignments from the RIRs? What
>> problem is that going to solve other than annoying the LIRs? Would you be
>> happier if Akamai announced 800 /32s instead?
>
> But if Akamai or some other organisation which has 800 publicly routed
> sites, then they're going to need 800 v6 prefixes. It is pointless to tell
> them that they need to use /32 for each just to get around peoples'
> filters. Insisting on /32 for each site is fixing the wrong problem.
>
> Also, please note ripe-555, section 4.
>
> Nick
>


jeroen at unfix

Aug 20, 2012, 1:07 PM

Post #14 of 53 (1033 views)
Permalink
Re: Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users [In reply to]

On 2012-08-20 19:45, Nick Hilliard wrote:
> On 20/08/2012 17:42, Jeroen Massar wrote:
>> Most of those are out of PA space that is announced inside the ISP that
>> that prefix is living in and thus which do not need to be seen in the
>> rest of the DFZ as they come out of a PA block.
>
> 2.16.0.0/13: 53 announcements
> 23.0.0.0/12: 110 announcements
> 23.32.0.0/11: 170 announcements
> 23.64.0.0/14: 54 announcements
>
> etc, etc, etc. I got bored at this stage and stopped totting up their
> larger allocations (there are lots more), but that's already nearly half
> the space they announce. It's PA space, but all allocated to Akamai and
> announced as smaller prefixes.

And likely if you traceroute the space it is located in the same network
as the covering prefix... thus why is the route there?

It is indeed nice and correct to have a separate inetnum there, but the
route is not needed.

> I.e. it's not PA space in the traditional
> sense of all being announced as a single aggregate from a single ASN. So
> why should Akamai's v6 allocation / assignment policies be different?

Hey, I fully agree, they and others should not have to do that.

PA address space though is not where that is intended for.
PI address space is expected to be chunked up though.

But maybe it is better to have a third one which basically means that
content-providers/CDN/etc that have a large address space can have a big
chunk of address space and then announce more specifics.

The policy, designation and name of the addresses used are the problem here.

>> See above about paying folks to get passed those filters.
>
> I thought about this for a while and figure that there are two likely
> scenarios for this sort of thing:

Yes, using moniez for this won't scale and especially not due to
scenarios you mentioned where one or the other is going to bite the dust.

[..]
>> And they could of course just use address space which is not meant for
>> de-aggregation...
>
> You mean, get 800 separate separate PI assignments from the RIRs? What
> problem is that going to solve other than annoying the LIRs? Would you be
> happier if Akamai announced 800 /32s instead?

PI's normally come in /48s and even an Akamai would likely be good with
a /48 per location. Those 800 /48's (or heck /40s) could all come out of
the same block, but from a larger block that is meant for it.

Greets,
Jeroen


jeroen at unfix

Aug 20, 2012, 1:20 PM

Post #15 of 53 (1035 views)
Permalink
Re: Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users [In reply to]

On 2012-08-20 19:44, Gert Doering wrote:> Hi,
>
> On Mon, Aug 20, 2012 at 06:42:34PM +0200, Jeroen Massar wrote:
>> I guess what the real thing is that the time is RIPE for a RIPE address
>> plan which is akin to ARIN's Micro Allocations.
>
> It's called "IPv6 PI" and we have that.

The blocks being used are from the PA pool, which is why it hurt in most
network's filters.

> (It doesn't matter for the amount of prefixes in the DFZ - but it does

Fully agree.

> make a difference if people filter out /48s from /32-range without
> checking with their IRRDB whether they might want to make an exception
> here - OTOH, if the /48s don't even *have* route6 objects, now that
> would be a good reason to bash Akamai)

As noted in the mail, there was no route6 for that prefix.

And from that perspective to make filtering networks happy would work if
three conditions for more specifics are met:
1) announce the covering prefix
2) have a route6 for that
3) have a route6 for the more specific.

Note that Akamai generally does this, but for these the route6
definitely where missing. And likely something else is very flaky on
that path, though can't tell is it is forward or reverse.

With the above, in case there is route explosion (or your gear is too
old to grok things), one could always ignore the more-specific route6's
and filter those out thus making lots of folks around the world much
happier.

Greets,
Jeroen


nick at foobar

Aug 20, 2012, 1:30 PM

Post #16 of 53 (1031 views)
Permalink
Re: Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users [In reply to]

On 20/08/2012 21:07, Jeroen Massar wrote:
> And likely if you traceroute the space it is located in the same network
> as the covering prefix...

Have you looked at the entries for each of the smaller prefixes? They go
all over the place. And there are no covering prefixes for the PA
allocations. The longest prefix is /20.

> But maybe it is better to have a third one which basically means that
> content-providers/CDN/etc that have a large address space can have a big
> chunk of address space and then announce more specifics.

There are sufficiently few large CDNs that this probably isn't required.

> PI's normally come in /48s and even an Akamai would likely be good with
> a /48 per location. Those 800 /48's (or heck /40s) could all come out of
> the same block, but from a larger block that is meant for it.

What you're saying here is that they should have been given space from a
different range so that you could continue with a /32 netmask filter.

RIPE-555 suggests that your netmask filter should be /48. The days of
filtering at /32 are already over, at least in the RIPE region. If you
continue to filter at /32, you should expect connectivity problems.

Nick


jeroen at unfix

Aug 20, 2012, 1:31 PM

Post #17 of 53 (1031 views)
Permalink
Re: Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users [In reply to]

On 2012-08-20 21:44, Patrick W. Gilmore wrote:
> Hi everyone. How's it hangin'? Seems like we have once again
> stirred up a lot of dust unintentionally.

Well nope, I did. I should have spammed you and/or the noc directly
instead ;)

> I'm going to try to clear
> up a few things here, so please pardon the length of this response.
> Feel free to let me know if anything was not clear.

Thanks for this!

> First, some housekeeping: Akamai should have route6 objects for all
> our announcements. I'll have someone get on that. Mea Culpa, please
> forgive any inconvenience. (I thought we already cleaned that up
> after the last thread, so this is especially embarrassing.)

Shit happens. I am sure it will be resolved soon.

> Either way, since Akamai has no backbone, each node with unique,
> Akamai-owned IP space must (obviously) announce its block
> independently. (If anyone suggests something like GRE tunnels, I
> will ridicule them in public. :)
>
>
> Now on to the issue at hand. As per an earlier thread here, we have
> three /32s. We deaggregate and hand out /48s to our individual
> nodes. We also announce the aggregate, as shown earlier in this
> thread. If you have trouble reaching a node, please email
> noc [at] akamai (24/7), or NetSupport-tix [at] akamai (M-F biz-hours++,
> but likely a better place to get your problem solved).

And I will do that the next time, will owe whiskey if I forget that,
before anyone thinks I am specifically targeting Akamai, which I am not,
especially as they are doing a good job in getting IPv6 out there.

> The interesting thing below is that things "sometimes" work. If a
> prefix / path is unavailable, it should not work, full stop.

I think what is happening is that due to the missing route6 object some
filtering is over-aggressive and failing some kind of RPF check or
something else. But only way to find this out is doing checks from both
sides.

> And
> assuming you are using a topologically close recursive name server,
> our system should see the disconnectivity and not return that AAAA
> when you resolve v6 hostnames. I'm not sure what is wrong with this
> particular situation, but we'll be looking into it. Probably some
> weirdness around SIXXS which I cannot grok in my massively sleep
> deprived condition, but that's another matter.

I'll check again tomorrow how broken it is and follow up on the
noc [at] akama address on this.

Note that SixXS is effectively networked in the same way as Akamai
nodes: each PoP uses address space and hardware from the ISP hosting it.
For this you will find no more specifics though (at least you should
not) as they are fully integrated inside the AS.

> As for whether we should deaggregate PA space, I'm afraid that
> decision is already made. We are not asking for 1000+ /32s from the
> RIRs, and there really isn't another good solution to this problem
> AFAIK. We are not trying to cause problems, but we have constraints
> in which we must work as well.

Fully aware. I think the policy should be updated for these kind of
network situatuations.

Note that I am not suggesting 1000+ /32's, I am suggesting that address
space used in this way comes out of a block that is properly filterable,
thus marked as 'will be de-aggregated'.

See also my note in the other message about the covering prefix along
with the route6, which is what Akamai does.

I am also heavily of the opinion that we should rename the "Aggregated"
portion of the PA naming to solve part of this issue.

[..]
> It seems more than obvious to me the
> real danger is creating obstacles to distributing traffic more
> widely, not whether we have a few thousand more or even a couple
> orders of magnitude more prefixes in the v6 DFZ.

Please note that the problem is not the proper-use of these routing
slots, you have IMHO valid reasoning to do so.

The problem is that some sites filter as they expect no longer prefixes
than /32 from the PA address space used by the RIRs and that this breaks
things that they expected from the last 10+ years of doing IPv6 already.

Greets,
Jeroen


nick at foobar

Aug 20, 2012, 1:32 PM

Post #18 of 53 (1031 views)
Permalink
Re: Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users [In reply to]

On 20/08/2012 21:20, Jeroen Massar wrote:
> 1) announce the covering prefix

This doesn't scale.

Nick


jeroen at unfix

Aug 20, 2012, 1:40 PM

Post #19 of 53 (1032 views)
Permalink
Re: Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users [In reply to]

On 2012-08-20 22:32, Nick Hilliard wrote:
> On 20/08/2012 21:20, Jeroen Massar wrote:
>> 1) announce the covering prefix
>
> This doesn't scale.

That extra route should not be a problem when there are 800
more-specifics below it.

But, yes, indeed, it should be effectively required that people who do
BGP use something like irrtoolset and run it regularly.

As as you are noticing, people do not update their filters, neither in
IPv4 and neither in IPv6...

Greets,
Jeroen


pekkas at netcore

Aug 20, 2012, 1:42 PM

Post #20 of 53 (1031 views)
Permalink
Re: Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users [In reply to]

On Mon, 20 Aug 2012, Nick Hilliard wrote:
> There is no tragedy of the commons here.

Someone once said (and wrote), "call me when we have 1000 prefixes."
Well, now we have 10000 prefixes and half of them are crap. There's no
response when I try to call. I love the smell of crappy prefixes in
the morning... it smells like.. ipv6 deployment!

The moral of this story? There are plenty. You get to pick.

--
Pekka Savola "You each name yourselves king, yet the
Netcore Oy kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings


roger at jorgensen

Aug 20, 2012, 1:42 PM

Post #21 of 53 (1028 views)
Permalink
Re: Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users [In reply to]

<snip>

> As for whether we should deaggregate PA space, I'm afraid that decision is
> already made. We are not asking for 1000+ /32s from the RIRs, and there
> really isn't another good solution to this problem AFAIK. We are not
> trying to cause problems, but we have constraints in which we must work as
> well.
>
> If this does not fit your view of how the world should work, I am afraid
> we shall have to agree to disagree - unless you can come up with a better
> solution than asking for 1000+ /32s.

there is another way to look at this, consider geobased routing... but
that's like swearing in any church/temple.

Not to forget it will likely create atleast a dozen other issues we have
to deal with... like how and who should manage it, and how should it be
routed etc. Someone will either way have to carry the "covering" prefix
for any region. But who said moving mountains (or moving the wind) should
be easy?

Either way, it is another way to look at this :-)



--
------------------------------
Roger Jorgensen | - ROJO9-RIPE - RJ1866P-NORID
roger [at] jorgensen | - The Future is IPv6
-------------------------------------------------------

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?


laurent at guerby

Aug 20, 2012, 2:01 PM

Post #22 of 53 (1031 views)
Permalink
Re: Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users [In reply to]

On Mon, 2012-08-20 at 18:45 +0100, Nick Hilliard wrote:
> > I guess what the real thing is that the time is RIPE for a RIPE address
> > plan which is akin to ARIN's Micro Allocations.
>
> The current RIPE recommendations are to filter on /48 for v6. No clue
> about Arinland or Apnicistan.

Hi,

The RIPE document:

http://www.ripe.net/ripe/docs/ripe-532
<<
RIPE Routing Working Group Recommendations on IPv6 Route Aggregation
07 Nov 2011
(...)
Recommendation

It is suggested that prefix filters allow for prudent subdivision of an
IPv6 allocation. The operator community will ultimately decide what
degree of subdivision is supportable, but the majority of ISPs accept
prefixes up to a length of /48 within PA space.

Advertisement of more specific prefixes should not be used unless
absolutely necessary and, where sensible, a covering aggregate should
also be advertised. Further, LIRs should use BGP methods such as
NO_EXPORT [RFC-1997] and NOPEER [RFC-3765] or provider-specific
communities, as described in RIPE-399 to limit the propagation of more
specific prefixes in the routing table.

Operators should register appropriate "route6" objects in their
preferred routing registry, or ROAs in the RPKI, to reflect any more
specific advertisements.
(...)
>>

About 1% of AS do not implement this recommendation and
filter at /32 in PA space.

Not having route6 is clearly not good but /48 in RIPE PA space
shouldn't be an issue here for use by CDN like Akamai.

Sincerely,

Laurent


nick at foobar

Aug 20, 2012, 2:06 PM

Post #23 of 53 (1037 views)
Permalink
Re: Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users [In reply to]

On 20/08/2012 21:40, Jeroen Massar wrote:
> On 2012-08-20 22:32, Nick Hilliard wrote:
>> On 20/08/2012 21:20, Jeroen Massar wrote:
>>> 1) announce the covering prefix
>>
>> This doesn't scale.
>
> That extra route should not be a problem when there are 800
> more-specifics below it.

let me clarify this statement then. Akamai claims to deliver ~20% of the
world's internet traffic. While it's possible that announcing their three
/32s would work right now because there's so little ipv6 traffic, in future
times if 1% of the world's internet traffic were to default to using the
three /32 announcements, then that would end up being a very large amount
of traffic indeed, e.g. an order of magnitude or two larger than the median
size of an Akamai node.

This argument isn't going to apply equally to every organisation. Most
organisations don't have a very large distributed network structure like
Akamai, but in Akamai's case it would seem to me to be a very bad idea to
announce their entire /32 allocations.

The 1% figure comes from here:

> https://labs.ripe.net/Members/emileaben/ripe-atlas-a-case-study-of-ipv6-48-filtering

[.Incidentally, a couple of months ago when this same discussion came up
regarding Cloudflare, I took a different approach about announcing only
/48s and no supernet. As Cloudflare are significantly smaller than Akamai,
it makes a lot more sense for them to announce a supernet. Also, Emile's
report changed my mind on the importance of announcing /48s without a
covering /32.]

> But, yes, indeed, it should be effectively required that people who do
> BGP use something like irrtoolset and run it regularly.

I wouldn't force irrtoolset on my worst enemy, but I agree that filters
need to be updated regularly.

> As as you are noticing, people do not update their filters, neither in
> IPv4 and neither in IPv6...

they do, but very slowly. If Akamai or other large CDNs break, they sit up
and take notice.

Nick


nick at foobar

Aug 20, 2012, 2:08 PM

Post #24 of 53 (1030 views)
Permalink
Re: Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users [In reply to]

On 20/08/2012 21:42, Pekka Savola wrote:
> Someone once said (and wrote), "call me when we have 1000 prefixes." Well,
> now we have 10000 prefixes and half of them are crap. There's no response
> when I try to call. I love the smell of crappy prefixes in the morning...
> it smells like.. ipv6 deployment!
>
> The moral of this story? There are plenty. You get to pick.

The ratio of prefixes to unique ASNs is still pretty good though. This is
a better metric of crappiness than the raw number of prefixes.

Nick


tore.anderson at redpill-linpro

Aug 20, 2012, 2:24 PM

Post #25 of 53 (1031 views)
Permalink
Re: Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users [In reply to]

* Patrick W. Gilmore

> Hi everyone. How's it hangin'?

Hi - happily accepting your /48s, thanks for asking!

Reading your message left me with a few questions, though...

> We prefer to use a block from the ISP hosting the node

So you prefer getting individual assignments for each of your nodes.
Okay, with you so far.

> When necessary, [...], we have to make a decision whether the node is
> worth using our own IP space.

Hmm. So if you prefer getting individual assignments in the first place,
why don't get individual PI assignments as a second best option in this
case?

> i.e. when the hosting ISP cannot give us space

Say *what*? Do these really exist - [IPv6 enabled] ISPs that can not
provide their customers with [IPv6] address space? That sounds like
something out of a thedailywtf.com post to me.

> [...] come up with a better solution than asking for 1000+ /32s.

So you haven't just come across *one* such absurd ISP, but over a
thousand? That's good news for IPv6 deployment, I suppose - here I
thought that I would be hard pressed to find over a thousand
IPv6-enabled ISPs full stop - but these weird ISPs of yours must surely
be outnumbered greatly by the normal ones that happily assign you
address space...right? Perhaps you could just take your business to
those instead? I've got space for you, racks AND addresses, if you need
it... ;-)

Seriously though, you *can* go to the RIPE NCC and say in one single
request «I've got 1000+ sites, please give me a /48 for each of them». I
can't see any reason why such a request would be rejected. You'd
probably get a nice contiguous /38 (shorter if you document a growth
expectation) from the PI range, from which people that filter strictly
allow /48s. Win-win.

Best regards,
--
Tore Anderson
Redpill Linpro AS - http://www.redpill-linpro.com/

First page Previous page 1 2 3 Next page Last page  View All nsp ipv6 RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.