Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: NANOG: users

ISP port blocking practice

 

 

First page Previous page 1 2 3 4 5 6 Next page Last page  View All NANOG users RSS feed   Index | Next | Previous | View Threaded


jmaimon at ttec

Oct 25, 2009, 5:52 PM

Post #51 of 126 (760 views)
Permalink
Re: ingress filtering and multiple Internet conenctions [In reply to]

Joe Greco wrote:
>> Joe Greco wrote:

>>
>> Right now things still suck pretty hard, depending on what you are using.
>
> Who defines what "The Right Thing" is?

The right thing is to allow the operator to twiddle the knobs so that
everything works properly with multiple internet connections
specifically when the ISP is using BCP38.


> Basically, allowing this dooms BCP38.
>
> ... JG

Basically getting this wrong dooms BCP38.


owen at delong

Oct 25, 2009, 8:51 PM

Post #52 of 126 (759 views)
Permalink
Re: ingress filtering and multiple Internet conenctions [In reply to]

On Oct 25, 2009, at 4:05 PM, Joe Maimon wrote:

>
>
> Joe Greco wrote:
>
>> There's a problem: I can validly emit a variety of other
>> addresses, in
>> particular any address in 206.55.64.0/20 and some other networks.
>> I am
>> not "forging" packets if I emit 206.55.64.0/20-sourced addresses
>> down a
>> Comcast pipe.
>> How many people realistically have this problem? Well, potentially,
>> lots. Anyone who uses a VPN could have a legitimate IP address on
>> their
>> machine; because of BCP38 (and other security policy) it is common
>> for a VPN setup to forward Internet-bound traffic back to the VPN
>> server rather than directly out the Internet. In some cases, one
>> could
>> reasonably argue that this is undesirable.
>
>
> I would like to take the opportunity to urge vendors of routers and
> firewalls to take extra special care and attention to make sure that
> The Right Thing can always happen whenever multiple egress services
> are employed.
>
> This means that policy routing for network AND ALL locally generated
> traffic should be available and work as the operator intends it to.
>
This includes the ability to turn OFF stateful inspection in all cases
if desired, and, full ability to
support asymmetrical (or Triangle) routing in cases where it is desired.

Also, not breaking PMTU-D would be good.

> Right now things still suck pretty hard, depending on what you are
> using.
>
Indeed.

Owen
Attachments: smime.p7s (2.06 KB)


owen at delong

Oct 25, 2009, 8:52 PM

Post #53 of 126 (758 views)
Permalink
Re: ingress filtering and multiple Internet conenctions [In reply to]

On Oct 25, 2009, at 4:58 PM, Joe Greco wrote:

>> Joe Greco wrote:
>>> There's a problem: I can validly emit a variety of other
>>> addresses, in
>>> particular any address in 206.55.64.0/20 and some other networks.
>>> I am
>>> not "forging" packets if I emit 206.55.64.0/20-sourced addresses
>>> down a
>>> Comcast pipe.
>>>
>>> How many people realistically have this problem? Well, potentially,
>>> lots. Anyone who uses a VPN could have a legitimate IP address on
>>> their
>>> machine; because of BCP38 (and other security policy) it is common
>>> for a VPN setup to forward Internet-bound traffic back to the VPN
>>> server rather than directly out the Internet. In some cases, one
>>> could
>>> reasonably argue that this is undesirable.
>>
>> I would like to take the opportunity to urge vendors of routers and
>> firewalls to take extra special care and attention to make sure
>> that The
>> Right Thing can always happen whenever multiple egress services are
>> employed.
>>
>> This means that policy routing for network AND ALL locally generated
>> traffic should be available and work as the operator intends it to.
>>
>> Right now things still suck pretty hard, depending on what you are
>> using.
>
> Who defines what "The Right Thing" is?
>
> Allowing (what are to the service provider) random IP's inbound, even
> if there's some mechanism to limit it, means that the ISP now has some
> additional responsibilities to be able to transport packets for space
> that isn't theirs; a transit upstream or peer might filter, especially
> for smaller service providers.
>
> Basically, allowing this dooms BCP38.
>
Allowing the operator the configuration OPTION in all cases is good.
Rational defaults in favor of BCP-38 are acceptable. The inability to
override those defaults is bad.

Owen
Attachments: smime.p7s (2.06 KB)


nanog-post at rsuc

Oct 26, 2009, 3:03 AM

Post #54 of 126 (749 views)
Permalink
Re: ISP port blocking practice [In reply to]

[tangent of interst for the archives]

On Sat, Oct 24, 2009 at 02:07:42PM -0500, Joe Greco wrote:
[snip]
> If I'm assigned 24.1.2.3 by Comcast, and Comcast filters my ingress to
> prevent me from emitting other addresses, you claim that's fine because
> it's BCP38.
>
> There's a problem: I can validly emit a variety of other addresses, in
> particular any address in 206.55.64.0/20 and some other networks. I am
> not "forging" packets if I emit 206.55.64.0/20-sourced addresses down a
> Comcast pipe.

Only in your service agreement allows this. Most folks realized both
- the bad guys figured out this 'triangle routing' ages ago (was common
to send bulk abuse traffic down broadband and receive the ack stream
on dialup Back In The Day) and specificlly disallow it.
- such hacks to attempt multihoming without BGP fail in spectacular
ways nd can't be reled on for any real traffic.

So while you may have an allocation and therefore not be 'forging' by
strict definitions, you are injecting martian traffic as far as the
resi broadband provider is concerned and it should be dropped.

--
RSUC / GweepNet / Spunk / FnB / Usenix / SAGE


rbonica at juniper

Nov 3, 2009, 12:21 PM

Post #55 of 126 (718 views)
Permalink
Re: ISP port blocking practice [In reply to]

Folks,

I would love to see the IETF OPSEC WG publish a Best Common Practices
document on ISP Port filtering. The document would capture information
similar to that offered by Justin.

Would anybody on this list be willing to author an Internet Draft?

Ron
(co-director IETF O&M Area)

Justin Shore wrote:
> Zhiyun Qian wrote:
>> Hi all,
>>
>> What is the common practice for enforcing port blocking policy (or what
>> is the common practice for you and your ISP)? More specifically, when
>> ISPs try to block certain outgoing port (port 25 for instance), they
>> could do two rules:
>> 1). For any outgoing traffic, if the destination port is 25, then drop
>> the packets.
>> 2). For any incoming traffic, if the source port is 25, then drop the
>> packets.
>
> I block on both generally. I block inbound and outbound for residential
> customers in dynamic pools. I block inbound only for residential with
> statically-assigned IPs. That way a customer can request (and pay for)
> a static IP and be able to get around out outbound SMTP block. Few
> companies use the MSP port (tcp/587). I'm not sure why more don't make
> the effort but most don't. To make up for that we allow static
> residential customers to evade that filter with a static IP. We still
> block inbound though. We also allow them to use our SMTP servers and
> SmartHosts if they want with no requirements on source domains (like
> some providers require, essentially requiring the customer to advertise
> for you). The inbound block isn't really all that useful as you elude
> to. However I use it more often than not to look for people scanning
> out ranges for open relays. I use that data for feed my RTBH trigger
> router and drop the spammer's traffic on the floor (or the poor,
> unfortunate owner of the compromised PC that's been 0wned.
>
> We block several other things too. Netbios traffic gets dropped both
> ways. MS-SQL traffic gets dropped both ways (a few users have
> complained about this but very few stick to their guns when you point
> out that their traffic is traversing the web completely unencrypted). I
> block default and common proxy ports such as 3128, 7212, and 6588 in
> both directions. Squid is too easy to misconfigure (done it myself).
> GhostSurf and WinGate have both been abusable as open proxies in various
> releases. I also block 8000, 8080 and 8081 towards the customers.
> These are some of our most commonly scanned ports (usually all 3 at once
> plus some or all of the 80xx ones). I've encountered many compromised
> residential CPEs that the users either enabled themselves or were
> enabled by default. I don't block those 3 ports on outbound flows
> though; too many false positives.
>
> And finally we also block several different types of ICMPs. First off
> we block ICMP fragments. Then we permit echo, echo-reply,
> packet-too-big, and time-exceeded. The rest get explicitly dropped.
> IPv6 will change this list dramatically. I haven't had time to research
> ICMPv6 thoroughly enough to say any more than that.
>
> Basically I just pick out some of the really bad ports and block them.
> This gives me a wealth of data with which to null-route compromised PCs
> scanning my networks.
>
>> Also, is it common that the rules are based on tcp flags (e.g. SYN,
>> SYN-ACK)? One would think block SYN packet is good enough.
>
> I don't get that deep into it. Forged packets of types other than SYN
> can still reek havoc on existing flows. I think it's better to block
> all and move on.
>
> Justin
>
>
> .
>


mark at edgewire

Nov 3, 2009, 5:51 PM

Post #56 of 126 (717 views)
Permalink
Re: ISP port blocking practice [In reply to]

Hi all,

Just out of curiosity for those whom may manage Hotel Wifi networks (I
know I know, not really ISP level but since we're on the topic of port
blocking). Does anyone actually make an effort to be blocking port
443? I've had that experience at a few Hotels in Philippines and I
can't think of a valid reason as to why those ports would be dropping
traffic. Would like to hear from anyone whom has had this experience.

Best regards,

Mark

On 04-Nov-2009, at 4:21 AM, Ron Bonica wrote:

> Folks,
>
> I would love to see the IETF OPSEC WG publish a Best Common Practices
> document on ISP Port filtering. The document would capture information
> similar to that offered by Justin.
>
> Would anybody on this list be willing to author an Internet Draft?
>
> Ron
> (co-director IETF O&M Area)
>
> Justin Shore wrote:
>> Zhiyun Qian wrote:
>>> Hi all,
>>>
>>> What is the common practice for enforcing port blocking policy (or
>>> what
>>> is the common practice for you and your ISP)? More specifically,
>>> when
>>> ISPs try to block certain outgoing port (port 25 for instance), they
>>> could do two rules:
>>> 1). For any outgoing traffic, if the destination port is 25, then
>>> drop
>>> the packets.
>>> 2). For any incoming traffic, if the source port is 25, then drop
>>> the
>>> packets.
>>
>> I block on both generally. I block inbound and outbound for
>> residential
>> customers in dynamic pools. I block inbound only for residential
>> with
>> statically-assigned IPs. That way a customer can request (and pay
>> for)
>> a static IP and be able to get around out outbound SMTP block. Few
>> companies use the MSP port (tcp/587). I'm not sure why more don't
>> make
>> the effort but most don't. To make up for that we allow static
>> residential customers to evade that filter with a static IP. We
>> still
>> block inbound though. We also allow them to use our SMTP servers and
>> SmartHosts if they want with no requirements on source domains (like
>> some providers require, essentially requiring the customer to
>> advertise
>> for you). The inbound block isn't really all that useful as you
>> elude
>> to. However I use it more often than not to look for people scanning
>> out ranges for open relays. I use that data for feed my RTBH trigger
>> router and drop the spammer's traffic on the floor (or the poor,
>> unfortunate owner of the compromised PC that's been 0wned.
>>
>> We block several other things too. Netbios traffic gets dropped both
>> ways. MS-SQL traffic gets dropped both ways (a few users have
>> complained about this but very few stick to their guns when you point
>> out that their traffic is traversing the web completely
>> unencrypted). I
>> block default and common proxy ports such as 3128, 7212, and 6588 in
>> both directions. Squid is too easy to misconfigure (done it myself).
>> GhostSurf and WinGate have both been abusable as open proxies in
>> various
>> releases. I also block 8000, 8080 and 8081 towards the customers.
>> These are some of our most commonly scanned ports (usually all 3 at
>> once
>> plus some or all of the 80xx ones). I've encountered many
>> compromised
>> residential CPEs that the users either enabled themselves or were
>> enabled by default. I don't block those 3 ports on outbound flows
>> though; too many false positives.
>>
>> And finally we also block several different types of ICMPs. First
>> off
>> we block ICMP fragments. Then we permit echo, echo-reply,
>> packet-too-big, and time-exceeded. The rest get explicitly dropped.
>> IPv6 will change this list dramatically. I haven't had time to
>> research
>> ICMPv6 thoroughly enough to say any more than that.
>>
>> Basically I just pick out some of the really bad ports and block
>> them.
>> This gives me a wealth of data with which to null-route compromised
>> PCs
>> scanning my networks.
>>
>>> Also, is it common that the rules are based on tcp flags (e.g. SYN,
>>> SYN-ACK)? One would think block SYN packet is good enough.
>>
>> I don't get that deep into it. Forged packets of types other than
>> SYN
>> can still reek havoc on existing flows. I think it's better to block
>> all and move on.
>>
>> Justin
>>
>>
>> .
>>
>


jared at puck

Nov 3, 2009, 6:13 PM

Post #57 of 126 (717 views)
Permalink
Re: ISP port blocking practice [In reply to]

On Nov 3, 2009, at 8:51 PM, mark [at] edgewire wrote:

> Hi all,
>
> Just out of curiosity for those whom may manage Hotel Wifi networks
> (I know I know, not really ISP level but since we're on the topic of
> port blocking). Does anyone actually make an effort to be blocking
> port 443? I've had that experience at a few Hotels in Philippines
> and I can't think of a valid reason as to why those ports would be
> dropping traffic. Would like to hear from anyone whom has had this
> experience.

I've found that some public (eg: Hospital) networks have very
draconian security policies on their guest wireless. The University
of Michigan hospitals block IMAP over SSL (tcp/993), SMTP-Submit (tcp/
587) and all the vpn software I had at my disposal.

This blocking is becoming more common to force people to HTTP/HTTPS
ONLY based systems. They make utilizing these networks from a mobile
device hard, and quickly forget your mac authentication quickly and
are overall poorly run (no feedback loop to get things unblocked that
are legit).

I have found that I am having to vpn-out more often from these 'guest'
networks to obtain "real" internet access. I recommend running a few
gateways (eg: pptp, ipsec, openvpn) to get around these issues.

(I have found some well run hotel networks that intercept tcp/3128 and
send it to a local squid cache).

- Jared


zhiyunq at umich

Sep 2, 2010, 2:59 PM

Post #58 of 126 (468 views)
Permalink
Re: ISP port blocking practice [In reply to]

Sorry for bringing this old topic back. But we have made some academic effort investigating the spamming behaviors using assymetric routing (we named it "triangualr spamming"). This work appeared in this year's IEEE Security & Privacy conference. You can take a look at it if you are interested (and feedbacks are welcome):

http://www.eecs.umich.edu/~zhiyunq/pub/oakland10_triangular-spamming.pdf

One of the high-level findings is that we developed probing techniques to verify that indeed most ISPs are only blocking 1) "outgoing traffic of destination port 25" instead of 2) "incoming traffic with source port 25", which means that these ISPs are vulnerable to the assymetric routing attack.

Finally, many thanks to all your useful inputs :)

Regards.
-Zhiyun
On Oct 22, 2009, at 12:33 PM, Valdis.Kletnieks [at] vt wrote:

> On Thu, 22 Oct 2009 13:22:17 EDT, Zhiyun Qian said:
>> Hi all,
>>
>> What is the common practice for enforcing port blocking policy (or
>> what is the common practice for you and your ISP)? More specifically,
>> when ISPs try to block certain outgoing port (port 25 for instance),
>> they could do two rules:
>> 1). For any outgoing traffic, if the destination port is 25, then drop
>> the packets.
>> 2). For any incoming traffic, if the source port is 25, then drop the
>> packets.
>>
>> Note that either of the rule would be able to block outgoing port 25
>> traffic since each rule essentially represent one direction in a TCP
>> flow. Of course, they could apply both rules. However, based on our
>> measurement study, it looks like most of the ISPs are only using rule
>> 1). Is there any particular reason why rule 1) instead of rule 2)? Or
>> maybe both?
>
> Note that some spammers use assymetric routing with forged packets - they
> spew out a stream of crafted packets from a compromised machine, showing
> a different source IP. The claimed source IP is also under the spammer's
> control, and just basically has to catch the inbound SYN/ACK and forward
> it to the spam-sender (and, if wanted, sink the other ACKs and forward the
> SMTP replies from the server to the real sender). So you can have a big
> sending pipe that doesn't get ingress-filtered(*) sending the spam, and do the
> control from a throwaway that may have a small pipe.
>
> (*) Of course it's not ingress-filtered - if somebody is selling a spammer
> a big pipe for this, they can arrange to fail to filter. ;)
>
> The upshot is, of course, that you want to do (1) because you never actually
> see the (2) packets, they're going someplace else...


bill at herrin

Sep 2, 2010, 3:45 PM

Post #59 of 126 (467 views)
Permalink
Re: ISP port blocking practice [In reply to]

On Thu, Sep 2, 2010 at 5:59 PM, Zhiyun Qian <zhiyunq [at] umich> wrote:
> http://www.eecs.umich.edu/~zhiyunq/pub/oakland10_triangular-spamming.pdf
>
> One of the high-level findings is that we developed probing techniques
> to verify that indeed most ISPs are only blocking 1) "outgoing traffic
> of destination port 25" instead of 2) "incoming traffic with source
> port 25", which means that these ISPs are vulnerable to the
> assymetric routing attack.

If I understand your idea correctly:

1. GoodNet filters TCP destination port 25 packets from his customer
PwndBox, preventing PwndBox from spamming.

2. BadGuy on BadNet sends a forged TCP SYN packet to SpamVictim
allegedly from PwndBox on GoodNet.

3. PwndBox receives the response packets from SpamVictim and tunnels
them to BadGuy allowing BadGuy to complete the spam.

4. GoodNet didn't stop it because PwndBox never sent any packets to TCP port 25.

5. Since the IP address used was GoodNet's, GoodNet's reputation is damaged.

6. GoodNet could prevent this attack vector by also blocking packets
with TCP source port 25 sent -to- PwndBox.

Is that correct?

I observe that if PwndBox is behind a stateful firewall such as a COTS
NAT box, that also prevents this attack.

Regards,
Bill Herrin



--
William D. Herrin ................ herrin [at] dirtside  bill [at] herrin
3005 Crane Dr. ...................... Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004


zhiyunq at umich

Sep 2, 2010, 4:05 PM

Post #60 of 126 (467 views)
Permalink
Re: ISP port blocking practice [In reply to]

You are exactly right. We also talked about stateful firewall that can protect the GoodNet. For NAT box, depends on the type of NAT, it is possible to setup port forwarding on the router (mostly home routers) via uPnP without any authentication (I think many home routers are like this by default). And since the machine in GoodNet is also compromised, it would not be difficult to achieve this.

Regards.
-Zhiyun
On Sep 2, 2010, at 5:45 PM, William Herrin wrote:

> On Thu, Sep 2, 2010 at 5:59 PM, Zhiyun Qian <zhiyunq [at] umich> wrote:
>> http://www.eecs.umich.edu/~zhiyunq/pub/oakland10_triangular-spamming.pdf
>>
>> One of the high-level findings is that we developed probing techniques
>> to verify that indeed most ISPs are only blocking 1) "outgoing traffic
>> of destination port 25" instead of 2) "incoming traffic with source
>> port 25", which means that these ISPs are vulnerable to the
>> assymetric routing attack.
>
> If I understand your idea correctly:
>
> 1. GoodNet filters TCP destination port 25 packets from his customer
> PwndBox, preventing PwndBox from spamming.
>
> 2. BadGuy on BadNet sends a forged TCP SYN packet to SpamVictim
> allegedly from PwndBox on GoodNet.
>
> 3. PwndBox receives the response packets from SpamVictim and tunnels
> them to BadGuy allowing BadGuy to complete the spam.
>
> 4. GoodNet didn't stop it because PwndBox never sent any packets to TCP port 25.
>
> 5. Since the IP address used was GoodNet's, GoodNet's reputation is damaged..
>
> 6. GoodNet could prevent this attack vector by also blocking packets
> with TCP source port 25 sent -to- PwndBox.
>
> Is that correct?
>
> I observe that if PwndBox is behind a stateful firewall such as a COTS
> NAT box, that also prevents this attack.
>
> Regards,
> Bill Herrin
>
>
>
> --
> William D. Herrin ................ herrin [at] dirtside bill [at] herrin
> 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/>
> Falls Church, VA 22042-3004
>
>


ops.lists at gmail

Sep 2, 2010, 6:19 PM

Post #61 of 126 (468 views)
Permalink
Re: ISP port blocking practice [In reply to]

Zhiyun, this is by far the most comprehensive paper I've seen on
asymmetric routing spam .. a technique that's as old as, for example,
Alan Ralsky. So been around for about a decade.

Congratulations, great effort. Do you have more results available (in
more detail than were published in this paper)? Should be worth
seeing.

thanks
--srs

On Fri, Sep 3, 2010 at 3:29 AM, Zhiyun Qian <zhiyunq [at] umich> wrote:
> Sorry for bringing this old topic back. But we have made some academic effort investigating the spamming behaviors using assymetric routing (we named it "triangualr spamming"). This work appeared in this year's IEEE Security & Privacy conference. You can take a look at it if you are interested (and feedbacks are welcome):
>
> http://www.eecs.umich.edu/~zhiyunq/pub/oakland10_triangular-spamming.pdf



--
Suresh Ramasubramanian (ops.lists [at] gmail)


zhiyunq at umich

Sep 2, 2010, 7:17 PM

Post #62 of 126 (468 views)
Permalink
Re: ISP port blocking practice [In reply to]

Suresh, thanks for your interest. I see you've had a lot of experience in fighting spam, so you must have known this. Yes, I know this spamming technique has been around for a while. But it's surprising to see that the majority of the ISPs that we studied are still vulnerable to this attack. That probably indicates that it is not as widely known as we would expect. So I thought it would be beneficial to raise the awareness of the problem.

In terms of more results, the paper is the most detailed document we have. Otherwise, if you interested in the data that we collected (which ISPs or IP ranges are vulnerable to this attack). We can chat offline.

Regards.
-Zhiyun
On Sep 2, 2010, at 8:19 PM, Suresh Ramasubramanian wrote:

> Zhiyun, this is by far the most comprehensive paper I've seen on
> asymmetric routing spam .. a technique that's as old as, for example,
> Alan Ralsky. So been around for about a decade.
>
> Congratulations, great effort. Do you have more results available (in
> more detail than were published in this paper)? Should be worth
> seeing.
>
> thanks
> --srs
>
> On Fri, Sep 3, 2010 at 3:29 AM, Zhiyun Qian <zhiyunq [at] umich> wrote:
>> Sorry for bringing this old topic back. But we have made some academic effort investigating the spamming behaviors using assymetric routing (we named it "triangualr spamming"). This work appeared in this year's IEEE Security & Privacy conference. You can take a look at it if you are interested (and feedbacks are welcome):
>>
>> http://www.eecs.umich.edu/~zhiyunq/pub/oakland10_triangular-spamming.pdf
>
>
>
> --
> Suresh Ramasubramanian (ops.lists [at] gmail)
>
>


ops.lists at gmail

Sep 2, 2010, 7:20 PM

Post #63 of 126 (468 views)
Permalink
Re: ISP port blocking practice [In reply to]

BCP38 / RFC2827 were created specifically to address some quite
similar problems. And googling either of those two strings on nanog
will get you a lot of griping and/or reasons as to why these aren't
being more widely adopted :)

--srs

On Fri, Sep 3, 2010 at 7:47 AM, Zhiyun Qian <zhiyunq [at] umich> wrote:
> Suresh, thanks for your interest. I see you've had a lot of experience in fighting spam, so you must have known this. Yes, I know this spamming technique has been around for a while. But it's surprising to see that the majority of the ISPs that we studied are still vulnerable to this attack.  That probably indicates that it is not as widely known as we would expect. So I thought it would be beneficial to raise the awareness of the problem.
>
> In terms of more results, the paper is the most detailed document we have. Otherwise, if you interested in the data that we collected (which ISPs or IP ranges are vulnerable to this attack). We can chat offline.
>
> Regards.
> -Zhiyun


zhiyunq at umich

Sep 2, 2010, 7:23 PM

Post #64 of 126 (468 views)
Permalink
Re: ISP port blocking practice [In reply to]

Great. Thanks for the information.

-Zhiyun
On Sep 2, 2010, at 9:20 PM, Suresh Ramasubramanian wrote:

> BCP38 / RFC2827 were created specifically to address some quite
> similar problems. And googling either of those two strings on nanog
> will get you a lot of griping and/or reasons as to why these aren't
> being more widely adopted :)
>
> --srs
>
> On Fri, Sep 3, 2010 at 7:47 AM, Zhiyun Qian <zhiyunq [at] umich> wrote:
>> Suresh, thanks for your interest. I see you've had a lot of experience in fighting spam, so you must have known this. Yes, I know this spamming technique has been around for a while. But it's surprising to see that the majority of the ISPs that we studied are still vulnerable to this attack. That probably indicates that it is not as widely known as we would expect. So I thought it would be beneficial to raise the awareness of the problem.
>>
>> In terms of more results, the paper is the most detailed document we have. Otherwise, if you interested in the data that we collected (which ISPs or IP ranges are vulnerable to this attack). We can chat offline.
>>
>> Regards.
>> -Zhiyun
>
>


zhiyunq at umich

Sep 2, 2010, 7:55 PM

Post #65 of 126 (468 views)
Permalink
Re: ISP port blocking practice [In reply to]

I skimmed through these specs. They are useful but seems only related specific to IP spoofing prevention. I see that IP spoofing is part of the asymmetric routing story. But I was more thinking that given that IP spoofing is not widely adopted, the other defenses that they can more perhaps more easily implement is to block incoming traffic with source port 25 (if they already decided to block outgoing traffic with destination port 25). But according to our study, most of the ISPs didn't do that at the time of study (probably still true today).

-Zhiyun
On Sep 2, 2010, at 9:20 PM, Suresh Ramasubramanian wrote:

> BCP38 / RFC2827 were created specifically to address some quite
> similar problems. And googling either of those two strings on nanog
> will get you a lot of griping and/or reasons as to why these aren't
> being more widely adopted :)
>
> --srs
>
> On Fri, Sep 3, 2010 at 7:47 AM, Zhiyun Qian <zhiyunq [at] umich> wrote:
>> Suresh, thanks for your interest. I see you've had a lot of experience in fighting spam, so you must have known this. Yes, I know this spamming technique has been around for a while. But it's surprising to see that the majority of the ISPs that we studied are still vulnerable to this attack. That probably indicates that it is not as widely known as we would expect. So I thought it would be beneficial to raise the awareness of the problem.
>>
>> In terms of more results, the paper is the most detailed document we have. Otherwise, if you interested in the data that we collected (which ISPs or IP ranges are vulnerable to this attack). We can chat offline.
>>
>> Regards.
>> -Zhiyun
>
>


dts at senie

Sep 2, 2010, 8:04 PM

Post #66 of 126 (469 views)
Permalink
Re: ISP port blocking practice [In reply to]

Ingress filtering is the correct tool for the job. The whole point here is that packets are coming from somewhere they should not, and they are thus spoofed. The tools have been in place to deal with this for a very long time now. The drafts that became RFC 2267 (precursor of RFC 2827 / BCP38) date from mid-1996. Paul and I wrote the original drafts to solve something else, but the issue is the same. Solving the vector you're concerned about doesn't need another layer of implementation in the mail servers. The packet routing fabric needs to handle it, and doing so addresses far more than just the email situation. I agree it'd be nice to get the asymmetric attack stopped, but disagree we need yet another mechanism to do it.

- Dan


On Sep 2, 2010, at 10:55 PM, Zhiyun Qian wrote:

> I skimmed through these specs. They are useful but seems only related specific to IP spoofing prevention. I see that IP spoofing is part of the asymmetric routing story. But I was more thinking that given that IP spoofing is not widely adopted, the other defenses that they can more perhaps more easily implement is to block incoming traffic with source port 25 (if they already decided to block outgoing traffic with destination port 25). But according to our study, most of the ISPs didn't do that at the time of study (probably still true today).
>
> -Zhiyun
> On Sep 2, 2010, at 9:20 PM, Suresh Ramasubramanian wrote:
>
>> BCP38 / RFC2827 were created specifically to address some quite
>> similar problems. And googling either of those two strings on nanog
>> will get you a lot of griping and/or reasons as to why these aren't
>> being more widely adopted :)
>>
>> --srs
>>
>> On Fri, Sep 3, 2010 at 7:47 AM, Zhiyun Qian <zhiyunq [at] umich> wrote:
>>> Suresh, thanks for your interest. I see you've had a lot of experience in fighting spam, so you must have known this. Yes, I know this spamming technique has been around for a while. But it's surprising to see that the majority of the ISPs that we studied are still vulnerable to this attack. That probably indicates that it is not as widely known as we would expect. So I thought it would be beneficial to raise the awareness of the problem.
>>>
>>> In terms of more results, the paper is the most detailed document we have. Otherwise, if you interested in the data that we collected (which ISPs or IP ranges are vulnerable to this attack). We can chat offline.
>>>
>>> Regards.
>>> -Zhiyun
>>
>>
>
>


owen at delong

Sep 2, 2010, 8:48 PM

Post #67 of 126 (451 views)
Permalink
Re: ISP port blocking practice [In reply to]

We should be seeking to stop damaging the network for ineffective anti spam measures (blocking outbound 25 for example) rather than to expand this practice to bidirectional brokenness.

Owen

Sent from my iPad

On Sep 3, 2010, at 12:25 PM, Zhiyun Qian <zhiyunq [at] umich> wrote:

> I skimmed through these specs. They are useful but seems only related specific to IP spoofing prevention. I see that IP spoofing is part of the asymmetric routing story. But I was more thinking that given that IP spoofing is not widely adopted, the other defenses that they can more perhaps more easily implement is to block incoming traffic with source port 25 (if they already decided to block outgoing traffic with destination port 25). But according to our study, most of the ISPs didn't do that at the time of study (probably still true today).
>
> -Zhiyun
> On Sep 2, 2010, at 9:20 PM, Suresh Ramasubramanian wrote:
>
>> BCP38 / RFC2827 were created specifically to address some quite
>> similar problems. And googling either of those two strings on nanog
>> will get you a lot of griping and/or reasons as to why these aren't
>> being more widely adopted :)
>>
>> --srs
>>
>> On Fri, Sep 3, 2010 at 7:47 AM, Zhiyun Qian <zhiyunq [at] umich> wrote:
>>> Suresh, thanks for your interest. I see you've had a lot of experience in fighting spam, so you must have known this. Yes, I know this spamming technique has been around for a while. But it's surprising to see that the majority of the ISPs that we studied are still vulnerable to this attack. That probably indicates that it is not as widely known as we would expect. So I thought it would be beneficial to raise the awareness of the problem.
>>>
>>> In terms of more results, the paper is the most detailed document we have. Otherwise, if you interested in the data that we collected (which ISPs or IP ranges are vulnerable to this attack). We can chat offline.
>>>
>>> Regards.
>>> -Zhiyun
>>
>>
>


patrick at ianai

Sep 2, 2010, 8:54 PM

Post #68 of 126 (451 views)
Permalink
Re: ISP port blocking practice [In reply to]

On Sep 2, 2010, at 11:48 PM, Owen DeLong wrote:

> We should be seeking to stop damaging the network for ineffective anti spam measures (blocking outbound 25 for example) rather than to expand this practice to bidirectional brokenness.

Since at least part of your premise ('ineffective anti-spam measures') has been objectively proven false to fact for many years, I guess we can ignore the rest of your note.

But thanx for playing. :)

Also, just so everyone doesn't think I'm in favor of "damaging" the network, I would much prefer a completely open 'Net. Who wouldn't? Since that is not possible, we have to do what we can to damage the network as little as possible. Port 25 blocking is completely unnoticeable to something on the order of 5-nines worth of users, and the rest should know how to get around it with a minimum of fuss (including things like "ask your provider to unblock" in many cases).

--
TTFN,
patrick


> On Sep 3, 2010, at 12:25 PM, Zhiyun Qian <zhiyunq [at] umich> wrote:
>
>> I skimmed through these specs. They are useful but seems only related specific to IP spoofing prevention. I see that IP spoofing is part of the asymmetric routing story. But I was more thinking that given that IP spoofing is not widely adopted, the other defenses that they can more perhaps more easily implement is to block incoming traffic with source port 25 (if they already decided to block outgoing traffic with destination port 25). But according to our study, most of the ISPs didn't do that at the time of study (probably still true today).
>>
>> -Zhiyun
>> On Sep 2, 2010, at 9:20 PM, Suresh Ramasubramanian wrote:
>>
>>> BCP38 / RFC2827 were created specifically to address some quite
>>> similar problems. And googling either of those two strings on nanog
>>> will get you a lot of griping and/or reasons as to why these aren't
>>> being more widely adopted :)
>>>
>>> --srs
>>>
>>> On Fri, Sep 3, 2010 at 7:47 AM, Zhiyun Qian <zhiyunq [at] umich> wrote:
>>>> Suresh, thanks for your interest. I see you've had a lot of experience in fighting spam, so you must have known this. Yes, I know this spamming technique has been around for a while. But it's surprising to see that the majority of the ISPs that we studied are still vulnerable to this attack. That probably indicates that it is not as widely known as we would expect. So I thought it would be beneficial to raise the awareness of the problem.
>>>>
>>>> In terms of more results, the paper is the most detailed document we have. Otherwise, if you interested in the data that we collected (which ISPs or IP ranges are vulnerable to this attack). We can chat offline.
>>>>
>>>> Regards.
>>>> -Zhiyun
>>>
>>>
>>
>


franck at genius

Sep 2, 2010, 8:56 PM

Post #69 of 126 (453 views)
Permalink
Re: ISP port blocking practice [In reply to]

Blocking outbound port 25 in certain conditions (mainly anything with a dynamic IPv4), is a recommended practice from MAAWG.org and others, they have a few useful documents for ISPs to deal with their network.

----- Original Message -----
From: "Owen DeLong" <owen [at] delong>
To: "Zhiyun Qian" <zhiyunq [at] umich>
Cc: "NANOG list" <nanog [at] nanog>
Sent: Friday, 3 September, 2010 3:48:20 PM
Subject: Re: ISP port blocking practice

We should be seeking to stop damaging the network for ineffective anti spam measures (blocking outbound 25 for example) rather than to expand this practice to bidirectional brokenness.

Owen

Sent from my iPad

On Sep 3, 2010, at 12:25 PM, Zhiyun Qian <zhiyunq [at] umich> wrote:

> I skimmed through these specs. They are useful but seems only related specific to IP spoofing prevention. I see that IP spoofing is part of the asymmetric routing story. But I was more thinking that given that IP spoofing is not widely adopted, the other defenses that they can more perhaps more easily implement is to block incoming traffic with source port 25 (if they already decided to block outgoing traffic with destination port 25). But according to our study, most of the ISPs didn't do that at the time of study (probably still true today).
>
> -Zhiyun
> On Sep 2, 2010, at 9:20 PM, Suresh Ramasubramanian wrote:
>
>> BCP38 / RFC2827 were created specifically to address some quite
>> similar problems. And googling either of those two strings on nanog
>> will get you a lot of griping and/or reasons as to why these aren't
>> being more widely adopted :)
>>
>> --srs
>>
>> On Fri, Sep 3, 2010 at 7:47 AM, Zhiyun Qian <zhiyunq [at] umich> wrote:
>>> Suresh, thanks for your interest. I see you've had a lot of experience in fighting spam, so you must have known this. Yes, I know this spamming technique has been around for a while. But it's surprising to see that the majority of the ISPs that we studied are still vulnerable to this attack. That probably indicates that it is not as widely known as we would expect. So I thought it would be beneficial to raise the awareness of the problem.
>>>
>>> In terms of more results, the paper is the most detailed document we have. Otherwise, if you interested in the data that we collected (which ISPs or IP ranges are vulnerable to this attack). We can chat offline.
>>>
>>> Regards.
>>> -Zhiyun
>>
>>
>


jbates at brightok

Sep 2, 2010, 9:08 PM

Post #70 of 126 (451 views)
Permalink
Re: ISP port blocking practice [In reply to]

Patrick W. Gilmore wrote:
>> We should be seeking to stop damaging the network for ineffective anti spam measures (blocking outbound 25 for example) rather than to expand this practice to bidirectional brokenness.
>
> Since at least part of your premise ('ineffective anti-spam measures') has been objectively proven false to fact for many years, I guess we can ignore the rest of your note.

He's right though. tcp/25 blocks are a hack. Easy man's way out.
Honestly, it'd be nicer if edge or even core systems could easily handle
higher level filtering for things like this. There's plenty of systems
that watch traffic patterns and issue blocks based on those patterns.

I was working with a hotel today concerning just that. They were only
doing a generic 500 connections in x period, block mac. They are now
adding a tighter rule for 15 tcp/25 connections in 1 minute, block
tcp/25 (or mac, doesn't matter to me). Of course, we didn't see valid
reasons for mail blasts to be leaving a hotel and 15/minute is plenty of
grace for a normal user. At an ISP level, it would work fine, though
methods for determining exceptions would have to be planned (though that
could easily be handled by customer classifications like everything else).

> Also, just so everyone doesn't think I'm in favor of "damaging" the network, I would much prefer a completely open 'Net. Who wouldn't? Since that is not possible, we have to do what we can to damage the network as little as possible. Port 25 blocking is completely unnoticeable to something on the order of 5-nines worth of users, and the rest should know how to get around it with a minimum of fuss (including things like "ask your provider to unblock" in many cases).
>

Blocking inbound vs outbound is another story, though. Getting people to
implement spoof protections is more useful. I'd be interested to see
your data for concluding 5-nines of users, or did you just make that up?


Jack


franck at genius

Sep 2, 2010, 10:41 PM

Post #71 of 126 (440 views)
Permalink
Re: ISP port blocking practice [In reply to]

Have you heard of the submission port?

Why Clients of an hotel would run a MTA anyhow?

----- Original Message -----
From: "Jack Bates" <jbates [at] brightok>
To: "NANOG list" <nanog [at] nanog>
Sent: Friday, 3 September, 2010 4:08:54 PM
Subject: Re: ISP port blocking practice

Patrick W. Gilmore wrote:
>> We should be seeking to stop damaging the network for ineffective anti spam measures (blocking outbound 25 for example) rather than to expand this practice to bidirectional brokenness.
>
> Since at least part of your premise ('ineffective anti-spam measures') has been objectively proven false to fact for many years, I guess we can ignore the rest of your note.

He's right though. tcp/25 blocks are a hack. Easy man's way out.
Honestly, it'd be nicer if edge or even core systems could easily handle
higher level filtering for things like this. There's plenty of systems
that watch traffic patterns and issue blocks based on those patterns.

I was working with a hotel today concerning just that. They were only
doing a generic 500 connections in x period, block mac. They are now
adding a tighter rule for 15 tcp/25 connections in 1 minute, block
tcp/25 (or mac, doesn't matter to me). Of course, we didn't see valid
reasons for mail blasts to be leaving a hotel and 15/minute is plenty of
grace for a normal user. At an ISP level, it would work fine, though
methods for determining exceptions would have to be planned (though that
could easily be handled by customer classifications like everything else).

> Also, just so everyone doesn't think I'm in favor of "damaging" the network, I would much prefer a completely open 'Net. Who wouldn't? Since that is not possible, we have to do what we can to damage the network as little as possible. Port 25 blocking is completely unnoticeable to something on the order of 5-nines worth of users, and the rest should know how to get around it with a minimum of fuss (including things like "ask your provider to unblock" in many cases).
>

Blocking inbound vs outbound is another story, though. Getting people to
implement spoof protections is more useful. I'd be interested to see
your data for concluding 5-nines of users, or did you just make that up?


Jack


owen at delong

Sep 3, 2010, 5:12 AM

Post #72 of 126 (430 views)
Permalink
Re: ISP port blocking practice [In reply to]

On Sep 2, 2010, at 8:54 PM, Patrick W. Gilmore wrote:

> On Sep 2, 2010, at 11:48 PM, Owen DeLong wrote:
>
>> We should be seeking to stop damaging the network for ineffective anti spam measures (blocking outbound 25 for example) rather than to expand this practice to bidirectional brokenness.
>
> Since at least part of your premise ('ineffective anti-spam measures') has been objectively proven false to fact for many years, I guess we can ignore the rest of your note.
>
Really? So, since so many ISPs are blocking port 25, there's lots less spam hitting our networks?
That's really news to me... I'm still seeing an ever increasing number of attempts to deliver spam on my mailservers.

I'd say that it has been pretty ineffective.

> Also, just so everyone doesn't think I'm in favor of "damaging" the network, I would much prefer a completely open 'Net. Who wouldn't? Since that is not possible, we have to do what we can to damage the network as little as possible. Port 25 blocking is completely unnoticeable to something on the order of 5-nines worth of users, and the rest should know how to get around it with a minimum of fuss (including things like "ask your provider to unblock" in many cases).
>
Not really true. First, i dispute your 5-nines figure, second, yes, i can usually get around it, but seems each network requires a different workaround. Since, like many of us, I use a lot of transient networks, having to reconfigure for each unique set of brokenness is actually wasting more of my time than the spam this brokenness was alleged to prevent.

I suppose I should just shut up and run an instance of my SMTP daemon on port 80. After all, since IPv4 addresses are so abundant, rather than use port numbers for services, let's use IP addresses and force everything to ports 80 and 443.

Owen


owen at delong

Sep 3, 2010, 5:15 AM

Post #73 of 126 (430 views)
Permalink
Re: ISP port blocking practice [In reply to]

On Sep 2, 2010, at 9:08 PM, Jack Bates wrote:

> Patrick W. Gilmore wrote:
>>> We should be seeking to stop damaging the network for ineffective anti spam measures (blocking outbound 25 for example) rather than to expand this practice to bidirectional brokenness.
>> Since at least part of your premise ('ineffective anti-spam measures') has been objectively proven false to fact for many years, I guess we can ignore the rest of your note.
>
> He's right though. tcp/25 blocks are a hack. Easy man's way out. Honestly, it'd be nicer if edge or even core systems could easily handle higher level filtering for things like this. There's plenty of systems that watch traffic patterns and issue blocks based on those patterns.
>
> I was working with a hotel today concerning just that. They were only doing a generic 500 connections in x period, block mac. They are now adding a tighter rule for 15 tcp/25 connections in 1 minute, block tcp/25 (or mac, doesn't matter to me). Of course, we didn't see valid reasons for mail blasts to be leaving a hotel and 15/minute is plenty of grace for a normal user. At an ISP level, it would work fine, though methods for determining exceptions would have to be planned (though that could easily be handled by customer classifications like everything else).
>
This seems to me like it would be much more effective and less damaging without being significantly harder to implement.

>> Also, just so everyone doesn't think I'm in favor of "damaging" the network, I would much prefer a completely open 'Net. Who wouldn't? Since that is not possible, we have to do what we can to damage the network as little as possible. Port 25 blocking is completely unnoticeable to something on the order of 5-nines worth of users, and the rest should know how to get around it with a minimum of fuss (including things like "ask your provider to unblock" in many cases).
>
> Blocking inbound vs outbound is another story, though. Getting people to implement spoof protections is more useful. I'd be interested to see your data for concluding 5-nines of users, or did you just make that up?
>
I'm all for anti-spoof (BCP-38) and strict/loose (as appropriate) RPF. I implement those things on networks I run. That's not damage, that's blocking things that shouldn't happen.

I'd also like to see his data to support his claim that it is somehow effective at reducing spam.

Owen


owen at delong

Sep 3, 2010, 5:18 AM

Post #74 of 126 (430 views)
Permalink
Re: ISP port blocking practice [In reply to]

It may be a recommended practice from MAAWG, but, it's still damage to the network which is often routed around. It's a minor inconvenience to spammers and a slightly bigger problem for legitimate
users. I don't see the win here. Just because they recommend it doesn't make it a good recommendation.
MAAWG appears to have a single priority... Reducing spam by whatever means possible, regardless
of cost or efficacy. Some of their recommendations (most, even) are good and useful. Some are
easy to implement, ineffective, and ill-conceived. Outbound blocking of port 25 from people attempting
to reach their home MTA/MSA with TLS and SMTP-AUTH just because they don't have a static address
is an example of easy to implement, ineffective, and ill-conceived.

Owen

On Sep 2, 2010, at 8:56 PM, Franck Martin wrote:

> Blocking outbound port 25 in certain conditions (mainly anything with a dynamic IPv4), is a recommended practice from MAAWG.org and others, they have a few useful documents for ISPs to deal with their network.
>
> ----- Original Message -----
> From: "Owen DeLong" <owen [at] delong>
> To: "Zhiyun Qian" <zhiyunq [at] umich>
> Cc: "NANOG list" <nanog [at] nanog>
> Sent: Friday, 3 September, 2010 3:48:20 PM
> Subject: Re: ISP port blocking practice
>
> We should be seeking to stop damaging the network for ineffective anti spam measures (blocking outbound 25 for example) rather than to expand this practice to bidirectional brokenness.
>
> Owen
>
> Sent from my iPad
>
> On Sep 3, 2010, at 12:25 PM, Zhiyun Qian <zhiyunq [at] umich> wrote:
>
>> I skimmed through these specs. They are useful but seems only related specific to IP spoofing prevention. I see that IP spoofing is part of the asymmetric routing story. But I was more thinking that given that IP spoofing is not widely adopted, the other defenses that they can more perhaps more easily implement is to block incoming traffic with source port 25 (if they already decided to block outgoing traffic with destination port 25). But according to our study, most of the ISPs didn't do that at the time of study (probably still true today).
>>
>> -Zhiyun
>> On Sep 2, 2010, at 9:20 PM, Suresh Ramasubramanian wrote:
>>
>>> BCP38 / RFC2827 were created specifically to address some quite
>>> similar problems. And googling either of those two strings on nanog
>>> will get you a lot of griping and/or reasons as to why these aren't
>>> being more widely adopted :)
>>>
>>> --srs
>>>
>>> On Fri, Sep 3, 2010 at 7:47 AM, Zhiyun Qian <zhiyunq [at] umich> wrote:
>>>> Suresh, thanks for your interest. I see you've had a lot of experience in fighting spam, so you must have known this. Yes, I know this spamming technique has been around for a while. But it's surprising to see that the majority of the ISPs that we studied are still vulnerable to this attack. That probably indicates that it is not as widely known as we would expect. So I thought it would be beneficial to raise the awareness of the problem.
>>>>
>>>> In terms of more results, the paper is the most detailed document we have. Otherwise, if you interested in the data that we collected (which ISPs or IP ranges are vulnerable to this attack). We can chat offline.
>>>>
>>>> Regards.
>>>> -Zhiyun
>>>
>>>
>>
>


owen at delong

Sep 3, 2010, 5:22 AM

Post #75 of 126 (430 views)
Permalink
Re: ISP port blocking practice [In reply to]

On Sep 2, 2010, at 10:41 PM, Franck Martin wrote:

> Have you heard of the submission port?
>
Yes... Many of the idiots that block outbound 25 also block outbound 587 and sometimes 465.

> Why Clients of an hotel would run a MTA anyhow?
>
Huh? I think you misunderstand... The problem is hotels blocking people from submitting outbound
messages to their home MTA, not people trying to run an MTA inside their hotel room. NAT pretty
much guarantees running an MTA inside the hotel room is impossible in most circumstances.
(That might improve when IPv6 starts hitting hotel rooms, but, for now, it's just not there).
Yes, some hotels offer you the option of a public IP (and usually when I take that option, I have
fewer network problems in general. One of the reasons I tend to prefer Hilton brands when possible.)

Owen

> ----- Original Message -----
> From: "Jack Bates" <jbates [at] brightok>
> To: "NANOG list" <nanog [at] nanog>
> Sent: Friday, 3 September, 2010 4:08:54 PM
> Subject: Re: ISP port blocking practice
>
> Patrick W. Gilmore wrote:
>>> We should be seeking to stop damaging the network for ineffective anti spam measures (blocking outbound 25 for example) rather than to expand this practice to bidirectional brokenness.
>>
>> Since at least part of your premise ('ineffective anti-spam measures') has been objectively proven false to fact for many years, I guess we can ignore the rest of your note.
>
> He's right though. tcp/25 blocks are a hack. Easy man's way out.
> Honestly, it'd be nicer if edge or even core systems could easily handle
> higher level filtering for things like this. There's plenty of systems
> that watch traffic patterns and issue blocks based on those patterns.
>
> I was working with a hotel today concerning just that. They were only
> doing a generic 500 connections in x period, block mac. They are now
> adding a tighter rule for 15 tcp/25 connections in 1 minute, block
> tcp/25 (or mac, doesn't matter to me). Of course, we didn't see valid
> reasons for mail blasts to be leaving a hotel and 15/minute is plenty of
> grace for a normal user. At an ISP level, it would work fine, though
> methods for determining exceptions would have to be planned (though that
> could easily be handled by customer classifications like everything else).
>
>> Also, just so everyone doesn't think I'm in favor of "damaging" the network, I would much prefer a completely open 'Net. Who wouldn't? Since that is not possible, we have to do what we can to damage the network as little as possible. Port 25 blocking is completely unnoticeable to something on the order of 5-nines worth of users, and the rest should know how to get around it with a minimum of fuss (including things like "ask your provider to unblock" in many cases).
>>
>
> Blocking inbound vs outbound is another story, though. Getting people to
> implement spoof protections is more useful. I'd be interested to see
> your data for concluding 5-nines of users, or did you just make that up?
>
>
> Jack
>

First page Previous page 1 2 3 4 5 6 Next page Last page  View All NANOG users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.