Mailing List Archive: NANOG: users

Abuse procedures... Reality Checks

1 2 3 4

View All

Index | Next | Previous | View Threaded

dotis at mail-abuse

Apr 9, 2007, 3:04 PM

Post #51 of 83 (4197 views)
Permalink

Re: Abuse procedures... Reality Checks [In reply to]

On Apr 8, 2007, at 9:03 PM, Paul Vixie wrote:
> dotis [at] mail-abuse (Douglas Otis) writes:
>
>> Good advise. For various reasons, a majority of IP addresses
>> within a CIDR of any size being abusive is likely to cause the
>> CIDR to be blocked. While a majority could be considered as being
>> half right, the existence of the "bad neighborhood" demonstrates a
>> lack of oversight for the entire CIDR, which is also fairly
>> predictive of future abuse.
>
> that sounds like a continuum, but my experience requires more
> dimensions than you're describing. for example, this weekend two /
> 24's were hijacked and used for spam spew.

Agreed.

This was expressed recently as well.

http://www.merit.edu/mail.archives/nanog/msg05351.html

CIDRs should also conform with ASN boundaries and reputation tracks
with announcements.

Unfortunately an effort to create a black-hole operator's BCP failed
to consider these issues. Many building their own reputation
histories will also likely ignore this concern. This means John's
advice remains valid, whether fair or not. Adopting transient
tracking methods cope with this problem.

-Doug

chris at westnet

Apr 9, 2007, 4:49 PM

Post #52 of 83 (4215 views)
Permalink

Re: Abuse procedures... Reality Checks [In reply to]

On Mon, 9 Apr 2007, Paul Vixie wrote:

>
> than you're describing. for example, this weekend two /24's were hijacked
> and used for spam spew. as my receivebot started blackholing /32's, the

Why do you think they were hijacked ? At least for your second block:

> 1 71.6.213.103
> ....

I've had that /24 blocked since 4/4/07. I have spam attempts for that domain
going back to Feb 13 2007, but it didn't have reverse DNS set up until 4/4
so nothing got through.

==========================================================
Chris Candreva -- chris [at] westnet -- (914) 948-3162
WestNet Internet Services of Westchester
http://www.westnet.com/

michael.dillon at bt

Apr 10, 2007, 2:30 AM

Post #53 of 83 (4204 views)
Permalink

RE: Abuse procedures... Reality Checks [In reply to]

> I have to disagree. SWIP is not meaningless.
>
> In my company some functions related to sending a SWIP are
> automated, but my company has people on staff who know that
> it is happening and what it means.
>
> And I talk with plenty of other companies that fall into the
> same boat.
>
> In short I find this one comment below to be argumentive and
> full of conjecture.

No more argumentative and full of conjecture than your posting. I said
that there were SOME companies where SWIP is just a mysterious automated
process and nobody on staff fully understands the meaning of it, beyond
the fact that it needs to be done to help get approval for that next
allocation request.

The fact that SOME companies do have a process for managing SWIP as they
understand it, does not mean that there are no delinquents.

I also find it curious that you claim to have people on staff at your
company who know what SWIP means. Perhaps you could ask them to share
that information with us since I have never seen this documented
anywhere. Do they really know what you claim they know?

--Michael Dillon

rsk at gsp

Apr 10, 2007, 5:09 AM

Post #54 of 83 (4196 views)
Permalink

Re: Abuse procedures... Reality Checks [In reply to]

On Sat, Apr 07, 2007 at 09:50:34PM +0000, Fergie wrote:
> I would have to respectfully disagree with you. When network
> operators do due diligence and SWIP their sub-allocations, they
> (the sub-allocations) should be authoritative in regards to things
> like RBLs.

After thinking it over: I partly-to-mostly agree. In principal, yes.
In practice, however, [some] negligent network operators have built
such long and pervasive track records of large-scale abuse that their
allocations can be classified into two categories:

1. Those that have emitted lots of abuse.
2. Those that are going to emit lots of abuse.

In such cases, I'm not inclined to wait for (2) to become reality.

---Rsk

rsk at gsp

Apr 10, 2007, 5:26 AM

Post #55 of 83 (4206 views)
Permalink

Re: Abuse procedures... Reality Checks [In reply to]

On Sat, Apr 07, 2007 at 04:20:59PM -0500, Frank Bulk wrote:
> Define network operator: the AS holder for that space or the operator of
> that smaller-than-slash-24 sub-block? If the problem consistently comes
> from /29 why not just leave the block in and be done with it?

Because experience...long, bitter experience...strongly indicates that
what happens today often merely presages what will happen tomorrow.

Because I haven't got unlimited time. Or money. Or resources.

Because I haven't got unlimited WHOIS queries. (Although I and everyone
else *should* have those. There are no valid reasons to rate-limit any
form of WHOIS query.)

Because there are way, WAY too many incompetently-managed networks whose
operators can often be heard complaining about the abuse inbound to them
at the same time they fail to take rudimentary measures to control the
abuse outbound from them. <cough> port 25 blocking <cough>

Because I was more patient for the first decade or two, and it proved
to be a losing strategy.

Because This Is Not My Problem. If by chance someone benign has chosen
to locate their operation in known-hostile, known-negligently-operated
network space, then their failure to perform due diligence may have
consequences for them.

> I guess this begs the question: Is it best to block with a /32, /24, or some
> other range? Sounds a lot like throwing something against the wall and
> seeing what sticks. Or vigilantism.

1. Gratuitously labeling carefully-considered measures as random is not a
route to productive conversation.

2. It is hardly "vigilantism" to take passive measures to protect one's
network/systems/users from hostile activity. Doubly so when those measures
consist merely of a refusal to grant a *privilege* after it's been repeatedly,
systemically abused.

---Rsk

frnkblk at iname

Apr 10, 2007, 5:44 AM

Post #56 of 83 (4196 views)
Permalink

RE: Abuse procedures... Reality Checks [In reply to]

Comcast is known to emit lots of abuse -- are you blocking all their
networks today?

Frank

-----Original Message-----
From: Frank Bulk
Sent: Tuesday, April 10, 2007 7:43 AM
To: nanog [at] merit
Subject: Re: Abuse procedures... Reality Checks

On Sat, Apr 07, 2007 at 09:50:34PM +0000, Fergie wrote:
> I would have to respectfully disagree with you. When network
> operators do due diligence and SWIP their sub-allocations, they
> (the sub-allocations) should be authoritative in regards to things
> like RBLs.

After thinking it over: I partly-to-mostly agree. In principal, yes.
In practice, however, [some] negligent network operators have built
such long and pervasive track records of large-scale abuse that their
allocations can be classified into two categories:

1. Those that have emitted lots of abuse.
2. Those that are going to emit lots of abuse.

In such cases, I'm not inclined to wait for (2) to become reality.

---Rsk

michael.dillon at bt

Apr 10, 2007, 7:11 AM

Post #57 of 83 (4214 views)
Permalink

RE: Abuse procedures... Reality Checks [In reply to]

> Because I haven't got unlimited WHOIS queries. (Although I
> and everyone
> else *should* have those. There are no valid reasons to
> rate-limit any
> form of WHOIS query.)

Yes there are. The current whois returns way more information on a query
than you need for network operations. That's because the current whois
was designed back in the 1970's so that ARPANET network managers could
identify all the users of the network in order to help them make the
business case for their budget requests to cover the cost of high-speed
56k frame relay links.

There is no good reason to rate-limit a query that takes an IP address
(or IP address range or CIDR block) and returns with a list of database
record identifiers for the enclosing blocks. The record identifiers for
organizations who directly received an allocation or assignment from
ARIN would be their org-id. The other ones, SWIP records, would have
some fixed database key like REASG20060000000022812536. If no
REASsiGnment record exists, you now have the orgid to contact and have
no need to do an additional query if they are a known organization. If
the REASiGnment records do exist, you can look them up in your own
database to see if they are a re-offender. And if you really need to,
then you can do a RATE-LIMITED lookup of contact info.

One type of query is justifiably rate limited to prevent DB scraping by
spammers et al. The other type is not, however it does not currently
exist because the RIR whois directory was not created for network
operations support nor is it designed to do this job. You can hack
together all kinds of mashups that sort of work if you squint the right
way, but the bottom-line is that whois does not do the job that many
network operators think it does or would like it to do.

> Because This Is Not My Problem. If by chance someone benign
> has chosen
> to locate their operation in known-hostile, known-negligently-operated
> network space, then their failure to perform due diligence may have
> consequences for them.

It would be interesting if you, and other like-minded hard-nosed network
admins would get together and write a requirements document for a whois
type directory lookup that would actually support you in what you are
trying to do while minimizing collateral damage. The only caveat is that
it must be legal to implement in the USA, i.e. you will never get GPS
coordinates and a photo of the registrant in such a system.

In my opinion, the purpose and scope of such a directory is to provide
contact info for people who are ready, willing and able to communicate
regarding network operations and interconnect issues and who are able to
act on that communication. All contact info should be verified with the
contactee who must EXPLICITLY agree to have the info published. All
contact info will be verified periodically (maybe every 4 months?) by
out-of band means, i.e. the directory operator will keep track of
individual email addresses and phone numbers for role account managers.

If such a directory did exist, then it would be smaller than whois. You
would get many more failures on a quick query which is a good thing. It
means that the network operator did not make it a contractual
requirement for their customer to maintain an up-to-date network
contact. In that case, the network operator is not just morally
responsible for abuse, they are contractually responsible.

Or maybe you could come up with something better?

> 1. Gratuitously labeling carefully-considered measures as
> random is not a
> route to productive conversation.

Agreed. I think a lot of the problem stems from assumptions. People make
a lot of assumptions on what whois does based on the net folklore that
was handed down to them when they "joined" the Internet. Few people seem
to question such folklore and few people notice that not everybody
shares the same understanding. However, it is a lot easier for people to
notice that your carefully-considered measures look like a lot like a
crude weapon that causes lots of collateral damage. They feel that you
could do better and attack you rather than attacking their own
assumptions which are the real root of the problem. If you had better
data to work with, then your carefully-considered measures would evolve
to appear highly sophisticated wisdom, and would also cause little
collateral damage.

--Michael Dillon

jsdy at center

Apr 10, 2007, 7:34 AM

Post #58 of 83 (4201 views)
Permalink

Re: Abuse procedures... Reality Checks [In reply to]

On Tue, Apr 10, 2007 at 03:11:31PM +0100, michael.dillon [at] bt wrote:
...
> Yes there are. The current whois returns way more information on a query
> than you need for network operations. That's because the current whois
> was designed back in the 1970's so that ARPANET network managers could
> identify all the users of the network in order to help them make the
> business case for their budget requests to cover the cost of high-speed
> 56k frame relay links.

Mike, that's twice in two days that you've made that assertion. I don't
remember any financial administrator in those days that would have
accepted WHOIS output as justification for anything. I do remember,
however, that those "high-speed" 9600 baud and 56Kb links were point-to-
point and went down a lot. And so what I remember the WHOIS entries
being used for was:

...
> In my opinion, the purpose and scope of such a directory is to provide
> contact info for people who are ready, willing and able to communicate
> regarding network operations and interconnect issues and who are able to
> act on that communication. All contact info should be verified with the
> contactee who must EXPLICITLY agree to have the info published. All
> contact info will be verified periodically (maybe every 4 months?) by
> out-of band means, i.e. the directory operator will keep track of
> individual email addresses and phone numbers for role account managers.
...

so that we could contact the person at the other end who was responsible
for and knowledgable of their side of the network connection, to fix it.
At o-dark-thirty, if necessary.

Unfortunately, the way WHOIS is maintained these days, this can no
longer be trusted.

Note: at the time, I was a bit younger and did not often encounter
financial managers, so it's possible some might have accepted WHOIS
output. But most people thought computers were some weird thing out
THERE [point in random direction], and would sooner have accepted a
hand-written note than one printed on a TTY33 or chain printer.

--
Joe Yao
Analex Contractor

jsdy at center

Apr 10, 2007, 8:07 AM

Post #59 of 83 (4199 views)
Permalink

Re: Abuse procedures... Reality Checks [In reply to]

On Tue, Apr 10, 2007 at 10:30:32AM +0100, michael.dillon [at] bt wrote:
...
> I also find it curious that you claim to have people on staff at your
> company who know what SWIP means. Perhaps you could ask them to share
> that information with us since I have never seen this documented
> anywhere. Do they really know what you claim they know?
...

http://www.swip.com/: Scottish Widows Investment Partnership
http://www.uh.edu/~cfreelan/SWIP/: Society for Women in Philosophy
http://www.sat-tel.com/Swip.html: Shared WHOIS Project
http://www.swip.net/: The Swedish IP Network

Note that there are far more entries for chapters of SWIP #2 than for
any others. But one may assume that you refer to SWIP #3.

Definitions on the Web found by Google do vary slightly. The referenced
InterNIC policy appears to no longer be available on the InterNIC Web
site. However,
<http://www.arin.net/registration/guidelines/report_reassign.html>
will do.

There seem to have been more proposals on how to produce a better WHOIS
then one can assume in a reasonable amount of time. ;-]

--
Joe Yao
Analex Contractor

list at satchell

Apr 10, 2007, 5:34 PM

Post #60 of 83 (4222 views)
Permalink

Re: Abuse procedures... Reality Checks [In reply to]

michael.dillon [at] bt wrote:

> I also find it curious that you claim to have people on staff at your
> company who know what SWIP means. Perhaps you could ask them to share
> that information with us since I have never seen this documented
> anywhere. Do they really know what you claim they know?
>
> --Michael Dillon
>

Google is your friend.

http://www.arin.net/registration/guidelines/report_reassign.html

Shared WHOIS Project (SWIP)

"SWIP is a process used by organizations to submit information about
downstream customer's address space reassignments to ARIN for inclusion
in the WHOIS database. Its goal is to ensure the effective and efficient
maintenance of records for IP address space.

"SWIP is intended to:

* Provide information to identify the organizations utilizing each
subdelegated IP address block.
* Provide registration information for each IP address block.
* Track utilization of allocated IP address blocks to determine if
additional allocations may be justified.

"For IPv4, organizations can use the Reassign-Simple, Reassign-Detailed,
Reallocate, and Network-Modification templates to report SWIP information.

"Organizations reporting IPv6 reassignment information can use the IPv6
Reassign, IPv6 Reallocate, and IPv6 Modify templates.

"Organizations may only submit reassignment data for records within
their allocated blocks. ARIN reserves the right to make changes to these
records upon the organization's approval. Up to 10 templates may be
submitted as part of a single e-mail."

SWIPs are required for reallocations of /29 and larger if the allocation
owner does not operate a RWhoIs server.

Of course, SWIP is a ARIN thing, and you work for BRITISH
TELECOMMUNICATIONS PLC. As a US network operator, I was well aware of
the requirements for SWIP, because ARIN rules make it clear that, as a
netblock owner of an ARIN allocation, I'm required to do it.

Which numbering authority do you work with day to day?

sil at infiltrated

Apr 11, 2007, 4:07 AM

Post #61 of 83 (4195 views)
Permalink

Re: Abuse procedures... Reality Checks [In reply to]

Stephen Satchell wrote:
>
> SWIPs are required for reallocations of /29 and larger if the
> allocation owner does not operate a RWhoIs server.
>
> Of course, SWIP is a ARIN thing, and you work for BRITISH
> TELECOMMUNICATIONS PLC. As a US network operator, I was well aware of
> the requirements for SWIP, because ARIN rules make it clear that, as a
> netblock owner of an ARIN allocation, I'm required to do it.
>

Being I work at a US network operator and others who've been
attacking my hosts come from US network operators, who can
I complain to when some of the bigger fish not complying with
these so called rules? Many network operators are required to
do a lot of things, one of these things should be the
mitigation of malicious traffic from LEAVING their network.

If some of these companies can't follow the rules, then I see
no need for me to discontinue "punishing" allocations on their
CIDRs whenever my network is attacked since it seems to be the
only method I found to 1) protect my networks and clients and
2) to get someone's attention.

> Which numbering authority do you work with day to day?
>
Me? I work for an authority that many bigger provider should be
following its guidelines and setting examples for smaller
network operators. I shouldn't have to do the work for some of
these bigger operators. I shouldn't have to send emails making
them aware that 40 hosts on their /24 are sending out malicious
traffic.

Maybe ARIN staff should start re-writing policies and
implementing out punishments. Guarantee you if operators were
penalized for not following rules, for allowing filth to leave
their networks, I bet you many maladies on the net would be
cut substantially.

Not going to be a popular stance to most of the bigger fish, but
lets get real here, looking at normal everyday life, if a
country were shipping rotten products, don't you think those
in government would call for measures to halt these products
else no business would occur with said country. Why not
re-write policies to do the same with networks.

I will always point to dampening/flapping on BGP as a baseline...
Company X violates, null route them for a second or two until
they comply. They still don't listen double the penalty and
null route them twice the amount. Once their pockets start
hurting, they'll get a clue. And if their engineers still
don't get it, then management of that company would be fools
to keep their lazy asses around.

--
====================================================
J. Oquendo
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x1383A743
sil . infiltrated @ net http://www.infiltrated.net

The happiness of society is the end of government.
John Adams

Attachments:

smime.p7s (5.04 KB)

michael.dillon at bt

Apr 11, 2007, 5:20 AM

Post #62 of 83 (4218 views)
Permalink

RE: Abuse procedures... Reality Checks [In reply to]

> "SWIP is a process used by organizations to submit information about
> downstream customer's address space reassignments to ARIN for
> inclusion
> in the WHOIS database. Its goal is to ensure the effective
> and efficient
> maintenance of records for IP address space.

Lovely language but it ignores the existence of Rwhois and does not
explain by what standard the effectiveness and efficiency is judged.

> "SWIP is intended to:
> * Provide information to identify the organizations
> utilizing each
> subdelegated IP address block.
> * Provide registration information for each IP address block.
> * Track utilization of allocated IP address blocks to
> determine if
> additional allocations may be justified.

This clearly omits any mention of network abuse. It doesn't even
directly mention that contact information is supplied or what the
contact info may/should be used for. It is heavily slanted towards a
bureaucratic process for counting addresses to support decision-making
about applications for additional address space.

> Of course, SWIP is a ARIN thing, and you work for BRITISH
> TELECOMMUNICATIONS PLC. As a US network operator,

BT is also a US network operator. And a global network operator and a
global network and security consulting firm. And some other stuff too
like the project to run the entire UK telephone network over IP, 21CN.

>I was well
> aware of
> the requirements for SWIP, because ARIN rules make it clear
> that, as a
> netblock owner of an ARIN allocation, I'm required to do it.
>
> Which numbering authority do you work with day to day?

ARIN. I have a long history with ARIN predating the existence of the
organization and I was one of the founding members of the ARIN Advisory
Council. I was not asking a typical dumb question here.

The fact is that nobody really has a clear idea what SWIP is, why it
exists, what it is for. What is the purpose and meaning of SWIP? Why is
it different from RIPE or APNIC? All the answers I have ever seen boil
down to "It's traditional!". And I have spent a lot of effort in trying
to track down older documents to see if there was any more clarity back
in the early days of SWIP and whois, but I failed to find anything other
than some references to budget justifications by ealry ARPANET managers.

On two occasions I tried to address this by proposing some policy
language to ARIN which would define the purpose and scope of the whois
directory but the members were not interested in messing with tradition.

The fact is that SWIP/whois/rwhois suck badly. Different groups of
people have different ideas of what these things mean and the different
ideas do not match. If I ask a waitress for two eggs over-easy I do not
want to receive a slice of Quiche Lorraine. But in the world of
SWIP/whois/rwhois, this is what we deal with every day.

Network operators have a CRYING need for a database to identify contacts
for dealing with network abuse issues. They try to use the whois
directory for this, but too often it fails them because the people
stuffing the info into the directory are merely following tradition to
make sure that the numbers come up right the next time they apply for
additional IP addresses.

By the way, as a holder of an ARIN netblock allocation, you are *NOT*
required to do SWIP. That is just another myth propogated by the holders
of tradition and net folklore. Whenever you ask "Why?" and someone
says, "Because you are required to do it.", they are really telling you
not to think. You pointed me to a page written by ARIN staff as
justification for your views about SWIP but you somehow missed the line
which said:

SWIPs are required for reallocations of /29 and larger if the
allocation owner does not operate a RWhoIs server.

But, I take it a step further. Why should I believe what ARIN staff have
written and why should I do what they tell me to do? What is their
justification for writing this page? If you look in the ARIN policies it
always uses the term SWIP in the context of "efficient utilization". So
why do they publish it in the whois directory? Why do people think that
whois contains valid contact info? Why do people think that whois should
contain contacts who are ready, willing and able to act on network abuse
issues? The only reason people think these things is because it is
traditonal net folklore. It was never part of the purpose and scope of
SWIP/whois/Rwhois.

--Michael Dillon

michael.dillon at bt

Apr 11, 2007, 5:27 AM

Post #63 of 83 (4210 views)
Permalink

RE: Abuse procedures... Reality Checks [In reply to]

> Maybe ARIN staff should start re-writing policies and
> implementing out punishments. Guarantee you if operators were
> penalized for not following rules, for allowing filth to leave
> their networks, I bet you many maladies on the net would be
> cut substantially.

Sorry, that's not their job. That is *YOUR* job!
http://lists.arin.net/mailman/listinfo/ppml
Join the list and propose the new policy.

And ARIN will never mete out punishments or act as a police force in any
way because that is not in ARIN's charter. However, it could operate a
whois directory that meets the needs of network operators fighting
abuse, if said network operators would get off their butts, agree on a
policy describing such a whois directory, and propose it to ARIN.

It's like a lot of those people who complain about the Bush
administration. If you asked them whether they voted Democrat in the
last election, they often say no, they didn't vote at all. Well, you not
only get what you vote for, but you also get what you don't vote
against. Network operators who don't participate in ARIN policy
development don't deserve to complain about anything ARIN-related.

--Michael Dillon

Valdis.Kletnieks at vt

Apr 11, 2007, 8:21 AM

Post #64 of 83 (4208 views)
Permalink

Re: Abuse procedures... Reality Checks [In reply to]

On Wed, 11 Apr 2007 07:07:19 EDT, "J. Oquendo" said:
> these so called rules? Many network operators are required to
> do a lot of things, one of these things should be the
> mitigation of malicious traffic from LEAVING their network.

And I want a pony.

We don't even do a (near) universal job of filtering rfc1918 addresses
and spoofed addresses. We aren't filtering obvious bogon packets, how
do you propose we filter less obvious malicious traffic (is that SYN
packet legit, or part of a DDOS, or just a slashdotting of a suddenly
popular site?).

sil at infiltrated

Apr 11, 2007, 8:28 AM

Post #65 of 83 (4205 views)
Permalink

Re: Abuse procedures... Reality Checks [In reply to]

Valdis.Kletnieks [at] vt wrote:
> * PGP Signed by an unverified key: 04/11/07 at 11:21:15
>
> On Wed, 11 Apr 2007 07:07:19 EDT, "J. Oquendo" said:
>
>> these so called rules? Many network operators are required to
>> do a lot of things, one of these things should be the
>> mitigation of malicious traffic from LEAVING their network.
>>
>
> And I want a pony.
>
> We don't even do a (near) universal job of filtering rfc1918 addresses
> and spoofed addresses. We aren't filtering obvious bogon packets, how
> do you propose we filter less obvious malicious traffic (is that SYN
> packet legit, or part of a DDOS, or just a slashdotting of a suddenly
> popular site?).
>
>
> * Valdis Kletnieks <valdis.kletnieks [at] vt>
> * 0xB4D3D7B0 - Unverified
>
When you say we, speak for yourself and your own networks. There ARE some
people who do take the time to properly design their networks. It is the
same "Well since Billy didn't do it neither will I" attitude that makes
me never think twice about blocking CIDR's.

Since 'THEY' (your "WE") didn't properly configure their network, why
should I think twice about letting it into my backyard. I guess its calling
for too much for network operators to actually do their work though and I
guess considering IPv6 is like how many years away now, I can expect that
much of a wait for people to implement what should have been done from the
onset.

I don't care how filtering gets done from someone else. Like I said if I
can watch and control what comes out of my networks using raw tools on
nix machines, you cannot with a straight face/typing method tell me that
someone at one of these big providers can't clue themselves in to getting
malicious traffic controlled.

Should someone want to comment about "oh golly the cost is outrageous"
I say bs... Its utter laziness from my eyes. So here I go politely
pointing it out... If I can do it with a couple of thousand machines on
my VERY OWN, not a "team", not a "department" but me, in a matter of
minutes, situate my network to not send out crap, then why can't these
companies? I'd like to here something logical, not someone's opinion.
Something like "According to ARIN/IEEE specifications of foobarfoo,
operators are not allowed to view traffic entering or leaving their
networks" which hinders this. There is no reason I could think of,
no scenario I could imagine, that would prohibit network operators
from putting the nail in the coffin with stuff LEAVING THEIR NETS.

Note the word LEAVING now. If it doesn't leave, you wouldn't have
complaints from some other operator now would you.

--
====================================================
J. Oquendo
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x1383A743
sil . infiltrated @ net http://www.infiltrated.net

The happiness of society is the end of government.
John Adams

Attachments:

smime.p7s (5.04 KB)

warren at kumari

Apr 11, 2007, 10:32 AM

Post #66 of 83 (4213 views)
Permalink

Re: Abuse procedures... Reality Checks [In reply to]

On Apr 11, 2007, at 11:28 AM, J. Oquendo wrote:

> Valdis.Kletnieks [at] vt wrote:
>> * PGP Signed by an unverified key: 04/11/07 at 11:21:15
>>
>> On Wed, 11 Apr 2007 07:07:19 EDT, "J. Oquendo" said:
>>
>>> these so called rules? Many network operators are required to
>>> do a lot of things, one of these things should be the
>>> mitigation of malicious traffic from LEAVING their network.
>>>
>>
>> And I want a pony.
>>
>> We don't even do a (near) universal job of filtering rfc1918
>> addresses
>> and spoofed addresses. We aren't filtering obvious bogon packets,
>> how
>> do you propose we filter less obvious malicious traffic (is that SYN
>> packet legit, or part of a DDOS, or just a slashdotting of a suddenly
>> popular site?).
>>
>>
>> * Valdis Kletnieks <valdis.kletnieks [at] vt>
>> * 0xB4D3D7B0 - Unverified
>>
> When you say we, speak for yourself and your own networks.
> There ARE some
> people who do take the time to properly design their networks.

And I would suggest that Valdis is one of them....

From my reading of his message I understood that:
A: Some people filter bad stuff.
B: Some people don't.

I don't think that it is unreasonable that he used "we " to include
all network engineers -- "we" as a community does include A and B

> It is the
> same "Well since Billy didn't do it neither will I" attitude that
> makes
> me never think twice about blocking CIDR's.

So, I have always wondered -- how do you customers really react when
they can no longer reach www.example.com, a site hosted a few IPs
away from www.badevilphisher.net? And do you really think that you
blocking them is going to make example.com contact their provider to
get things fixed?

>
> Since 'THEY' (your "WE") didn't properly configure their network, why
> should I think twice about letting it into my backyard. I guess its
> calling
> for too much for network operators to actually do their work though

Have you considered that being a little politer and not insulting
everyone on the list might be a more constructive way of getting your
point across -- if I were to call you a "big, fat, doodoo head" you
would probably be less receptive than if I didn't...

> and I
> guess considering IPv6 is like how many years away now, I can
> expect that
> much of a wait for people to implement what should have been done
> from the
> onset.
>
> I don't care how filtering gets done from someone else. Like I said
> if I
> can watch and control what comes out of my networks using raw tools on
> nix machines, you cannot with a straight face/typing method tell me
> that
> someone at one of these big providers can't clue themselves in to
> getting
> malicious traffic controlled.
>
> Should someone want to comment about "oh golly the cost is outrageous"
> I say bs... Its utter laziness from my eyes. So here I go politely
> pointing it out... If I can do it with a couple of thousand
> machines on
> my VERY OWN, not a "team", not a "department" but me, in a matter of
> minutes, situate my network to not send out crap, then why can't these
> companies?

Yes, it is great that you are doing your bit to help keep the net
clean. Congratulations and thank you. Perhaps you could write a nice,
simple, friendly guide explaining how you ensure that your network is
never the source of malicious traffic? And how this can be scaled up
to work in a large, backbone network where? Perhaps you could
politely contact those who are not doing their bit and, in a helpful
manner explain how they could improve -- educating and encouraging
change in those who are not doing their bit is much more likely to
make things better than screaming "You suck, I'm not going to accept
your packets, nah nah nah."

> I'd like to here something logical, not someone's opinion.
> Something like "According to ARIN/IEEE specifications of foobarfoo,
> operators are not allowed to view traffic entering or leaving their
> networks" which hinders this. There is no reason I could think of,
> no scenario I could imagine, that would prohibit network operators
> from putting the nail in the coffin with stuff LEAVING THEIR NETS.
>
> Note the word LEAVING now. If it doesn't leave, you wouldn't have
> complaints from some other operator now would you.
>
>
>
> --
> ====================================================
> J. Oquendo
> http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x1383A743
> sil . infiltrated @ net http://www.infiltrated.net
> The happiness of society is the end of government.
> John Adams
>

I suspect that I should have just stayed out of this thread....
W
--
"Go on, prove me wrong. Destroy the fabric of the universe. See if I
care." -- Terry Prachett

sil at infiltrated

Apr 11, 2007, 10:49 AM

Post #67 of 83 (4227 views)
Permalink

Re: Abuse procedures... Reality Checks [In reply to]

Warren Kumari wrote:
>
> So, I have always wondered -- how do you customers really react when
> they can no longer reach www.example.com, a site hosted a few IPs away
> from www.badevilphisher.net? And do you really think that you blocking
> them is going to make example.com contact their provider to get things
> fixed?
>
You confused two things.

1) I do my best to stop malicious traffic from leaving my network. With
this said, if someone cannot get out somewhere, they're obviously going
to get in touch with me as to why. Once this is done, it is explained
to them that either their machine, or a machine on their network was
doing something fuzzy therefore they were blocked. Most are actually
thankful that it was pointed out to them as opposed to having to wait
for Security Company X to update its virus/spamware definitions.

2) I do not block getting TO company X at first signs of garbage coming
into my network from them. I've always contacted someone to some degree
so don't misconstrue my actions as "I block the first packets I see."
On the contrary I only block CIDR's after about 3 attempts at getting
someone to assess their network. After that, I begin with services.
This is my network so this is how it pans out... Spam? A CIDR to my
email ports are blocked. SSH brute forcing, etc., those ports are
blocked. Network who's blocked on ports continues, everything is then
blocked.

>
> Have you considered that being a little politer and not insulting
> everyone on the list might be a more constructive way of getting your
> point across -- if I were to call you a "big, fat, doodoo head" you
> would probably be less receptive than if I didn't...
>
What does being polite and "matter of factly" have to do with
administrators cleaning up their networks? Should I beg an
administrator of some network to be polite and not refer me to their
generic abuse desk who'll do nothing about the issue?

I actually am a little too polite in the fact that 1) I'm doing
network operators a favor pointing them out to rogue hosts on
THEIR networks not mines. If they want to continue hosting said
rogue idiots, their problem. I won't be allowing it into my range.
If you knew me personally, or have dealt with me, I can guarantee
you within minutes of you contacting me for something I would be
on it. I as an admin/engineer whatever you want to call me would
want to make sure that nothing internal to me is affecting anyone
else since it is likely to make things more difficult for me if
left unchecked.

So on issues of politeness, I am being polite contacting people.
I'm being double polite posting evil doing networks on my personal
site so others can be aware that "These networks are infected.
Here are there hosts if you want to block them." I do this on my
own spare time, my own expense, and my own filtering of the
denials of service that ensue when some botnet reject sees me
post a percentage of his botnet. So please don't my messages as
anything other than "Hey... When is someone going to deal with
this?" frustration targeted at those with the power to do actually
something about it instead of waiting for someone else to take
the first move.

Analogy: You live in a house and sweep your property. Your
neighbors don't. Would you stop sweeping your house? Would you
keep your house dirty simply because the majority around you
do? I'm sure if you convinced the most visible neighbor to
make a change, the others would follow suit. Heck in some
areas those neighbors who didn't comply would face fines
after some point. Why not bring this chain of thought to a
network you maintain/manage.

As for documentation on this... There is PLENTY of it. Why should
I write another document no one would follow. If some can't follow
normal standards set by governmental bodies (for lack of better
terms), what makes you think someone would say "Gee... That
Oquendo sure wrote a nice document... Let me follow it" How
about following standards and using good old fashioned common
sense.

--
====================================================
J. Oquendo
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x1383A743
sil . infiltrated @ net http://www.infiltrated.net

The happiness of society is the end of government.
John Adams

Attachments:

smime.p7s (5.04 KB)

surfer at mauigateway

Apr 11, 2007, 11:53 AM

Post #68 of 83 (4201 views)
Permalink

Re: Abuse procedures... Reality Checks [In reply to]

: if someone cannot get out somewhere, they're obviously
: going to get in touch with me as to why. Once this is
: done, it is explained

: I've always contacted someone

: after about 3 attempts at getting someone to assess
: their network

I know from experience this doesn't scale into the hundreds of thousands of customers and can only imagine the big ass eyeball network's scalability issues...

scott

--- sil [at] infiltrated wrote:

From: "J. Oquendo" <sil [at] infiltrated>
To: nanog [at] merit
Cc: Warren Kumari <warren [at] kumari>
Subject: Re: Abuse procedures... Reality Checks
Date: Wed, 11 Apr 2007 13:49:40 -0400

Warren Kumari wrote:
>
> So, I have always wondered -- how do you customers really react when
> they can no longer reach www.example.com, a site hosted a few IPs away
> from www.badevilphisher.net? And do you really think that you blocking
> them is going to make example.com contact their provider to get things
> fixed?
>
You confused two things.

1) I do my best to stop malicious traffic from leaving my network. With
this said, if someone cannot get out somewhere, they're obviously going
to get in touch with me as to why. Once this is done, it is explained
to them that either their machine, or a machine on their network was
doing something fuzzy therefore they were blocked. Most are actually
thankful that it was pointed out to them as opposed to having to wait
for Security Company X to update its virus/spamware definitions.

2) I do not block getting TO company X at first signs of garbage coming
into my network from them. I've always contacted someone to some degree
so don't misconstrue my actions as "I block the first packets I see."
On the contrary I only block CIDR's after about 3 attempts at getting
someone to assess their network. After that, I begin with services.
This is my network so this is how it pans out... Spam? A CIDR to my
email ports are blocked. SSH brute forcing, etc., those ports are
blocked. Network who's blocked on ports continues, everything is then
blocked.

>
> Have you considered that being a little politer and not insulting
> everyone on the list might be a more constructive way of getting your
> point across -- if I were to call you a "big, fat, doodoo head" you
> would probably be less receptive than if I didn't...
>
What does being polite and "matter of factly" have to do with
administrators cleaning up their networks? Should I beg an
administrator of some network to be polite and not refer me to their
generic abuse desk who'll do nothing about the issue?

I actually am a little too polite in the fact that 1) I'm doing
network operators a favor pointing them out to rogue hosts on
THEIR networks not mines. If they want to continue hosting said
rogue idiots, their problem. I won't be allowing it into my range.
If you knew me personally, or have dealt with me, I can guarantee
you within minutes of you contacting me for something I would be
on it. I as an admin/engineer whatever you want to call me would
want to make sure that nothing internal to me is affecting anyone
else since it is likely to make things more difficult for me if
left unchecked.

So on issues of politeness, I am being polite contacting people.
I'm being double polite posting evil doing networks on my personal
site so others can be aware that "These networks are infected.
Here are there hosts if you want to block them." I do this on my
own spare time, my own expense, and my own filtering of the
denials of service that ensue when some botnet reject sees me
post a percentage of his botnet. So please don't my messages as
anything other than "Hey... When is someone going to deal with
this?" frustration targeted at those with the power to do actually
something about it instead of waiting for someone else to take
the first move.

Analogy: You live in a house and sweep your property. Your
neighbors don't. Would you stop sweeping your house? Would you
keep your house dirty simply because the majority around you
do? I'm sure if you convinced the most visible neighbor to
make a change, the others would follow suit. Heck in some
areas those neighbors who didn't comply would face fines
after some point. Why not bring this chain of thought to a
network you maintain/manage.

As for documentation on this... There is PLENTY of it. Why should
I write another document no one would follow. If some can't follow
normal standards set by governmental bodies (for lack of better
terms), what makes you think someone would say "Gee... That
Oquendo sure wrote a nice document... Let me follow it" How
about following standards and using good old fashioned common
sense.

--
====================================================
J. Oquendo
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x1383A743
sil . infiltrated @ net http://www.infiltrated.net

The happiness of society is the end of government.
John Adams

warren at kumari

Apr 11, 2007, 12:44 PM

Post #69 of 83 (4201 views)
Permalink

Re: Abuse procedures... Reality Checks [In reply to]

On Apr 11, 2007, at 2:53 PM, Scott Weeks wrote:

>
>
>
>
> : if someone cannot get out somewhere, they're obviously
> : going to get in touch with me as to why. Once this is
> : done, it is explained
>
> : I've always contacted someone
>
> : after about 3 attempts at getting someone to assess
> : their network
>
>
> I know from experience this doesn't scale into the hundreds of
> thousands of customers and can only imagine the big ass eyeball
> network's scalability issues...
>
> scott
>

Hear hear...

Scaling process and procedures is often as hard or harder than
scaling technical things...

Unfortunately, the lesson that scaling either is hard is only really
something that one can learn through experience -- I know that I for
one used to believe (as I would bet did most of us) that you could
scale just by buying a bigger X, where X could be a router, circuit,
etc. If that didn't work you could always just buy another X (or a
bunch more Xs) -- this strategy works up to a point, after which it
all goes pear-shaped. Until you have experienced this firsthand it
is hard to truly understand.

The same thing happens with things like abuse -- it is easy to deal
with abuse on a small scale. It is somewhat harder on a medium scale
and harder still on a large scale -- the progression from small to
medium to large is close to linear. At some point though the
difficulty suddenly hockey-sticks and becomes distinctly non-trivial
-- this doesn't mean that it is impossible, nor that you should give
up, but rather that a different approach is needed. Understanding
this is harder than understanding why you cannot grow your network
just by buying more X.

W

>
>
> --- sil [at] infiltrated wrote:
>
> From: "J. Oquendo" <sil [at] infiltrated>
> To: nanog [at] merit
> Cc: Warren Kumari <warren [at] kumari>
> Subject: Re: Abuse procedures... Reality Checks
> Date: Wed, 11 Apr 2007 13:49:40 -0400
>
> Warren Kumari wrote:
>>
>> So, I have always wondered -- how do you customers really react when
>> they can no longer reach www.example.com, a site hosted a few IPs
>> away
>> from www.badevilphisher.net? And do you really think that you
>> blocking
>> them is going to make example.com contact their provider to get
>> things
>> fixed?
>>
> You confused two things.
>
> 1) I do my best to stop malicious traffic from leaving my network.
> With
> this said, if someone cannot get out somewhere, they're obviously
> going
> to get in touch with me as to why. Once this is done, it is explained
> to them that either their machine, or a machine on their network was
> doing something fuzzy therefore they were blocked. Most are actually
> thankful that it was pointed out to them as opposed to having to wait
> for Security Company X to update its virus/spamware definitions.
>
> 2) I do not block getting TO company X at first signs of garbage
> coming
> into my network from them. I've always contacted someone to some
> degree
> so don't misconstrue my actions as "I block the first packets I see."
> On the contrary I only block CIDR's after about 3 attempts at getting
> someone to assess their network. After that, I begin with services.
> This is my network so this is how it pans out... Spam? A CIDR to my
> email ports are blocked. SSH brute forcing, etc., those ports are
> blocked. Network who's blocked on ports continues, everything is then
> blocked.
>
>>
>> Have you considered that being a little politer and not insulting
>> everyone on the list might be a more constructive way of getting your
>> point across -- if I were to call you a "big, fat, doodoo head" you
>> would probably be less receptive than if I didn't...
>>
> What does being polite and "matter of factly" have to do with
> administrators cleaning up their networks? Should I beg an
> administrator of some network to be polite and not refer me to their
> generic abuse desk who'll do nothing about the issue?
>
> I actually am a little too polite in the fact that 1) I'm doing
> network operators a favor pointing them out to rogue hosts on
> THEIR networks not mines. If they want to continue hosting said
> rogue idiots, their problem. I won't be allowing it into my range.
> If you knew me personally, or have dealt with me, I can guarantee
> you within minutes of you contacting me for something I would be
> on it. I as an admin/engineer whatever you want to call me would
> want to make sure that nothing internal to me is affecting anyone
> else since it is likely to make things more difficult for me if
> left unchecked.
>
> So on issues of politeness, I am being polite contacting people.
> I'm being double polite posting evil doing networks on my personal
> site so others can be aware that "These networks are infected.
> Here are there hosts if you want to block them." I do this on my
> own spare time, my own expense, and my own filtering of the
> denials of service that ensue when some botnet reject sees me
> post a percentage of his botnet. So please don't my messages as
> anything other than "Hey... When is someone going to deal with
> this?" frustration targeted at those with the power to do actually
> something about it instead of waiting for someone else to take
> the first move.
>
> Analogy: You live in a house and sweep your property. Your
> neighbors don't. Would you stop sweeping your house? Would you
> keep your house dirty simply because the majority around you
> do? I'm sure if you convinced the most visible neighbor to
> make a change, the others would follow suit. Heck in some
> areas those neighbors who didn't comply would face fines
> after some point. Why not bring this chain of thought to a
> network you maintain/manage.
>
> As for documentation on this... There is PLENTY of it. Why should
> I write another document no one would follow. If some can't follow
> normal standards set by governmental bodies (for lack of better
> terms), what makes you think someone would say "Gee... That
> Oquendo sure wrote a nice document... Let me follow it" How
> about following standards and using good old fashioned common
> sense.
>
> --
> ====================================================
> J. Oquendo
> http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x1383A743
> sil . infiltrated @ net http://www.infiltrated.net
>
> The happiness of society is the end of government.
> John Adams
>
>
>

--
After you'd known Christine for any length of time, you found
yourself fighting a desire to look into her ear to see if you could
spot daylight coming the other way.

-- (Terry Pratchett, Maskerade)t

dotis at mail-abuse

Apr 11, 2007, 2:02 PM

Post #70 of 83 (4204 views)
Permalink

Re: Abuse procedures... Reality Checks [In reply to]

On Apr 11, 2007, at 10:32 AM, Warren Kumari wrote:

> Perhaps you could write a nice, simple, friendly guide explaining
> how you ensure that your network is never the source of malicious
> traffic?

Identify your ownership, and ensure contact information is accurate
and well attended. Inconsiderate anonymous behavior is a typical
failing, where there is no excuse for remaining ignorant of abusive
activity.

-Doug

rsk at gsp

Apr 11, 2007, 2:48 PM

Post #71 of 83 (4219 views)
Permalink

Re: Abuse procedures... Reality Checks [In reply to]

On Tue, Apr 10, 2007 at 07:44:59AM -0500, Frank Bulk wrote:
> Comcast is known to emit lots of abuse -- are you blocking all their
> networks today?

All? No. But I shouldn't find it necessary to block ANY, and wouldn't,
if Comcast wasn't so appallingly negligent.

( I'm blocking huge swaths of Comcast space from port 25. This shouldn't
really surprise anyone; Comcast runs what may well be the most prolific
spam-spewing network in the world. I saw attempts from 80,000+ distinct
IP addresses during January 2007 alone -- to a *test* mail server.
I should have seen zero. The mitigation techniques for making that
happen are well-known, have been well-known for years, and can be
implemented easily by any competent organization.)

This, by the way, should not be taken as indicative of either what
I've done in the past or may do in the future. Nor should it be
taken as indicative of what decisions I've made in re other networks.

---Rsk

rsk at gsp

Apr 11, 2007, 2:55 PM

Post #72 of 83 (4198 views)
Permalink

Re: Abuse procedures... Reality Checks [In reply to]

On Wed, Apr 11, 2007 at 03:44:01PM -0400, Warren Kumari wrote:
> The same thing happens with things like abuse -- it is easy to deal
> with abuse on a small scale. It is somewhat harder on a medium scale
> and harder still on a large scale -- the progression from small to
> medium to large is close to linear.

First, I don't buy this. I think dealing with abuse is *much*
easier for large operations than small.

But suppose you're right. Let me concede that point for the purpose
of making my second point (and generic "you" throughout, BTW):

Second, I don't really care how hard it is. It's YOUR network, YOU
built it, YOU plugged it into our Internet: therefore, however hard
it is, it's YOUR problem. Fix it.

Or if you choose not to: at least stop whining about how much you
don't like the way in which other people try to partially compensate
for YOUR failure.

---Rsk

michael.dillon at bt

Apr 11, 2007, 4:11 PM

Post #73 of 83 (4193 views)
Permalink

RE: Abuse procedures... Reality Checks [In reply to]

> As for documentation on this... There is PLENTY of it. Why should
> I write another document no one would follow.

Because you might be a better writer than those other folks. You might
be able to present the right balance of technical detail and policy
goals to be understood by a larger number of people.

People often ask me to advise them which book they should buy to learn
language X fast. X being French or Russian or German etc. I always give
the same advice. Go to a good bookstore that stocks a large choice of
books in your chosen language. In some cities that means the local
university bookshop, in others there may even be a specialist bookshop
that sells just language books. The important thing is that you go and
look at several different books, compare them to one another and FIND
THE ONE WHOSE AUTHOR SPEAKS TO YOU. Find the writer whose writing
matches your way of thinking. Other than that, buy one dictionary that
you can carry with you all day long, one beginners book, and one graded
reader to start. Every 6 months, go back to this (or another) shop and
look over the selection again because you may have advanced to the point
where additional books/CDs will help. And always avoid beginners books
which do not use the native alphabet of the language you are learning, a
particular problem with Japanese.

In the masses of content that is indexed by Google, we need MORE
variety, not less. Please do try to write something if you can.

--Michael Dillon

michael.dillon at bt

Apr 11, 2007, 4:18 PM

Post #74 of 83 (4192 views)
Permalink

RE: Abuse procedures... Reality Checks [In reply to]

> > I know from experience this doesn't scale into the hundreds of
> > thousands of customers and can only imagine the big ass eyeball
> > network's scalability issues...

> Hear hear...
>
> Scaling process and procedures is often as hard or harder than
> scaling technical things...

It's true. But the big networks hire people who understand scaling
issues and know how to make things work. It's not up to us to solve
their scaling problem. If you can define a mechanism that will work on
smaller networks to achieve a goal, and if that goal is worthwhile
achieving, the the big networks will get their scalability networks to
scale it up. There is a similar problem in chemicals where researchers
create new compounds in the laboratory and then hand the details over to
scaling experts who know how to change the process to work on the scale
of a factory. And it's not unusual to see chemical factories that are
acres in size.

> The same thing happens with things like abuse -- it is easy to deal
> with abuse on a small scale. It is somewhat harder on a medium scale
> and harder still on a large scale -- the progression from small to
> medium to large is close to linear. At some point though the
> difficulty suddenly hockey-sticks and becomes distinctly non-trivial
> -- this doesn't mean that it is impossible, nor that you should give
> up, but rather that a different approach is needed. Understanding
> this is harder than understanding why you cannot grow your network
> just by buying more X.

Yes this is true. But the people who find different approaches need to
see how the smaller networks solve a problem. Their skill is not in
finding solutions to abuse, but in figuring out how to restructure an
abuse solution to work on a huge scale.

--Michael Dillon

surfer at mauigateway

Apr 11, 2007, 4:46 PM

Post #75 of 83 (4225 views)
Permalink

Re: Abuse procedures... Reality Checks [In reply to]

--- rsk [at] gsp wrote:
On Wed, Apr 11, 2007 at 03:44:01PM -0400, Warren Kumari wrote:

> The same thing happens with things like abuse -- it is easy to deal
> with abuse on a small scale. It is somewhat harder on a medium scale
> and harder still on a large scale -- the progression from small to
> medium to large is close to linear.

: First, I don't buy this. I think dealing with abuse is *much*
: easier for large operations than small.

The original email I sent was about *how* you deal with it. J. Oquendo vociferously defended his position when he finally got around to saying, "...if someone cannot get out somewhere, they're obviously going to get in touch with me as to why. Once this is done, it is explained [...] I've always contacted someone [...] after about 3 attempts at getting someone to assess their network..."

I said this doesn't scale even to hundreds of thousands of customers much less higher numbers. There are definitely scaling issues with this method of dealing with abuse. You can't just hire more phone monkeys linearly to the number of customers you have.

<snip>

: Second, I don't really care how hard it is. It's YOUR network, YOU
: built it, YOU plugged it into our Internet: therefore, however hard
: it is, it's YOUR problem. Fix it.

Not always. I have inherited various networks over the years that were already built by folks that didn't care. You do the best you can to get it to as good a network as possible, but you never completely reach the goal of "good".

Additionally, upper management gives or takes away manpower many times without the understanding of what 'should' be done to be a good netizen and this defines how much effort can be spent on fixing the problems. The only thing a person can really do is quit and move on. That's not always an option. There're very few interesting-to-operate networks here in Hawaii. So, you focus on the top priorities: keeping the current customers and getting more by operating the network in as efficient a manner as possible. Myself, I work outside business hours to try to be a good guy, fix stuff and serve the Hawaiian community in an altruistic manner, but there's only so much stuff one person can do.

<snip>

scott

1 2 3 4

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads