Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Wikipedia: Mediawiki

Malicious DDOS attacks to increase CPU usage on shared server

 

 

Wikipedia mediawiki RSS feed   Index | Next | Previous | View Threaded


danfisher261 at gmail

Aug 12, 2012, 1:34 PM

Post #1 of 5 (365 views)
Permalink
Malicious DDOS attacks to increase CPU usage on shared server

Like for many of us, my wiki is hosted on a shared server so I have to be
careful about CPU usage. There's a hacker/attacker who has been recently
flooding my wiki with malicious requests. His intentions may be to just
increase CPU usage, slow down the site, get it kicked off the server - or
all of those. The last attack started right on the hour and the requests
are as frequent as 4 per second. It can go on for hours. I've gotten CPU
usage alerts from the server company. If I could afford it I would get a
dedicated server, but being flooded maliciously would be a problem for any
server. Here's a sample from the Access logs:
---------------------------
x.x.x.x - - [8/Aug/2012:05:02:46 -0500] "GET /wiki/NON_EXISTING_PAGE
HTTP/1.1" 301 - "thewikisite.net" ""
x.x.x.x - - [8/Aug/2012:05:02:46 -0500] "GET /wiki/NON_EXISTING_PAGE
HTTP/1.1" 301 - "thewikisite.net" ""
x.x.x.x - - [8/Aug/2012:05:02:46 -0500] "GET /wiki/NON_EXISTING_PAGE
HTTP/1.1" 301 - "thewikisite.net" ""
x.x.x.x - - [8/Aug/2012:05:02:46 -0500] "GET /wiki/NON_EXISTING_PAGE
HTTP/1.1" 301 - "thewikisite.net" ""
x.x.x.x - - [8/Aug/2012:05:02:47 -0500] "GET /wiki/NON_EXISTING_PAGE
HTTP/1.1" 301 - "thewikisite.net" ""
----------------------------

For evidence that this is a malicious attack, you can see:
- the title of the non-existing page
- the fact that the requests started right on the hour (for example 5:00AM
or close to it)
- there's no browser information. Usually the browser agent is identified
at the end of each line but those quotes are empty for this case.
- There are many requests from the same IP (2 per second on average) and it
goes on for hours.
- IP Whois does not reveal a search engine IP so its not a search engine
spider. A spider would also request existing pages.

Seeing all these points, this is most likely a malicious script. The CPU
consumption goes up to 100% so if this happens frequently, the company
could shut off the site in the worst case or in the best case just slow it
down temporarily, which is what they've done in the past. Currently the
server company can somehow manage these attacks but it means about four
hours of increased CPU usage after which they start delaying the scripts
and somehow block the attacker's IP address. I'm doing this manually right
now in the HTACCESS file, by monitoring CPU usage, checking the IP in the
log and blocking it in htaccess.

Previously I've been successful in blocking malicious edit/move floods also
done by scripts so now I'm thinking I could try doing a "view/request"
flood protection as well. I'm thinking a scheme like:
- If there's more than X amount of requests from a single IP in 10 minutes,
block that IP for 3 hours

This may be simple to do, but here's two worst cases for CPU usage:
- if the hacker gets multiple IPs and starts flooding with requests to
existing pages (thus harder to detect the attack)
- if there's a genuine huge traffic surge, for example if a very popular
website linked to us for a day on their front page.
In this case I could count the total number of requests and if they exceed
a certain number for 15 minutes, I would deny requests for the next 20
seconds. Genuine visitors in these traffic surges may be denied traffic but
that's ok since CPU usage has to be kept low at any cost. Logged in editors
who are established users of the site (a minimum number of edits) would not
be denied access. This would keep CPU consumption low both for the
malicious case, and for the genuine case (a huge traffic surge). That is
the priority because we don't want to make the server company mad.

Does anyone have any suggestions on how to do this:
- detect and block single IP attacks
- manage huge traffic surges or malicious attacks using multiple IPs

DDOS attacks have been going on for a long time so perhaps people have
created solutions for them. Maybe there's a server level program that can
be installed? The server company may not install it but I could try asking
them.
If there's no automatic solution on the server/OS level, another solution
is for me to make an extension in Mediawiki to detect the attacks and edit
Htaccess automatically or deny the IP's some other way. The extension would
check every page view and keep records of IP addresses in tables. Maybe
there would be two tables, one for monitoring the traffic and that table
would be bigger and another smaller table for blocked IP's.
For genuine traffic surges, I have some questions on how to make the site
faster but will ask them in a later email. I've tried to enable caches and
so on.

If anyone has any suggestions for how to deal with these kinds of DDOS
attacks, I would be grateful.

thanks
Dan
_______________________________________________
MediaWiki-l mailing list
MediaWiki-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/mediawiki-l


Jan at Bytesmiths

Aug 13, 2012, 6:20 AM

Post #2 of 5 (309 views)
Permalink
Re: Malicious DDOS attacks to increase CPU usage on shared server [In reply to]

> Like for many of us, my wiki is hosted on a shared server so I have to be
> careful about CPU usage. There's a hacker/attacker who has been recently
> flooding my wiki with malicious requests. His intentions may be...

I feel your pain. Don't bother guessing intentions. Don't take it personally. It's most likely just a bot, following some algorithm. (Well, unless you've done something to piss someone off, or if your wiki is highly controversial.)

> ... somehow block the attacker's IP address. I'm doing this manually right
> now in the HTACCESS file, by monitoring CPU usage, checking the IP in the
> log and blocking it in htaccess.

Ugh. Not the best way to do it, as they've already caused significant CPU usage before you even figure out the IP.

Is this a virtual server? If so, where you need to block it is in the firewall, using ipfw(8). You can make it work so that there is no reply to packets at all, which has the significant advantage that you actually slow down the attacker, since they have to wait for a TCP lost packet timeout.

Anything else you do slows you down while your defensive measures are executing.

This is probably not the best place to get advice on ipfw(8), but take my word for it, that's the place to do it, at the TCP/UDP level. I'd google around for things like "ipfw denial of service attack" and such.

I've had great (but temporary) success blocking spam that way, buy using ipfw(8) to block port 25 access from huge address ranges from sections of the globe where I don't expect email from -- like China. If your wiki is English-oriented and non-global in nature, perhaps you can stop access to big foreign address ranges to ease the problem.

Good luck!

----------------
:::: Entirely new ways of living are necessary, and if we don't adobt them voluntarily, we or our children will eventually adopt them involuntarily, and probably with great pain and difficulty in the process. -- Thom Hartmann
:::: Jan Steinman, EcoReality Co-op ::::





_______________________________________________
MediaWiki-l mailing list
MediaWiki-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/mediawiki-l


danfisher261 at gmail

Aug 13, 2012, 7:39 PM

Post #3 of 5 (317 views)
Permalink
Re: Malicious DDOS attacks to increase CPU usage on shared server [In reply to]

>> I feel your pain. Don't bother guessing intentions. Don't take it
personally. It's most likely just a bot, following some algorithm. (Well,
unless you've done something to piss someone off, or if your wiki is highly
controversial.)
Yea, you got it, its a wiki that some people don't like. So I'm very sure
this is an intentional attack. But you're right, the intentions don't
matter in this case. I would still have to find a solution whether its a
misbehaving bot, a malicious script, a search engine spider, or a big surge
in traffic.
No its a not a virtual server although I've thought about that upgrade for
the future. Right now its a shared server.

>> Ugh. Not the best way to do it, as they've already caused significant
CPU usage before you even figure out the IP.

I could probably detect an attack in 15 minutes or less, as the same IP
makes 2 or 3 requests per second. I would say it would be unusual for a
genuine visitor to make that many requests: 15 x 60 x 2 = 1800 pages in 15
minutes. So yea I could probably detect an attack pretty quickly like this
if its coming from just one IP. After the detection comes enforcement, and
seeing how quick it can stop it. I would ideally do it the way you
suggested and slow the attacker if possible. I could refuse further
processing from within MW (block of code in an extension) and that wouldn't
be as fast the firewall or Htacess. I could use PHP to edit the Htaccess -
then detection would be slow, which is fine and enforcement would be quick.
When I manually blocked that IP in htaccess, CPU usage went back to normal
right away. Or I would ask for advice from server experts on how to block
effectively and they could give advice along the lines you gave. I would
see if I can do that on a shared server. I bet I wont be able to do the ipfw
you mentioned, but I will look for the best solution and work with server
support to see if its possible.

Yea we're english and non-global but the site is accessed/linked from all
over including by people speaking/writing in foreign languages. Last time
the attacker had an IP address from one continent and this time he had an
IP from another continent so I'm not able to know where it will come from
the next time.

Anyway, thanks for your advice! I have to learn a lot, will take the
information you gave and will also be looking at other sites/forums for
advice. I know as time goes and our site's traffic increases even more, the
chances of better quality and more frequent attacks will increase so the
sooner I can make this (if at all), the better.

Dan








On Mon, Aug 13, 2012 at 8:20 AM, Jan Steinman <Jan [at] bytesmiths> wrote:

> > Like for many of us, my wiki is hosted on a shared server so I have to be
> > careful about CPU usage. There's a hacker/attacker who has been recently
> > flooding my wiki with malicious requests. His intentions may be...
>
> I feel your pain. Don't bother guessing intentions. Don't take it
> personally. It's most likely just a bot, following some algorithm. (Well,
> unless you've done something to piss someone off, or if your wiki is highly
> controversial.)
>
> > ... somehow block the attacker's IP address. I'm doing this manually
> right
> > now in the HTACCESS file, by monitoring CPU usage, checking the IP in the
> > log and blocking it in htaccess.
>
> Ugh. Not the best way to do it, as they've already caused significant CPU
> usage before you even figure out the IP.
>
> Is this a virtual server? If so, where you need to block it is in the
> firewall, using ipfw(8). You can make it work so that there is no reply to
> packets at all, which has the significant advantage that you actually slow
> down the attacker, since they have to wait for a TCP lost packet timeout.
>
> Anything else you do slows you down while your defensive measures are
> executing.
>
> This is probably not the best place to get advice on ipfw(8), but take my
> word for it, that's the place to do it, at the TCP/UDP level. I'd google
> around for things like "ipfw denial of service attack" and such.
>
> I've had great (but temporary) success blocking spam that way, buy using
> ipfw(8) to block port 25 access from huge address ranges from sections of
> the globe where I don't expect email from -- like China. If your wiki is
> English-oriented and non-global in nature, perhaps you can stop access to
> big foreign address ranges to ease the problem.
>
> Good luck!
>
> ----------------
> :::: Entirely new ways of living are necessary, and if we don't adobt them
> voluntarily, we or our children will eventually adopt them involuntarily,
> and probably with great pain and difficulty in the process. -- Thom Hartmann
> :::: Jan Steinman, EcoReality Co-op ::::
>
>
>
>
>
> _______________________________________________
> MediaWiki-l mailing list
> MediaWiki-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
>
_______________________________________________
MediaWiki-l mailing list
MediaWiki-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/mediawiki-l


kevin.kueper at meypack

Aug 14, 2012, 5:40 AM

Post #4 of 5 (303 views)
Permalink
Re: Malicious DDOS attacks to increase CPU usage on shared server [In reply to]

Hi,

Fail2ban is a nice tool for reacting against attacks.

It can look into your Apache logfiles and search strange behaviour.

You can define cases digging for in those logfiles.

And fail2ban creates rules in iptables for blocking specific addresses.

 
http://www.fail2ban.org/wiki/index.php/Main_Page

 
_______________________________________________
MediaWiki-l mailing list
MediaWiki-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/mediawiki-l


danfisher261 at gmail

Aug 15, 2012, 6:44 PM

Post #5 of 5 (301 views)
Permalink
Re: Malicious DDOS attacks to increase CPU usage on shared server [In reply to]

Thanks! Looks like a great program/tool. I'll ask the server company.



On Tue, Aug 14, 2012 at 7:40 AM, Kevin Küper <kevin.kueper [at] meypack>wrote:

> Hi,
>
> Fail2ban is a nice tool for reacting against attacks.
>
> It can look into your Apache logfiles and search strange behaviour.
>
> You can define cases digging for in those logfiles.
>
> And fail2ban creates rules in iptables for blocking specific addresses.
>
>
> http://www.fail2ban.org/wiki/index.php/Main_Page
>
>
> _______________________________________________
> MediaWiki-l mailing list
> MediaWiki-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
>
_______________________________________________
MediaWiki-l mailing list
MediaWiki-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

Wikipedia mediawiki RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.