kenshaw at gmail
Jun 24, 2013, 1:54 AM
Post #5 of 5
(84 views)
Permalink
|
I updated the tests on libvmod-dns -- 'make check' should now work as
expected.
-Ken
On Mon, Apr 8, 2013 at 9:40 PM, Kenneth Shaw <kenshaw [at] gmail> wrote:
> Hi All,
>
> This has been successfully deployed in production, and the code (as-is) is
> handling "many thousands" of connections per second from fake and
> legitimate bots advertising themselves as Googlebot/Bingbot/etc with no
> apparent issues/problems. The configuration we've deployed is essentially
> the same as provided here (and in the code base).
>
> Anyway, if anyone else ends up finding libvmod-dns helpful, please
> consider it "emailware" -- ie, drop me an email and let me know
> (off-the-record, of course) how you're making use of it. I'm curious more
> than anything!
>
>
>
> -Ken
>
>
> On Mon, Apr 1, 2013 at 6:21 PM, Kenneth Shaw <kenshaw [at] gmail> wrote:
>
>> Hi,
>>
>> I spent a bit of time today developing a DNS module for Varnish.
>>
>> It is available here:
>>
>> https://github.com/kenshaw/libvmod-dns/
>>
>> The reason for this development is to cut off bots that abuse the
>> User-Agent string (ie, claiming to be Googlebot/bingbot/etc.) by doing a
>> reverse and then forward DNS against the client.ip/X-Forwarded-For header
>> and comparing with a regex against the resultant domain.
>>
>> The logic is meant to work something like this:
>>
>> sub vcl_recv {
>> # do a dns check on "good" crawlers
>> if (req.http.user-agent ~ "(?i)(googlebot|bingbot|slurp|teoma)") {
>> # do a reverse lookup on the client.ip (X-Forwarded-For) and
>> check that its in the allowed domains
>> set req.http.X-Crawler-DNS-Reverse =
>> dns.rresolve(req.http.X-Forwarded-For);
>>
>> # check that the RDNS points to an allowed domain -- 403 error if
>> it doesn't
>> if (req.http.X-Crawler-DNS-Reverse !~
>> "(?i)\.(googlebot\.com|search\.msn\.com|crawl\.yahoo\.net|ask\.com)$") {
>> error 403 "Forbidden";
>> }
>>
>> # do a forward lookup on the DNS
>> set req.http.X-Crawler-DNS-Forward =
>> dns.resolve(req.http.X-Crawler-DNS-Reverse);
>>
>> # if the client.ip/X-Forwarded-For doesn't match, then the
>> user-agent is fake
>> if (req.http.X-Crawler-DNS-Forward != req.http.X-Forwarded-For) {
>> error 403 "Forbidden";
>> }
>> }
>> }
>>
>> While this is not being used in production (yet), I plan to do so later
>> this week against a production system receiving ~10,000+ requests/sec. I
>> will report back afterwards.
>>
>> I realize the code currently has issues (memory, documentation, etc.),
>> which will be fixed in the near future.
>>
>> I also realize there are better ways to head malicious bots off at the
>> pass through DNS, etc (which we are doing as well). The largest issue here
>> for my purposes is that it is difficult / impossible to identify all
>> traffic. Additionally, it is nice to be able to monitor the actual traffic
>> coming through and not completely dropping it at the edge.
>>
>> Any input/comments against what I've written so far would be gladly
>> appreciated! Thanks!
>>
>> -Ken
>>
>
>
|
