Gossamer Forum
Home : Products : Links 2.0 : Discussions :

Help!!! How can I stop spammers???

(Page 1 of 2)
> >
Quote Reply
Help!!! How can I stop spammers???
Who knows how can I stop spammers that submit every day hundred of subpages??? It is very time-intensive to delete all their entries manually. Is there a possibility to stop these spammers and block their domains for all future submission??? What can I do?

------------------
Quote Reply
Re: Help!!! How can I stop spammers??? In reply to
Dear Bobsie,

thank you for your hint, but this does not help. They spam from our site or have configured their spam software to our site. There should be another possibility to block postings from certain domains AND e-mail addresses

------------------
Quote Reply
Re: Help!!! How can I stop spammers??? In reply to
In links.cfg there is a setting called @db_referers. If you set it to your domain, then the robots will not be able to submit unless it is done from your site. That is built into add.cgi.

I hope this helps.
Quote Reply
Re: Help!!! How can I stop spammers??? In reply to
perhaps if we could figure a way to use the latter suggestion to explicately state that one can only post from your specific ad/submit url web page. If this is not already possible. How about some mod code that perhaps reads a password file and sends that password along to one of our links modules with a compare and then kicks out as a rejectioned if no match.... Or something to that effect.
Quote Reply
Re: Help!!! How can I stop spammers??? In reply to
 
If you're being hit from the same folks we are -- 600 to 1000 submissions/day -- I don't think any source-blocking mod will make much of a difference. These are scripted submissions indistinguishable from a manual sub -- a point of which the provider/perpetrator of this software is immensely proud. I've been kicking around the idea of some sort of two-phase submission scheme. (That is, the submitter enters the info, and a keyed response returned to submitter's mailbox. Only by properly answering the "are you sure?" message could a submitter get his or her URL keyed for validation. This would filter out the slew of entries with bogus source addresses -- while filling my queue with UNDELs -- and might reduce the number of inappropriate submissions from honest folk.)

Vann
Quote Reply
Re: Help!!! How can I stop spammers??? In reply to
Hi,

Edit add.cgi and right before:

# Then mail it away!

at around line 161 add:

foreach (sort keys %ENV) {
$msg .= "$_ => $ENV{$_}\n";
}

which will get you all the environement information. Look for something common between all requests like a common IP number in REMOTE_ADDR or common host name in REMOTE_HOST or perhaps their spam tools labels it something in HTTP_USER_AGENT.

If you can find something that only the spammer is using, then edit add.cgi right before:

# Check the referer.

put:

if ($ENV{'SOMETHING'} eq 'SPAMMER') {
&site_html_add_failure ("Please contact System Admin");
return;
}

Let me know how it works out. If you find anything to stop this guy.

Cheers,

Alex


Quote Reply
Re: Help!!! How can I stop spammers??? In reply to
One of the things I did was modify add.cgi, sub send_email, to include all of the following information:

Code:
The following link is awaiting validation:

Title: $in{'Title'}
URL: $in{'URL'}
Category: $in{'Category'}
Description: $in{'Description'}
Contact Name: $in{'Contact Name'}
Contact Email: $in{'Contact Email'}
Receive Email: $in{'ReceiveMail'}

Remote Host: $ENV{'REMOTE_HOST'}
Remote Addr: $ENV{'REMOTE_ADDR'}
Referer: $ENV{'HTTP_REFERER'}
Submitted Using: $ENV{'HTTP_USER_AGENT'};

The Referer and Submitted Using tell me whether or not the link was added from my site and what was used (browser or robot) to add the link. I monitor the HTTP_USER_AGENT info so that I can update the following part of add.cgi, sub process_form. Right after the check referer code, I put:

Quote:
if (lc($ENV{'HTTP_USER_AGENT'}) =~ "submiteasy") {
&site_html_add_failure ("Resources cannot be added to this directory using $ENV{'HTTP_USER_AGENT'}. Please visit the site to add your entry.");
return;
}

When a new, non-browser user agent is used, I just update that code.

I hope this helps.


[This message has been edited by Bobsie (edited June 29, 1999).]
Quote Reply
Re: Help!!! How can I stop spammers??? In reply to
Hello Vann,

I like your idea. Have you already modified Links to be able to do this?

If not, what do others think of Vann's idea, and does anyone know how to implement it?

Thanks,
Densu
Quote Reply
Re: Help!!! How can I stop spammers??? In reply to
Both suggestions are not very useful. Most spammers use public accounts with dynamic IP addresses and they do not name their HTTP agent "Submiteasy" or similar. The only possibility would be to block the submitted domains totally and or/their e-mail address. I did not find any script here that is able to do this and add this feature to Links. Who can help? If there is no solution we have to shut Links down because we receive every day 1.000 submissions!!!!! Help!!!!

------------------
Quote Reply
Re: Help!!! How can I stop spammers??? In reply to
I hope Alex will find soon a solution to block the spamming domains and e-mail addresses, if not, we must delete our catalog because I am not able to delete hundreds of submissions (of subpages) daily.

------------------
Quote Reply
Re: Help!!! How can I stop spammers??? In reply to
Densu -

Unfortunately, I'm still in the process of converting the sites to Links. (No, more honestly, I'm still trying to convince the site owner to convert from the mess of PC- and Mac-based dBs, idiosyncratic Perl scripts, and miscellaneous web-design tools that keep them running today. I'm currently prototyping one of the smaller sites under Links -- don't want to do too much on spec, dontcha know -- and if that goes well, we'll try to move the others over. They're doing about 50K pageviews and 12K unique visits per day, but only around 3K search requests, so I expect the flatfile version will work for the prototype.)

In any case, the "two-stroke" links submission will have to wait for the conversion. I'm not sure what the best process might be, either (1) keyed message requiring response, a la listserv, or (2) some sort of registration (through email response) required to submit.

Any ideas?

Vann
Quote Reply
Re: Help!!! How can I stop spammers??? In reply to
Dear Alex,

we are on of your clients and I have contacted you already several times. Yesterday you had sent me a suggestion how we could spammers but this does also not work because most spammers use public Internet accounts with dynamic IP addresses and it does not make sense to block these dynamic adresses.

The only possibility would be to block the domain names and e-mail addresses of the spammers, i.e. http://www.subway.net, http://www.webereignisse.de, http://www.cd-online.ch and other spammers and their e-mail addresses.

I do not know how this can be added to one of the Links script. It is very urgent, because they send us hundreds of submissions of subpages every day!

If we do not find very soon a solution we must shut our catalog down. In addition it should be possible to delete several submission at a time (from the same domain) because we must remove all this spam from the database and this is very, very time-consuming.

Please help!!!


------------------
Quote Reply
Re: Help!!! How can I stop spammers??? In reply to
I symphatise with you. Indeed an annying problem. Allow me to suggest a simple, 'low-tech' solution that would solve your problem:
Rename you add.cgi! Do so often! If you want to 'pay back' on the spammers, put something nasty or at least annoying in the add.cgi which they link to - check http://www.gossamer-threads.com/scripts/forum/resources/Forum3/HTML/000328.html for an example (which is not that nasty actually, but annoying enough ...).

As to your solution for blocking domain names, I do not think that would solve your problem. Some of the submit programs out there are not webbased, and cannot be blocked simply. I fought a fight with one of these a while ago, and my simple mod mentioned above worked :-)

Good luck in fighting the spammers!
John
Quote Reply
Re: Help!!! How can I stop spammers??? In reply to
John, nice idea, but will not last very long. These spammers will know the name of the new add.cgi very soon and they will rename it in their spamming software. The best would be to block the spamming domain names (some sites submit thousands of not relevant subpages) and their e-mail addresses! But I do not know how this can be programmed because I am no programmer...
Who can help?

------------------
Quote Reply
Re: Help!!! How can I stop spammers??? In reply to
[Where is the glyph of a smiley face smacking itself in the forehead with the palm of its hand?]

John -

Terrific idea! Took 60 seconds and provided immediate relief. Now all I need to do is gen up a mod that renames the submission form to a random string (and fixes the references in the calling page) as part of the build process.

Of course, some day the software might start following links to get to the page, but by then I'll think of something on my own!

Thanks again,
Vann


Lengua -

The problem is that these aren't "spammers," per se; they're simply folks who purchased automated submission software. Sure, some of them blast repeated entries on a daily basis, but the more irritating ones (to me) are the vast majority who simply submit inappropriate link requests. I can't think of any way to block them; every day brings hundreds of submissions from previously unknown sources. And repeated requests to the software vendor have gone unanswered, of course.
Quote Reply
Re: Help!!! How can I stop spammers??? In reply to
lengua,

Quote:
they do not name their HTTP agent "Submiteasy" or similar.

How do you know this? Have you added the HTTP_USER_AGENT check to the email you receive when a new link is added? If they are using a submission utility to sumbit the link, then it will show up in that field in your email. Certainly people are not adding that many links per day by manually doing it from your site.

At one time, I was getting a lot of submissions just like you, although not nearly that many each day. My code stopped it dead in its tracks once I discovered these were not being submitted from my site but, instead, were being submitted by a program called SubmitEasy and all from the same IP address.

I suggest you investigate further instead of just saying it is not the problem 40 minutes after I posted my message. It HAS to be the problem. There may be multiple submission utilities being used but they each definitely have a name and it will be returned in HTTP_USER_AGENT, even if it is blank. You can just have it check for a blank along with other names. All browsers will send a valid user_agent name.

[This message has been edited by Bobsie (edited June 30, 1999).]
Quote Reply
Re: Help!!! How can I stop spammers??? In reply to
Bobsie, I HAVE added the HTTP_USER_AGENT check and how supposed - this is NOT the problem. We receive all spam from normal browser agents like:

HTTP_USER_AGENT => Mozilla/4.03 [en] (Win95; I)

In addition, they use public dynamic IP addresses.

I know, that it least one of the spammers USES submission software (because the told it me himself!), EuroPromote as a part of SubmitWolf of www.trellian.com. This submission software also shows the HTTP_USER_AGENT as a normal Netscape browser (masked).
I am very disappointed of the programmer Alex Krohn because I have invested already weeks to seek for a solution, written him a lot of e-mails but he has NOT found a working solution so far.

------------------
Quote Reply
Re: Help!!! How can I stop spammers??? In reply to
All I can say is, your system must work very much differently than mine does. Between the referer check (which blocks use of my add.cgi from outside my domain) and the user agent code I installed along with it, I get no more mass addition of links like I used to.

Quote Reply
Re: Help!!! How can I stop spammers??? In reply to
Bobsie, this does not help me.

------------------
Quote Reply
Re: Help!!! How can I stop spammers??? In reply to
My apologies then. Not every message I write has helpful information but I do try. This is, after all, a discussion forum and I am just discussing my experiences with the same problem you are having. So now, I will just bow out of the conversation since I am not being helpful.
Quote Reply
Re: Help!!! How can I stop spammers??? In reply to
OK, along the lines of Phillips post, and not knowing the ease of implementation, and assuming Bobsie's mods don't work for you, then:

Create a randnum.txt file in admin/data area that contains just a number.

Insert sub routines in add.cgi that both retrieve the number from randnum.txt and insert it in a hidden field in the form, and then check it upon submission to make sure that it matches the number in the randnum.txt file.

The trick then is to run a seperate CGI script via CRON that changes the number in randnum.txt every X days/hours/minutes, it's up to you.

This will only work if their software is doing a direct submission from their location. If it retrieves your submission form and automatically fills it out, then it won't.

And of course, every once in a while the number will be changed while a legitimate submitter is submitting, but if compared to 1000 bad submissions a day... ?

Will it work? Unless I'm missing something, it should. I just don't know how to write it!

[This message has been edited by joecarey (edited July 01, 1999).]
Quote Reply
Re: Help!!! How can I stop spammers??? In reply to
Hello Lengua!

From the form one can find it out the exact location of add.cgi etc.

Can one program a script like redirect.cgi that redirects the request of the local URL within the same directory to the real location of the location of the add.cgi "Without shouting in the Browser field Location" where it is leading to?

This can also be strongly programmed from a controlled behavior, that it checks for the @referrer + ENV

Once this "Shift Location" is eastabilished -- Within the Local directory -- then I beleive it should be less problem for the add.cgi to recognize the refferrer. I can be completely wrong, so pease do not debate on it.

This assumes the following for a solution :

1 - Redirect.cgi >>> A transfer cgi script that changes the request to anathor URL displaying

http://www.Website.com/cgi-bin/redirect.cgi?/add.cgi

In the print routine of this, one uses the URL cleaning method, if possible. It then cleans the directory path (Or to misguide peoplle, prints a wrong path!!!) and displays only the file name. Therefore the basic security is achieved.

2 - Modification of Add.cgi

There one inserts the BadURLs or Dissallow.txt routine. The add.cgi then looks for a list of all the words in this ASCI file and disallows if it finds out something comparable in there. Again it already has the @refferer tag in there which also again checks that only redirect.cgi is authorised to take add.cgi in arms and no one elso.

3 - Add.cgi can be made complex.
This can be modified to handle variables and print them again via web browser which then people have to add something like a password and submit or fill out an extra field. So an extra step making the like of mass submitter a bit more difficult, however making it only "One click + one word more difficult"

4 - Chaning the pattern of submission.
The add.cgi is not disclosed at all from the web. The submit form only takes an email address from submitter and then answers a reply the exact location of the Add.cgi. Then the person goes there to continue the submission in full. This can be done with the help of email printing where there is the Add.cgi location address. This can be changed every day. It only makes the life of a person more complicated who does mass emailing.

I have also been shouting enough, with all the possible positive intentions, that basic functionality needs to be re-questioned. I am also of the opinion that something needs to be done reading and looking at the problems like this. What you have is a web horror.

Quote:
I am very disappointed of the programmer Alex Krohn because I have invested already weeks to seek for a solution, written him a lot of e-mails but he has NOT found a working solution so far.

So you see, from my point of view, a strategic approach needs to be taken also. However if you ONLY depend on the programming it cannot help you. There are all kinds of minds around the web and one has to face them in a clever way as much as possible. Simply passing your frustration to the programmer is something I seriously condemn. How can you can say this? One can understand your frustrations and therefore one tries to help here, on this beautiful, healty and democratic platform "the programmer Alex Krohn" has created, people like you in a good way. The messages from above shows.

I am not an expert nor do I have any more knowledge of Links v2.0. All I have written above are simple basic idea with the spirit of help and not a "Return critism". If I cannot help you, sorry to make you visit here again because of the automatic emailing, and making this message heavy for its download.



------------------
rajani











Quote Reply
Re: Help!!! How can I stop spammers??? In reply to
Phoenix, very good advice. Can you imagine that I have to work and do not have the time to rename add.cgi every day.
By the way, I have BOUGHT and PAID the script. I think it should be possible to give support this is a normal thing for good software. I have posted my problem here several times. I have sent the programmer a lot of e-mail (by the way, I did 2 replies, but the suggestions there not working). I do not understand because it is not possible for the programmer to give me a mod that is able to block submissions with certain domain names and under and with certain e-mail addresses. If I have to work through thousands of e-mail every day I have to shut our catalog down. If you would be under such an amount of additional work you would understand this.

------------------
Quote Reply
Re: Help!!! How can I stop spammers??? In reply to
Hello,

How about a simple mod that uses the date and time of the system in a hidden field. If more than say 5 minutes have elapsed then redirect the user to an error page.

Another mod would track the number of submissions from a given IP address over a amount of time. Too many submissions from the same IP address would indicate a spammer.

Writing the submissions to a seperate holding DB also would protect the main links DB.

Just some ideas.

Thanks
Quote Reply
Re: Help!!! How can I stop spammers??? In reply to
lengua,

People are trying to help. You shoot down every suggestion made. You write a message about stopping spammers, and then you spam that message across every single Links forum. My sympathy for you is dwindling.

Why don't you just change the name of your add.cgi script and do so on a regular basis? If you change it often, it will take you way less time than deleting thousands of spammed links every day and it will solve the problem. Maybe it's not the best fix, but it will get rid of the problem and buy you time to find another solution without shouting at everyone.

Spam is not our fault. Spam is not Alex's fault or Links' fault. Switch to a different software and you'll still get the same problem. If normal means of preventing spammers that work for everyone else do not work for you, what on earth do you think anyone or any program can do to help?

Take the advice of changing the name of add.cgi. Change it every week. Change it every day if you want. Make it a number for every day of the year. Just take the simplest advice, relax a while, and look for another solution.

Or delete your database, which doesn't hurt us here at all.

Phoenix
> >