Mailing List Archive: Interchange: users

Robots

Index | Next | Previous | View Threaded

racke at linuxia

Nov 17, 2009, 4:43 AM

Post #1 of 8 (1705 views)
Permalink

Robots

Hello, Interchange enthusiasts!

Today I got a report about a customer who wasn't able to order in an
Interchange store.

It turned out that the user had the Ask toolbar installed and the robots.cfg
wasn't up-to-date, so the session was dropped.

You probably agree with me that preventing orders is really a
bad thing.

Recognition of robots by examining the UserAgent string is of course error
prone and false positives are possible.

Robots who successfully login to an Interchange store are unlikely to exist.
In this case Interchange should loudly scream into the logfiles about that
and leave an useful error message for the user.

Also Kevin Walsh used to manage the default settings in robots.cfg, but he
is not available anymore. Anyone else who would manage this task?

Regards
Racke

--
LinuXia Systems => http://www.linuxia.de/
Expert Interchange Consulting and System Administration
ICDEVGROUP => http://www.icdevgroup.org/
Interchange Development Team

_______________________________________________
interchange-users mailing list
interchange-users [at] icdevgroup
http://www.icdevgroup.org/mailman/listinfo/interchange-users

DB at M-and-D

Nov 17, 2009, 8:41 AM

Post #2 of 8 (1634 views)
Permalink

Re: Robots [In reply to]

> Hello, Interchange enthusiasts!
>
> Today I got a report about a customer who wasn't able to order in an
> Interchange store.
>
> It turned out that the user had the Ask toolbar installed and the robots.cfg
> wasn't up-to-date, so the session was dropped.
>
> You probably agree with me that preventing orders is really a
> bad thing.
>
> Recognition of robots by examining the UserAgent string is of course error
> prone and false positives are possible.
>
> Robots who successfully login to an Interchange store are unlikely to exist.
> In this case Interchange should loudly scream into the logfiles about that
> and leave an useful error message for the user.
>
> Also Kevin Walsh used to manage the default settings in robots.cfg, but he
> is not available anymore. Anyone else who would manage this task?
>
> Regards
> Racke
>
> --
> LinuXia Systems => http://www.linuxia.de/
> Expert Interchange Consulting and System Administration
> ICDEVGROUP => http://www.icdevgroup.org/
> Interchange Development Team

Hi - just for fun I visited
http://about.ask.com/en/docs/about/webmasters.shtml#8 which says that
Ask's current spider reports itself as

User-Agent: Mozilla/2.0 (compatible; Ask Jeeves/Teoma)

I do see such entires in my sites access log, but my current robots.cfg
appears to have no such entry. Perhaps someone who is able should make
that change?

DB

_______________________________________________
interchange-users mailing list
interchange-users [at] icdevgroup
http://www.icdevgroup.org/mailman/listinfo/interchange-users

icdev at mrlock

Nov 17, 2009, 9:04 AM

Post #3 of 8 (1629 views)
Permalink

Re: Robots [In reply to]

> Hi - just for fun I visited
> http://about.ask.com/en/docs/about/webmasters.shtml#8 which says that
> Ask's current spider reports itself as
>
> User-Agent: Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
>
>
> I do see such entires in my sites access log, but my current robots.cfg
> appears to have no such entry. Perhaps someone who is able should make
> that change?
>
> DB
>

There was a discussion recently about RobotUA, but I couldn't find the
reference.

Check here for the current github version:
http://github.com/interchange/interchange/tree/master/dist/

Steve

_______________________________________________
interchange-users mailing list
interchange-users [at] icdevgroup
http://www.icdevgroup.org/mailman/listinfo/interchange-users

ron at endpoint

Nov 18, 2009, 10:17 AM

Post #4 of 8 (1597 views)
Permalink

Re: Robots [In reply to]

Stefan Hornburg (Racke) wrote:
> Hello, Interchange enthusiasts!
>
> Today I got a report about a customer who wasn't able to order in an
> Interchange store.
>
> It turned out that the user had the Ask toolbar installed and the
> robots.cfg
> wasn't up-to-date, so the session was dropped.
>
> You probably agree with me that preventing orders is really a
> bad thing.
>
> Recognition of robots by examining the UserAgent string is of course error
> prone and false positives are possible.
>
> Robots who successfully login to an Interchange store are unlikely to
> exist.
> In this case Interchange should loudly scream into the logfiles about that
> and leave an useful error message for the user.
>
> Also Kevin Walsh used to manage the default settings in robots.cfg, but he
> is not available anymore. Anyone else who would manage this task?
>
> Regards
> Racke
>

Racke,

Do you happen to know what browser and version was installed along with
the Ask toolbar and how the UserAgent came across that caused the
session to be dropped? We are seeing session dropping periodically and
this could be the cause.

Thanks,

--
Ron Phipps
End Point Corporation
ron [at] endpoint

_______________________________________________
interchange-users mailing list
interchange-users [at] icdevgroup
http://www.icdevgroup.org/mailman/listinfo/interchange-users

mat at bibliopolis

Nov 18, 2009, 10:26 AM

Post #5 of 8 (1594 views)
Permalink

Re: Robots [In reply to]

There was a post about this issue back in June:

http://www.icdevgroup.org/pipermail/interchange-users/2009-June/050621.html

We had the same thing happen with some of our clients. Seemed to be IE with latest Ask Toolbar.

Mat

Ron Phipps wrote:
> Stefan Hornburg (Racke) wrote:
>> Hello, Interchange enthusiasts!
>>
>> Today I got a report about a customer who wasn't able to order in an
>> Interchange store.
>>
>> It turned out that the user had the Ask toolbar installed and the
>> robots.cfg
>> wasn't up-to-date, so the session was dropped.
>>
>> You probably agree with me that preventing orders is really a
>> bad thing.
>>
>> Recognition of robots by examining the UserAgent string is of course
>> error
>> prone and false positives are possible.
>>
>> Robots who successfully login to an Interchange store are unlikely to
>> exist.
>> In this case Interchange should loudly scream into the logfiles about
>> that
>> and leave an useful error message for the user.
>>
>> Also Kevin Walsh used to manage the default settings in robots.cfg,
>> but he
>> is not available anymore. Anyone else who would manage this task?
>>
>> Regards
>> Racke
>>
>
> Racke,
>
> Do you happen to know what browser and version was installed along with
> the Ask toolbar and how the UserAgent came across that caused the
> session to be dropped? We are seeing session dropping periodically and
> this could be the cause.
>
> Thanks,
>

_______________________________________________
interchange-users mailing list
interchange-users [at] icdevgroup
http://www.icdevgroup.org/mailman/listinfo/interchange-users

paul at gishnetwork

Nov 18, 2009, 10:33 AM

Post #6 of 8 (1596 views)
Permalink

Re: Robots [In reply to]

> Stefan Hornburg (Racke) wrote:
>> Hello, Interchange enthusiasts!
>>
>> Today I got a report about a customer who wasn't able to order in an
>> Interchange store.
>>
>> It turned out that the user had the Ask toolbar installed and the
>> robots.cfg
>> wasn't up-to-date, so the session was dropped.
>>
>> You probably agree with me that preventing orders is really a
>> bad thing.
>>
>> Recognition of robots by examining the UserAgent string is of course
>> error
>> prone and false positives are possible.
>>
>> Robots who successfully login to an Interchange store are unlikely to
>> exist.
>> In this case Interchange should loudly scream into the logfiles about
>> that
>> and leave an useful error message for the user.
>>
>> Also Kevin Walsh used to manage the default settings in robots.cfg, but
>> he
>> is not available anymore. Anyone else who would manage this task?
>>
>> Regards
>> Racke
>>
>
> Racke,
>
> Do you happen to know what browser and version was installed along with
> the Ask toolbar and how the UserAgent came across that caused the session
> to be dropped? We are seeing session dropping periodically and this could
> be the cause.

I am interested in the same. IIRC it was coming across as ASKTB5.1. I
remember looking into it and seeing that Ask Toolbar was no where close to
version 5.1 and just figured it was probably spyware, and not really the Ask
Toolbar. Of course, that is a pretty basic observation with many
assumptions, however I did not have proper access to the clients browser for
any real investigation. It was not wide spread enough for me to go the extra
mile and its expense at the time, and we've not seen any issues lately.

I don't believe I kept any of my reports, but I remember it being IE 7 and
8, on XP and Vista. I may even remember IE6 being in there.

Racke has access to our servers, and he is free us our logs if they can help
in any way.

So, I will say I may have experienced the issue presented in this thread,
and am interested in a solution too.

Paul

_______________________________________________
interchange-users mailing list
interchange-users [at] icdevgroup
http://www.icdevgroup.org/mailman/listinfo/interchange-users

racke at linuxia

Nov 18, 2009, 10:50 AM

Post #7 of 8 (1591 views)
Permalink

Re: Robots [In reply to]

Ron Phipps wrote:
> Stefan Hornburg (Racke) wrote:
>> Hello, Interchange enthusiasts!
>>
>> Today I got a report about a customer who wasn't able to order in an
>> Interchange store.
>>
>> It turned out that the user had the Ask toolbar installed and the
>> robots.cfg
>> wasn't up-to-date, so the session was dropped.
>>
>> You probably agree with me that preventing orders is really a
>> bad thing.
>>
>> Recognition of robots by examining the UserAgent string is of course
>> error
>> prone and false positives are possible.
>>
>> Robots who successfully login to an Interchange store are unlikely to
>> exist.
>> In this case Interchange should loudly scream into the logfiles about
>> that
>> and leave an useful error message for the user.
>>
>> Also Kevin Walsh used to manage the default settings in robots.cfg,
>> but he
>> is not available anymore. Anyone else who would manage this task?
>>
>> Regards
>> Racke
>>
>
> Racke,
>
> Do you happen to know what browser and version was installed along with
> the Ask toolbar and how the UserAgent came across that caused the
> session to be dropped? We are seeing session dropping periodically and
> this could be the cause.
>

The UserAgent was:

Mozilla/4.0 (compatible; MSIE 7.0; TOB 6.05; Windows NT 6.0; Trident/4.0;
GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 1.1.4322;
.NET CLR 3.5.30729; .NET CLR 3.0.30729; AskTB5.4)

I correlated "Successful logins" with nsession as session id
in the catalog log file with Apache access log entries.

Adding the NotRobotUA directive fixed the problem.

Regards
Racke

--
LinuXia Systems => http://www.linuxia.de/
Expert Interchange Consulting and System Administration
ICDEVGROUP => http://www.icdevgroup.org/
Interchange Development Team

_______________________________________________
interchange-users mailing list
interchange-users [at] icdevgroup
http://www.icdevgroup.org/mailman/listinfo/interchange-users

ron at endpoint

Nov 18, 2009, 11:38 AM

Post #8 of 8 (1592 views)
Permalink

Re: Robots [In reply to]

Stefan Hornburg (Racke) wrote:
> Ron Phipps wrote:
>> Stefan Hornburg (Racke) wrote:
>>> Hello, Interchange enthusiasts!
>>>
>>> Today I got a report about a customer who wasn't able to order in an
>>> Interchange store.
>>>
>>> It turned out that the user had the Ask toolbar installed and the
>>> robots.cfg
>>> wasn't up-to-date, so the session was dropped.
>>>
>>> You probably agree with me that preventing orders is really a
>>> bad thing.
>>>
>>> Recognition of robots by examining the UserAgent string is of course
>>> error
>>> prone and false positives are possible.
>>>
>>> Robots who successfully login to an Interchange store are unlikely to
>>> exist.
>>> In this case Interchange should loudly scream into the logfiles about
>>> that
>>> and leave an useful error message for the user.
>>>
>>> Also Kevin Walsh used to manage the default settings in robots.cfg,
>>> but he
>>> is not available anymore. Anyone else who would manage this task?
>>>
>>> Regards
>>> Racke
>>>
>>
>> Racke,
>>
>> Do you happen to know what browser and version was installed along
>> with the Ask toolbar and how the UserAgent came across that caused the
>> session to be dropped? We are seeing session dropping periodically
>> and this could be the cause.
>>
>
> The UserAgent was:
>
> Mozilla/4.0 (compatible; MSIE 7.0; TOB 6.05; Windows NT 6.0;
> Trident/4.0; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET
> CLR 1.1.4322; .NET CLR 3.5.30729; .NET CLR 3.0.30729; AskTB5.4)
>
> I correlated "Successful logins" with nsession as session id
> in the catalog log file with Apache access log entries.
>
> Adding the NotRobotUA directive fixed the problem.
>
> Regards
> Racke
>
>

Thanks Racke, I just verified that this was the issue we were seeing and
have resolved it for one specific client. Thanks to Matthew Jones for
posting the link to the previous discussion as well.

--
Ron Phipps
End Point Corporation
ron [at] endpoint

_______________________________________________
interchange-users mailing list
interchange-users [at] icdevgroup
http://www.icdevgroup.org/mailman/listinfo/interchange-users

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads