Mailing List Archive: Analog: Help

Remote and local statistics

Index | Next | Previous | View Threaded

bclanton at dupagels

Aug 12, 2008, 2:09 PM

Post #1 of 11 (1922 views)
Permalink

Remote and local statistics

I have Analog analyzing proxy server logs (Ezproxy) and I have two types of
log files: local and proxy.

Example:

proxy 66.158.45.2 - yvme9Cr1rT412ku [18/Jul/2008:10:29:11 -0600] "GET
http://search.ebscohost.com/login.aspx?authtype=ip,uid&profile=novelistk8
HTTP/1.1" 200 0

local 66.158.45.2 - - [23/Jul/2008:09:19:35 -0600] "GET
http://search.ebscohost.com/login.aspx?authtype=ip,uid&profile=novelistk8
HTTP/1.1" 301 567

The local logs are library patrons accessing resources from within the
library. Proxy requests are patrons accessing the resources from outside
the library.

I use the User Reports to show the remote patron requests total and the
REDIRECTED USER report to show the requests total for local users.

The problem is that the proxy server (Ezproxy) redirects the local access
requests (301) to save on bandwidth and only "proxies" the remote requests.
Therefore Analog only shows stats for the "successful" proxied requests in
the other subsequent reports, namely the REQUEST REPORT.

Is there a way to configure Analog to consider all requests, both local and
proxy, as legitimate requests so the REQUEST REPORT will show totals for
both local and remote patrons as well still have separate reports showing
total remote requests (User Report) and total local requests (REDIRECTED
USER REPORT)for a given time period?

+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

analog07 at eircom

Aug 12, 2008, 2:34 PM

Post #2 of 11 (1811 views)
Permalink

Re: Remote and local statistics [In reply to]

Brian Clanton <bclanton [at] dupagels> wrote:
> The problem is that the proxy server (Ezproxy) redirects the local
> access requests (301) to save on bandwidth and only "proxies" the
> remote requests. Therefore Analog only shows stats for the
> "successful" proxied requests in the other subsequent reports, namely
> the REQUEST REPORT.
>
> Is there a way to configure Analog to consider all requests, both
> local and proxy, as legitimate requests so the REQUEST REPORT will
> show totals for both local and remote patrons as well still have
> separate reports showing total remote requests (User Report) and
> total local requests (REDIRECTED USER REPORT)for a given time period?

The issue of treating 301's as Successful requests is delat with in the Analog FAQ:
http://analog.cx/docs/faq.html#faq181

I want to be able to count requests with status code 301 and
302 as successes, so that they appear in the Request Report.

No, you really don't, because that would lead to double counting
when a request for /dir (code 301) is redirected to /dir/ (code 200).
For CGI scripts etc. look in the Redirection Report instead of the
Request Report.

If you want all the entries to turn up in the Request Report, then you'll have to modify the Response Code to be 200.

Aengus

+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

bclanton at dupagels

Aug 12, 2008, 3:03 PM

Post #3 of 11 (1817 views)
Permalink

RE: Remote and local statistics [In reply to]

I believe my situation is a little different in that the Proxy server
(EZPROXY) that our library clients are using creates special log files that
only track when the database link is clicked on.

For instance, when a patron clicks on the link on the libray's website,
Ezproxy only creates ONE log and that entry is either a 200 remote or a 301
response.

Unless of course I am mistaken about this, but this is what it looks like is
happening in the log file. Here are a few log examples of from a LOCAL ip
address that is accessing different GROLER database:

local 66.158.45.2 - - [12/Aug/2008:14:45:05 -0600] "GET
http://atb.grolier.com/ HTTP/1.1" 301 467
local 66.158.45.2 - - [12/Aug/2008:14:45:22 -0600] "GET
http://gme.grolier.com HTTP/1.1" 301 465
local 66.158.45.2 - - [12/Aug/2008:14:45:41 -0600] "GET
http://lp.grolier.com HTTP/1.1" 301 463
local 66.158.45.2 - - [12/Aug/2008:14:46:06 -0600] "GET
http://ea.grolier.com/ HTTP/1.1" 301 465
local 66.158.45.2 - - [12/Aug/2008:14:46:54 -0600] "GET
http://atb.grolier.com/ HTTP/1.1" 301 467
local 66.158.45.2 - - [12/Aug/2008:14:47:05 -0600] "GET
http://ea.grolier.com/ HTTP/1.1" 301 465
local 66.158.45.2 - - [12/Aug/2008:14:47:16 -0600] "GET
http://gme.grolier.com HTTP/1.1" 301 465
local 66.158.45.2 - - [12/Aug/2008:14:47:29 -0600] "GET
http://lp.grolier.com HTTP/1.1" 301 463

These are the last logs of the day today and as you can see, no additional
logs are generated once the log has been redirected. If you are familiar
with Ezproxy the logformat command within the Ezproxy config file is:

LogSPU spu.log %{ezproxy-spuaccess}i %h %l %u %t "%r" %s %b

This log file format allows for differentiating between local and proxy
logs, which the libraries want statistics for.

Anyway, to me it seems I need to include the redirected logs since these are
the only accounts of local access requests from patrons that are in the log
file.

Am I correct in saying this?

> -----Original Message-----
> From: analog-help-bounces [at] lists
> [mailto:analog-help-bounces [at] lists] On Behalf Of Aengus
> Sent: Tuesday, August 12, 2008 4:34 PM
> To: Support for analog web log analyzer
> Subject: Re: [analog-help] Remote and local statistics
>
> Brian Clanton <bclanton [at] dupagels> wrote:
> > The problem is that the proxy server (Ezproxy) redirects the local
> > access requests (301) to save on bandwidth and only "proxies" the
> > remote requests. Therefore Analog only shows stats for the
> > "successful" proxied requests in the other subsequent
> reports, namely
> > the REQUEST REPORT.
> >
> > Is there a way to configure Analog to consider all requests, both
> > local and proxy, as legitimate requests so the REQUEST REPORT will
> > show totals for both local and remote patrons as well still have
> > separate reports showing total remote requests (User
> Report) and total
> > local requests (REDIRECTED USER REPORT)for a given time period?
>
> The issue of treating 301's as Successful requests is delat
> with in the Analog FAQ:
> http://analog.cx/docs/faq.html#faq181
>
> I want to be able to count requests with status code 301 and
> 302 as successes, so that they appear in the Request Report.
>
> No, you really don't, because that would lead to double counting
> when a request for /dir (code 301) is redirected to /dir/
> (code 200).
> For CGI scripts etc. look in the Redirection Report instead of the
> Request Report.
>
>
> If you want all the entries to turn up in the Request Report,
> then you'll have to modify the Response Code to be 200.
>
> Aengus
>
> +-------------------------------------------------------------
> ----------
> +-
> | TO UNSUBSCRIBE from this list:
> | http://lists.meer.net/mailman/listinfo/analog-help
> |
> | Analog Documentation: http://analog.cx/docs/Readme.html List
> | archives: http://www.analog.cx/docs/mailing.html#listarchives
> | Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
> +-------------------------------------------------------------
> ----------
> +-
>

+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

bclanton at dupagels

Aug 12, 2008, 3:07 PM

Post #4 of 11 (1805 views)
Permalink

RE: Remote and local statistics [In reply to]

If it helps explain my situation at all, here is a test results page I have
been working on

http://proxy.dupagels.lib.il.us/ob_proxyStats.html

> -----Original Message-----
> From: analog-help-bounces [at] lists
> [mailto:analog-help-bounces [at] lists] On Behalf Of Brian Clanton
> Sent: Tuesday, August 12, 2008 5:04 PM
> To: 'Support for analog web log analyzer'
> Subject: RE: [analog-help] Remote and local statistics
>
> I believe my situation is a little different in that the Proxy server
> (EZPROXY) that our library clients are using creates special
> log files that only track when the database link is clicked on.
>
> For instance, when a patron clicks on the link on the
> libray's website, Ezproxy only creates ONE log and that entry
> is either a 200 remote or a 301 response.
>
> Unless of course I am mistaken about this, but this is what
> it looks like is happening in the log file. Here are a few
> log examples of from a LOCAL ip address that is accessing
> different GROLER database:
>
> local 66.158.45.2 - - [12/Aug/2008:14:45:05 -0600] "GET
> http://atb.grolier.com/ HTTP/1.1" 301 467 local 66.158.45.2
> - - [12/Aug/2008:14:45:22 -0600] "GET http://gme.grolier.com
> HTTP/1.1" 301 465 local 66.158.45.2 - -
> [12/Aug/2008:14:45:41 -0600] "GET http://lp.grolier.com
> HTTP/1.1" 301 463 local 66.158.45.2 - -
> [12/Aug/2008:14:46:06 -0600] "GET http://ea.grolier.com/
> HTTP/1.1" 301 465 local 66.158.45.2 - -
> [12/Aug/2008:14:46:54 -0600] "GET http://atb.grolier.com/
> HTTP/1.1" 301 467 local 66.158.45.2 - -
> [12/Aug/2008:14:47:05 -0600] "GET http://ea.grolier.com/
> HTTP/1.1" 301 465 local 66.158.45.2 - -
> [12/Aug/2008:14:47:16 -0600] "GET http://gme.grolier.com
> HTTP/1.1" 301 465 local 66.158.45.2 - -
> [12/Aug/2008:14:47:29 -0600] "GET http://lp.grolier.com
> HTTP/1.1" 301 463
>
> These are the last logs of the day today and as you can see,
> no additional logs are generated once the log has been
> redirected. If you are familiar with Ezproxy the logformat
> command within the Ezproxy config file is:
>
> LogSPU spu.log %{ezproxy-spuaccess}i %h %l %u %t "%r" %s %b
>
> This log file format allows for differentiating between local
> and proxy logs, which the libraries want statistics for.
>
> Anyway, to me it seems I need to include the redirected logs
> since these are the only accounts of local access requests
> from patrons that are in the log file.
>
> Am I correct in saying this?
>
> > -----Original Message-----
> > From: analog-help-bounces [at] lists
> > [mailto:analog-help-bounces [at] lists] On Behalf Of Aengus
> > Sent: Tuesday, August 12, 2008 4:34 PM
> > To: Support for analog web log analyzer
> > Subject: Re: [analog-help] Remote and local statistics
> >
> > Brian Clanton <bclanton [at] dupagels> wrote:
> > > The problem is that the proxy server (Ezproxy) redirects
> the local
> > > access requests (301) to save on bandwidth and only "proxies" the
> > > remote requests. Therefore Analog only shows stats for the
> > > "successful" proxied requests in the other subsequent
> > reports, namely
> > > the REQUEST REPORT.
> > >
> > > Is there a way to configure Analog to consider all requests, both
> > > local and proxy, as legitimate requests so the REQUEST
> REPORT will
> > > show totals for both local and remote patrons as well still have
> > > separate reports showing total remote requests (User
> > Report) and total
> > > local requests (REDIRECTED USER REPORT)for a given time period?
> >
> > The issue of treating 301's as Successful requests is delat with in
> > the Analog FAQ:
> > http://analog.cx/docs/faq.html#faq181
> >
> > I want to be able to count requests with status code 301 and
> > 302 as successes, so that they appear in the Request Report.
> >
> > No, you really don't, because that would lead to double counting
> > when a request for /dir (code 301) is redirected to /dir/ (code
> > 200).
> > For CGI scripts etc. look in the Redirection Report instead of the
> > Request Report.
> >
> >
> > If you want all the entries to turn up in the Request Report, then
> > you'll have to modify the Response Code to be 200.
> >
> > Aengus
> >
> > +-------------------------------------------------------------
> > ----------
> > +-
> > | TO UNSUBSCRIBE from this list:
> > | http://lists.meer.net/mailman/listinfo/analog-help
> > |
> > | Analog Documentation: http://analog.cx/docs/Readme.html List
> > | archives: http://www.analog.cx/docs/mailing.html#listarchives
> > | Usenet version:
> news://news.gmane.org/gmane.comp.web.analog.general
> > +-------------------------------------------------------------
> > ----------
> > +-
> >
>
> +-------------------------------------------------------------
> ----------
> +-
> | TO UNSUBSCRIBE from this list:
> | http://lists.meer.net/mailman/listinfo/analog-help
> |
> | Analog Documentation: http://analog.cx/docs/Readme.html List
> | archives: http://www.analog.cx/docs/mailing.html#listarchives
> | Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
> +-------------------------------------------------------------
> ----------
> +-
>

+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

bclanton at dupagels

Aug 12, 2008, 3:22 PM

Post #5 of 11 (1816 views)
Permalink

RE: Remote and local statistics [In reply to]

One problem I just thought of....

When I generate monthly reports, I could use the stream editor command (sed)
and replace all 301 replys with 200, but then I would not be able to
differentiate between local and proxy anymore using the REDIRECT REPORT for
local users and have separate request results for each.....would I?

> -----Original Message-----
> From: analog-help-bounces [at] lists
> [mailto:analog-help-bounces [at] lists] On Behalf Of Brian Clanton
> Sent: Tuesday, August 12, 2008 5:08 PM
> To: 'Support for analog web log analyzer'
> Subject: RE: [analog-help] Remote and local statistics
>
> If it helps explain my situation at all, here is a test
> results page I have been working on
>
> http://proxy.dupagels.lib.il.us/ob_proxyStats.html
>
>
>
>
> > -----Original Message-----
> > From: analog-help-bounces [at] lists
> > [mailto:analog-help-bounces [at] lists] On Behalf Of
> Brian Clanton
> > Sent: Tuesday, August 12, 2008 5:04 PM
> > To: 'Support for analog web log analyzer'
> > Subject: RE: [analog-help] Remote and local statistics
> >
> > I believe my situation is a little different in that the
> Proxy server
> > (EZPROXY) that our library clients are using creates
> special log files
> > that only track when the database link is clicked on.
> >
> > For instance, when a patron clicks on the link on the libray's
> > website, Ezproxy only creates ONE log and that entry is
> either a 200
> > remote or a 301 response.
> >
> > Unless of course I am mistaken about this, but this is what
> it looks
> > like is happening in the log file. Here are a few log examples of
> > from a LOCAL ip address that is accessing different GROLER database:
> >
> > local 66.158.45.2 - - [12/Aug/2008:14:45:05 -0600] "GET
> > http://atb.grolier.com/ HTTP/1.1" 301 467 local 66.158.45.2
> > - - [12/Aug/2008:14:45:22 -0600] "GET
> http://gme.grolier.com HTTP/1.1"
> > 301 465 local 66.158.45.2 - -
> > [12/Aug/2008:14:45:41 -0600] "GET http://lp.grolier.com
> HTTP/1.1" 301
> > 463 local 66.158.45.2 - -
> > [12/Aug/2008:14:46:06 -0600] "GET http://ea.grolier.com/
> HTTP/1.1" 301
> > 465 local 66.158.45.2 - -
> > [12/Aug/2008:14:46:54 -0600] "GET http://atb.grolier.com/ HTTP/1.1"
> > 301 467 local 66.158.45.2 - -
> > [12/Aug/2008:14:47:05 -0600] "GET http://ea.grolier.com/
> HTTP/1.1" 301
> > 465 local 66.158.45.2 - -
> > [12/Aug/2008:14:47:16 -0600] "GET http://gme.grolier.com
> HTTP/1.1" 301
> > 465 local 66.158.45.2 - -
> > [12/Aug/2008:14:47:29 -0600] "GET http://lp.grolier.com
> HTTP/1.1" 301
> > 463
> >
> > These are the last logs of the day today and as you can see, no
> > additional logs are generated once the log has been redirected. If
> > you are familiar with Ezproxy the logformat command within
> the Ezproxy
> > config file is:
> >
> > LogSPU spu.log %{ezproxy-spuaccess}i %h %l %u %t "%r" %s %b
> >
> > This log file format allows for differentiating between local and
> > proxy logs, which the libraries want statistics for.
> >
> > Anyway, to me it seems I need to include the redirected logs since
> > these are the only accounts of local access requests from
> patrons that
> > are in the log file.
> >
> > Am I correct in saying this?
> >
> > > -----Original Message-----
> > > From: analog-help-bounces [at] lists
> > > [mailto:analog-help-bounces [at] lists] On Behalf Of Aengus
> > > Sent: Tuesday, August 12, 2008 4:34 PM
> > > To: Support for analog web log analyzer
> > > Subject: Re: [analog-help] Remote and local statistics
> > >
> > > Brian Clanton <bclanton [at] dupagels> wrote:
> > > > The problem is that the proxy server (Ezproxy) redirects
> > the local
> > > > access requests (301) to save on bandwidth and only
> "proxies" the
> > > > remote requests. Therefore Analog only shows stats for the
> > > > "successful" proxied requests in the other subsequent
> > > reports, namely
> > > > the REQUEST REPORT.
> > > >
> > > > Is there a way to configure Analog to consider all
> requests, both
> > > > local and proxy, as legitimate requests so the REQUEST
> > REPORT will
> > > > show totals for both local and remote patrons as well
> still have
> > > > separate reports showing total remote requests (User
> > > Report) and total
> > > > local requests (REDIRECTED USER REPORT)for a given time period?
> > >
> > > The issue of treating 301's as Successful requests is
> delat with in
> > > the Analog FAQ:
> > > http://analog.cx/docs/faq.html#faq181
> > >
> > > I want to be able to count requests with status code 301 and
> > > 302 as successes, so that they appear in the Request Report.
> > >
> > > No, you really don't, because that would lead to double counting
> > > when a request for /dir (code 301) is redirected to /dir/ (code
> > > 200).
> > > For CGI scripts etc. look in the Redirection Report
> instead of the
> > > Request Report.
> > >
> > >
> > > If you want all the entries to turn up in the Request
> Report, then
> > > you'll have to modify the Response Code to be 200.
> > >
> > > Aengus
> > >
> > > +-------------------------------------------------------------
> > > ----------
> > > +-
> > > | TO UNSUBSCRIBE from this list:
> > > | http://lists.meer.net/mailman/listinfo/analog-help
> > > |
> > > | Analog Documentation: http://analog.cx/docs/Readme.html List
> > > | archives: http://www.analog.cx/docs/mailing.html#listarchives
> > > | Usenet version:
> > news://news.gmane.org/gmane.comp.web.analog.general
> > > +-------------------------------------------------------------
> > > ----------
> > > +-
> > >
> >
> > +-------------------------------------------------------------
> > ----------
> > +-
> > | TO UNSUBSCRIBE from this list:
> > | http://lists.meer.net/mailman/listinfo/analog-help
> > |
> > | Analog Documentation: http://analog.cx/docs/Readme.html List
> > | archives: http://www.analog.cx/docs/mailing.html#listarchives
> > | Usenet version:
> news://news.gmane.org/gmane.comp.web.analog.general
> > +-------------------------------------------------------------
> > ----------
> > +-
> >
>
> +-------------------------------------------------------------
> ----------
> +-
> | TO UNSUBSCRIBE from this list:
> | http://lists.meer.net/mailman/listinfo/analog-help
> |
> | Analog Documentation: http://analog.cx/docs/Readme.html List
> | archives: http://www.analog.cx/docs/mailing.html#listarchives
> | Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
> +-------------------------------------------------------------
> ----------
> +-
>

+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

analog07 at eircom

Aug 12, 2008, 3:27 PM

Post #6 of 11 (1814 views)
Permalink

Re: Remote and local statistics [In reply to]

Brian Clanton <bclanton [at] dupagels> wrote:
>
> Anyway, to me it seems I need to include the redirected logs since
> these are the only accounts of local access requests from patrons
> that are in the log file.
>
> Am I correct in saying this?

It doesn't make much difference whether you're correct or not. Analog won't treat log entries with a 301 response code as "Successful requests". If you want those log entries to show up in the Request Report rather than the Redirection Report, then you'll have to either modify the logs, modify Analog's source code, or modify EZProxy.

You might have some success by ignoring the Status Code altogether (marking it as %j in the LOGFORMAT), but that depends on whether you are paying any attention to requests other than 200s and 301s.

Aengus

+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

analog-author at lists

Aug 12, 2008, 3:41 PM

Post #7 of 11 (1815 views)
Permalink

Re: Remote and local statistics [In reply to]

2008/8/12 Brian Clanton <bclanton [at] dupagels>:
>
> The local logs are library patrons accessing resources from within the
> library. Proxy requests are patrons accessing the resources from outside
> the library.
>
> I use the User Reports to show the remote patron requests total and the
> REDIRECTED USER report to show the requests total for local users.
>
> The problem is that the proxy server (Ezproxy) redirects the local access
> requests (301) to save on bandwidth and only "proxies" the remote requests.
> Therefore Analog only shows stats for the "successful" proxied requests in
> the other subsequent reports, namely the REQUEST REPORT.
>
> Is there a way to configure Analog to consider all requests, both local and
> proxy, as legitimate requests so the REQUEST REPORT will show totals for
> both local and remote patrons as well still have separate reports showing
> total remote requests (User Report) and total local requests (REDIRECTED
> USER REPORT)for a given time period?
>

I think I understand now. There is no way to treat 301 as a success
(without preprocessing the log), but you can look at the Redirection
Report which gives the same information as the Request Report, but for
redirects.

--
Stephen Turner
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------

bclanton at dupagels

Aug 12, 2008, 3:55 PM

Post #8 of 11 (1807 views)
Permalink

RE: Remote and local statistics [In reply to]

My problem is that the REQUEST REPORT does not include the redirected logs
(301).

Libraries want the following information about their online databases:
- How many local users are using the resources
- How many remote users are using the resources
- How much are the resources being used (both local and remote combined)

The REQUEST REPORT does not include the local "redirected" requests. The
REDIRECTED REPORT will give a total of "requests" made, but not what
databases were accessed.

Example:
http://proxy.dupagels.lib.il.us/ob_proxyStats.html

What about this idea......

What if I change the response codes in the log files from redirected to
successful (301 to 200) which will give me all local and proxy requests in
the REQUEST REPORT.

Then if I want to differentiate between how many local requests as opposed
to proxy requests, I can somhow use the first part of the log entery (proxy
or local)

local 66.158.45.2 - - [15/Jul/2008:18:08:00 -0600] "HEAD
http://firstsearch.oclc.org/FSIP HTTP/1.1" 301 485

I don't necessarily need a graph for local and proxy users, just how many
requests per the designated time period.

The problem is, I'm not sure if there is a report that I can use from the
first part of this log entry.

My analog logformat is this:

APACHELOGFORMAT (%u %h %l %w %t "%r" %s %b)
APACHELOGFORMAT (%u %h %w %l %w %t "%r" %s %b)
APACHELOGFORMAT (%u %w %h %w %l %w %t "%r" %s %b)

I have to have the extra formats because for some reason certain log entries
have extra spacing and are recognised as corrupted.

Anyway, is it possible to generate a report based on the %u log variable to
show local and remote usage?

> -----Original Message-----
> From: analog-help-bounces [at] lists
> [mailto:analog-help-bounces [at] lists] On Behalf Of
> Stephen Turner
> Sent: Tuesday, August 12, 2008 5:42 PM
> To: Support for analog web log analyzer
> Subject: Re: [analog-help] Remote and local statistics
>
> 2008/8/12 Brian Clanton <bclanton [at] dupagels>:
> >
> > The local logs are library patrons accessing resources from
> within the
> > library. Proxy requests are patrons accessing the resources from
> > outside the library.
> >
> > I use the User Reports to show the remote patron requests total and
> > the REDIRECTED USER report to show the requests total for
> local users.
> >
> > The problem is that the proxy server (Ezproxy) redirects the local
> > access requests (301) to save on bandwidth and only
> "proxies" the remote requests.
> > Therefore Analog only shows stats for the "successful" proxied
> > requests in the other subsequent reports, namely the REQUEST REPORT.
> >
> > Is there a way to configure Analog to consider all requests, both
> > local and proxy, as legitimate requests so the REQUEST REPORT will
> > show totals for both local and remote patrons as well still have
> > separate reports showing total remote requests (User
> Report) and total
> > local requests (REDIRECTED USER REPORT)for a given time period?
> >
>
> I think I understand now. There is no way to treat 301 as a
> success (without preprocessing the log), but you can look at
> the Redirection Report which gives the same information as
> the Request Report, but for redirects.
>
> --
> Stephen Turner
> +-------------------------------------------------------------
> ----------
> +-
> | TO UNSUBSCRIBE from this list:
> | http://lists.meer.net/mailman/listinfo/analog-help
> |
> | Analog Documentation: http://analog.cx/docs/Readme.html List
> | archives: http://www.analog.cx/docs/mailing.html#listarchives
> | Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
> +-------------------------------------------------------------
> ----------
> +-
>

jeremy at 7simplemachines

Aug 12, 2008, 4:31 PM

Post #9 of 11 (1813 views)
Permalink

RE: Remote and local statistics [In reply to]

Try this:

* Covert 301 request to 200 (you can "inline" the sed command using the UNCOMPRESS command<http://analog.cx/docs/logfile.html#UNCOMPRESS> if you want)
* USER ON
* USERALIAS proxy remote

You should then get something like this:

Users

(Go To: Top<http://proxy.dupagels.lib.il.us/ob_proxyStats.html#Top> | Monthly Report<http://proxy.dupagels.lib.il.us/ob_proxyStats.html#month> | Remote Users | Local Users<http://proxy.dupagels.lib.il.us/ob_proxyStats.html#rediuser> | Databases Accessed<http://proxy.dupagels.lib.il.us/ob_proxyStats.html#req>)

This report lists remote users who requested online resources, if users have been authenticated.

Listing remote users, sorted by the number of requests.
reqs

user

212

remote

186

local

--

Jeremy Wadsack
Seven Simple Machines

________________________________
From: analog-help-bounces [at] lists [mailto:analog-help-bounces [at] lists] On Behalf Of Brian Clanton
Sent: Tuesday, August 12, 2008 3:56 PM
To: 'Support for analog web log analyzer'
Subject: RE: [analog-help] Remote and local statistics

My problem is that the REQUEST REPORT does not include the redirected logs (301).

Libraries want the following information about their online databases:
- How many local users are using the resources
- How many remote users are using the resources
- How much are the resources being used (both local and remote combined)

The REQUEST REPORT does not include the local "redirected" requests. The REDIRECTED REPORT will give a total of "requests" made, but not what databases were accessed.

Example:
http://proxy.dupagels.lib.il.us/ob_proxyStats.html

What about this idea......

What if I change the response codes in the log files from redirected to successful (301 to 200) which will give me all local and proxy requests in the REQUEST REPORT.

Then if I want to differentiate between how many local requests as opposed to proxy requests, I can somhow use the first part of the log entery (proxy or local)

local 66.158.45.2 - - [15/Jul/2008:18:08:00 -0600] "HEAD http://firstsearch.oclc.org/FSIP HTTP/1.1" 301 485

I don't necessarily need a graph for local and proxy users, just how many requests per the designated time period.

The problem is, I'm not sure if there is a report that I can use from the first part of this log entry.

My analog logformat is this:

APACHELOGFORMAT (%u %h %l %w %t "%r" %s %b)
APACHELOGFORMAT (%u %h %w %l %w %t "%r" %s %b)
APACHELOGFORMAT (%u %w %h %w %l %w %t "%r" %s %b)

I have to have the extra formats because for some reason certain log entries have extra spacing and are recognised as corrupted.

Anyway, is it possible to generate a report based on the %u log variable to show local and remote usage?

> -----Original Message-----
> From: analog-help-bounces [at] lists
> [mailto:analog-help-bounces [at] lists] On Behalf Of
> Stephen Turner
> Sent: Tuesday, August 12, 2008 5:42 PM
> To: Support for analog web log analyzer
> Subject: Re: [analog-help] Remote and local statistics
>
> 2008/8/12 Brian Clanton <bclanton [at] dupagels>:
> >
> > The local logs are library patrons accessing resources from
> within the
> > library. Proxy requests are patrons accessing the resources from
> > outside the library.
> >
> > I use the User Reports to show the remote patron requests total and
> > the REDIRECTED USER report to show the requests total for
> local users.
> >
> > The problem is that the proxy server (Ezproxy) redirects the local
> > access requests (301) to save on bandwidth and only
> "proxies" the remote requests.
> > Therefore Analog only shows stats for the "successful" proxied
> > requests in the other subsequent reports, namely the REQUEST REPORT.
> >
> > Is there a way to configure Analog to consider all requests, both
> > local and proxy, as legitimate requests so the REQUEST REPORT will
> > show totals for both local and remote patrons as well still have
> > separate reports showing total remote requests (User
> Report) and total
> > local requests (REDIRECTED USER REPORT)for a given time period?
> >
>
> I think I understand now. There is no way to treat 301 as a
> success (without preprocessing the log), but you can look at
> the Redirection Report which gives the same information as
> the Request Report, but for redirects.
>
> --
> Stephen Turner
> +-------------------------------------------------------------
> ----------
> +-
> | TO UNSUBSCRIBE from this list:
> | http://lists.meer.net/mailman/listinfo/analog-help
> |
> | Analog Documentation: http://analog.cx/docs/Readme.html List
> | archives: http://www.analog.cx/docs/mailing.html#listarchives
> | Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
> +-------------------------------------------------------------
> ----------
> +-
>

bclanton at dupagels

Aug 13, 2008, 9:55 AM

Post #10 of 11 (1778 views)
Permalink

RE: Remote and local statistics [In reply to]

Just want to tip my hat and say thanks all who gave input on this issue.

After just doing the conversion from 301 to 200 response code in the log
file, I didnt' have to do anything else. The User Report, which was already
on, was able to recognise the local and proxy users in the log entry and
generate a graph and my REQUEST REPORT aka DATABASES ACCESSED reflects both
local and remote access. Ya Hoo!

http://proxy.dupagels.lib.il.us/ob_proxyStats.html

Now, all I need to do is run each report for only the previous month.

The current time frame is:
Analysed requests from Tue-15-Jul-2008 18:07 to Wed-13-Aug-2008 10:22 (28.68
days).

which is the begining and end of the log file. I've gone over the docs a
couple of times, but I can't find any commands that alter this time frame.

Is this possible?

_____

From: analog-help-bounces [at] lists
[mailto:analog-help-bounces [at] lists] On Behalf Of Jeremy Wadsack
Sent: Tuesday, August 12, 2008 6:32 PM
To: Support for analog web log analyzer
Subject: RE: [analog-help] Remote and local statistics

Try this:

* Covert 301 request to 200 (you can "inline" the sed command using
the UNCOMPRESS <http://analog.cx/docs/logfile.html#UNCOMPRESS> command if
you want)

* USER ON

* USERALIAS proxy remote

You should then get something like this:

Users

(Go To: Top <http://proxy.dupagels.lib.il.us/ob_proxyStats.html#Top> |
Monthly <http://proxy.dupagels.lib.il.us/ob_proxyStats.html#month> Report |
Remote Users | Local
<http://proxy.dupagels.lib.il.us/ob_proxyStats.html#rediuser> Users |
Databases <http://proxy.dupagels.lib.il.us/ob_proxyStats.html#req>
Accessed)

This report lists remote users who requested online resources, if users have
been authenticated.

Listing remote users, sorted by the number of requests.

reqs

user

212

remote

186

local

--

Jeremy Wadsack

Seven Simple Machines

_____

From: analog-help-bounces [at] lists
[mailto:analog-help-bounces [at] lists] On Behalf Of Brian Clanton
Sent: Tuesday, August 12, 2008 3:56 PM
To: 'Support for analog web log analyzer'
Subject: RE: [analog-help] Remote and local statistics

My problem is that the REQUEST REPORT does not include the redirected logs
(301).

Libraries want the following information about their online databases:
- How many local users are using the resources
- How many remote users are using the resources
- How much are the resources being used (both local and remote combined)

The REQUEST REPORT does not include the local "redirected" requests. The
REDIRECTED REPORT will give a total of "requests" made, but not what
databases were accessed.

Example:
<http://proxy.dupagels.lib.il.us/ob_proxyStats.html>
http://proxy.dupagels.lib.il.us/ob_proxyStats.html

What about this idea......

What if I change the response codes in the log files from redirected to
successful (301 to 200) which will give me all local and proxy requests in
the REQUEST REPORT.

Then if I want to differentiate between how many local requests as opposed
to proxy requests, I can somhow use the first part of the log entery (proxy
or local)

local 66.158.45.2 - - [15/Jul/2008:18:08:00 -0600] "HEAD
<http://firstsearch.oclc.org/FSIP> http://firstsearch.oclc.org/FSIP
HTTP/1.1" 301 485

I don't necessarily need a graph for local and proxy users, just how many
requests per the designated time period.

The problem is, I'm not sure if there is a report that I can use from the
first part of this log entry.

My analog logformat is this:

APACHELOGFORMAT (%u %h %l %w %t "%r" %s %b)
APACHELOGFORMAT (%u %h %w %l %w %t "%r" %s %b)
APACHELOGFORMAT (%u %w %h %w %l %w %t "%r" %s %b)

I have to have the extra formats because for some reason certain log entries
have extra spacing and are recognised as corrupted.

Anyway, is it possible to generate a report based on the %u log variable to
show local and remote usage?

> -----Original Message-----
> From: analog-help-bounces [at] lists
> [ <mailto:analog-help-bounces [at] lists>
mailto:analog-help-bounces [at] lists] On Behalf Of
> Stephen Turner
> Sent: Tuesday, August 12, 2008 5:42 PM
> To: Support for analog web log analyzer
> Subject: Re: [analog-help] Remote and local statistics
>
> 2008/8/12 Brian Clanton <bclanton [at] dupagels>:
> >
> > The local logs are library patrons accessing resources from
> within the
> > library. Proxy requests are patrons accessing the resources from
> > outside the library.
> >
> > I use the User Reports to show the remote patron requests total and
> > the REDIRECTED USER report to show the requests total for
> local users.
> >
> > The problem is that the proxy server (Ezproxy) redirects the local
> > access requests (301) to save on bandwidth and only
> "proxies" the remote requests.
> > Therefore Analog only shows stats for the "successful" proxied
> > requests in the other subsequent reports, namely the REQUEST REPORT.
> >
> > Is there a way to configure Analog to consider all requests, both
> > local and proxy, as legitimate requests so the REQUEST REPORT will
> > show totals for both local and remote patrons as well still have
> > separate reports showing total remote requests (User
> Report) and total
> > local requests (REDIRECTED USER REPORT)for a given time period?
> >
>
> I think I understand now. There is no way to treat 301 as a
> success (without preprocessing the log), but you can look at
> the Redirection Report which gives the same information as
> the Request Report, but for redirects.
>
> --
> Stephen Turner
> +-------------------------------------------------------------
> ----------
> +-
> | TO UNSUBSCRIBE from this list:
> | <http://lists.meer.net/mailman/listinfo/analog-help>
http://lists.meer.net/mailman/listinfo/analog-help
> |
> | Analog Documentation: <http://analog.cx/docs/Readme.html>
http://analog.cx/docs/Readme.html List
> | archives: <http://www.analog.cx/docs/mailing.html#listarchives>
http://www.analog.cx/docs/mailing.html#listarchives
> | Usenet version: <news://news.gmane.org/gmane.comp.web.analog.general>
news://news.gmane.org/gmane.comp.web.analog.general
> +-------------------------------------------------------------
> ----------
> +-
>

jeremy at 7simplemachines

Aug 13, 2008, 10:12 AM

Post #11 of 11 (1779 views)
Permalink

RE: Remote and local statistics [In reply to]

If you have separate log files the most efficient way is to specify to Analog which log files to read for the month - probably through shell commands, but possibly through Analog's %Y and %M operators for the LOGFILE command. See http://analog.cx/docs/logfile.html for details.

If it's one log file or the files overlap month boundaries you can use the FROM and TO commands to limit the time period.

FROM -00-0001
TO -00-0031

See http://analog.cx/docs/include.html#FROMTO for details.

--

Jeremy Wadsack
Seven Simple Machines

________________________________
From: analog-help-bounces [at] lists [mailto:analog-help-bounces [at] lists] On Behalf Of Brian Clanton
Sent: Wednesday, August 13, 2008 9:55 AM
To: 'Support for analog web log analyzer'
Subject: RE: [analog-help] Remote and local statistics

Just want to tip my hat and say thanks all who gave input on this issue.

After just doing the conversion from 301 to 200 response code in the log file, I didnt' have to do anything else. The User Report, which was already on, was able to recognise the local and proxy users in the log entry and generate a graph and my REQUEST REPORT aka DATABASES ACCESSED reflects both local and remote access. Ya Hoo!

http://proxy.dupagels.lib.il.us/ob_proxyStats.html

Now, all I need to do is run each report for only the previous month.

The current time frame is:
Analysed requests from Tue-15-Jul-2008 18:07 to Wed-13-Aug-2008 10:22 (28.68 days).

which is the begining and end of the log file. I've gone over the docs a couple of times, but I can't find any commands that alter this time frame.

Is this possible?

________________________________
From: analog-help-bounces [at] lists [mailto:analog-help-bounces [at] lists] On Behalf Of Jeremy Wadsack
Sent: Tuesday, August 12, 2008 6:32 PM
To: Support for analog web log analyzer
Subject: RE: [analog-help] Remote and local statistics
Try this:

* Covert 301 request to 200 (you can "inline" the sed command using the UNCOMPRESS command<http://analog.cx/docs/logfile.html#UNCOMPRESS> if you want)
* USER ON
* USERALIAS proxy remote

You should then get something like this:

Users

(Go To: Top<http://proxy.dupagels.lib.il.us/ob_proxyStats.html#Top> | Monthly Report<http://proxy.dupagels.lib.il.us/ob_proxyStats.html#month> | Remote Users | Local Users<http://proxy.dupagels.lib.il.us/ob_proxyStats.html#rediuser> | Databases Accessed<http://proxy.dupagels.lib.il.us/ob_proxyStats.html#req>)

This report lists remote users who requested online resources, if users have been authenticated.

Listing remote users, sorted by the number of requests.
reqs

user

212

remote

186

local

--

Jeremy Wadsack
Seven Simple Machines

________________________________
From: analog-help-bounces [at] lists [mailto:analog-help-bounces [at] lists] On Behalf Of Brian Clanton
Sent: Tuesday, August 12, 2008 3:56 PM
To: 'Support for analog web log analyzer'
Subject: RE: [analog-help] Remote and local statistics

My problem is that the REQUEST REPORT does not include the redirected logs (301).

Libraries want the following information about their online databases:
- How many local users are using the resources
- How many remote users are using the resources
- How much are the resources being used (both local and remote combined)

The REQUEST REPORT does not include the local "redirected" requests. The REDIRECTED REPORT will give a total of "requests" made, but not what databases were accessed.

Example:
http://proxy.dupagels.lib.il.us/ob_proxyStats.html

What about this idea......

What if I change the response codes in the log files from redirected to successful (301 to 200) which will give me all local and proxy requests in the REQUEST REPORT.

Then if I want to differentiate between how many local requests as opposed to proxy requests, I can somhow use the first part of the log entery (proxy or local)

local 66.158.45.2 - - [15/Jul/2008:18:08:00 -0600] "HEAD http://firstsearch.oclc.org/FSIP HTTP/1.1" 301 485

I don't necessarily need a graph for local and proxy users, just how many requests per the designated time period.

The problem is, I'm not sure if there is a report that I can use from the first part of this log entry.

My analog logformat is this:

APACHELOGFORMAT (%u %h %l %w %t "%r" %s %b)
APACHELOGFORMAT (%u %h %w %l %w %t "%r" %s %b)
APACHELOGFORMAT (%u %w %h %w %l %w %t "%r" %s %b)

I have to have the extra formats because for some reason certain log entries have extra spacing and are recognised as corrupted.

Anyway, is it possible to generate a report based on the %u log variable to show local and remote usage?

> -----Original Message-----
> From: analog-help-bounces [at] lists
> [mailto:analog-help-bounces [at] lists] On Behalf Of
> Stephen Turner
> Sent: Tuesday, August 12, 2008 5:42 PM
> To: Support for analog web log analyzer
> Subject: Re: [analog-help] Remote and local statistics
>
> 2008/8/12 Brian Clanton <bclanton [at] dupagels>:
> >
> > The local logs are library patrons accessing resources from
> within the
> > library. Proxy requests are patrons accessing the resources from
> > outside the library.
> >
> > I use the User Reports to show the remote patron requests total and
> > the REDIRECTED USER report to show the requests total for
> local users.
> >
> > The problem is that the proxy server (Ezproxy) redirects the local
> > access requests (301) to save on bandwidth and only
> "proxies" the remote requests.
> > Therefore Analog only shows stats for the "successful" proxied
> > requests in the other subsequent reports, namely the REQUEST REPORT.
> >
> > Is there a way to configure Analog to consider all requests, both
> > local and proxy, as legitimate requests so the REQUEST REPORT will
> > show totals for both local and remote patrons as well still have
> > separate reports showing total remote requests (User
> Report) and total
> > local requests (REDIRECTED USER REPORT)for a given time period?
> >
>
> I think I understand now. There is no way to treat 301 as a
> success (without preprocessing the log), but you can look at
> the Redirection Report which gives the same information as
> the Request Report, but for redirects.
>
> --
> Stephen Turner
> +-------------------------------------------------------------
> ----------
> +-
> | TO UNSUBSCRIBE from this list:
> | http://lists.meer.net/mailman/listinfo/analog-help
> |
> | Analog Documentation: http://analog.cx/docs/Readme.html List
> | archives: http://www.analog.cx/docs/mailing.html#listarchives
> | Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
> +-------------------------------------------------------------
> ----------
> +-
>

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads