Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Analog: Help

multi-line log format

 

 

Analog help RSS feed   Index | Next | Previous | View Threaded


rinehart at brb

Feb 17, 1999, 7:00 AM

Post #1 of 6 (172 views)
Permalink
multi-line log format

I have a FileMaker Pro database published on our agency web site, and have
not been able to configure Analog to read its log. I think the main problem
is that each "hit" spreads out over 3 or 4 lines, depending on whether it
includes just a request for a file or image, or includes a database search
request.

If the FMPro database is set to brief logging, LogTrans will format it so
that Analog can read it. However, we lose too much valuable information by
using brief log format, so that is not an option. (I've thoroughly
questioned everyone on the FMP/Web Publishing list about this, search all
applicable mailing list archives, and no one seems to have been able to get
Analog to read these log files.)

In the analog documentation, NetPresenz is acknowledged as having entries
spread over several lines, but it includes the host name on each line.
FileMaker Pro does not, and Analog definitely does not like that.

Two sample hits from the FMP log look like this:

2/1/99 11:01:48 AM 204.65.5.15 /schools/searchschool.htm
2/1/99 11:01:48 AM /schools/searchschool.htm
2/1/99 11:01:48 AM Sending reply. Elapsed time: 48 millisec.

2/1/99 11:01:59 AM 204.65.5.15 /schools/FMPro
2/1/99 11:01:59 AM
-db=MainSchool.fp3&-lay=DebtRatios&-error=errors.htm&-op=contains&ISD_Name=a
bilene&-op=equals&County=&-max=10&-format=formats.htm&-find=Start+Search
2/1/99 11:01:59 AM Sending reply. Elapsed time: 77 millisec.

Each hit starts out with the line including the host IP address, and ends
with the line including Sending Reply and Elapsed time.

The log formats I've tried using include:
LOGFORMAT (%m/%d/%y %h:%n:%j %a %S %r)
LOGFORMAT (%m/%d/%y %h:%n:%j %a %r)
LOGFORMAT (%m/%d/%y %h:%n:%j %a %j %j %j)
LOGFORMAT (%m/%d/%y %h:%n:%j %a %r)
LOGFORMAT (%m/%d/%y %h:%n:%j %a )
LOGFORMAT (%m/%d/%y %h:%n:%j %a %j %j %w %j %j %j %j)

- and -

DEFAULTLOGFORMAT (%m/%d/%y %h:%n:%j %a %S %r)
DEFAULTLOGFORMAT (%m/%d/%y %h:%n:%j %a %r)
DEFAULTLOGFORMAT (%m/%d/%y %h:%n:%j %a Got search argument.)
DEFAULTLOGFORMAT (%m/%d/%y %h:%n:%j %a %r)
DEFAULTLOGFORMAT (%m/%d/%y %h:%n:%j %a )
DEFAULTLOGFORMAT (%m/%d/%y %h:%n:%j %a Sending reply. %w Elapsed time: %j
millisec.)

The FileMaker Pro Web Development community will be much obliged if we can
come up with a solution for this. Thanks in advance for your help.

Ruth Rinehart
Texas Bond Review Board Web Team
http://204.65.5.2:590


--------------------------------------------------------------------
This is the analog-help mailing list. To unsubscribe from this
mailing list, send mail to analog-help-request [at] lists
with "unsubscribe analog-help" in the main BODY OF THE MESSAGE.
--------------------------------------------------------------------


jason at summary

Feb 17, 1999, 1:20 PM

Post #2 of 6 (168 views)
Permalink
multi-line log format [In reply to]

On 2/17/99 10:00 AM Ruth Rinehart (rinehart [at] brb) wrote:

>Two sample hits from the FMP log look like this:
>
>2/1/99 11:01:48 AM 204.65.5.15 /schools/searchschool.htm
>2/1/99 11:01:48 AM /schools/searchschool.htm
>2/1/99 11:01:48 AM Sending reply. Elapsed time: 48 millisec.
>
>2/1/99 11:01:59 AM 204.65.5.15 /schools/FMPro
>2/1/99 11:01:59 AM
>-db=MainSchool.fp3&-lay=DebtRatios&-error=errors.htm&-op=contains&ISD_Name=a
>bilene&-op=equals&County=&-max=10&-format=formats.htm&-find=Start+Search
>2/1/99 11:01:59 AM Sending reply. Elapsed time: 77 millisec.
>
>Each hit starts out with the line including the host IP address, and ends
>with the line including Sending Reply and Elapsed time.
>
>The log formats I've tried using include:
>LOGFORMAT (%m/%d/%y %h:%n:%j %a %S %r)
>LOGFORMAT (%m/%d/%y %h:%n:%j %a %r)
>LOGFORMAT (%m/%d/%y %h:%n:%j %a %j %j %j)
>LOGFORMAT (%m/%d/%y %h:%n:%j %a %r)
>LOGFORMAT (%m/%d/%y %h:%n:%j %a )
>LOGFORMAT (%m/%d/%y %h:%n:%j %a %j %j %w %j %j %j %j)

I can give you a hint but not the complete solution. You need to use \n's
in the LOGFORMAT command so that the entire multi-line entity is read by
a single LOGFORMAT. They might go something like:

LOGFORMAT (%m/%d/%y %h:%n:%j %a %S %r\n%j/%j/%j %j:%j:%j %j %r\n%jElapsed
%j)

You will need a line like that for every possible configuration of lines
in the log file. Putting the "Elapsed " in makes sure it only is used if
it can match the end of transaction exactly, too many %j's and you can
get out of sync.

Good Luck
Jason

-----------------
Jason [at] Summary
-----------------
Dr. Seuss books . . . can be read and enjoyed on several levels. For
example, 'One Fish Two Fish, Red Fish Blue Fish' can be deconstructed
as a searing indictment of the narrow-minded binary counting system.
-- Peter van der Linden, Expert C Programming, Deep C Secrets


--------------------------------------------------------------------
This is the analog-help mailing list. To unsubscribe from this
mailing list, send mail to analog-help-request [at] lists
with "unsubscribe analog-help" in the main BODY OF THE MESSAGE.
--------------------------------------------------------------------


rinehart at brb

Feb 25, 1999, 1:18 AM

Post #3 of 6 (169 views)
Permalink
multi-line log format [In reply to]

Back on the Filemaker Pro multi-line log format: Using the following
LOGFORMAT, Analog processes with no errors except corrupt lines, with no
output in the report. These logformat lines are long, because FMP produces
either 3 or 4 lines of log to each hit.

Is there too much %j here for Analog to handle?

Another possibility: I am naming the database search argument as a file,
because it contains all the pertinent data that we want to capture. Does
Analog have a hard time recognizing this long query string as a file? I
thought about using %q for query, but there is no "?" in the string.

Any help on this matter would benefit the Filemaker Pro Web Development
community greatly!

LOGFORMAT (%m/%d/%y %h:%n:%j %a %S %r\n%j/%j/%j %j:%j:%j %j %j\n%j/%j/%j
%j:%j:%j %j Sending reply. %w Elapsed time: %j millisec.)
LOGFORMAT (%m/%d/%y %h:%n:%j %a %S %j\n%j/%j/%j %j:%j:%j %j Got search
argument.\n%j/%j/%j %j:%j:%j %j %r\n%j/%j/%j %j:%j:%j %j Sending reply. %w
Elapsed time: %j millisec.)

----------
>From: Jason Linhart <jason [at] summary>

>On 2/17/99 10:00 AM Ruth Rinehart (rinehart [at] brb) wrote:
>
>>Two sample hits from the FMP log look like this:
>>
>>2/1/99 11:01:48 AM 204.65.5.15 /schools/searchschool.htm
>>2/1/99 11:01:48 AM /schools/searchschool.htm
>>2/1/99 11:01:48 AM Sending reply. Elapsed time: 48 millisec.
>>
>>2/1/99 11:01:59 AM 204.65.5.15 /schools/FMPro
>>2/1/99 11:01:59 AM Got search argument.
>>2/1/99 11:01:59 AM
>>-db=MainSchool.fp3&-lay=DebtRatios&-error=errors.htm&-op=contains&ISD_Name
=a
>>bilene&-op=equals&County=&-max=10&-format=formats.htm&-find=Start+Search
>>2/1/99 11:01:59 AM Sending reply. Elapsed time: 77 millisec.
>>
>>Each hit starts out with the line including the host IP address, and ends
>>with the line including Sending Reply and Elapsed time.
>>
>>The log formats I've tried using include:
>>LOGFORMAT (%m/%d/%y %h:%n:%j %a %S %r)
>>LOGFORMAT (%m/%d/%y %h:%n:%j %a %r)
>>LOGFORMAT (%m/%d/%y %h:%n:%j %a %j %j %j)
>>LOGFORMAT (%m/%d/%y %h:%n:%j %a %r)
>>LOGFORMAT (%m/%d/%y %h:%n:%j %a )
>>LOGFORMAT (%m/%d/%y %h:%n:%j %a %j %j %w %j %j %j %j)
>
>I can give you a hint but not the complete solution. You need to use \n's
>in the LOGFORMAT command so that the entire multi-line entity is read by
>a single LOGFORMAT. They might go something like:
>
>LOGFORMAT (%m/%d/%y %h:%n:%j %a %S %r\n%j/%j/%j %j:%j:%j %j %r\n%jElapsed
>%j)
>
>You will need a line like that for every possible configuration of lines
>in the log file. Putting the "Elapsed " in makes sure it only is used if
>it can match the end of transaction exactly, too many %j's and you can
>get out of sync.
>
>Good Luck
>Jason


--------------------------------------------------------------------
This is the analog-help mailing list. To unsubscribe from this
mailing list, send mail to analog-help-request [at] lists
with "unsubscribe analog-help" in the main BODY OF THE MESSAGE.
--------------------------------------------------------------------


jason at summary

Feb 26, 1999, 10:53 PM

Post #4 of 6 (167 views)
Permalink
multi-line log format [In reply to]

I have some suggested improvements. I don't have enough examples of
FileMaker Pro logs to be sure of anything, but I can improve on your
attempts. Try:

LOGFORMAT (%m/%d/%y %h:%n:%j %a %S %r\n%j/%j/%j %j:%j:%j %j %j\n%j/%j/%j
%j:%j:%j %jElapsed%j)
LOGFORMAT (%m/%d/%y %h:%n:%j %a %S %r\n%j/%j/%j %j:%j:%j %jGot
search%j\n%j/%j/%j %j:%j:%j %j %q\n%j/%j/%j %j:%j:%j %jSending reply%j)

Changes:

In the second format, the request (%r) is still on the first line, it is
the query string (%q) on the third line.

You can have %j match more that just one token. A %j will continue until
it gets to the character immediately following it. So %j) will match
anything to the end of the line. I tried to reduce the amount of literal
matching and let the %j's eat a little more. You want enough literal
matching to be sure of the format but not so much that minor changes or
mistyping ruin everything.

When debugging formats you need to be careful of the case where you don't
match anything. You don't make it clear if all of the lines were corrupt
or if only some of them were and there just wasn't anything in the
report. The formats could be just plain wrong and not matching anything.
Also keep in mind that you may need to duplicate each format for a
DEFAULTLOGFORMAT command in addition to the LOGFORMAT command to support
command line (drag and drop on the Mac) log files.

If every line is corrupt, it may be that we are missing some important
field. I believe Analog requires a status code field be present. Did you
look closely at your Analog error messages? It would say if it was
ignoring these LOGFORMAT lines for some reason. I wonder where we can get
a status code out of this?

Good Luck
Jason


On 2/25/99 4:18 AM Ruth Rinehart (rinehart [at] brb) wrote:

>Back on the Filemaker Pro multi-line log format: Using the following
>LOGFORMAT, Analog processes with no errors except corrupt lines, with no
>output in the report. These logformat lines are long, because FMP produces
>either 3 or 4 lines of log to each hit.
>
>Is there too much %j here for Analog to handle?
>
>Another possibility: I am naming the database search argument as a file,
>because it contains all the pertinent data that we want to capture. Does
>Analog have a hard time recognizing this long query string as a file? I
>thought about using %q for query, but there is no "?" in the string.
>
>Any help on this matter would benefit the Filemaker Pro Web Development
>community greatly!
>
>LOGFORMAT (%m/%d/%y %h:%n:%j %a %S %r\n%j/%j/%j %j:%j:%j %j %j\n%j/%j/%j
>%j:%j:%j %j Sending reply. %w Elapsed time: %j millisec.)
>LOGFORMAT (%m/%d/%y %h:%n:%j %a %S %j\n%j/%j/%j %j:%j:%j %j Got search
>argument.\n%j/%j/%j %j:%j:%j %j %r\n%j/%j/%j %j:%j:%j %j Sending reply. %w
>Elapsed time: %j millisec.)
>
>----------
>>From: Jason Linhart <jason [at] summary>
>
>>On 2/17/99 10:00 AM Ruth Rinehart (rinehart [at] brb) wrote:
>>
>>>Two sample hits from the FMP log look like this:
>>>
>>>2/1/99 11:01:48 AM 204.65.5.15 /schools/searchschool.htm
>>>2/1/99 11:01:48 AM /schools/searchschool.htm
>>>2/1/99 11:01:48 AM Sending reply. Elapsed time: 48 millisec.
>>>
>>>2/1/99 11:01:59 AM 204.65.5.15 /schools/FMPro
>>>2/1/99 11:01:59 AM Got search argument.
>>>2/1/99 11:01:59 AM
>>>-db=MainSchool.fp3&-lay=DebtRatios&-error=errors.htm&-op=contains&ISD_Name
>=a
>>>bilene&-op=equals&County=&-max=10&-format=formats.htm&-find=Start+Search
>>>2/1/99 11:01:59 AM Sending reply. Elapsed time: 77 millisec.
>>>
>>>Each hit starts out with the line including the host IP address, and ends
>>>with the line including Sending Reply and Elapsed time.
>>>
>>>The log formats I've tried using include:
>>>LOGFORMAT (%m/%d/%y %h:%n:%j %a %S %r)
>>>LOGFORMAT (%m/%d/%y %h:%n:%j %a %r)
>>>LOGFORMAT (%m/%d/%y %h:%n:%j %a %j %j %j)
>>>LOGFORMAT (%m/%d/%y %h:%n:%j %a %r)
>>>LOGFORMAT (%m/%d/%y %h:%n:%j %a )
>>>LOGFORMAT (%m/%d/%y %h:%n:%j %a %j %j %w %j %j %j %j)
>>
>>I can give you a hint but not the complete solution. You need to use \n's
>>in the LOGFORMAT command so that the entire multi-line entity is read by
>>a single LOGFORMAT. They might go something like:
>>
>>LOGFORMAT (%m/%d/%y %h:%n:%j %a %S %r\n%j/%j/%j %j:%j:%j %j %r\n%jElapsed
>>%j)
>>
>>You will need a line like that for every possible configuration of lines
>>in the log file. Putting the "Elapsed " in makes sure it only is used if
>>it can match the end of transaction exactly, too many %j's and you can
>>get out of sync.


-----------------
Jason [at] Summary
-----------------
Dr. Seuss books . . . can be read and enjoyed on several levels. For
example, 'One Fish Two Fish, Red Fish Blue Fish' can be deconstructed
as a searing indictment of the narrow-minded binary counting system.
-- Peter van der Linden, Expert C Programming, Deep C Secrets


--------------------------------------------------------------------
This is the analog-help mailing list. To unsubscribe from this
mailing list, send mail to analog-help-request [at] lists
with "unsubscribe analog-help" in the main BODY OF THE MESSAGE.
--------------------------------------------------------------------


jfoley at sk

Feb 27, 1999, 2:24 AM

Post #5 of 6 (169 views)
Permalink
multi-line log format [In reply to]

Subject: Re: [analog-help] multi-line log format
Date sent: Sat, 27 Feb 1999 01:53:29 -0500
From: Jason Linhart <jason [at] summary>
To: <analog-help [at] lists>
Send reply to: analog-help [at] lists

> If every line is corrupt, it may be that we are missing some important
> field. I believe Analog requires a status code field be present. Did you
> look closely at your Analog error messages? It would say if it was

Actually it doesn't require a status code. I am using a log file
created by a script, for when my host server has days missing. It
has no status code and no bytes field. I do get a warning about the
bytes field not being present and the log file has to be
preprocessed, but it does work.

Take Care


Jim Foley
The Information Wizard
http://momp.hypermart.net/
jfoley [at] momp
--------------------------------------------------------------------
This is the analog-help mailing list. To unsubscribe from this
mailing list, send mail to analog-help-request [at] lists
with "unsubscribe analog-help" in the main BODY OF THE MESSAGE.
--------------------------------------------------------------------


rinehart at brb

Mar 1, 1999, 1:16 AM

Post #6 of 6 (170 views)
Permalink
multi-line log format [In reply to]

I've stripped my log file so that each hit now only takes 2 lines, instead
of the inconsistent 3 or 4 lines previously. The new log file hits look like
this:

2/1/99 11:01:47 AM 204.65.5.15 /schools/images/check.gif
2/1/99 11:01:47 AM /schools/images/check.gif
2/1/99 11:01:48 AM 204.65.5.15 /schools/images/powered_by_fmp.gif
2/1/99 11:01:48 AM /schools/images/powered_by_fmp.gif
2/1/99 11:01:48 AM 204.65.5.15 /schools/images/bar.gif
2/1/99 11:01:48 AM /schools/images/bar.gif
2/1/99 11:01:59 AM 204.65.5.15 /schools/FMPro
2/1/99 11:01:59 AM
-db=MainSchool.fp3&-lay=DebtRatios&-error=errors.htm&-op=contains&ISD_Name=a
bilene&-op=equals&County=&-max=10&-format=formats.htm&-find=Start+Search
2/1/99 11:03:12 AM 204.65.5.15 /schools/FMPro
2/1/99 11:03:12 AM
-db=MainSchool.fp3&-lay=DebtRatios&-error=errors.htm&-op=contains&ISD_Name=a
bilene&-op=equals&County=&-max=10&-format=formats.htm&-find=Start+Search


Each hit takes only 2 lines, and I can identify the filename after the IP
address as junk, because when it isn't a query string, the filename is
repeated on the 2nd line.

HERE'S A QUESTION THAT I HAVEN'T SEEN AN ANSWER TO: Is the database query
string
too much for Analog to handle? Can Analog recognize the FileMaker Pro query
string as a request (%r)?

Jason, I appreciate your help here. I got this same basic error processing
message using your most recent suggestions (but below example is my latest
shot at it, with just 2 lines of log to process):

****************************
Processing...
analog: Warning M: Logfile %%% contains lines with no referrers, which are
being filtered
(For help on all errors and warnings, see docs/errors.html)
analog: Warning M: Logfile %%% contains lines with no bytes: byte counts may
be low
analog: Warning L: Large number of corrupt lines in logfile %%%: try
different
LOGFORMAT
Current logfile format:
%m/%d/%y %h:%n:%j %a %S %j\n%j/%j/%j %j:%j:%j %j %r\n
analog: Warning R: Turning off empty time reports
analog: Warning R: Turning off empty Request Report
analog: Warning R: Turning off empty Failure Report
analog: Warning R: Turning off empty Host Report
analog: Warning R: Turning off empty Browser Report
analog: Warning R: Turning off empty Browser Summary
Complete!
************************************




--------------------------------------------------------------------
This is the analog-help mailing list. To unsubscribe from this
mailing list, send mail to analog-help-request [at] lists
with "unsubscribe analog-help" in the main BODY OF THE MESSAGE.
--------------------------------------------------------------------

Analog help RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.