Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Varnish: Misc

unique request ids

 

 

Varnish misc RSS feed   Index | Next | Previous | View Threaded


mit at stagename

Nov 2, 2011, 12:03 PM

Post #1 of 7 (947 views)
Permalink
unique request ids

Hello!

For many reasons (QA, performance analytics, etc) we log all of our proxy
traffic into a database.

We do this with varnish by collecting a text log via varnishncsa -F and a
CSV formatted string, and regularly run a batch process that inserts this
CSV data into a database.

I was wondering... does varnish internally generate a reliably random
unique value for each request (similar to mod_unique_id in apache or a
uuid/guid), and if so is there a way that i can write it out to my logs,
perhaps via an insertion into a request header and using the %{HEADER}i
macro in varnishncsa?

-Mit


straightflush at gmail

Nov 2, 2011, 12:54 PM

Post #2 of 7 (917 views)
Permalink
Re: unique request ids [In reply to]

we do something similar. My suggestion,

md5(logentry + currentTimeInMilliseconds) as your key as you loop through
the logfile



On Wed, Nov 2, 2011 at 3:03 PM, Mit Rowe <mit [at] stagename> wrote:

> Hello!
>
> For many reasons (QA, performance analytics, etc) we log all of our proxy
> traffic into a database.
>
> We do this with varnish by collecting a text log via varnishncsa -F and a
> CSV formatted string, and regularly run a batch process that inserts this
> CSV data into a database.
>
> I was wondering... does varnish internally generate a reliably random
> unique value for each request (similar to mod_unique_id in apache or a
> uuid/guid), and if so is there a way that i can write it out to my logs,
> perhaps via an insertion into a request header and using the %{HEADER}i
> macro in varnishncsa?
>
> -Mit
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc [at] varnish-cache
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>


mit at stagename

Nov 2, 2011, 1:25 PM

Post #3 of 7 (923 views)
Permalink
Re: unique request ids [In reply to]

for our purposes, it would have to be done at the point of log writing
(primarily, but not limited to, the need to ensure that if for whatever
reason we have to re-process a log file that we can quickly identify lines
already written to the db via a unique index).



On Wed, Nov 2, 2011 at 3:54 PM, AD <straightflush [at] gmail> wrote:

> we do something similar. My suggestion,
>
> md5(logentry + currentTimeInMilliseconds) as your key as you loop through
> the logfile
>
>
>
> On Wed, Nov 2, 2011 at 3:03 PM, Mit Rowe <mit [at] stagename> wrote:
>
>> Hello!
>>
>> For many reasons (QA, performance analytics, etc) we log all of our proxy
>> traffic into a database.
>>
>> We do this with varnish by collecting a text log via varnishncsa -F and a
>> CSV formatted string, and regularly run a batch process that inserts this
>> CSV data into a database.
>>
>> I was wondering... does varnish internally generate a reliably random
>> unique value for each request (similar to mod_unique_id in apache or a
>> uuid/guid), and if so is there a way that i can write it out to my logs,
>> perhaps via an insertion into a request header and using the %{HEADER}i
>> macro in varnishncsa?
>>
>> -Mit
>>
>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc [at] varnish-cache
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
>


--
Will 'Mit' Rowe
Stagename*
*1-866-326-3098
mit [at] stagename <josh [at] stagename>
www.stagename.com
Twitter: @stagename

*The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of this
information by persons or entities other than the intended recipient is
prohibited. If you received this transmission in error, please contact the
sender and delete all material contained herein from your computer.*


perbu at varnish-software

Nov 2, 2011, 1:27 PM

Post #4 of 7 (918 views)
Permalink
Re: unique request ids [In reply to]

Hi Mit,

On Wed, Nov 2, 2011 at 8:03 PM, Mit Rowe <mit [at] stagename> wrote:

>
> I was wondering... does varnish internally generate a reliably random
> unique value for each request (similar to mod_unique_id in apache or a
> uuid/guid), and if so is there a way that i can write it out to my logs,
> perhaps via an insertion into a request header and using the %{HEADER}i
> macro in varnishncsa?
>

There is a request ID, in Varnishlog you'll see it as the last number in
the ReqStart lines ie.

1932 ReqStart c 127.0.0.1 54471 1236808814

You could probably extract it from the X-Varnish header, it will be the
first of one or two numbers in that header.

--
Per Buer, CEO
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>


mit at stagename

Nov 2, 2011, 1:43 PM

Post #5 of 7 (918 views)
Permalink
Re: unique request ids [In reply to]

Hi Per,

That's looks promising, i could save the X-Varnish header in the logs and
during the import strip off the second number.

Is that a sequential number or a random number?
What is its range?





On Wed, Nov 2, 2011 at 4:27 PM, Per Buer <perbu [at] varnish-software> wrote:

> Hi Mit,
>
> On Wed, Nov 2, 2011 at 8:03 PM, Mit Rowe <mit [at] stagename> wrote:
>
>>
>> I was wondering... does varnish internally generate a reliably random
>> unique value for each request (similar to mod_unique_id in apache or a
>> uuid/guid), and if so is there a way that i can write it out to my logs,
>> perhaps via an insertion into a request header and using the %{HEADER}i
>> macro in varnishncsa?
>>
>
> There is a request ID, in Varnishlog you'll see it as the last number in
> the ReqStart lines ie.
>
> 1932 ReqStart c 127.0.0.1 54471 1236808814
>
> You could probably extract it from the X-Varnish header, it will be the
> first of one or two numbers in that header.
>
> --
> Per Buer, CEO
> Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
> *Varnish makes websites fly!*
> Whitepapers <http://www.varnish-software.com/whitepapers> | Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
> Twitter <https://twitter.com/varnishsoftware>
>
>
>


--
Will 'Mit' Rowe
Stagename*
*1-866-326-3098
mit [at] stagename <josh [at] stagename>
www.stagename.com
Twitter: @stagename

*The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of this
information by persons or entities other than the intended recipient is
prohibited. If you received this transmission in error, please contact the
sender and delete all material contained herein from your computer.*


perbu at varnish-software

Nov 2, 2011, 2:05 PM

Post #6 of 7 (934 views)
Permalink
Re: unique request ids [In reply to]

On Wed, Nov 2, 2011 at 9:43 PM, Mit Rowe <mit [at] stagename> wrote:

> Hi Per,
>
> That's looks promising, i could save the X-Varnish header in the logs and
> during the import strip off the second number.
>
> Is that a sequential number or a random number?
>

sequential.


> What is its range?
>

I'm guessing it is a 64bit int that would be unique for that instance of
Varnish. i.e. all bets are off if you restart or are running Varnish on
multiple servers. You might want to have a look at the source - look for
xid. Not sure where it is actually constructed.


--
Per Buer, CEO
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>


tfheen at varnish-software

Nov 3, 2011, 5:34 AM

Post #7 of 7 (912 views)
Permalink
Re: unique request ids [In reply to]

]] Per Buer

| > What is its range?
|
| I'm guessing it is a 64bit int that would be unique for that instance of
| Varnish. i.e. all bets are off if you restart or are running Varnish on
| multiple servers. You might want to have a look at the source - look for
| xid. Not sure where it is actually constructed.

It's seeded as a random number and then increased (albeit unlocked) so
you can end up with duplicates.

It's also a 32 bit number, so it will eventually wrap (about 10 days
wrap time if you average 5kr/s).

Cheers,
--
Tollef Fog Heen
Varnish Software
t: +47 21 98 92 64

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Varnish misc RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.