Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Varnish: Bugs

#501: Stats variable to monitor how many threads are actually doing something (n_wrk_busy): busy threads monitoring

 

 

Varnish bugs RSS feed   Index | Next | Previous | View Threaded


varnish-bugs at projects

Apr 28, 2009, 8:25 PM

Post #1 of 8 (279 views)
Permalink
#501: Stats variable to monitor how many threads are actually doing something (n_wrk_busy): busy threads monitoring

#501: Stats variable to monitor how many threads are actually doing something
(n_wrk_busy): busy threads monitoring
-------------------------+--------------------------------------------------
Reporter: stockrt | Owner: phk
Type: enhancement | Status: new
Priority: normal | Milestone:
Component: varnishd | Version: 2.0
Severity: normal | Keywords: busy threads monitoring
-------------------------+--------------------------------------------------
Hi!


I made this patch for implementing a new varnishstat variable, called
'n_wrk_busy'.


With this new variable we can now know for sure how many of our active
threads are actually doing something, so the system can be better
dimensioned in order to know if we are getting near the max threads
utilization: if we are, we can increase the number of max threads in order
to instatiate more, if the system can handle more threads, if not, go for
more hardware.


This is good in the cases when we configure thread_pool_min ==
thread_pool_max and do not have to know how many of the threads are
actually used for something.
I believe the same applies when we have thread_pool_min == 1 and
thread_pool_max == MAX_WISHED. In this case we can know how many threads
are idle (waiting for thread_pool_timeout to expire) and how many are
actually working.


This is good for monitoring and I hope this can help someone else to know
what is happening under the hood a little better.


Best regards,

Rog?rio Schneider

--
Ticket URL: <http://varnish.projects.linpro.no/ticket/501>
Varnish <http://varnish.projects.linpro.no/>
The Varnish HTTP Accelerator


varnish-bugs at projects

Apr 28, 2009, 8:49 PM

Post #2 of 8 (275 views)
Permalink
#501: Stats variable to monitor how many threads are actually doing something (n_wrk_busy): busy threads monitoring [In reply to]

#501: Stats variable to monitor how many threads are actually doing something
(n_wrk_busy): busy threads monitoring
-------------------------------------+--------------------------------------
Reporter: stockrt | Owner: phk
Type: enhancement | Status: new
Priority: normal | Milestone:
Component: varnishd | Version: 2.0
Severity: normal | Resolution:
Keywords: busy threads monitoring |
-------------------------------------+--------------------------------------
Comment (by stockrt):

I am not sure if this initialisation of 'n_wrk_busy = 0' is required. I
can see that in r4040 (post 2.0.4 release) when we declare a new stats
variable, there is a field that seems to be the initial value. This does
not exist for 2.0.4 release.

So, this needs checks in my patches:

VSL_stats->n_wrk_busy = 0;

I also can see that sometimes variables are not initialised in 2.0.4
release, but I prefer to publish my code the way I place it here,
initialising this var. If you thing it is better not, please remove these
extra lines.

Regards,

Rog?rio Schneider

--
Ticket URL: <http://varnish.projects.linpro.no/ticket/501#comment:1>
Varnish <http://varnish.projects.linpro.no/>
The Varnish HTTP Accelerator


varnish-bugs at projects

Apr 29, 2009, 2:51 AM

Post #3 of 8 (271 views)
Permalink
#501: Stats variable to monitor how many threads are actually doing something (n_wrk_busy): busy threads monitoring [In reply to]

#501: Stats variable to monitor how many threads are actually doing something
(n_wrk_busy): busy threads monitoring
-------------------------------------+--------------------------------------
Reporter: stockrt | Owner: phk
Type: enhancement | Status: new
Priority: normal | Milestone:
Component: varnishd | Version: 2.0
Severity: normal | Resolution:
Keywords: busy threads monitoring |
-------------------------------------+--------------------------------------
Comment (by phk):

The reason we do not have that variable presently, is that it will quite
soon become useless if it is not properly locked.

Properly locking it is not trivial.

--
Ticket URL: <http://varnish.projects.linpro.no/ticket/501#comment:2>
Varnish <http://varnish.projects.linpro.no/>
The Varnish HTTP Accelerator


varnish-bugs at projects

Apr 29, 2009, 10:42 AM

Post #4 of 8 (271 views)
Permalink
#501: Stats variable to monitor how many threads are actually doing something (n_wrk_busy): busy threads monitoring [In reply to]

#501: Stats variable to monitor how many threads are actually doing something
(n_wrk_busy): busy threads monitoring
-------------------------------------+--------------------------------------
Reporter: stockrt | Owner: phk
Type: enhancement | Status: new
Priority: normal | Milestone:
Component: varnishd | Version: 2.0
Severity: normal | Resolution:
Keywords: busy threads monitoring |
-------------------------------------+--------------------------------------
Comment (by stockrt):

Replying to [comment:2 phk]:
> The reason we do not have that variable presently, is that it will quite
soon become useless if it is not properly locked.
>
> Properly locking it is not trivial.
----
Yep, I just saw it with more load.

Anyway, I am attaching another version, that, I believe, should be less
error prone, but yet should suffer for concurrency problems.

If this is not good yet (need more testing) we can forget it for a while.

Thanks,

Rog?rio Schneider

--
Ticket URL: <http://varnish.projects.linpro.no/ticket/501#comment:3>
Varnish <http://varnish.projects.linpro.no/>
The Varnish HTTP Accelerator


varnish-bugs at projects

May 2, 2009, 10:45 AM

Post #5 of 8 (281 views)
Permalink
#501: Stats variable to monitor how many threads are actually doing something (n_wrk_busy): busy threads monitoring [In reply to]

#501: Stats variable to monitor how many threads are actually doing something
(n_wrk_busy): busy threads monitoring
-------------------------------------+--------------------------------------
Reporter: stockrt | Owner: phk
Type: enhancement | Status: new
Priority: normal | Milestone:
Component: varnishd | Version: 2.0
Severity: normal | Resolution:
Keywords: busy threads monitoring |
-------------------------------------+--------------------------------------
Comment (by stockrt):

This new patch I uploaded seems to be very consistent. Could you take a
look at it?

Cheers,

Rog?rio Schneider

--
Ticket URL: <http://varnish.projects.linpro.no/ticket/501#comment:4>
Varnish <http://varnish.projects.linpro.no/>
The Varnish HTTP Accelerator


varnish-bugs at varnish-cache

Mar 7, 2011, 4:21 AM

Post #6 of 8 (274 views)
Permalink
Re: #501: Stats variable to monitor how many threads are actually doing something (n_wrk_busy): busy threads monitoring [In reply to]

#501: Stats variable to monitor how many threads are actually doing something
(n_wrk_busy): busy threads monitoring
-------------------------+--------------------------------------------------
Reporter: stockrt | Owner: phk
Type: enhancement | Status: new
Priority: normal | Milestone: Varnish 3.0 dev
Component: varnishd | Version: trunk
Severity: normal | Keywords: busy threads monitoring
-------------------------+--------------------------------------------------
Changes (by kristian):

* version: 2.0 => trunk
* milestone: => Varnish 3.0 dev


Old description:

> Hi!
>

> I made this patch for implementing a new varnishstat variable, called
> 'n_wrk_busy'.
>

> With this new variable we can now know for sure how many of our active
> threads are actually doing something, so the system can be better
> dimensioned in order to know if we are getting near the max threads
> utilization: if we are, we can increase the number of max threads in
> order to instatiate more, if the system can handle more threads, if not,
> go for more hardware.
>

> This is good in the cases when we configure thread_pool_min ==
> thread_pool_max and do not have to know how many of the threads are
> actually used for something.
> I believe the same applies when we have thread_pool_min == 1 and
> thread_pool_max == MAX_WISHED. In this case we can know how many threads
> are idle (waiting for thread_pool_timeout to expire) and how many are
> actually working.
>

> This is good for monitoring and I hope this can help someone else to know
> what is happening under the hood a little better.
>

> Best regards,
>
> Rogério Schneider

New description:

Hi!


I made this patch for implementing a new varnishstat variable, called
'n_wrk_busy'.


With this new variable we can now know for sure how many of our active
threads are actually doing something, so the system can be better
dimensioned in order to know if we are getting near the max threads
utilization: if we are, we can increase the number of max threads in order
to instatiate more, if the system can handle more threads, if not, go for
more hardware.


This is good in the cases when we configure thread_pool_min ==
thread_pool_max and do not have to know how many of the threads are
actually used for something.
I believe the same applies when we have thread_pool_min == 1 and
thread_pool_max == MAX_WISHED. In this case we can know how many threads
are idle (waiting for thread_pool_timeout to expire) and how many are
actually working.


This is good for monitoring and I hope this can help someone else to know
what is happening under the hood a little better.


Best regards,

Rogério Schneider

--

Comment:

This ties in with proposed scoreboard-functionality, but is not going to
make it for 3.0.

Sorry for the horrendous response-time.... Ugh.

--
Ticket URL: <http://www.varnish-cache.org/trac/ticket/501#comment:5>
Varnish <http://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs [at] varnish-cache
http://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs


varnish-bugs at varnish-cache

Mar 7, 2011, 4:21 AM

Post #7 of 8 (274 views)
Permalink
Re: #501: Stats variable to monitor how many threads are actually doing something (n_wrk_busy): busy threads monitoring [In reply to]

#501: Stats variable to monitor how many threads are actually doing something
(n_wrk_busy): busy threads monitoring
-------------------------+--------------------------------------------------
Reporter: stockrt | Owner: phk
Type: enhancement | Status: new
Priority: normal | Milestone: Later
Component: varnishd | Version: trunk
Severity: normal | Keywords: busy threads monitoring
-------------------------+--------------------------------------------------
Changes (by kristian):

* milestone: Varnish 3.0 dev => Later


--
Ticket URL: <http://varnish-cache.org/trac/ticket/501#comment:6>
Varnish <http://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs [at] varnish-cache
http://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs


varnish-bugs at varnish-cache

Aug 15, 2011, 4:10 AM

Post #8 of 8 (213 views)
Permalink
Re: #501: Stats variable to monitor how many threads are actually doing something (n_wrk_busy): busy threads monitoring [In reply to]

#501: Stats variable to monitor how many threads are actually doing something
(n_wrk_busy): busy threads monitoring
-------------------------------------+--------------------------------------
Reporter: stockrt | Owner: phk
Type: enhancement | Status: closed
Priority: normal | Milestone: Later
Component: varnishd | Version: trunk
Severity: normal | Resolution: worksforme
Keywords: busy threads monitoring |
-------------------------------------+--------------------------------------
Changes (by phk):

* status: new => closed
* resolution: => worksforme


Comment:

We revisted the oldest tickets on the bug-wash today, and finally(!)
made our mind up about this one:

The central concern here is that the locking necessary for such a counter
would be expensive, performance wise, because it would serialize all
sessions on that lock.

The plans presently is to make the thread-pools even more independent and
as a result of that, we will get per-thread-pool statistics. For each
thread-pool, having a counter of number of busy treads is actually pretty
cheap, so that we can do.

What is needed then is for libvarnishapi or varnishstat to tally up the
global sum for all the threadpools.

I'm closing this ticket, because it does not really serve any function
going forward, seeing that the ultimate solution is much different from
what is proposed.

Sorry for taking so long to resolve this.

--
Ticket URL: <http://www.varnish-cache.org/trac/ticket/501#comment:7>
Varnish <http://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs

Varnish bugs RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.