Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Varnish: Misc

Is LCI on the radar?

 

 

Varnish misc RSS feed   Index | Next | Previous | View Threaded


mls at pooteeweet

May 30, 2011, 3:23 PM

Post #1 of 10 (3380 views)
Permalink
Is LCI on the radar?

Hi,

I assume some of you have stumbled over LCI by now:
http://www.ietf.org/id/draft-nottingham-linked-cache-inv-00.txt

This is actually quite interesting. For an application we are building we are looking to create an invalidation service to which the various independent frontend server applications can register and which gets notified by the backend. Of course the frontends then have to figure out which pages all need to be invalidated. The original article will be easy. Some of the category overviews will also be easy to delete. What will already get harder is invalidating all articles that reference the given article and worse yet would be if we start caching search results.

So I am wondering if you guys are looking at LCI for a future varnish impovement and if someone has build something like this on top of varnish today already that could maybe help us here.

regards,
Lukas Kahwe Smith
mls [at] pooteeweet




_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc


perbu at varnish-software

May 31, 2011, 2:34 AM

Post #2 of 10 (3258 views)
Permalink
Re: Is LCI on the radar? [In reply to]

Hi

On Tue, May 31, 2011 at 12:23 AM, Lukas Kahwe Smith <mls [at] pooteeweet>wrote:

> Hi,
>
> I assume some of you have stumbled over LCI by now:
> http://www.ietf.org/id/draft-nottingham-linked-cache-inv-00.txt
>
> This is actually quite interesting. For an application we are building we
> are looking to create an invalidation service to which the various
> independent frontend server applications can register and which gets
> notified by the backend. Of course the frontends then have to figure out
> which pages all need to be invalidated. The original article will be easy.
> Some of the category overviews will also be easy to delete. What will
> already get harder is invalidating all articles that reference the given
> article and worse yet would be if we start caching search results.
>
> So I am wondering if you guys are looking at LCI for a future varnish
> impovement and if someone has build something like this on top of varnish
> today already that could maybe help us here.
>

I'm pretty sure this can be implemented in VCL. No need to place it on the
radar. I have an upcoming blog-post describing something similar. It might
get a bit hairy with all the regular expression so it might be cleaner in a
module.


--
Per Buer, CEO
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>


l at lrowe

May 31, 2011, 3:58 AM

Post #3 of 10 (3265 views)
Permalink
Re: Is LCI on the radar? [In reply to]

On 31 May 2011 10:34, Per Buer <perbu [at] varnish-software> wrote:

> Hi
>
> On Tue, May 31, 2011 at 12:23 AM, Lukas Kahwe Smith <mls [at] pooteeweet>wrote:
>
>> Hi,
>>
>> I assume some of you have stumbled over LCI by now:
>> http://www.ietf.org/id/draft-nottingham-linked-cache-inv-00.txt
>>
>> This is actually quite interesting. For an application we are building we
>> are looking to create an invalidation service to which the various
>> independent frontend server applications can register and which gets
>> notified by the backend. Of course the frontends then have to figure out
>> which pages all need to be invalidated. The original article will be easy.
>> Some of the category overviews will also be easy to delete. What will
>> already get harder is invalidating all articles that reference the given
>> article and worse yet would be if we start caching search results.
>>
>> So I am wondering if you guys are looking at LCI for a future varnish
>> impovement and if someone has build something like this on top of varnish
>> today already that could maybe help us here.
>>
>
> I'm pretty sure this can be implemented in VCL. No need to place it on the
> radar. I have an upcoming blog-post describing something similar. It might
> get a bit hairy with all the regular expression so it might be cleaner in a
> module.
>

I experimented with something that sounds similar. Each page set a header
recording the the content item ids that were used in rendering the page.
They could then be purged with a regex including any dependents id.
http://dev.plone.org/collective/browser/experimental.depends/trunk/varnish.vcl

It works when you update or delete a content item, but it can't help the
case where you add a new content item and want that to appear in listing.

Laurence


phk at phk

May 31, 2011, 6:17 AM

Post #4 of 10 (3263 views)
Permalink
Re: Is LCI on the radar? [In reply to]

In message <4E0D7A8A-2219-4B7E-BBD5-5BF5DBC54047 [at] pooteeweet>, Lukas Kahwe S
mith writes:
>Hi,
>
>I assume some of you have stumbled over LCI by now:
>http://www.ietf.org/id/draft-nottingham-linked-cache-inv-00.txt
>
>This is actually quite interesting.

And unfortunately very very very troublesome. I'm still composing
my reply to it.

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk [at] FreeBSD | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc


mls at pooteeweet

Jun 27, 2011, 7:41 AM

Post #5 of 10 (3199 views)
Permalink
Re: Is LCI on the radar? [In reply to]

On 31.05.2011, at 12:58, Laurence Rowe wrote:

> On 31 May 2011 10:34, Per Buer <perbu [at] varnish-software> wrote:
> Hi
>
> On Tue, May 31, 2011 at 12:23 AM, Lukas Kahwe Smith <mls [at] pooteeweet> wrote:
> Hi,
>
> I assume some of you have stumbled over LCI by now:
> http://www.ietf.org/id/draft-nottingham-linked-cache-inv-00.txt
>
> This is actually quite interesting. For an application we are building we are looking to create an invalidation service to which the various independent frontend server applications can register and which gets notified by the backend. Of course the frontends then have to figure out which pages all need to be invalidated. The original article will be easy. Some of the category overviews will also be easy to delete. What will already get harder is invalidating all articles that reference the given article and worse yet would be if we start caching search results.
>
> So I am wondering if you guys are looking at LCI for a future varnish impovement and if someone has build something like this on top of varnish today already that could maybe help us here.
>
> I'm pretty sure this can be implemented in VCL. No need to place it on the radar. I have an upcoming blog-post describing something similar. It might get a bit hairy with all the regular expression so it might be cleaner in a module.
>
> I experimented with something that sounds similar. Each page set a header recording the the content item ids that were used in rendering the page. They could then be purged with a regex including any dependents id. http://dev.plone.org/collective/browser/experimental.depends/trunk/varnish.vcl
>
> It works when you update or delete a content item, but it can't help the case where you add a new content item and want that to appear in listing.

So we are looking to implement this. However I have one question:
How well does this perform if you start to have 100k, 1M or more objects in your varnish cache?
Does varnish create some sort of index of all headers? I assume even if it did, it cant really leverage it with a regexp.

Just dont want to kill my varnish servers CPU/harddrive when I start to purge stuff. If someone could give me some indication of what to expect it would be very good. We will in the end of course have to do our own benchmarks, but it would be good to be able to control expectations :)

I guess in the long run if one would want to properly implement LCI it would be necessary to maybe use an sqlite DB and some inline C magic to parse the relevant headers in there and then use that for lookups when doing a PURGE.

Does anyone have an idea how much effort it would be to properly implement LCI in Varnish and how we could maybe organize funding among all interested parties?

regards,
Lukas Kahwe Smith
mls [at] pooteeweet




_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc


l at lrowe

Jun 27, 2011, 8:45 AM

Post #6 of 10 (3196 views)
Permalink
Re: Is LCI on the radar? [In reply to]

On 27 June 2011 15:41, Lukas Kahwe Smith <mls [at] pooteeweet> wrote:
>
> On 31.05.2011, at 12:58, Laurence Rowe wrote:
>
>> On 31 May 2011 10:34, Per Buer <perbu [at] varnish-software> wrote:
>> Hi
>>
>> On Tue, May 31, 2011 at 12:23 AM, Lukas Kahwe Smith <mls [at] pooteeweet> wrote:
>> Hi,
>>
>> I assume some of you have stumbled over LCI by now:
>> http://www.ietf.org/id/draft-nottingham-linked-cache-inv-00.txt
>>
>> This is actually quite interesting. For an application we are building we are looking to create an invalidation service to which the various independent frontend server applications can register and which gets notified by the backend. Of course the frontends then have to figure out which pages all need to be invalidated. The original article will be easy. Some of the category overviews will also be easy to delete. What will already get harder is invalidating all articles that reference the given article and worse yet would be if we start caching search results.
>>
>> So I am wondering if you guys are looking at LCI for a future varnish impovement and if someone has build something like this on top of varnish today already that could maybe help us here.
>>
>> I'm pretty sure this can be implemented in VCL. No need to place it on the radar. I have an upcoming blog-post describing something similar. It might get a bit hairy with all the regular expression so it might be cleaner in a module.
>>
>> I experimented with something that sounds similar. Each page set a header recording the the content item ids that were used in rendering the page. They could then be purged with a regex including any dependents id. http://dev.plone.org/collective/browser/experimental.depends/trunk/varnish.vcl
>>
>> It works when you update or delete a content item, but it can't help the case where you add a new content item and want that to appear in listing.
>
> So we are looking to implement this. However I have one question:
> How well does this perform if you start to have 100k, 1M or more objects in your varnish cache?
> Does varnish create some sort of index of all headers? I assume even if it did, it cant really leverage it with a regexp.
>
> Just dont want to kill my varnish servers CPU/harddrive when I start to purge stuff. If someone could give me some indication of what to expect it would be very good. We will in the end of course have to do our own benchmarks, but it would be good to be able to control expectations :)
>
> I guess in the long run if one would want to properly implement LCI it would be necessary to maybe use an sqlite DB and some inline C magic to parse the relevant headers in there and then use that for lookups when doing a PURGE.
>
> Does anyone have an idea how much effort it would be to properly implement LCI in Varnish and how we could maybe organize funding among all interested parties?

This type of purge (which in Varnish 3 is renames 'ban') adds the
expression to the ban list. When any object is found in the cache the
expressions in the ban list are checked to decide whether to call
vcl_hit or vcl_miss. To prevent the ban list getting too long another
thread periodically works its way through all objects in the cache
removing those that have been banned and updating the pointer to the
place in the ban list it has checked so on subsequent requests fewer
ban expressions need to be checked.

I've not done any benchmarking on this, but for me even thousands of
regular expression checks will be much faster than re-requesting the
page from the CMS. So with my Varnish config, a PURGE request is cheap
(it only results in adding an entry to the ban list, not checking
against the entire contents of the cache) and the additional cost of
checking each GET request is not noticeable. So I don't really see the
need for LCI support given the existing support for bans.

Laurence

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc


mls at pooteeweet

Jun 27, 2011, 10:22 AM

Post #7 of 10 (3202 views)
Permalink
Re: Is LCI on the radar? [In reply to]

On 27.06.2011, at 17:45, Laurence Rowe wrote:

> This type of purge (which in Varnish 3 is renames 'ban') adds the
> expression to the ban list. When any object is found in the cache the
> expressions in the ban list are checked to decide whether to call
> vcl_hit or vcl_miss. To prevent the ban list getting too long another
> thread periodically works its way through all objects in the cache
> removing those that have been banned and updating the pointer to the
> place in the ban list it has checked so on subsequent requests fewer
> ban expressions need to be checked.

Ok, that sounds pretty much like genius to me :)

> I've not done any benchmarking on this, but for me even thousands of
> regular expression checks will be much faster than re-requesting the
> page from the CMS. So with my Varnish config, a PURGE request is cheap
> (it only results in adding an entry to the ban list, not checking
> against the entire contents of the cache) and the additional cost of
> checking each GET request is not noticeable. So I don't really see the
> need for LCI support given the existing support for bans.


well LCI support would just mean Varnish plays along with a published standard for this kind of feature, which is always a plus.

regards,
Lukas Kahwe Smith
mls [at] pooteeweet




_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc


phk at phk

Jun 28, 2011, 1:42 AM

Post #8 of 10 (3195 views)
Permalink
Re: Is LCI on the radar? [In reply to]

In message <4B5D88B4-F620-43C0-AB8A-DE19565B22DF [at] pooteeweet>, Lukas Kahwe S
mith writes:

>> I assume some of you have stumbled over LCI by now:
>> http://www.ietf.org/id/draft-nottingham-linked-cache-inv-00.txt

It's not a proposal I particular is happy about because it is very
expensive to implement one way or the other.

But I made a quick attempt and you can implement this using VCL and bans
in Varnish already, so I don't expect to do more about it.

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk [at] FreeBSD | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc


mls at pooteeweet

Jun 28, 2011, 1:49 AM

Post #9 of 10 (3205 views)
Permalink
Re: Is LCI on the radar? [In reply to]

On 28.06.2011, at 10:42, Poul-Henning Kamp wrote:

> In message <4B5D88B4-F620-43C0-AB8A-DE19565B22DF [at] pooteeweet>, Lukas Kahwe S
> mith writes:
>
>>> I assume some of you have stumbled over LCI by now:
>>> http://www.ietf.org/id/draft-nottingham-linked-cache-inv-00.txt
>
> It's not a proposal I particular is happy about because it is very
> expensive to implement one way or the other.
>
> But I made a quick attempt and you can implement this using VCL and bans
> in Varnish already, so I don't expect to do more about it.


I would appreciate it a lot if your tests and Laurence's solution [1] could somehow we documented on the wiki. We will happily beta test it :)

regards,
Lukas Kahwe Smith
mls [at] pooteeweet

[1] http://dev.plone.org/collective/browser/experimental.depends/trunk/varnish.vcl


_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc


mls at pooteeweet

Sep 2, 2011, 12:52 AM

Post #10 of 10 (3047 views)
Permalink
Re: Is LCI on the radar? [In reply to]

On Jun 28, 2011, at 10:49 , Lukas Kahwe Smith wrote:

>
> On 28.06.2011, at 10:42, Poul-Henning Kamp wrote:
>
>> In message <4B5D88B4-F620-43C0-AB8A-DE19565B22DF [at] pooteeweet>, Lukas Kahwe S
>> mith writes:
>>
>>>> I assume some of you have stumbled over LCI by now:
>>>> http://www.ietf.org/id/draft-nottingham-linked-cache-inv-00.txt
>>
>> It's not a proposal I particular is happy about because it is very
>> expensive to implement one way or the other.
>>
>> But I made a quick attempt and you can implement this using VCL and bans
>> in Varnish already, so I don't expect to do more about it.
>
>
> I would appreciate it a lot if your tests and Laurence's solution [1] could somehow we documented on the wiki. We will happily beta test it :)
>
> regards,
> Lukas Kahwe Smith
> mls [at] pooteeweet
>
> [1] http://dev.plone.org/collective/browser/experimental.depends/trunk/varnish.vcl


ok .. it looks like there is a good chance we will work on something based on the above for a large news site which is drives multiple separate frontends (each with their own caching) from a single backend. here is my game plan:
http://pooteeweet.org/blog/1979

regards,
Lukas Kahwe Smith
mls [at] pooteeweet




_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Varnish misc RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.