Mailing List Archive: Varnish: Misc

Sharding with Varnish

Index | Next | Previous | View Threaded

enno at tty

Aug 1, 2012, 5:46 AM

Post #1 of 3 (238 views)
Permalink

Sharding with Varnish

Hey guys,

I was wondering if any of you have implemented sharding using varnish.

We're considering the following setup:

Let's say we have 3 shards, where each shard consists of 2 servers.
So this would mean 3 directors with 2 backends each.
In VCL I want to determine which shard to use, using a simple ID mod N.

1. Client requests a file
2. Cache Miss
3. Calculate which shard must handle the request
4. Set backend to correct shard
5. Serve the file

I'm pretty confident this setup works, the only complicated issue is
determining the correct shard.
Has anyone done anything like that yet? With inline C it shouldn't be too
hard, or maybe someone has already written a VMOD?
Any examples would be greatly appreciated.

Thanks in advance

Enno van Amerongen

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

phk at phk

Aug 1, 2012, 8:46 AM

Post #2 of 3 (224 views)
Permalink

Re: Sharding with Varnish [In reply to]

In message <50192529.6010808 [at] tty>, Enno van Amerongen writes:

>3. Calculate which shard must handle the request

Look at the "hash" director.

You can also implement your own algorithm in a VMOD

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk [at] FreeBSD | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

_______________________________________________
varnish-misc mailing list
varnish-misc [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

enno at tty

Aug 1, 2012, 12:46 PM

Post #3 of 3 (222 views)
Permalink

Re: Sharding with Varnish [In reply to]

From: Poul-Henning Kamp [mailto:phk [at] phk]

In message <50192529.6010808 [at] tty>, Enno van Amerongen writes:

>3. Calculate which shard must handle the request

Look at the "hash" director.

The problem with the hash director in my case is, is that I think it's hard to predict, if possible at all. I need to know on which servers to store which files.
You can also implement your own algorithm in a VMOD

I think that's the direction for us, so that we can control the dispatching to several clusters of backends by an id somewhere in the url.
Funny that we discussed VMODs at VUG2 in Amsterdam before they existed ;)
Been a Varnish fanboy ever since, it's done wonders for me (and billions of requests). Excellent job mate!

Enno

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads