Mailing List Archive: Varnish: Dev

Varnish as a redirector for a millions URLs

Index | Next | Previous | View Threaded

aleemb at gmail

May 23, 2013, 4:57 AM

Post #1 of 6 (176 views)
Permalink

Varnish as a redirector for a millions URLs

Hi,

I currently use nginx + redis + lua module to setup redirects for close to
a million old URLs that have been ported to a new platform.

Is it possible to achieve the same with varnish (maybe replay logs for all
redirects and set a max TTL. My cache size is more than enough to prevent
LRU from erasing these entries). However, I am not sure if that's a good
approach since Varnish would still require a backend through which it
populates the redirects.

Is this recommended through Varnish and is there any better approach?

Aleem

phk at phk

May 23, 2013, 6:37 AM

Post #2 of 6 (170 views)
Permalink

Re: Varnish as a redirector for a millions URLs [In reply to]

In message <CA+=Uhp_pepXsb6Q_ORQpYe+AhWSh_yytRNjKBzBBA_FxwJb28Q [at] mail>

>I currently use nginx + redis + lua module to setup redirects for close to
>a million old URLs that have been ported to a new platform.

Are the redirects based on rules ?

Ie, something like:

/article?ID=##### -> /#####

or is it a database lookup to figure out the mapping ?

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk [at] FreeBSD | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

_______________________________________________
varnish-dev mailing list
varnish-dev [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

aleemb at gmail

May 23, 2013, 6:41 AM

Post #3 of 6 (166 views)
Permalink

Re: Varnish as a redirector for a millions URLs [In reply to]

It's based on a DB lookup. Currently I am using NGINX + Redis and there are
about a million URLs.

Aleem

On Thu, May 23, 2013 at 6:37 PM, Poul-Henning Kamp <phk [at] phk>wrote:

> In message <CA+=
> Uhp_pepXsb6Q_ORQpYe+AhWSh_yytRNjKBzBBA_FxwJb28Q [at] mail>
>
> >I currently use nginx + redis + lua module to setup redirects for close to
> >a million old URLs that have been ported to a new platform.
>
> Are the redirects based on rules ?
>
> Ie, something like:
>
> /article?ID=##### -> /#####
>
> or is it a database lookup to figure out the mapping ?
>
>
>
> --
> Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
> phk [at] FreeBSD | TCP/IP since RFC 956
> FreeBSD committer | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
>

Raul.Rangel at disney

May 23, 2013, 6:57 AM

Post #4 of 6 (169 views)
Permalink

RE: Varnish as a redirector for a millions URLs [In reply to]

Can you make a backend that takes a url, looks it up in the database and returns a 301? You could then cache the 301 in varnish. Thought tht will only work for GETs. Once you get into POST requests it's another story.

Raul

From: varnish-dev-bounces [at] varnish-cache [mailto:varnish-dev-bounces [at] varnish-cache] On Behalf Of Aleem B
Sent: Thursday, May 23, 2013 7:41 AM
To: Poul-Henning Kamp
Cc: varnish-dev [at] varnish-cache
Subject: Re: Varnish as a redirector for a millions URLs

It's based on a DB lookup. Currently I am using NGINX + Redis and there are about a million URLs.

Aleem

On Thu, May 23, 2013 at 6:37 PM, Poul-Henning Kamp <phk [at] phk<mailto:phk [at] phk>> wrote:
In message <CA+=Uhp_pepXsb6Q_ORQpYe+AhWSh_yytRNjKBzBBA_FxwJb28Q [at] mail<mailto:Uhp_pepXsb6Q_ORQpYe%2BAhWSh_yytRNjKBzBBA_FxwJb28Q [at] mail>>

>I currently use nginx + redis + lua module to setup redirects for close to
>a million old URLs that have been ported to a new platform.
Are the redirects based on rules ?

Ie, something like:

/article?ID=##### -> /#####

or is it a database lookup to figure out the mapping ?

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk [at] FreeBSD<mailto:phk [at] FreeBSD> | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

phk at phk

May 23, 2013, 7:45 AM

Post #5 of 6 (170 views)
Permalink

Re: Varnish as a redirector for a millions URLs [In reply to]

In message <CA+=Uhp8Eb6a6tgERfoMpiDg1Od15QAw+ZLBsuU+zTa++s4W8OQ [at] mail>
, Aleem B writes:

>It's based on a DB lookup. Currently I am using NGINX + Redis and there are
>about a million URLs.

Then you can either have a backend server do the lookup and have
varnish cache the result, or you can write a VMOD for varnish which
does the lookup and generate the redirects that way.

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk [at] FreeBSD | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

_______________________________________________
varnish-dev mailing list
varnish-dev [at] varnish-cache
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

aleemb at gmail

May 23, 2013, 8:37 AM

Post #6 of 6 (166 views)
Permalink

Re: Varnish as a redirector for a millions URLs [In reply to]

>
> Then you can either have a backend server do the lookup and have varnish
> cache the result, or you can write a VMOD for varnish which does the lookup
> and generate the redirects that way.

The database of URLS won't change (these are legacy domains). Is there any
way manually insert a 301 redirect entry to the varnish cache? Can the VMOD
do that? If yes, is there anyway for the VMOD to protect entries from
getting wiped out by LRU (make them static)?

Thanks,
Aleem

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads