Truly dynamic backend definition in Varnish

Truly dynamic backend definition in Varnish

If you have a really big web farm you'll have lots of backends and some sort of complex mapping between virtual hosts and web servers.

Currently you can choose to have some script that builds the backend definitions and then include that in your VCL or use the DNS director. Neither are ideal. Changing VCL is a hassle. The DNS director, while being built for exactly that purpose, requires you to shove everything into one network and then maintain a zone. Besides, the DNS director doesn't work very well with IPv6. 

Both lack the power and flexibility that is the hallmark of Varnish and for quite some time we've been thinking of a way to improve the situation. They are not very varnishy.

So, for some time we've been trying to figure our something new. These have been the requirements, contributed by various people we've met over the last couple of years.

  • Backends should be defined in a lazy fashion. Unless you need it it shouldn't need to be created.
  • It should be flexible and powerful. Backend definitions should be provided by a VMOD, which in turn can get data from almost anything.
  • There must be some built in caching of definitions. Calling out to external programs or network services is dead slow. 
  • Which means there must be some way to re-validate backends. DNS is a good example here. Amazon S3 hosts change all the time.

Proposed solution

Lets have a backend type "dynamic".

director dyn1 dynamic {
# some defaults could be overridden here
}

Now, that doesn't give us much. Let me explain what is going on behind the curtain. Each dynamic director contains a data structure (an alist or a hash or something similar) that bind the backend to a name. The name is a arbitrary string, unique for each backend in that director. At least initially, at some point we might want Varnish to balance load between two backends and giving them the same name might be a good way to signal your intent. 

We also need some functions to do some operations in VCL. In order to contain the function calls that deal with dynamic backends somewhat we prefix stuff with dyn. So we'll need the following functions in VCL:

  • dyn.create(director, name, IP, TTL)
  • dyn.delete(director, name)

Then there might be a couple of functions that might be useful:

  • dyn.exists(director, name)
  • dyn.set_ttl(director,name, TTL)
  • dyn.get_ttl(director,name)
  • (..)

The missing piece is how to tie the requests to the correct backend. I suggest we introduce a new variable on each requests. req.routing or similar. The director will look at req.routing and compare it to the strings identifying each backend. If one matches, great. If there is no match then we use the default (if any) or throw an error. 

The really cool thing

about this is that we can get the definition from any source. If we build a DNS VMOD, which would be quite simple, defining a backend might look like this:

dyn.create(dyn1, "foo", dns.lookup("foo"), dns.lookup_ttl("foo"));

or we could get the data from /etc/hosts, Berkeley DB, CDB files, memcache, redis or some awful XML file if that what is what you like. I talked to someone having the host mapping in some proprietary database being accessed through multicast - it would probably be a day or two of work to get Varnish to understand that. 

A more complete VCL example

vcl_recv {
# www.foo.com ==> foo
set req.routing = regsub(req.http.host, "(?i:)(www.)(.*).com","\2");
	set req.http.x-routing = req.routing;
	if (! dyn.exists(d1, req.routing) {
# Create “foo” backend here with a TTL of 3600 secs.
dynbackend.create(d1, req.routing,
dns.lookup(req.routing),
					3600 );
	if (! dyn.exists(d1, req.routing) {
		error;

}

   }

Current status

This is currently not implemented. We're currently looking for companies to work together with in order to make this a reality. If your company needs this please send me an email at perbu@varnish-software.com or leave a comment. Feedback is more than welcome.

 

The image is (c) 2008 bdesham and used under CC BY-SA 2.0 license.

Add comment

Filtered HTML

  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
Type the characters you see in this picture. (verify using audio)
Type the characters you see in the picture above; if you can't read them, submit the form and a new image will be generated. Not case sensitive.

Comments

We can totally use this. We change VCL every time we push out new code to flip servers around. It's a hassle, for sure.

-david

Not only dynamic backend creation would be nice, but dynamic backen selection as well. That would let override any existint logic (round robin, random, etc) with another one fit to any situation.

I don't really need dynamic backend creation but I definitely need dynamic backend selection.  In my case, it would be ideal to pass through backend selection using switch or pass-through matching style on the requested VHOST.

I.e.:

Suppose I have 3 backends (or groups of backends) called dev, staging, and live.  I'd like to route requests to backends based on VHOST matching so that

  • *.dev.mydevsite.com (and dev.mydevsite.com) requests go to the dev backend group
  • *.mydevsite.com (and mydevsite.com) requests go to the staging backend group (note the use of a superset of the previous matching rules)
  • any remaining unmatched VHOSTS go to the live backend group

With a configuration like that, I could set it and forget it, never again having to add anything for new VHOSTS except apache configs and in the case of arbitrary domains a new DNS entry.  The setup would essentially mimmick the same flexible options DNS resolution supports, going a step further since we can already assume everything going through Varnish has already chosen a public IP destination.

I'll be keeping an eye out for news on this front.  If such a configuration becomes possible, I'll definitely be switching to Varnish.

Per Buer's picture

@Evan King:

Unless there is something I'm missing - that should be quite easy to do today. In vcl_recv:

if (req.http.host ~ ".*\.dev\.mydevsite\.com") {
   set req.backend = dev;
} elsif (req.http.host ~"mydevsite\.com") {
  set req.backend = staging;
}

 

I do need dynamic backend creation and selection. In my case, I have a growing number of backend servers, with domain names:

node1.src.foo.com
node2.src.foo.com 
... 

 

While users access data with frontend domain names:

node1.foo.com 
node2.foo.com 
... 

 

Each time when I add a backend node, I have to modify vlc on all frontend nodes. I would prefer a regex based fasion of creating and selecting backend server, such like:

set req.backend.host = regsub("(node.*).foo.com", "\1.foo.com")