Using multiple backends in Varnish

Tags: vcl (15) backend (2)

The most common Varnish setups will feature a single backend, as illustrated in the VCL snippet below:

vcl 4.1;

backend default {
    .host = "127.0.0.1";
    .port = "8080";
}
However it is also possible to define multiple backends and to use VCL logic to decide which backend to use.

VCL example

The example below has 2 backends:

  • default that connects to backend.example.com on port 80
  • image that connects to image.example.com on port 80

The image backend in the example is only used when the URL ends with .png, .jpg or .svg.

By conditionally assigning a value to the req.backend_hint variable in VCL, a backend is selected for that specific condition.

In the example below, every cache miss will be sent to the default backend, except the cache misses for the images:

vcl 4.1;

backend default {
    .host = "backend.example.com";
    .port = "80";
}

backend image {
    .host = "image.example.com";
    .port = "80";
}

sub vcl_recv {
    if(req.url ~ "\.(png|jpg|svg)$") {
        set req.backend_hint = image;
    }    
}

Beware of unused backends

If you’re using multiple backends, make sure all the defined backends are in use. When a backend is not in use, Varnish will return the following error when it compiles the VCL file:

Error:
Message from VCC-compiler:
Unused backend image, defined:
('/etc/varnish/default.vcl' Line 7 Pos 9)
backend image {
--------######--

Running VCC-compiler failed, exited with 2
VCL compilation failed

Varnish complains that the image backend was defined but not used, as you can see in the VCL example below:

vcl 4.1;

backend default {
    .host = "backend.example.com";
    .port = "80";
}

backend image {
    .host = "image.example.com";
    .port = "80";
}
The default backend will automatically be used because it was defined first. But without additional backend allocation logic Varnish’s VCC compiler will throw an error.

The solution is to either remove the image backend or to assign the backend in our VCL code as we have done in the initial VCL example.

Loadbalancing requests between multiple backends

In the VCL example we assigned backends based on routing decisions: certain content had to be served from a specific backend.

Although the use of multiple backends will offload some of the server load from the primary backend, the distribution is far from predictable.

For actual loadbalancing between multiple backends you should leverage vmod_directors. Here’s an example of round-robin based load distribution across 3 backends:

vcl 4.1;

import directors;

backend backend1 {
    .host = "backend1.example.com";
    .port = "80";
}

backend backend2 {
    .host = "backend2.example.com";
    .port = "80";
}

backend backend3 {
    .host = "backend3.example.com";
    .port = "80";
}

sub vcl_init {
    new vdir = directors.round_robin();
    vdir.add_backend(backend1);
    vdir.add_backend(backend2);
    vdir.add_backend(backend3);    
}

sub vcl_recv {
    set req.backend_hint = vdir.backend();
}

But for more in-depth information about loadbalancing in Varnish, please read the dedicated loadbalancing tutorial.