Business Insider scales with Varnish - no matter what
Business Insider (BI) is an online business news site that serves 150 million pages and has around 24 million unique visitors every month. BI was launched in 2007 and has been growing fast since. Varnish has been an imperative element for scalability and web performance at BI since they first installed it in 2011. We recently talked to Pax Dickinson, Business Insider’s Chief Technology Officer, and asked him a couple of questions on how he’s been using Varnish to keep the news site stable.
What challenges did you experience on businessinsider.com before you deployed Varnish?
“As is typically the case for online news providers our traffic is very spiky. So to give you a concrete example of why we needed Varnish, we posted a video of Christina Aguilera singing at the 2011 Super Bowl and getting the words of the national anthem wrong. We were the first ones to publish the video, and The Drudge Report and other social news websites quickly posted links to it which resulted in so much traffic that our site went down. That was the last time BI was down because I instituted Varnish a couple of weeks later. We’ve never had a problem since. Today we can have several popular sites linking to articles on BI without the spike in traffic affecting us. It goes to the cached pages and our four web servers remain stable. Varnish allows us to scale no matter what.”
Are you happy with Varnish as it is today?
“Yes, it’s great! We couldn’t be happier with it. It has done everything we wanted it to do and more. We have two Varnish servers up and running but I only need the second one in case the first one crashes and that hasn’t happened to this day.”
Business Insider promises to deliver news “at rapid-fire pace”, how does Varnish help you deliver on that promise?
“Really well, our pages load very fast because we are not dependent on the back-end for speed. I use Catchpoint to monitor the speed of my frontend and from there it’s obvious that our speed bottlenecks come from ads, third party images, third party tracking widgets and all that other stuff. The Business Insider site is never slowed down by the backend.”
What does the Business Insider stack look like? Where is Varnish?
“We have a load balancer at our hosting company. Requests come in through the load balancer and go to Nginx which handles my SSL and compression and then Varnish is situated behind Nginx. Behind Varnish are four web servers and then we have our database clusters behind those again. I am still using Varnish 2.1.5 which is why I use Nginx for compression.”
Why did Business Insider select Varnish as a caching solution?
“I worked with this Drupal consultant for a short while that was very high on Varnish. He got me interested in it and after I read up on it the choice became obvious. For us, Varnish is basically how you scale. We used to use a different caching solution but after I deployed Varnish our CPU load dropped dramatically. I only use four web servers with 32.000 concurrent readers on the site. Varnish does a pretty amazing job for us!”
Have you experienced problems with Varnish since you deployed it?
“Not really, I’m well acquainted with HTTP protocol so it was easy for me to pick up and learn to use Varnish. The better you know how HTTP works at a low level the better you are going to be at building your Varnish VCL.”
Pax Dickinson is Chief Technology Officer at Business Insider. He manages a team of 15 developers and system administrators. Pax initiated the Varnish community in New York and has now folded that into the NYC Web Scaling Group which he is heading. He believes that anyone interested in scaling needs to know about Varnish. So if you are interested in joining the Varnish community in NY feel free to contact Pax to hear more or follow and sign up for the planned meetups here.