One of the longest-lived problems with Zuul is how many connections it opened to back-end servers. We recently invested in modifying our connection pooling logic to make use of subsetting and fix these problems. It’s worth reading the blog post if for nothing other than the phenomenal improvement graphs. Here’s a sneak peek of the total connection counts per region falling off a cliff: Another…
Over the years, Netflix has evolved from a streaming company to a full-fledged studio. Being a tech company, we wanted to leverage that expertise to build a studio of the future. In this future world, there is an application for every part of the process – from creative, to production, to marketing, and eventually all the way to playback. We call it the “Studio in the Cloud.” As you can imagine,…
Running Zuul at Netflix, we often see traffic spikes. Typically the spikes are caused by retry storms because of bugs on device builds or service deploys, and more rarely due to DDoS attacks or network failures. Zuul needs to maintain availability for both itself and its backends when faced with unsustainable load. As a result, we need to reduce the load, meaning customer requests will be…
I gave another Strange Loop talk this year. It was about our real-time anomaly detection system called Raju. Here is the abstract: In a large scale distributed system, detecting and pinpointing failures gets exponentially harder as an architecture gets more complex. Netflix’s cloud architecture is composed of thousands of services and hundreds of thousands of VMs and containers. Failures can…
Today we officially announced the release of Zuul 2 . It’s an exciting day for the team and the blog post linked hightlights some of the other work we’ve been doing. Here’s a summary of the major features included in the open source release: Today we are releasing many core features. Here are the ones we’re most excited about: Server Protocols HTTP/2 — full server support for inbound HTTP/2…
This is a talk I gave at Strange Loop about how we rebuilt Zuul on Netty and RxJava. The main themes of the talk are: what it took to rebuild it, what it’s been like operating it in production and how to make the right decision about going blocking vs non-blocking. Here is the abstract: Zuul 2.0 is the latest iteration of the gateway application fronting Netflix’s API and underlying microservices.…
It’s only a matter of time before Java shop managers are faced with eager developers trying to push a new language on them. It’s a hard sell. Java has mature tooling, tons of documentation, countless stackoverflow articles, and lots of senior developers available to fill vacancies. Combine that with your team’s expertise in Java and the rampup cost for a new language, on a project that was…
It’s a framework we developed almost a year ago, that has been running in production for over 10 months. It’s so simple and effective that we have barely had to make any commits to it since. That framework is Objectify.scala . Objectify was inspired by James Golick’s framework, Objectify . He introduced it by describing why Rails needs to move away from fat models, and create a service layer. In a…
We’ve recently started on a new product at work using the AngularJS framework . One of the most useful resources when starting out was the #angularjs room on freenode . There seems to be a really large, vibrant community around AngularJS and that’s always helpful when starting from scratch on a new framework. One of the best resources in the room was David Mosher , who helped us get set up with…
A common model for implementing EIP in Java, is to use Apache Camel , with some sort of MQ or JMS backing. One of the more interesting MQ frameworks in use today is the Erlang -based RabbitMQ . Camel is a very widely used integration platform . It’s very flexible and takes care of wiring up services, mediating messages, transforming data, among many other useful things. It also comes with tons of…