100 Terabytes, 5 Billion Documents, 10 Billion Parameters, 1 Billion Inserts/s

We’ve been busy building the next generation of a Parameter Server and it’s finally ready. It’s quite different from our previous designs, the main improvements being fault tolerance and self repair, a much improved network protocol, flexible consistency models, and a much more general interface.

In the next few posts I’ll explain the engineering decisions that went into this system which is capable of solving problems as diverse as very high throughput sketching, topic models, and optimization. And yes, it will be open source so you can build your own algorithms on top of it.