More and more companies are realising they have a wealth of data available to them, or they have the opportunity to aggregate information from multiple sources in near real time.

While big data and batch processing systems like Hadoop are part of the tool kit of many teams, more and more are realising that a continually updating system of pipes and filters – a fire hose – of their data is a powerful and flexible tool.

In this talk we will look at the challenges of building this type of system, the patterns and techniques that can help simplify it, and the difficulties of managing such a system when it’s running.