Building on open source

The betting site has used Apache Kakfa to track the navigation of customers, for buffering the data. It provides data to a huge number of subsystems within William

Hill’s Omni architecture, to enable information to be consumed as quickly as possible, which the company needed for its recommendation engine.

“All the information we collect is stored in Kafka, where it is then distributed,” said Di Loreto. “The data is sent to two layers – a speed layer and a batch layer. The speed layer is used to calculate ‘best bets’. The data is also sent to the batch layer, which is used for long-term business intelligence to create a better profile of the user.”

The Omni platform was built using a Java-like programming language called Scala. It was selected because it offers a functional programming model, which Di Loreto said better suited the highly distributed nature of William Hill’s platform. “We felt it would let us handle all the data wrangling and logic required to manage the data streams on our users,” he added.

Di Loreto said Scala offers a different paradigm of programming, enabling developers to use object-oriented programming for web services and Soap, interpretative programming and also functional programming, where appropriate.

Read more about open source in the enterprise

The open-source Apache Spark general engine for big data processing from Berkeley University’s Algorithms, Machines and People (AMP) lab was used on top of Scala to handle complex, associative data queries. Spark Streaming, which is built into Spark, enables William Hill to handle hundreds of thousands of data operations distributed across multiple processors.

Information on users is organised into timelines stored in the Cassandra open-source distributed database management system.

Di Loreto said William Hill also used the open-source Akka toolkit, which provides a transparent processing framework for all requests, to enable the Spark Streaming real-time data processing tool to process the data feed from Kafta.

With this architecture, Di Loreto said William Hill has used logical reasoning to present relevant content to users. “Because you always place a bet on Liverpool, then you are likely to be a Liverpool supporter,” he said.

The site has also been designed for failure using the open-source Docker container technology, which provides a more lightweight way to run applications compared with a virtual machine. Di Loreto said: “If an application crashes in a virtual machine, it could corrupt the whole VM, including all other applications. But if you have a failure in a containerised Docker application, there is no impact on the rest of the system.”