The era of Big Data is upon us – the Volume, Velocity and Variety of enterprise and scientific data are growing at an exponential rate and will continue to do so for the foreseeable future. Hidden inside a flood of heterogeneous raw data is the knowledge capable of having transformative impacts across virtually every area of society. Raw data must be converted into knowledge and understanding in order to stimulate scientific discoveries, economic growth and industrial innovation.
With the advent of cloud computing, resizable infrastructure for data analysis is now available to everyone via an on-demand pay-per-use model. Cloud computing provides one solution to Big Data infrastructure requirements, however writing scalable applications and algorithms that utilise cloud infrastructure is an extremely demanding and error prone task. Engineers still have no off-the shelf solutions that they can readily use to manage, move, analyse and visualise complex distributed data sets.
In order to unlock the potential of Big Data we need to overcome a significant number of research challenges including: managing diverse sources of unstructured data with no common schema, removing the complexity of writing auto-scaling algorithms, real time analytics, suitable visualisation techniques for Petabyte scale data sets etc.
Big Data Lab’s research mission is to identify, engineer and evaluate innovative technologies that address current and future data-intensive challenges.
Our data-intensive research agenda is pursued by engineering technologies that extend the state-of-the-art and importantly address real problems in the Big Data space. We follow agile development practices and conduct weekly stand-up meetings to report progress, plan ahead and foster collaboration. Much of our work follows a multi-disciplinary approach; we work on problems at the intersection of Science, Engineering, Finance, and Computer Science. Big Data Lab is currently funded through generous grants from the EPSRC, European Research Council (ERC), Royal Society, Google, Royal Society of Edinburgh, SICSA, Impact Acceleration Account (IAA), and Amazon Web Services. Our research was recently featured in the MIT Technology Review and The Guardian.