Labour leadership sentiment analysis

Following the initial kick-off of the Labour Leadership campaign, we are monitoring how the candidates are performing against each other in terms of social media sentiment. 

The base tool, built on Skyscape, was configured in under two weeks to analyse social media data and present the findings live. 

Labour Leadership candidates:
Positive minus negative sentiment

Jeremy Corbyn

Owen Smith

How did we do it?

OvervIew

We've previously successfully built rapid prototypes to monitor Brexit and the Apple Developer's Conference. Both were built on the AWS platform, and only took a few days to build. The results were great, with social media producing better forecasts than the pollsters!

Although great prototypes, one of our clients challenged us to do the same on the ukcloud platform.

Proving that we're not a one-trick pony and that we're technology agnostic, we set about rapidly building a sentiment analysis tool for the Labour Party leadership, using this public sector cloud solution.

Day 1

  • Planned and agreed desired outcome Agreed on feature prioritisation and set deadlines for our minimum viable product (MVP).
  • Commenced UX Research to create initial wireframes.

Day 2-4

  • Planned the Skyscape architecture.
  • Gained access to and became accustomed to Cloud Foundry, Skyscape Portal and Skyscape Virtual Data Centre.
  • Created and configured networks, firewall and NAT rules in the Virtual Data Centre (VDC).
  • Created Linux VMs and installed necessary software and PostgreSQL database.
  • Adapted and setup the core social analytics engine to search Twitter for specific keywords (around 15 keywords referenced to the target candidates).
  • Configured security on the VMs.
  • Analysed Twitter, searching for specific keywords (around 15 keywords referenced to the target candidates).
  • Input sentiment analysis of the Twitter results, based on a sentiment lexicon. Then, calculated on a per-tweet bases immediately after being collected.
  • Saved data into a database (Skyscape) including: date of tweet, language, author, content and of course, sentiment.

Day 5-6

  • Began streaming tweets to our script, running the sentiment analysis on them, and storing the results in our database.
  • Installed and configured NGINX to serve content to the front-end site.
  • Set up jobs to periodically poll the database and populate a JSON file, stored on the NGINX server.

Day 7

  • Created interactive front end UI from UX wireframes.
  • Integrated front end code into SPARCK Live site.

Day 8

  • Linked front end code to the live system, and tested against a live stream of >100 Tweets per second to test integrity under load.
  • Launched SPARCK LAB to the public through http://sparck.io/lab

What technologies did we use?

Cloud foundry

Cloud Foundry is an open source cloud computing platform as a service (PaaS). Cloud Foundry supports the full lifecycle, from initial development, through all testing stages, to deployment. For our purposes, it is integrated with the Skyscape Cloud solution - therefore making it an ideal part of the process journey for developing our solution. The Cloud Foundry command line interface (CF) requires a little upskilling, but is well documented through Skyscape's built-in portal.

vmware vcloud director

vCloud Director orchestrates the provisioning of the virtual data centre upon which the Skyscape VMs are built. It allows you to manage multiple VM's and configurations from one location, with varied access controls. The Skyscape instance of this tool allows a user to quickly provision complete and operationally ready infrastructure without worrying about the physical configuration of hardware. For the development of a lean product, this is a useful feature.

NGINX

We use NGINX in our solution to serve JSON files containing the underlying graph data to the front-end site. With its small footprint, ability to serve static content quickly, and relative ease of configuration, it was the obvious choice of web server to use for this project.