G2G - Datawarehouse

2016-2018

Introduction

G2G online is a marketing agency that manages the online channels for some internationally operating companies. In 2016, they realized a revenue of €100 million for their clients. As a freelance web developer, I worked on a project with the objective of improving the data and information required by online marketers to optimize campaign performance.

Briefing

At G2G online, marketers used to make their decisions based on information from several sources, such as Google Analytics, Google Ads, Facebook Ads, Taboola Ads and CRM tools. The data was often incomplete and inaccurate. Furthermore, it took a long time for online marketers to retrieve all the data from different sources to get the overview they needed.

The objective of the project was to improve the accuracy of the data and to eliminate the manual operations required to gather that data. One specific requirement of the organization was to store all data on premises, and not to use cloud providers for data storage.

Result

We decided to use a microservice architecture in which each microservice is responsible for the collection of a specific type of data. This allowed us to build independent and well-maintainable applications. In August 2017, we finalized enough microservices to make the project invaluable to business users. In the months that followed, components were extended and new microservices were added with the aim of gathering more data to support online marketers in managing their campaigns.

Overview microservices

UML sequence diagram of asynchronous tracking of page views with Node Express and RabbitMQ

UML activity diagram of lead registration

In this project, my tasks included: creating an application architecture, coding, and project management. This project made me more experienced in working with Docker, Microservice architecture, Node.js (10), PHP (7.2), Yii2, RabbitMQ, ETL, TDD and SCRUM.

Tools and languages used: PHP (7.2), Yii2, Docker, Docker Compose, Microservice architecture, UML modeling, RabbitMQ, MySQL (5.7), MySQL Workbench, Node.js (10), MongoDB, Redis, Bitbucket Pipelines, Semaphore CI, Papertrail, New Relic (APM & Infrastructure), unit-testing, PHPUnit, Codeception, Mockery, Mocha, Chai, Sinon.JS, SCRUM, code reviewing, GIT, Bitbucket, SSL with Let's Encrypt, Teamwork Projects, Draw.io, REST APIs, Xplenty (ETL service), Stitch (Data extraction service), Facebook Graph API, Google Ads API, Bing Ads API, Outbrain Amplify API, Taboola Backstage API, Google Apps Script, Google Analytics server side tracking, Google BigQuery, Microsoft Power BI, Google Data Studio.