Automated Data Extraction
Go back to all case studies

Automated Data Extraction

Custom Data Skraping extensions for Dexi.io platform

About

Trusted by fortune 500 companies, Dexi.io helps solve real market challenges and execute detailed background checks, monitor brands and is being used as a powerful research instrument. BGO Software managed to get the most out of this powerful platform for our clients, by adding accurate technical knowledge and proven business practices for a number of tailor-made software solutions. 

BGO Software successfully tutored and assembled an entire team responsible for building data scrapers/extractor through the Dexi.io platform. Over a period of one year, our Web academy delivered a significant amount of specifically trained IT specialists for developing complex data crawling bots, using the wide range of options provided by the leading Enterprise Web Scraping Software – Dexi. 

 

Customer

Dexi

Location

Denmark

Developers Team

15 IT specialist

Project Timeline

During the whole 2017

Business Model

Dedicated team

Presentation

Involvement

Project Management Methodology

Custom Agile, SCRUM

Scrum is one of the most popular Agile methodologies. It is an adaptive, iterative, fast, flexible, and effective methodology designed to deliver significant value quickly and throughout a project. Scrum ensures transparency in communication and creates an environment of collective accountability and continuous progress. The Scrum framework is structured in such a way that it supports product and service development in all types of industries and in any type of project, irrespective of its complexity.

Additionally to SCRUM, our Project managers combined some Kanban and Waterfall elements in the workflow, because of the complexity of the whole project. 

 

Challenge

The main challenges during this project were focused mainly on the proper organization of  the whole workflow:

– Gathering requirements

– Target site analysis

– Architecting the data scraper

– Implementing the scraper within the Dexi platform

– Execute and do postprocessing and validation of the gathered data.

Solution

In order for a project of this scale to work properly, our managing team had to define a blazing fast and 100% practically oriented onboarding procedure, so the team (built from scratch) became highly proficient in less than a month:

  • BGO invented and implemented a very effective pipeline and tasks processing workflow that results in high throughput of “unit-to-delivery”
  • We developed separate tools for automated data validation additional data enrichment (geolocation, addresses, specific calculations, etc)
  • Added data accordingly, following the clients’ specifications
  • Managed to assemble an efficient “drag and drop” configuration with block structure 
  • We developed separate tools for
    – automated data validation
    – additional data enrichment (geolocation, addresses, specific calculations,etc)

Results

During the project agreed period, BGO Software managed to deliver about 520 different “data scraper robots” with a variety of custom extensions for specific data enrichment and extracting.  

More case studies with similar tech