Custom Data Skraping extensions for Dexi.io platform
About the solution
Trusted by fortune 500 companies, Dexi.io helps solve real market challenges and execute detailed background checks, monitor brands and is being used as a powerful research instrument. BGO Software managed to get the most out of this powerful platform for our clients, by adding accurate technical knowledge and proven business practices for a number of tailor-made software solutions.
BGO Software successfully tutored and assembled an entire team responsible for building data scrapers/extractor through the Dexi.io platform. Over a period of one year, our Web academy delivered a significant amount of specifically trained IT specialists for developing complex data crawling bots, using the wide range of options provided by the leading Enterprise Web Scraping Software – Dexi.
Scrum is one of the most popular Agile methodologies. It is an adaptive, iterative, fast, flexible, and effective methodology designed to deliver significant value quickly and throughout a project. Scrum ensures transparency in communication and creates an environment of collective accountability and continuous progress. The Scrum framework is structured in such a way that it supports product and service development in all types of industries and in any type of project, irrespective of its complexity.
Additionally to SCRUM, our Project managers combined some Kanban and Waterfall elements in the workflow, because of the complexity of the whole project.
Challenge
The main challenges during this project were focused mainly on the proper organization of the whole workflow:
– Gathering requirements
– Target site analysis
– Architecting the data scraper
– Implementing the scraper within the Dexi platform
– Execute and do postprocessing and validation of the gathered data.
Solution
In order for a project of this scale to work properly, our managing team had to define a blazing fast and 100% practically oriented onboarding procedure, so the team (built from scratch) became highly proficient in less than a month:
BGO invented and implemented a very effective pipeline and tasks processing workflow that results in high throughput of “unit-to-delivery”
We developed separate tools for automated data validation additional data enrichment (geolocation, addresses, specific calculations, etc)
Added data accordingly, following the clients’ specifications
Managed to assemble an efficient “drag and drop” configuration with block structure
We developed separate tools for
– automated data validation
– additional data enrichment (geolocation, addresses, specific calculations,etc)
Results
During the project agreed period, BGO Software managed to deliver about 520 different “data scraper robots” with a variety of custom extensions for specific data enrichment and extracting.
