Importing Data from Any Data Source to a Data Warehouse

ЗАДАЧА
A federal retail network faced a problem with collecting information from different Excel-files and data sources and importing them to a Data Warehouse.

The client had no single data source for business intelligence and data visualization tools, because employees collected data about sales, competitors’ activities, promotions, etc in non-structured Excel-files and difference data bases on their computers.

The client asked Byndyusoft to develop a tool for continuously importing any data source to a Data Warehouse and tune it through a web interface.

РЕШЕНИЕ

Byndyusoft worked together with the client’s Reporting and Analysis Department. We found that almost all data sources can export data in an Excel format so this format was chosen as a default for the data transfer.

In our system, an uploaded Excel file or a linked data source is parsed using a particular template. The template contains rules with constraints and metadata about internal data. The system can check an Excel file, show errors and suggest solutions for fixing the problem. After uploading and checking, the system put the data in a structured format and sent it to a Data Warehouse for further analysis and reporting.

Byndyusoft developed a web interface on React.js and Redux.js and solved the problem of processing more than 500,000 rows in a web browser. A back-end was developed on .NET Core by using Docker.

Byndyusoft создали для моего подразделения систему импорта данных из любого источника в DWH. Первый прототип был создан всего за две недели. Они отработали согласно роадмапу продукта, а первый старт рабочей версии сделали уже через месяц после начала работ.
Сергей Орешкин, директор систем отчета и анализа данных в 585 Золото
РЕЗУЛЬТАТ

Byndyusoft designed and developed a system that can upload, analyze and import data from any data source.

Any employee can link his data source to a Data Warehouse which will lead to quickly sharing important information within the company.

500,000+ rows
of data can be parsed in a web browser
50 data sources
have already been added by the client’s team
Infinite numbers
of further connected data sources
Data Warehouse · Reporting · Analysis