The solution helps large companies collect, sort, and analyze data by category in one system, as well as format, run and import files automatically at any convenient time. Businesses get an estimate of data quality and a history of how data has changed by month or day. The quality shows the error rate, the total amount of data. The project uses a fully distributed cloud architecture, which allows individual services to be developed, tested, and scaled independently of each other. On this project, our team's task was to migrate the existing customer system to an external platform and create a data validation. The challenge was to collect the existence of a huge amount of data that was inconvenient to manage and manipulate.
Also maybe interesting
What Was Done
In order to automate the process of transferring data to an external service and organize the information, we developed filters for quick search and sorting. With the help of filters, the system selects valid data and helps to find anomalies and errors in huge data sets.
The data is downloaded automatically through the user interface and can send files to sources or to the cloud. The user can upload all types of data formats to the application: Excel, JSON, etc. Users can also set up a schedule for launching files (once a week, or on certain days at certain times).
The system is divided into microservices. All the parts of the microservices are independent of each other. There is the site itself, which sends commands between the microservices. The data is uploaded not just to the database, but to a search index, and because of this, we have both search and validation in seconds. Thanks to the intelligent design of the solution, users can use only 10% of the functionality without having to run the whole system. Also, they can scale separate parts. For example, start 2 validation modules and they will work simultaneously and turn on/off when it needs to save money.
We also developed a visual display of information and data in a form of various graphs.
Project team: 2 developers, 1 PM.
Technology: Azure Search Index, .NET web API, React, Azure Functions, Azure Service Bus, Azure Queue, Azure Search index.