The System collects data from more than 40 000 sites every 60 minutes and processes more than 1 million posts a day in total, providing detailed information on the audience, geography, sites etc. where a particular object is mentioned. About 50 indicators, charts and diagrams for analysis of the observed object are available in total.
The customer requested expert review from Byndyusoft after one of the leading developers of custom designed software in the Eastern Europe failed to design and implement a scalable solution with sufficient performance.
Works on version 1.0 were performed by a team of 6–10
After Byndyusoft took over the development, the tasks for the first 3 months were stated as follows:
Conclusions, drawn by the Byndyusoft team from mistakes, which had been made by the previous team in the design of the system, wide experience in development of high load systems and professional usage of flexible development techniques allowed to create a new version of the System in compliance with all customer’s requirements in the shortest time possible.
The System must meet the highest requirements with regard to ability to deal with high load -ability to process up to 1,5 million posts per day, taking into account highlighting of object in the text, combining of messages into semantic clusters and aggregation of data about users. Therefore to decrease design risks about 10 prototypes with different architectures were created, each of which was tested under load. For each of them there had to be a possibility of horizontal scaling.
A version of architecture, which complied with all requirements, was discovered by a team of 3 persons in 2 months.
Data flows were moved to message queues. Further on, the queue could scale the processing for any amount of services and clients. A separate database with several tables, designed for complex selections, was created for each client.
Small DB and services can be located at large quantity of cheap servers, which allows horizontal expansion of power of the whole system.
From organizational point of view, work related to creation of new version of the System was based on the iterative and incremental approaches.
Results of work, provided by the Byndyusoft team had the following advantages for the customer: