The System collects data from more than 40 000 sites every 60 minutes and processes more than 1 million posts a day in total, providing detailed information on the audience, geography, sites etc. where a particular object is mentioned. About 50 indicators, charts and diagrams for analysis of the observed object are available in total.

The customer requested expert review from Byndyusoft after one of the leading developers of custom designed software in the Eastern Europe failed to design and implement a scalable solution with sufficient performance.

Problems of version 1.0:

1. Processing of data lagged behind the input of new data.
2. New data was received by the users with significant delay.
3. Preparation of data for new users took several hours instead of several minutes, which did not suit the requirements of the customer.
4. The System was absolutely non-scalable and refused to operate under load of only 10 users.

Reasons:

1. The architecture was based around Shared DB and triggers, which made impossible horizontal scaling of data processing.
2. Development was stuck because of inadequate covering of the system with autotests (about 10%), lack of UI tests and large number of regression errors.

Works on version 1.0 were performed by a team of 6–10 high-priced IT specialists for about a year.

After Byndyusoft took over the development, the tasks for the first 3 months were stated as follows:

1. Completely rewrite the mechanism of report generation (calculation of assays)
2. Completely rewrite the user interface of data presentation and handling of reports.
3. Improve data processing so that the system would be ready for submission to beta-testing with a load of 50 users and volume of processed data equal to 250 000 posts per day.

Conclusions, drawn by the Byndyusoft team from mistakes, which had been made by the previous team in the design of the system, wide experience in development of high load systems and professional usage of flexible development techniques allowed to create a new version of the System in compliance with all customer’s requirements in the shortest time possible.

The System must meet the highest requirements with regard to ability to deal with high load -ability to process up to 1,5 million posts per day, taking into account highlighting of object in the text, combining of messages into semantic clusters and aggregation of data about users. Therefore to decrease design risks about 10 prototypes with different architectures were created, each of which was tested under load. For each of them there had to be a possibility of horizontal scaling.

A version of architecture, which complied with all requirements, was discovered by a team of 3 persons in 2 months.

Data flows were moved to message queues. Further on, the queue could scale the processing for any amount of services and clients. A separate database with several tables, designed for complex selections, was created for each client.

Small DB and services can be located at large quantity of cheap servers, which allows horizontal expansion of power of the whole system.

From organizational point of view, work related to creation of new version of the System was based on the iterative and incremental approaches.

1. SCRUM framework was utilized with 1 week iterations.
2. Functionality was implemented in the system by parts in accordance with priorities, defined by the customer.
3. At the end of each iteration the customer received the System, fully operational in the part, taken for implementation.
4. Releases for beta-testing took place every 2–3 iterations.
5. The code was completely covered with autotests (back-end and front-end), opening the door for Continuous Integration. Working parts of the product were automatically delivered and integrated at the server for development after each modification in the code.

Results of work, provided by the Byndyusoft team had the following advantages for the customer:

• Byndyusoft spent half as much time and half as many resources, being approximately more than five times more cost-efficient than the well-known vendor.
• Byndyusoft was able to meet the deadlines and made it possible to timely start the product sales to real customers.
• Thanks to the utmost technological competence Byndyusoft was able to design the system, which meets the client’s performance, fail-safety and scalability requirements for several years ahead
• Due to correctly arranged communications with the customer at all levers — from business users to technical experts — Byndyusoft was able to take into consideration wishes of all interested persons during the development in a timely fashion, not wasting precious time of the customer.