We have been helping entrepreneurs & business mitigate the risks out of new product development efforts. We do it in many ways. One of which is to help them prototype quickly to come up with a Minimum Viable Product (MVP).

Such an MVP can be used to gather feedback from actual users and to test the viability and acceptability of the product. Our involvement varies from consulting to doing part of or the complete discovery, design & prototyping.

We recently completed an MVP for an established organization in India which is looking to develop an analytics solution aimed at a particular use case. The project involved the development of front-end and backend to upload large files, parse and transform it and then present analytics on the front-end. We thought of sharing the process, technological approach, and our experience with the community in the hopes that it can benefit someone looking to do something similar. Read on for the details.

The front-end

We have been using Angular for quite some time for our design & prototyping efforts. Given the ease of development, the effectiveness of Angular architecture and our familiarity, that was the obvious choice for the front-end development. We also used Angular Material as the design framework.

For the upload part, we generally use the ng2-file-upload component. This component is competent with almost everything that you require to include file uploads as part of your project. We went ahead and used the same for this project too.

The Upload backend

We developed the backend using Node.js/Express combination. To support large file uploads, we found Formidable to be the best-suited one. The decision was primarily due to the following two factors.

Formidable is fast multipart parser (~500 Mb/sec as per the documentation)
Better memory management since we were dealing with large sizes.
Just for the reference, here is the comparison between different Node multipart parsers.

Formidable has a default file upload limit of 200MB. You can set the limit something like this;

var form = new formidable.IncomingForm();
form.maxFileSize = 1000 * 1024 * 1024;

Data parsing & transformation

The requirement was to start the parsing & transforming the data as soon as the upload completes. If this code runs on the primary process, it’s going to block the application until the entire file is parsed and transformed, which runs into hours. Though Node.js runs in a single thread, you can use the child_process module to provide scalability. Read this excellent article for more understanding. All our parsing and transformation work was transferred to a child process, leaving the rest of the application to be responsive as expected.

Pushing to database

Generally, we would have preferred a NoSQL database (MongoDB), but MySql was mandatory in this case. We prefer NoSQL in general for early-stage applications which are yet to be validated by the markets. This choice helps on both accounts, costs, and flexibility to change. However, in this case, MySQL was mandatory. Of course, then we had to convert the data into the relational format, but the real challenge was pushing a large amount of data (millions of records per file). Running individual insert queries is impossible in this case. So we put the transformed data into another flat file and then used LOAD DATA INFLIE  syntax. We had to use –local option since the files were getting stored on the API server.


We used Chart.js to present the results of logical operations while developing the required algorithms ourselves. D3.js is usually our choice for such applications once they have reached a certain maturity. However, Chart.js is good enough to start with before the requirements become too complicated.

Here is what the process looks like;




We started with rough napkin sketches and very lo-fi wireframes and then did the UI on the go. It’s very minimal, but we indeed focus on the UX part even for prototypes and MVP. We finalized the sketches in 1.5 days, and then we completed the development and deployment in 10 days. We spent the bulk of the time on the parsing and transformation and the analytical queries. The rest of the infrastructure was ready in 2 days. These activities were done with 1 single senior developer, assisted part-time by a project manager. Here is an excerpt from the client feedback;

“Organizational processes and extraneous demands on the time of developers takes away a lot of development time. We could save about 40% of the cost that we would have incurred otherwise, and around 60% of the time. Our developers did not spend time on learning & implementing things which might not be useful when we go for the final product development. That would have added to the demands on their time & energy. Working with you allowed us to reduce some load on our developers, which goes a long way towards keeping them effective in the long run.”


When it comes to prototypes and MVP, there are a lot of tread-offs to be considered. Such tread-offs determine your design, toolset & approach. Of course, the kind of architecture employed ring now is not the right one for the actual production system. In fact, for every context, and the stage of the production, the design, toolset, and approach would vary. We will discuss these tradeoffs in the next article. Stay tuned!