We have been helping entrepreneurs & business mitigate the risks out of new product development efforts. We do it in many ways, one of which is to help them design, develop and prototype quickly to come up with a Minimum Viable Product (MVP) that can be used to gather feedback from actual users and to test the viability and acceptability of the product. Our involvement varies from consulting to actually doing part of or the complete discovery, design & prototyping.

We recently completed an MVP for an established organization in India which is looking to develop an analytics solution aimed at a very specific use case. The project involved the development of front-end and backend to upload large files, parse and transform it and then present analytics on the front-end. We thought of sharing the process, technological approach and our experience with the community in the hopes that it can benefit someone looking to do something similar. Read on for the details.

The front-end

We have been using Angular for quite some time for our design & prototyping efforts. Given the ease of development, the effectiveness of Angular architecture and our familiarity, that was the obvious choice for the front-end development. We also used Angular Material as the design framework.

For the upload part, we generally use the ng2-file-upload component. This is a very capable component with almost everything that you require to include file uploads as part of your project. We went ahead and used the same for this project too.

The Upload backend

Our backend is developed on Node.js/Express combination. To support large file uploads, we found Formidable to be the best-suited one. The decision was primarily due to two factors; 1) Formidable is really fast multipart parser (~500 Mb/sec as per the documentation) and 2) Better memory management since we were dealing with large sizes. Just for the reference, here is the comparison between different Node multipart parsers.

Formidable has a default file upload limit of 200MB. You can set the limit something like this;

var form = new formidable.IncomingForm();
form.maxFileSize = 1000 * 1024 * 1024;

Data parsing & transformation

The requirement was to start the parsing & transforming the data as soon as the upload completes. It’s perhaps obvious that if this is run on the same process that the rest of the application runs, it’s going to block the application till the entire file is parsed and transformed, which runs into hours. Though Node.js runs in a single thread, you can actually use the child_process module to provide scalability. Read this excellent article for more understanding. All our parsing and transformation work was transferred to a child process, leaving the rest of the application to be responsive as expected.

Pushing to database

Generally, we would have preferred a NoSQL database (MongoDB etc.), but MySql was mandated in this case. We prefer NoSQL in general for early-stage applications which are yet to be validated by the markets. This helps on both accounts, costs, and flexibility to change. But in this case, MySQL was mandatory. Of course, then we had to convert the data into the relational format, but the real challenge was pushing a large amount of data (millions of records per file). Running individual insert queries is impossible in this case. So we put the transformed data into another flat file and then used LOAD DATA INFLIE  syntax. We had to use –local option since the files were getting stored on the API server.

Analytics

We used Chart.js to present the results of analytical operations while developing the required algorithms ourselves. D3.js is usually our choice for such applications once they have reached a certain maturity. But Chart.js is good enough to start with before the requirements become too complex.

Here is what the process looks like;

ocspprocess

Execution

We started with rough napkin sketches and very lo-fi wireframes and then did the UI on the go. It’s very minimal, but we certainly focus on the UX part even for prototypes and MVP. The sketches were finalized in 1.5 days and then we completed the development and deployment in 10 days. The bulk of the time was spent on the parsing and transformation and the analytical queries. The rest of the infrastructure was ready in 2 days. This was done with 1 single senior developer, assisted part-time by a project manager. Here is an excerpt from the client feedback;

“Given organizational processes and extraneous demands on the time of developers (meetings are the biggest time consumers), we could get this done from you in about 40% of the cost that we would have incurred otherwise, and in around 60% of the time. You have saved us both. What is also important that our developer would have had to spend time on learning & implementing things which might not be even used when we go for the final product development. That would have added to the demands on their time & energy. Working with you allowed us to reduce some load on our developers, which goes a long way towards keeping them effective in the long run.”

Conclusion

When it comes to prototypes and MVP, there are a lot of tread-offs to be considered. Those tread-offs will determine your design, toolset & approach. Of course, the kind of architecture employed will not be the right one for the actual production system. In fact, for every context, and for the stage of the production, the design, toolset, and approach would vary. We will discuss these tradeoffs in the next article. Stay tuned!