What are the steps to successfully create a data project?
Over the course of developing new products, we identify three distinct phases: Exploring, productionalizing, and scaling & optimizating. All phases have four different steps: Refinement, building a walking skeleton, developing a solution(s), and documentation & delivery.
Explore new data product
Project’s use case is identified and a proof-of-concept (POC) is developed to demonstrate the potential solution(s). This includes data collection and analysis, feature engineering and model development. The goal of this phase is to validate the feasibility of the proposed solution.
Productionalize data product
Building on the POC, this phase focuses on developing a minimal viable product (MVP). This includes further refinement of the model, implementation of the necessary infrastructure and integration with other systems. The MVP is a functional solution that can be tested with real users in production.
Scale & optimize data product
In the final phase, the focus is on scaling the MVP to handle an increased workload and optimize the model for performance. This includes fine-tuning the model, implementing data pipelines, monitoring and testing to ensure the solution is stable. The goal is to ensure the solution can handle the intended usage and deliver the desired business value continuously.