Anyone who got their hands dirty managing software development and computer infrastructure projects probably appreciates the fundamental differences between the two. Most often than not, they are piled up together because they both involve computers (albeit differently).
Developing computer software is typically an incremental process where a prototype is developed first. The prototype can be discarded after proofing the concept (throw-away approach) or converted into the final product (evolutionary approach). Various factors influence the methodology selection, but I reckon the clarity of the requirements and the quality of the prototype itself are the most important.
The IT infrastructure is a different beast. Prototyping is not generally an option and incremental build up can be very risky and pricey. The requirements must be clear and precise because once an investment has been made (i.e. hardware’s procured), it is very difficult to go back. The challenge here is to ensure that new infrastructure components will integrate properly and not create more problems than they will solve.
Machine Learning (and AI) Projects
This is where software development and IT infrastructure overlap. Read on..
Nearly every large scale Machine Learning project involves external data sources that the organisation has no control over. The quality of such data cannot be assured. This is also true for the completion and correctness of the data.
One might argue that these data issues can be detected early on and dealt with. This is not untrue, and there are techniques and tools out there that can help detect these issues. However, there are other type of problems that are harder to detect until late stages, such as the degree of bias in the data.
It is not easy to estimate the computing resources early in the project, and testing with smaller data sample can be misleading. For instance, a model that converges in 10 hours with 10% of the data may not necessarily converge in 100 hours with the full dataset (on the same platform). In fact, the growth in computing requirement and time can be exponential.
How Good is “Good Enough”?
Achieving perfection in ML/AI projects is still a dream, and will be so for a while so that’s something we need to live up with. Therefore, what is hoped to achieve is a system that is “good enough” and “fit for purpose”. The 80/20 role really applies here but it’s the remaining 20% (or rather 19%) that takes up most of the resources.
Arguably, the most expensive resource is the data scientist time with average pay of £80k ~ £100k in the UK. In fixed contracts, the ML/AI vendor would want to finish the project as early as possible. In pay-per-day arrangements, however, the client would want to have the project delivered ASAP. What this means is that there is tendency for ML/AI projects to end prematurely as soon as they are deemed “good enough”.
Can’t Reach the Destination? Change the Journey
Of all possibilities, however, the most troublesome one is when the objective changes somewhere in the middle of the project. As weird as it may sound, it is not that uncommon and can be caused by various reasons. For instance, if the parties realised that the project would require double the amount of resources originally allocated to it. They may decide to lower the objective to avoid wasting the resources used so far. Something is better than nothing after all!
So Why ML/AI projects fail?
From my personal experience, I think that failing to manage expectations is one of the most prominent reasons. There is so much hype out there about ML/AI at the moment, blurring the boundary between what can be achieved, and what can’t. On the flip side, it is the hype that is fuelling the demand with no sign to stop for the time being!
The other reason is the failure to properly integrate ML/AI into the workflow (after successful delivery). Unfortunately, many managers perceive ML/AI projects as “IT things” for the IT guys to do. There is no such a thing as “IT project”. Rather, there is “Business Change” project that involves IT.
Conclusion: No One Size Fits All
The field of Software Engineering is relatively new and far from maturity. Methodologies and frameworks continue to emerge and evolve. This is also true for Machine Learning and Artificial Intelligence projects, which fall under the umbrella of Software Engineering (not perfectly though).
Managing expectations and sound business change management are keys to success. As for the development of ML/AI, the reliance on experienced project managers will continue until the whole process matures enough. Whether you are an organisation interested in ML/AI or a developer, my recommendation is to start with a small, easy-to-manage and low-risk project and move on as your experience and confidence levels build up.