How Machine Learning Differs from Traditional Software
Machine learning projects ultimately come down to delivering and implementing computer code. It’s no surprise, then, that the management of these projects is similar to traditional software development. Software project management is a well understood, well-documented field. If you’re going to do a machine learning project, it’s logical that you’d start with a software development mindset.
In fact, machine learning and traditional software projects are similar in many ways. But the differences are significant enough to completely derail ML projects that hold tightly to a traditional software mentality. Some of these differences are subtle enough that a project manager might not be aware of a misstep until long after it’s occurred.
The following sections describe key differences in traditional software development and machine learning development across several aspects of a project.
Problem Scope
All projects require a problem, a solution, and a definition of success. Machine learning is no exception — but project leaders may be surprised to find that all three can appear a bit hazy. Let’s compare:
Software Development
In traditional software development, participants know a realistic version of all three at the beginning of a project. For example:
Problem: I want to offer places to stay on a vacation that are not hotels.
Solution: Build a marketplace for temporary home rentals (Airbnb, VRBO, etc).
Success: A working software platform for renting homes.
Although all projects encounter unexpected speed bumps and requirements can change over time, the finish line is generally visible from the start. Even if every facet of the tool is not explicitly stated upfront, both stakeholders and developers know the tool’s general function and basic features. The solution is clarified through iterations. When feature ideas arise, they go into a backlog that is relatively easy to prioritize based on the underlying goal of the project.
Overall, any stakeholder is capable of judging the problem, solution, and success metrics, at least at a high level.
Machine Learning
The problem, solution, and success definition of a machine learning project can all be significantly murkier when a project begins. In some cases, the problem itself may need to change as the project progresses. A stakeholder may define the problem they think they have, but it might not be the problem that A) they really have, B) would be most helpful to their business, or C) machine learning can actually solve.
Even if the problem can be defined upfront, the solution may be harder to pin down, requiring iterative exploration of data and processes. For instance, the solution to the generic problem of “saving money by automating manual tasks” could take many forms in many areas of a company’s business.
And then even when stakeholders think they know the general solution to the problem, an exploration of the data often reveals roadblocks. Stakeholders may not anticipate these challenges when a project is initially being discussed or a Statement of Work is being negotiated — they are often very optimistic about their data. But invariably, the data discovery process will lead to unexpected questions and issues. It might take a month or two for the stakeholder and the data science team to get on the same page for what the actual solution should be.
So, while data scientists should pay attention to what the stakeholder has said they wanted, there may be more useful — or more possible — things to do with the data.
Instead of a straightforward race to the finish line, the project can feel more like a safari or a ride through the Wild West. Instead of a backlog of new features, exploration can lead to a backlog of new, different statements of work, or even new products.
This is the nature of machine learning. The scope of the project can (and sometimes should!) change as data scientists explore the stakeholder’s data and business needs. The end goal shifts by design. Stakeholders might imagine that data scientists are race car drivers who, when they run out of gas, get out of their cars and then paint the finish line on the ground behind them. But sometimes, data scientists really do have to design the course while they’re driving. With this kind of freedom and responsibility, data science efforts must be highly focused on business value rather than interesting data for its own sake.
When it comes to success metrics, data science teams often have to steer stakeholders to create them. Stakeholders often aren’t able to give specific guidance on their own. While they likely have overall goals — more automation, less cost, more efficiency — they may lack a clear idea of how to define success, or how to give constructive feedback along the way.
What they can scrutinize is the final product — and if their knowledge of ML comes from reading headlines, they may be comparing the result to their unrealistic — and perhaps unstated — expectations. 85% accuracy could be a fantastic result, especially if it improves on human performance. But stakeholders might expect an “intelligent” machine to work at 100% accuracy.
One critical place to probe for misalignment is whether the stakeholder expects an integrated software product with a user interface — and whether that is, in fact, what the data science team believes it is building. User interfaces and APIs are a great way to describe what the touchpoint looks like between your ML solution and the rest of the software/team/organization. They are, just as their names indicate, interfaces or the layer in between two things.
Without clear alignment, it’s possible that stakeholders will imagine an interface as the project’s endpoint, while data scientists believe the project ends when they demonstrate that the algorithm works. In short, stakeholders could be expecting a deployable solution, while data scientists may believe they’re working on a pilot project.
Even if everyone is aligned and no one expects an interface, the absence of one can make it harder to judge a project’s success unless there are other agreed-upon measures. It’s easier to see something working. It’s less satisfying to simply be told that it works.
Timelines
Project managers are nothing if not professional timeline makers. But even those who are used to missed deadlines might find their organizational predictive skills are tested on an ML project.
Software Development
While project slowdowns can happen, projects move in one predictable direction: forward. Wise project managers can anticipate a few delays and build them into their plans.
Then, once a project is done, it’s done; teams can move on to the next thing.
Machine Learning
Machine learning projects require a special degree of patience and flexibility. This is true even if the scope doesn’t change as described above.
If a project is tightly scheduled but the data isn’t ready to go, the real work of machine learning could be delayed by a month or more just waiting for the data that was supposed to be available from the start. Once the data is ready, algorithm development may also take longer than anticipated, as data scientists try a number of things that don’t work before finding a solution. Such delays can cause a ripple effect — if a three-month project takes four months, what happens to the project that was supposed to start after month three?
Unlike “set it and forget it” traditional software projects based on explicit rules, it’s important to keep an eye on machine learning implementations even after the bulk of the work is done. After an algorithm is trained and deployed, it will likely need to be monitored, optimized, and eventually retrained after it encounters new data. Project planning that doesn’t account for this could result in team members being dragged back into projects unexpectedly.
Resourcing
Picking the right team is essential — but what if you’re not entirely sure what kind of team you need?
Software Development
Projects that are relatively straightforward have relatively predictable resourcing needs. Project managers are less likely to scramble for more people at the tail end of a project.
Machine Learning
Scope changes and project shifts can mean that resourcing needs change as well. This can be especially true of engineering resources, which may need to be increased once deliverable and deployment needs (and possibilities) become clearer.
Stakeholders
Stakeholders always play a key role in any project. They are, after all, who the project is ultimately for. On machine learning projects, their involvement is especially essential to making progress.
Software Development
Since software can be developed on a separate system before it gets integrated and deployed, a technical team can get to work almost immediately after a project kickoff without much heavy lifting from stakeholders. Developers can use placeholder content and begin creating site architecture, wireframes, and general design templates. They can then build software on their own system and have a reasonably clear idea of what will happen when they integrate it with stakeholder data.
Stakeholders can be highly involved in software development projects as needed, but they’ll also be in a position of reacting to and guiding the steady progress that the development team can make on their own.
Machine Learning
The beginning of a project can be slow because data scientists need to get access to data. Typically, this involves setting up a virtual machine (VM) to view the data in a stakeholder’s system, as opposed to getting the data directly. Sometimes, the machine learning model will be built inside the VM as well, since the data cannot leave. Setting up this VM requires stakeholder participation.
Overall, data scientists rely heavily on stakeholders to understand and access data and to help them explore what problems are solvable and important. The first several months of a project may be more about asking questions than going off to make rapid progress. In fact, a machine learning project that doesn’t start with a lot of thoughtful questions should stoke stakeholder suspicions.
Guiding Stakeholders
While project management is never passive, machine learning projects require especially proactive management. If a stakeholder is new to machine learning, they’ll need extra guidance on how to move forward.
Software Development
Project managers are often in the position of executing stakeholder guidance and priorities, rather than actively shaping those things. Furthermore, stakeholders may feel empowered to scrutinize the budget and timeline, questioning why a task projected to take a week can’t be done in an hour. In other words, they feel they know enough to question the manner in which a project is carried out.
Machine Learning
Even though ML projects can be more dependent on stakeholder actions than traditional software projects (see previous section), they also require more proactive project management on the ML side.
Project managers often need to guide stakeholders through development, testing, and deployment. Because ML is new and has a conceptual learning curve, stakeholders look to the ML team for guidance. They may not question tactical issues like timelines or budgets.
This might seem like a golden moment for ML project managers. After all, who hasn’t wished for less scrutiny from stakeholders?? But the current situation has several potential downsides. For one thing, as long as machine learning seems magically inexplicable, stakeholders may have unrealistic assumptions about what it can accomplish (as opposed to their unrealistic assumptions about how it can be accomplished in traditional software projects).
Furthermore, the team members on the stakeholder side who need to deliver data or set up software integrations may be slow to do so because of the actual or perceived difficulty of a machine learning project as compared to more straightforward software tasks they are used to.
Of course, as stakeholders get more experience with machine learning, and as software tools make it less mysterious and more accessible, stakeholders may begin to offer better guidance and bring a deeper understanding of the tactical decisions that data scientists make. But for now, project managers who are not proactively defining tasks, setting up meetings, and asking questions may get a false sense of security. Stakeholders won’t know to raise certain questions or red flags, and so major issues may not be discovered until long after they should have been dealt with. If a machine learning stakeholder is along for the ride but isn’t driving, the ride may be a smooth one right up until the wheels fall off.
Guiding Technical Teams
ML project leaders may also need to be especially proactive with their own teammates. Working with developers can be a bit different than working with data scientists.
Software Development
Project managers may play more of a support role for their technical teams. Developers can move quickly through tasks until a project is done and may do so with minimal guidance.
Machine Learning
Technical teams may need special oversight for machine learning projects. Data scientists are used to conducting research for extended periods. While research is a necessary part of the machine learning process, scientists may sometimes need to hear that it’s time to move on. There’s always another efficiency gain to chase from another algorithm or another tweak to the data — but sometimes, from a business perspective, those efforts have diminishing returns.
Showing Progress
In machine learning, it can be harder to show that progress is being made than on a traditional software project.
Software Development
Developers can show steady progress on a periodic basis. If a project is meant to have eight eventual features, it’s possible to demo each new feature as they are built. As work moves forward, developers can show user flow diagrams, wireframes, and polished designs that stakeholders can react to.
Machine Learning
Stakeholder meetings often have less “demo-able” information. The meetings may consist more of questions, clarifications, process logistics, and quantitative results that may not lend themselves to straightforward visualizations or a sense of consistent progress. Moreover, team tasks can involve more research than action. For example, teams may need time to explore which data collection sources to use; the output of that research could be a discussion document rather than the action itself. The action comes later once the decision is made.
During this slower-seeming period, it is important for data scientists and ML project managers to clearly communicate that the apparent lack of progress is normal. Encouragement like “This is typical of any ML project” and “We made really great headway today with a better understanding of your data thanks to your clarifications” can provide much needed emotional momentum, even when it seems like a project is stalled. These early days are also a good time to align or realign on evaluation and success metrics before results come in.
Even as the project continues, ML progress reports can still be murky. We know a stakeholder who, at the end of an hour-long technical presentation, asked the team how they felt about the project. She was trying to translate the deep dive into a clear sense of whether the project was going well.
Fundamentally, machine learning projects require stakeholder trust in the data scientists that a project is really working, and that the data scientists didn’t just pull results manually.
Testing Results
Testing software can be straightforward. Testing a machine learning project can be much more complex.
Software Development
Software testing often involves making sure the software works, checking for bugs, and making needed tweaks. Developers and stakeholders can both do this work. After weeks or months of steady progress and testing, it’s unlikely that fundamental issues will suddenly arise as the project comes to a close. Once a software project is complete, the results that it delivers rarely change in a significant way.
Machine Learning
Testing a machine learning model can have the feel of running an experiment with a control group, a test group, a baseline ground truth, a hypothesis, and actual outcomes.
It’s important to have early alignment with stakeholders on how tests will be conducted and evaluated, perhaps by presenting stakeholders with different options to choose from. The proposed test plan needs to account for factors like evaluator bias, speeds at which different evaluators work, and time to get used to a new process. The idea is not to simply see if the model “works” in general, but whether it is useful. In other words, whether it is better than the current process it’s meant to replace or upgrade.
It’s possible for the test to not to give the results one wants or expects, especially when it comes in contact with new data. The evaluation process may reveal lessons that need to be applied back to the model to improve results, or even go back to the drawing board and make fundamental changes.
Deployment and Beyond
While the process of implementing and maintaining traditional software has its challenges, putting a machine learning project into real-world use takes special consideration.
Software Development
Traditional software is built to be deployed, and customers are generally eager to implement it as soon as possible. In fact, they may push developers to get a project done by a deadline, even if it means shipping without every desired feature.
After deployment, it’s well known that software maintenance costs can be larger than the initial investment. Maintenance tasks will be fairly straightforward debugging work, as well as fixing challenges involved with integrating additional software into the system as time goes on. The core nature of the solution is fairly static, however — given certain inputs, it’s always going to return expected outputs. There’s little risk that the system will dramatically change.
Machine Learning
Deployment can be a large obstacle in ML projects. Stakeholders may not be ready to immediately deploy the model they’ve built. This may be because they’re not technically ready to do so, employees aren’t ready to use it, or they didn’t think about how to reorganize the company based on the new capability. Non-technical management factors can increase the time it takes to realize hoped-for savings or efficiencies as a company adopts new technology.
To quote Tom Davenport and Randy Bean in the MIT Sloan Management Review:
If the AI initiative actually changes the relevant business process and the skills necessary to perform it, that raises another barrier to full implementation — the old bugaboo “change management.” Most AI systems still involve some interaction with human workers, and educating those workers on new tasks and new skills can be time-consuming and expensive.
Unlike traditional software, ML projects don’t necessarily just add another tool in a well-understood toolbox. They can bring large-scale organizational transformation. And change is hard, especially (and perhaps ironically) for the kinds of large organizations who actually have the resources to do ML projects. Machine learning practitioners may need to go the extra mile to get stakeholders to implement a model, such as a building a user interface or helping facilitate a pilot program so they can better understand its potential.
Once an ML solution is deployed, the cost of ongoing maintenance could be higher or lower than the initial investment, varying based on the nature of the new data the system takes in and the way the machine learns as a result. Models need to be monitored to avoid and adjust for drift, which can happen when new data skews models in a sub-optimal way.
Summary
Let’s face it — for many businesses, software development projects are important, they rarely have the capacity to be fundamentally transformative. While machine learning projects have their unique issues, they also come with the potential for massive reward at scale in the form of transformative business impact.
Key ways that machine learning projects differ from traditional software projects include the following:
1. The problem, solution, and success conditions are harder to define.
2. Timelines are more variable.
3. Resourcing needs may shift.
4. ML projects are more reliant on stakeholder actions, such as setting up a VM to grant data access.
5. ML projects require more proactive project management to guide stakeholders through a less-familiar process.
6. Technical teams may need guidance toward action rather than research (even though research is very important).
7. Progress can be harder to track periodically — the work might not lend itself to visualization or steady improvement.
8. Testing can seem experimental and could send teams back to the drawing board.
9. Stakeholders may not be ready or able to implement results right away.
Still, despite their difference, it’s worth noting that there’s a reason we explain machine learning projects in the context of software projects. The software mindset is a good starting point for machine learning project planning. The problem is not going beyond it.
Robbie Allen is a Senior Advisor to Infinia ML, a team of data scientists, engineers, and business experts putting machine learning to work.
