Development Process for AI Apllications

Process for engineering AI software

By Published On: September 16th, 2021Categories: AI, machine learning, Research and Development, software engineering

What is the Motivation for a Development Process for AI Projects?

AI applications often involve not only classical application engineering but also elements of research (Fig. 1). Sometimes it is not clear from the start which approach will be better and one needs to conduct experiments to evaluate multiple approaches. For example if we are building a machine learning model we would need to evaluate and experiment with different features until we find an optimal feature set. Further if we are building machine learning models usually debugging is not an easy task. Also in many cases it is not trivial to evaluate the performance of statistical models and how this performance will translate to business value.

All these factors can add an additional layer of complexity the engineering teams need to cope with.

Fig.1: AI systems are at the intersection between Application Engineering and Research.

 

Development Process for Engineering AI Applications

One way to manage the inherent complexity in AI projects could be to introduce a clear methodology in the development process. An easy to understand process usually helps members of the engineering team and also business stakeholders to be on the same page where the projects is at the moment and also what are the next steps. A well defined process should help with the construction of a technical road-map and also help in raising the transparency in general.

Since AI projects can be seen as located at the intersection between application development and scientific research, we argue that these projects can benefit from some practices commonly used in research and academia. In this post we will present an AI Engineering Cycle. The cycle is motivated by the work presented in the following paper [1] and it is very much related to the idea of Falsifiable Hypothesis introduced by Karl Popper.

The AI Engineering Cycle can be seen in Fig. 2. Some steps probably could be seen also as one logical step, however we have split them here for more clarity. Further the scope of this process is related to experimentation. It is not capturing all activities – e.g. gathering of test data is not depicted, however still needed.

Fig.2: AI Engineering Cycle for constructing AI systems.

In my experience one of the key points to have clarity in AI projects is to always start an experiment with a clear falsifiable hypothesis, i.e. hypotheses which can be proven to be wrong. If the engineering team does not have a clear hypothesis in mind to start with, there is the risk that the team looses itself in experimentation – a problem which can be compared with chasing one’s tail.

After starting with a clear hypothesis and going through the experimentation cycle one can evaluate the results and prove or disprove the hypothesis. With the new insights the engineering team can then create a new hypothesis and start the next experimentation cycle.

Example

For example, if we want to evaluate if a new feature will improve the performance of a machine learning model one run through the AI Engineering cycle could look as:

  1. Falsifiable Hypothesis: The new feature X will not improve the performance of the model with the data set T.
  2. Design an Experiment: in this case the design of the experiment is quite simple. We just need to add the new feature in the training process, train the model and evaluate with the test data set.
  3. Implement the Experiment: add the new feature in the Training/Validation/Test data set and in the training process.
  4. Run the experiment: run the training to produce a model.
  5. Evaluate results: evaluate the model, one needs to evaluate with the same evaluation metrics as the baseline model in order to have comparability of course.
  6. Draw conclusions about the hypothesis: if the evaluation metrics indicate a better performance we have disproved the negative hypothesis. The feature does improve the model with the given data set T. As next experiment we could for example test if the feature is still beneficial with a larger data set.

 

References:

[1] Sanders P. (2009) Algorithm Engineering – An Attempt at a Definition. In: Albers S., Alt H., Näher S. (eds) Efficient Algorithms. Lecture Notes in Computer Science, vol 5760. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-03456-5_22

[2] The Logic of Scientific Discovery, Karl Popper

Do you have questions about this article or just want to discuss the topic? Do not hesitate to contact us at

— — —

We put a lot of effort in the content creation in our blog. Multiple information sources are used, we do our own analysis and always double check what we have written down. However, it is still possible that factual or other mistakes occur. If you choose to use what is written on our blog in your own business or personal activities, you do so at your own risk. Be aware that Perelik Soft Ltd. is not liable for any direct or indirect damages you may suffer regarding the use of the content of our blog.

Author: Luben Alexandrov

Luben is the main consultant for software engineering and software architecture questions at Perelik Soft. Always looking for ways to improve the efficiency of the software teams.

Share this story

You might be interested

  • Process for engineering embedded software

    In modern technologies, where the trend is towards mobility and compactness, embedded systems find growing number of applications. [...]

  • Checklist for an efficient code review

    Why is code review important? Peer code review is a widely used technique by software engineering teams. The [...]

  • 5 tangible metrics for the speed of software engineering teams

    Firstly,  since we want to discuss the topic of metrics, what does it mean for a software engineering [...]