Evaluating feedback mechanisms in the automatic creation of assembly instructions using retrieval-augmented generation

In recent years, Retrieval Augmented Generation (RAG) has emerged as a promising technology for designing application-specific chatbots. RAG combines a Large Language Model (LLM) with application-specific information provided via a database. This opens up the possibility for companies to at least partially automate existing business processes using RAG. One use case relates to work preparation at a mechanical engineering company. Due to the high complexity of the products and the wide variety of variants, creating assembly instructions for individual assemblies and modules involves a considerable amount of administrative work. The use of RAG can help to significantly improve the efficiency of this process. However, a key challenge is to ensure high quality of automatically generated assembly instructions. While existing research has focused on feedback mechanisms for improving LLMs, the effectiveness of different prompt-based feedback types on RAG-generated technical documentation quality has not yet been sufficiently investigated. This article therefore explores how quality of assembly instructions can be improved by integrating feedback mechanisms into a RAG system. For this purpose, two different feedback mechanisms were used in a laboratory experiment. On the one hand, an exclusively quantitative feedback mechanism was used. On the other hand, quantitative and qualitative feedback were combined. The results demonstrate that feedback mechanisms contribute to improving quality of information in assembly instructions. In addition, it was found that exclusively quantitative feedback leads to similarly good results as more complex combined feedback.