AI-based quality monitoring of auto-body weld joints in real time

Share this on social media:


Christian Knaak and Peter Abels, of Fraunhofer ILT, explain how imaging and machine learning can be used to identify weld imperfections with high accuracy

With regard to current trends such as Industry 4.0 in the field of industrial production, the laser is proving to be the ideal partner for the automation of production processes.

In automotive production the laser has gained popularity due to its precision, flexibility and speed. Nevertheless, even with the further development of laser machines and systems, faulty products resulting from irregularities in production processes are still a problem.

The demand for high-quality, high-end products requires modern, process-oriented monitoring methods in order to maintain low rejection rates and high quality standards. In the application of laser beam welding, the use of synergy effects in the optical set-up enables the simple integration of complex sensors such as cameras or spectrometers into the processing head. The use of such sensors during laser welding enables the in-situ monitoring of the production process for the detection of component-critical quality deviations, such as lack of fusion, pores or cosmetic seam irregularities.

In addition to real-time data processing, a special challenge is the generation of a meaningful fingerprint of the current process status. If the process zone is observed with a camera, for example, the question arises as to what information in the captured data stream is relevant and thus describes the current process state as precisely as possible. In practice, several different sensor data streams during the process often have to be analysed in order to enable the current process status to be classified into a defined quality category.

At the Fraunhofer Institute of Laser Technology (ILT), we have not only answered this question, but through using a combination of imaging sensors and machine learning, have developed a system capable of identifying seam defects during the welding of galvanised steel sheets with a success rate of more than 99 per cent.


Firstly, in order to detect welding defects during processing, an infrared camera system was integrated into our chosen processing optic for in-situ inspection. Due to the short interaction time between the laser and material, cameras with a low frame rate would have been unsuitable for monitoring the welding process, as they cannot adequately display short-term process fluctuations, for example spatters. The camera used therefore has a high recording speed of 1kHz, in addition to a spectral sensitivity in the range AI of 1-5µm. With the aid of optical components, the thermal radiation emitted from the laser interaction zone could be imaged coaxially to the laser beam on the sensor of the thermal imaging camera. The thermal image acquired in this way, shown in figure 1, provides information about the keyhole – a vapour capillary in the material induced by the laser radiation – which characterises the laser beam’s deep penetration in the welding process. In addition, the thermal image shows the weld pool and its cooling behaviour.

Figure 1: Thermal image of the laser interaction zone and characteristic features. 

We then derived characteristic features from the captured image data and combined them into a process fingerprint. Here, image processing methods are suitable for describing the geometry of the vapour capillary and the melt pool with the aid of simple geometric shapes such as an ellipse.

In order to gain additional information about the heat distribution with respect to the melt pool surface, it is possible to determine statistical key figures such as variance, curvature, skewness and average value in the different image areas. A further source for suitable process characteristics is the temporal course or change over time of certain image characteristics. For example, the temporal variance of the vapour capillary size can be a measure of its stability and provide information about the formation of pore nests.

The process fingerprint is composed of characteristics such as the cooling rate of the liquid melt, the melt pool surface geometry as well as the shape of the vapour capillary and its change over time. The determination of the characteristics is computationally intensive, and for real-time application is realised with a field-programmable gate array (FPGA), in which several calculations can be executed in parallel. Immediately after image acquisition, the FPGA calculates the process fingerprint for each captured thermal image.

In the next step, welding tests were carried out in the laboratory, in which different seam imperfections occurred during welding, which we organised into different categories:

  • Category 1: Seam OK (no abnormalities)
  • Category 2: Seam collapse
  • Category 3: Lack of fusion (‘false friend’)
  • Category 4: Seam width increased
  • Category 5: No seam

During the welding experiment, lack of fusion, so-called ‘false friends’, and seam collapses were specifically created. The extracted process fingerprints were then manually categorised by welding experts according to their associated weld seam quality.

By using a specific procedure known as ‘Sequential Forward Floating Selection’, the importance of the features used were also determined. In addition to sorting out redundant and unimportant process features, this also allowed an estimation of the type of information from the different areas of the process zone that is important for the detection of seam imperfections. Figure 2 shows this assessment for the features extracted from the thermal image data. It can clearly be seen that at least six features were required to achieve a detection rate of over 99 per cent in the laboratory set-up. The method classifies features from the keyhole range as the most relevant features in comparison.

Figure 2: Evaluation of image features based on ‘Sequential Forward Floating Selection’.

The learning algorithm ‘k-nearest neighbours’ was then used with this data to derive rules for classifying the process fingerprints into a quality category. The rules could then be used, in the form of a trained classification model, to evaluate further welding processes. Figure 3 shows such an evaluation of a weld seam consisting of two galvanised cold forming steel sheets (0.9mm thickness per sheet, gap 0.190mm). The classification algorithm recognises the quality state at specific locations on the seam based on about 4,000 process fingerprints extracted from the thermal image data. For example, lack of fusion defects (red) were detected, which are not visible in the top view, but can be displayed in a corresponding longitudinal section. Further imperfections such as seam collapse and the absence of the seam can also be detected in a differentiated way by the developed system.

Figure 3: Quality assessment of a weld using the proposed machine learning approach.

Results and discussion

The in-situ process observation by means of high-speed thermal imaging enables the detailed observation of the interaction zone of the laser welding process. In combination with machine learning methods, it is possible to identify different process faults during the manufacturing process. A machine learning system was trained successfully to distinguish process imperfections on the basis of their specific fingerprint. The average accuracy of the classification result reaches up to 99.56 per cent for laboratory samples. During the development of the system, 16 characteristics were determined based on the image information of different process areas. It has been shown that using at least six features from the keyhole and melting bath area enables a recognition rate of over 99 per cent for this application example.

The field of application of AI-based process monitoring within the scope of this application example is the industrial laser beam welding of automotive components. Here it enables the automated, real-time evaluation of weld seams. The method of measurement technology, as well as the evaluation strategy under consideration of learning algorithms, can be transferred to many machining processes within and outside of laser technology. In addition to the detection of process imperfections, it offers the possibility of documenting production steps, increasing process understanding and can be the foundation for adaptive control strategies.

Christian Knaak is a research associate in the Process Control and System Technology Group at Fraunhofer ILT.

Peter Abels is the head of the Process Control and System Technology Group at Fraunhofer ILT.