Analyzing metallographic images with manageable effort is possible with the use of artificial intelligence, especially when it comes to distinguishing between grain boundaries and grinding marks, for example. A microscope manufacturer explains how such a neural network is constructed and used.

Conventional threshold-based analyses are carried out as a standard method in digital image processing. These analyses are often specified as a requirement in standards for the microstructure analysis of metals, alloys, ceramics, composites, and other materials. While it is an established method, image thresholding has some limitations.

Thresholding does not detect specific structures in the images. It detects multiple objects at once without distinguishing between them. Analytical algorithms such as thresholding can use additional approaches, such as edge enhancement filters, shading correction, and morphological analyses, to find specific structures. However, these approaches require programming skills and effort to enable automated analyses. Further, some problems may not be solvable with these approaches, taking into account the potentially huge number of special cases and exceptions.[1] In contrast, machine learning forms rules for object detection based on multiple examples of representatives of objects of interest.

Applying AI-Assisted Image Analysis to Metallography and Materialography

Image analysis assisted by artificial intelligence (AI) promises to solve many of the issues associated with analytical approaches. Automated evaluation, supported by deep artificial neural networks that have learned to classify image areas independently of previously set threshold values in the image, opens a new world of easier and more accurate image analysis. In metallography and materialography laboratories, AI-assisted image analysis revolutionizes everyday work.

The differences between the threshold method and an automated evaluation with AI is shown when we look at a metallographic sample with grain boundaries, polishing marks, and dust. With the simple threshold setting, the image analysis software cannot clearly distinguish the grain boundaries from the polishing marks and dust. This results in an incorrect grain size measurement since it is impossible to detect only the grain boundaries.

With the assistance of AI, traces of grinding, polishing, dust, and residues can be distinguished from objects of interest, such as grain boundaries, in images of polished sections. AI-assisted image analysis can detect grain boundaries in microstructures that have very inhomogeneous grain structures with reliability and reproducibility. In addition, structural components can be classified with pixel accuracy.

The Standard Workflow for AI-Assisted Microstructure Analysis

Supervised learning, as a subfield of machine learning, is a term for methods designed to predict or classify an outcome of interest. With these methods, labels are used to clearly specify which features the learning model should detect.[2, 3] The resulting algorithm is based on the labeled data. Care must be ­taken so the algorithm does not fit the training data too closely. Otherwise, the algorithm may correctly detect objects in the training data but fail to perform with the same quality on new data.

For deep-learning-based image analysis, the labeling of data requires creating images with “ground truth,” the information with which the neural network is trained and evaluated. [4] This information must be marked in the images through image processing or hand labeling.

Further, the verification of the training data by specialists is essential. Only a specialist can define which data should be used for the training as it is the source the trained neural network uses for the analysis. The specialists must be experts in the material analysis application so they can make decisions about which details in the image should be detected.
Using the example of a metallographic section, the specialist might ask: When is the feature considered a grain boundary? How do we evaluate abnormalities? Importantly, the data must be representative of all expected objects and mappings within each class.

After this first phase, the second step is to select the optimal training configuration for the task. This is done using instructions for augmenting the training data [5] and selecting the training model. Augmenting the training data supports the training by giving the neural network model significantly more opportunities to learn and increase its reliability. The training data is multiplied by rotation, mirroring, and other image operations. It is important to note which augmentation method makes sense for the specific application. For example, rotation is useful for structures with no preferred direction but not useful for elongated materials such as rolled materials.

In deep learning, an artificial neural network with a given structure is created, but the decision-making process that the network uses is hidden. The network does not provide any analytical justification as to why a decision is made.
However, a specialist can validate the training success by seeing if the results from an analysis fit the expectations. It is possible to use validation data sets to compare how well the trained artificial neural network can recognize the specified image areas. The network can be used to generate a probability map which can be displayed during training on labeled validation images as an overlay. The validation data are not part of the data set that was used to train the network. To achieve a realistic assessment of the training status, the similarity between quality criteria, such as loss, evaluated on the training images and the validation images can be output numerically, and as a graph.

After a training that includes validation has been performed, a new data set is used to check whether the algorithm still works on representative new data: the test data set. This final test must be verified by a specialist, or ideally by multiple experts, to reduce the risk of misinterpretations of AI results due to human bias. [6] The trained neural network is now available as a segmentation method that can be applied to comparable images, such as images with similar light and exposure conditions. Applying a properly trained neural network can be realized easily and efficiently. With a single click, the network automatically segments the image and delivers reproducible results.

Conclusion

With manageable effort from a specialist, it is possible to use supervised deep learning in metallographic and materialographic image analysis for clearly defined tasks, such as image classification, object detection, and image segmentation. A specialist is essential for validating the marking of the ground truth in the training images, the test data set, and for checking the final evaluation test. Provided that the artificial neural network is well trained, AI offers powerful, reproducible image analysis on comparable images.

To learn more about the benefits of AI in microstructure analysis, go to olympus-ims.com/ landing/truai-technology.

References

[1] M. A. Nielsen. “Neural Networks and Deep Learning.” Determination Press, 2015.
[2] M. Iqbal, Z. Yan. “Supervised Machine Learning Approaches: A Survey.” International Journal of Soft Computing, 2015, 5, 946–952.
[3] T. Jiang, J. L. Gradus, A. J. Rosellini. “Supervised Machine Learning: A Brief Primer.” Behav Ther., 2020, 51, 675–687.
[4] A. A. Taha, A. Hanbury. “Metrics for Evaluating 3D Medical Image Segmentation: Analysis, Selection, and Tool.” BMC Med Imaging, 2015, 15, 29.
[5] C. Shorten, T. M. Khoshgoftaar. “A Survey on Image Data Augmentation for Deep Learning.” J Big Data, 2019, 6, 60.
[6] T. Kliegr, Š. Bahník, J. Fürnkranz. “A Review of Possible Effects of Cognitive Biases on Interpretation of Rule-Based Machine Learning Models.” Artificial Intelligence, Volume 295, June 2021, 103458.

Authors
Annegret Janovsky, Product Marketing Manager for Industrial Microscopy, EMEA, Evident
Dr. Christina Hesseling, Product Manager for Materials Science and Industrial Equipment, Global, Evident