Conference Agenda

The widespread use of large multimodal models (LMMs) and the ever-growing interest in their application across all fields raise questions about their reliability in areas such as mechanical design and CAD-related workflows. Before entrusting them with complex operations, it is important to establish whether the initial stage of interpreting drawings is robust enough and to evaluate their ability to correctly recognise and classify all the elements inside drawings. This paper describes an evaluation protocol implemented for three common LMMs in analysing mechanical drawings taken from textbooks and realised by means of CAD, as well as drawings created by hand by undergraduate students. The intention is not to determine which one is best, but rather to test their robustness in the initial stages of reading technical drawings. Each model is prompted to extract elements according to five categories: Views, Sections, Features, Dimensions, and Annotations. The results obtained are then compared with a ground truth, defined by experts, across four dimensions: Presence, Text, Type, and Context. This allows the overall results to be evaluated, as well as separate analyses of the five categories to consider factors such as false positives, omissions, element splitting, and uncertainty handling. The outcomes suggest that LMM-based drawing analysis should be reviewed using multidimensional protocols, as aggregate scores could mask differences in how models recognise and structure technical information.