Explainable AI (XAI) refers to the capacity of providing understandable and coherent explanations regarding the decisions and predictions of AI systems. In short, the goal of this research line is to make the models’ inner workings more transparent and interpretable by humans. There are several approaches to this, e.g., visualization, feature importance analysis or subgroup mining. Currently, my work focuses on explaining possible reasons that lead to anomalies in numerical errors using subgroup mining. These subgroups can then be used for critical decisions, such as choosing a model that better fits a set of input properties.