This paper investigates the use of eXplainable Artificial Intelligence (XAI) methods, Integrated Gradients (IG) and SHAP, to interpret fault diagnosis decisions made by a Long Short-Time Memory (LSTM) classifier in a chemical process. The study highlights how XAI can help identify the faulty subsystem, improving the trustworthiness of deep learning models in industrial settings.