Awesome-explainable-AI
A collection of research materials on explainable AI/ML
Stars: 1343
This repository contains frontier research on explainable AI (XAI), a hot topic in the field of artificial intelligence. It includes trends, use cases, survey papers, books, open courses, papers, and Python libraries related to XAI. The repository aims to organize and categorize publications on XAI, provide evaluation methods, and list various Python libraries for explainable AI.
README:
This repository contains the frontier research on explainable AI(XAI) which is a hot topic recently. From the figure below we can see the trend of interpretable/explainable AI. The publications on this topic are booming.
The figure below illustrates several use cases of XAI. Here we also divide the publications into serveal categories based on this figure. It is challenging to organise these papers well. Good to hear your voice!
Benchmarking and Survey of Explanation Methods for Black Box Models, DMKD 2023
Post-hoc Interpretability for Neural NLP: A Survey, ACM Computing Survey 2022
Explainable Artificial Intelligence (XAI): What we know and what is left to attain Trustworthy Artificial Intelligence, Information fusion 2023
Explainable Biometrics in the Age of Deep Learning, Arxiv preprint 2022
Explainable AI (XAI): Core Ideas, Techniques and Solutions, ACM Computing Survey 2022
A Review of Taxonomies of Explainable Artificial Intelligence (XAI) Methods, FaccT 2022
From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI, ArXiv preprint 2022. Corresponding website with collection of XAI methods
Interpretable machine learning:Fundamental principles and 10 grand challenges, Statist. Survey 2022
Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing, NeurlIPS 2021
Pitfalls of Explainable ML: An Industry Perspective, Arxiv preprint 2021
Explainable Machine Learning in Deployment, FAT 2020
The elephant in the interpretability room: Why use attention as explanation when we have saliency methods, EMNLP Workshop 2020
A Survey of the State of Explainable AI for Natural Language Processing, AACL-IJCNLP 2020
Interpretable Machine Learning – A Brief History, State-of-the-Art and Challenges, Communications in Computer and Information Science 2020
A brief survey of visualization methods for deep learning models from the perspective of Explainable AI, Information Visualization 2020
Explaining Explanations in AI, ACM FAT 2019
Machine learning interpretability: A survey on methods and metrics, Electronics, 2019
A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI, IEEE TNNLS 2020
Interpretable machine learning: definitions, methods, and applications, Arxiv preprint 2019
Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers, IEEE Transactions on Visualization and Computer Graphics, 2019
Explainable Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI, Information Fusion, 2019
Explanation in artificial intelligence: Insights from the social sciences, Artificial Intelligence 2019
Evaluating Explanation Without Ground Truth in Interpretable Machine Learning, Arxiv preprint 2019
Explanation in Human-AI Systems: A Literature Meta-Review Synopsis of Key Ideas and Publications and Bibliography for Explainable AI, DARPA XAI literature Review 2019
A survey of methods for explaining black box models, ACM Computing Surveys, 2018
Explaining Explanations: An Overview of Interpretability of Machine Learning, IEEE DSAA, 2018
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI), IEEE Access, 2018
Explainable artificial intelligence: A survey, MIPRO, 2018
The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery, ACM Queue 2018
How Convolutional Neural Networks See the World — A Survey of Convolutional Neural Network Visualization Methods, Mathematical Foundations of Computing 2018
Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models, Arxiv 2017
Towards A Rigorous Science of Interpretable Machine Learning, Arxiv preprint 2017
Explaining Explanation, Part 1: Theoretical Foundations, IEEE Intelligent System 2017
Explaining Explanation, Part 2: Empirical Foundations, IEEE Intelligent System 2017
Explaining Explanation, Part 3: The Causal Landscape, IEEE Intelligent System 2017
Explaining Explanation, Part 4: A Deep Dive on Deep Nets, IEEE Intelligent System 2017
An accurate comparison of methods for quantifying variable importance in artificial neural networks using simulated data, Ecological Modelling 2004
Review and comparison of methods to study the contribution of variables in artificial neural network models, Ecological Modelling 2003
Explainable Artificial Intelligence (xAI) Approaches and Deep Meta-Learning Models, Advances in Deep Learning Chapter 2020
Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, Springer 2019
Explanation in Artificial Intelligence: Insights from the Social Sciences, 2017 arxiv preprint
Visualizations of Deep Neural Networks in Computer Vision: A Survey, Springer Transparent Data Mining for Big and Small Data 2017
Explanatory Model Analysis Explore, Explain and Examine Predictive Models
Interpretable Machine Learning A Guide for Making Black Box Models Explainable
Limitations of Interpretable Machine Learning Methods
Interpretability and Explainability in Machine Learning, Harvard University
We mainly follow the taxonomy in the survey paper and divide the XAI/XML papers into the several branches.
Faithfulness Tests for Natural Language Explanations, ACL 2023
OpenXAI: Towards a Transparent Evaluation of Model Explanations, Arxiv 2022
From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI, ArXiv preprint 2022. Corresponding website with collection of XAI methods
Towards Better Understanding Attribution Methods, CVPR 2022
Evaluations and Methods for Explanation through Robustness Analysis, arxiv preprint 2020
Evaluating and Aggregating Feature-based Model Explanations, IJCAI 2020
Sanity Checks for Saliency Metrics, AAAI 2020
A benchmark for interpretability methods in deep neural networks, NIPS 2019
What Do Different Evaluation Metrics Tell Us About Saliency Models?, TPAMI 2018
Methods for interpreting and understanding deep neural networks, Digital Signal Processing 2017
Evaluating the visualization of what a Deep Neural Network has learned, IEEE Transactions on Neural Networks and Learning Systems 2015
AIF360: https://github.com/Trusted-AI/AIF360,
AIX360: https://github.com/IBM/AIX360,
Anchor: https://github.com/marcotcr/anchor, scikit-learn
Alibi: https://github.com/SeldonIO/alibi
Alibi-detect: https://github.com/SeldonIO/alibi-detect
BlackBoxAuditing: https://github.com/algofairness/BlackBoxAuditing, scikit-learn
Brain2020: https://github.com/vkola-lab/brain2020, Pytorch, 3D Brain MRI
Boruta-Shap: https://github.com/Ekeany/Boruta-Shap, scikit-learn
casme: https://github.com/kondiz/casme, Pytorch
Captum: https://github.com/pytorch/captum, Pytorch,
cnn-exposed: https://github.com/idealo/cnn-exposed, Tensorflow
ClusterShapley: https://github.com/wilsonjr/ClusterShapley, Sklearn
DALEX: https://github.com/ModelOriented/DALEX,
Deeplift: https://github.com/kundajelab/deeplift, Tensorflow, Keras
DeepExplain: https://github.com/marcoancona/DeepExplain, Tensorflow, Keras
Deep Visualization Toolbox: https://github.com/yosinski/deep-visualization-toolbox, Caffe,
dianna: https://github.com/dianna-ai/dianna, ONNX,
Eli5: https://github.com/TeamHG-Memex/eli5, Scikit-learn, Keras, xgboost, lightGBM, catboost etc.
explabox: https://github.com/MarcelRobeer/explabox, ONNX, Scikit-learn, Pytorch, Keras, Tensorflow, Huggingface
explainx: https://github.com/explainX/explainx, xgboost, catboost
ExplainaBoard: https://github.com/neulab/ExplainaBoard,
ExKMC: https://github.com/navefr/ExKMC, Python,
Facet: https://github.com/BCG-Gamma/facet, sklearn,
Grad-cam-Tensorflow: https://github.com/insikk/Grad-CAM-tensorflow, Tensorflow
GRACE: https://github.com/lethaiq/GRACE_KDD20, Pytorch
Innvestigate: https://github.com/albermax/innvestigate, tensorflow, theano, cntk, Keras
imodels: https://github.com/csinva/imodels,
InterpretML: https://github.com/interpretml/interpret
interpret-community: https://github.com/interpretml/interpret-community
Integrated-Gradients: https://github.com/ankurtaly/Integrated-Gradients, Tensorflow
Keras-grad-cam: https://github.com/jacobgil/keras-grad-cam, Keras
Keras-vis: https://github.com/raghakot/keras-vis, Keras
keract: https://github.com/philipperemy/keract, Keras
Lucid: https://github.com/tensorflow/lucid, Tensorflow
LIT: https://github.com/PAIR-code/lit, Tensorflow, specified for NLP Task
Lime: https://github.com/marcotcr/lime, Nearly all platform on Python
LOFO: https://github.com/aerdem4/lofo-importance, scikit-learn
modelStudio: https://github.com/ModelOriented/modelStudio, Keras, Tensorflow, xgboost, lightgbm, h2o
M3d-Cam: https://github.com/MECLabTUDA/M3d-Cam, PyTorch,
NeuroX: https://github.com/fdalvi/NeuroX, PyTorch,
neural-backed-decision-trees: https://github.com/alvinwan/neural-backed-decision-trees, Pytorch
Outliertree: https://github.com/david-cortes/outliertree, (Python, R, C++),
InterpretDL: https://github.com/PaddlePaddle/InterpretDL, (Python PaddlePaddle),
polyjuice: https://github.com/tongshuangwu/polyjuice, (Pytorch),
pytorch-cnn-visualizations: https://github.com/utkuozbulak/pytorch-cnn-visualizations, Pytorch
Pytorch-grad-cam: https://github.com/jacobgil/pytorch-grad-cam, Pytorch
PDPbox: https://github.com/SauceCat/PDPbox, Scikit-learn
py-ciu:https://github.com/TimKam/py-ciu/,
PyCEbox: https://github.com/AustinRochford/PyCEbox
path_explain: https://github.com/suinleelab/path_explain, Tensorflow
Quantus: https://github.com/understandable-machine-intelligence-lab/Quantus, Tensorflow, Pytorch
rulefit: https://github.com/christophM/rulefit,
rulematrix: https://github.com/rulematrix/rule-matrix-py,
Saliency: https://github.com/PAIR-code/saliency, Tensorflow
SHAP: https://github.com/slundberg/shap, Nearly all platform on Python
Shapley: https://github.com/benedekrozemberczki/shapley,
Skater: https://github.com/oracle/Skater
TCAV: https://github.com/tensorflow/tcav, Tensorflow, scikit-learn
skope-rules: https://github.com/scikit-learn-contrib/skope-rules, Scikit-learn
TensorWatch: https://github.com/microsoft/tensorwatch.git, Tensorflow
tf-explain: https://github.com/sicara/tf-explain, Tensorflow
Treeinterpreter: https://github.com/andosa/treeinterpreter, scikit-learn,
torch-cam: https://github.com/frgfm/torch-cam, Pytorch,
WeightWatcher: https://github.com/CalculatedContent/WeightWatcher, Keras, Pytorch
What-if-tool: https://github.com/PAIR-code/what-if-tool, Tensorflow
XAI: https://github.com/EthicalML/xai, scikit-learn
Xplique: https://github.com/deel-ai/xplique, Tensorflow,
https://github.com/jphall663/awesome-machine-learning-interpretability,
https://github.com/lopusz/awesome-interpretable-machine-learning,
https://github.com/pbiecek/xai_resources,
https://github.com/h2oai/mli-resources,
https://github.com/AstraZeneca/awesome-explainable-graph-reasoning,
https://github.com/utwente-dmb/xai-papers,
https://github.com/samzabdiel/XAI,
Need your help to re-organize and refine current taxonomy. Thanks very very much!
I appreciate it very much if you could add more works related to XAI/XML to this repo, archive uncategoried papers or anything to enrich this repo.
If any questions, feel free to drop me an email([email protected]). Welcome to discuss together.
For Tasks:
Click tags to check more tools for each tasksFor Jobs:
Alternative AI tools for Awesome-explainable-AI
Similar Open Source Tools
Awesome-explainable-AI
This repository contains frontier research on explainable AI (XAI), a hot topic in the field of artificial intelligence. It includes trends, use cases, survey papers, books, open courses, papers, and Python libraries related to XAI. The repository aims to organize and categorize publications on XAI, provide evaluation methods, and list various Python libraries for explainable AI.
Awesome-LLM4Graph-Papers
A collection of papers and resources about Large Language Models (LLM) for Graph Learning (Graph). Integrating LLMs with graph learning techniques to enhance performance in graph learning tasks. Categorizes approaches based on four primary paradigms and nine secondary-level categories. Valuable for research or practice in self-supervised learning for recommendation systems.
Awesome-Text2SQL
Awesome Text2SQL is a curated repository containing tutorials and resources for Large Language Models, Text2SQL, Text2DSL, Text2API, Text2Vis, and more. It provides guidelines on converting natural language questions into structured SQL queries, with a focus on NL2SQL. The repository includes information on various models, datasets, evaluation metrics, fine-tuning methods, libraries, and practice projects related to Text2SQL. It serves as a comprehensive resource for individuals interested in working with Text2SQL and related technologies.
adversarial-robustness-toolbox
Adversarial Robustness Toolbox (ART) is a Python library for Machine Learning Security. ART provides tools that enable developers and researchers to defend and evaluate Machine Learning models and applications against the adversarial threats of Evasion, Poisoning, Extraction, and Inference. ART supports all popular machine learning frameworks (TensorFlow, Keras, PyTorch, MXNet, scikit-learn, XGBoost, LightGBM, CatBoost, GPy, etc.), all data types (images, tables, audio, video, etc.) and machine learning tasks (classification, object detection, speech recognition, generation, certification, etc.).
fast-stable-diffusion
Fast-stable-diffusion is a project that offers notebooks for RunPod, Paperspace, and Colab Pro adaptations with AUTOMATIC1111 Webui and Dreambooth. It provides tools for running and implementing Dreambooth, a stable diffusion project. The project includes implementations by XavierXiao and is sponsored by Runpod, Paperspace, and Colab Pro.
BadukMegapack
BadukMegapack is an installer for various AI Baduk (Go) programs, designed for baduk players who want to easily access and use a variety of baduk AI programs without complex installations. The megapack includes popular programs like Lizzie, KaTrain, Sabaki, KataGo, LeelaZero, and more, along with weight files for different AI models. Users can update their graphics card drivers before installation for optimal performance.
Lumi-AI
Lumi AI is a friendly AI sidekick with a human-like personality that offers features like file upload and analysis, web search, local chat storage, custom instructions, changeable conversational style, enhanced context retention, voice query input, and various tools. The project has been developed with contributions from a team of developers, designers, and testers, and is licensed under Apache 2.0 and MIT licenses.
awesome-production-llm
This repository is a curated list of open-source libraries for production large language models. It includes tools for data preprocessing, training/finetuning, evaluation/benchmarking, serving/inference, application/RAG, testing/monitoring, and guardrails/security. The repository also provides a new category called LLM Cookbook/Examples for showcasing examples and guides on using various LLM APIs.
Embodied-AI-Guide
Embodied-AI-Guide is a comprehensive guide for beginners to understand Embodied AI, focusing on the path of entry and useful information in the field. It covers topics such as Reinforcement Learning, Imitation Learning, Large Language Model for Robotics, 3D Vision, Control, Benchmarks, and provides resources for building cognitive understanding. The repository aims to help newcomers quickly establish knowledge in the field of Embodied AI.
RAG-Survey
This repository is dedicated to collecting and categorizing papers related to Retrieval-Augmented Generation (RAG) for AI-generated content. It serves as a survey repository based on the paper 'Retrieval-Augmented Generation for AI-Generated Content: A Survey'. The repository is continuously updated to keep up with the rapid growth in the field of RAG.
Recommendation-Systems-without-Explicit-ID-Features-A-Literature-Review
This repository is a collection of papers and resources related to recommendation systems, focusing on foundation models, transferable recommender systems, large language models, and multimodal recommender systems. It explores questions such as the necessity of ID embeddings, the shift from matching to generating paradigms, and the future of multimodal recommender systems. The papers cover various aspects of recommendation systems, including pretraining, user representation, dataset benchmarks, and evaluation methods. The repository aims to provide insights and advancements in the field of recommendation systems through literature reviews, surveys, and empirical studies.
Interview-for-Algorithm-Engineer
This repository provides a collection of interview questions and answers for algorithm engineers. The questions are organized by topic, and each question includes a detailed explanation of the answer. This repository is a valuable resource for anyone preparing for an algorithm engineering interview.
fairseq
Fairseq is a sequence modeling toolkit that enables researchers and developers to train custom models for translation, summarization, language modeling, and other text generation tasks. It provides reference implementations of various sequence modeling papers covering CNN, LSTM networks, Transformer networks, LightConv, DynamicConv models, Non-autoregressive Transformers, Finetuning, and more. The toolkit supports multi-GPU training, fast generation on CPU and GPU, mixed precision training, extensibility, flexible configuration based on Hydra, and full parameter and optimizer state sharding. Pre-trained models are available for translation and language modeling with a torch.hub interface. Fairseq also offers pre-trained models and examples for tasks like XLS-R, cross-lingual retrieval, wav2vec 2.0, unsupervised quality estimation, and more.
EvalAI
EvalAI is an open-source platform for evaluating and comparing machine learning (ML) and artificial intelligence (AI) algorithms at scale. It provides a central leaderboard and submission interface, making it easier for researchers to reproduce results mentioned in papers and perform reliable & accurate quantitative analysis. EvalAI also offers features such as custom evaluation protocols and phases, remote evaluation, evaluation inside environments, CLI support, portability, and faster evaluation.
lobe-icons
Lobe Icons is a collection of popular AI / LLM Model Brand SVG logos and icons. It features lightweight and scalable icons designed with highly optimized scalable vector graphics (SVG) for optimal performance. The collection is tree-shakable, allowing users to import only the icons they need to reduce the overall bundle size of their projects. Lobe Icons has an active community of designers and developers who can contribute and seek support on platforms like GitHub and Discord. The repository supports a wide range of brands across different models, providers, and applications, with more brands continuously being added through contributions. Users can easily install Lobe UI with the provided commands and integrate it with NextJS for server-side rendering. Local development can be done using Github Codespaces or by cloning the repository. Contributions are welcome, and users can contribute code by checking out the GitHub Issues. The project is MIT licensed and maintained by LobeHub.
Awesome-Graph-LLM
Awesome-Graph-LLM is a curated collection of research papers exploring the intersection of graph-based techniques with Large Language Models (LLMs). The repository aims to bridge the gap between LLMs and graph structures prevalent in real-world applications by providing a comprehensive list of papers covering various aspects of graph reasoning, node classification, graph classification/regression, knowledge graphs, multimodal models, applications, and tools. It serves as a valuable resource for researchers and practitioners interested in leveraging LLMs for graph-related tasks.
For similar tasks
Awesome-explainable-AI
This repository contains frontier research on explainable AI (XAI), a hot topic in the field of artificial intelligence. It includes trends, use cases, survey papers, books, open courses, papers, and Python libraries related to XAI. The repository aims to organize and categorize publications on XAI, provide evaluation methods, and list various Python libraries for explainable AI.
pytorch-grad-cam
This repository provides advanced AI explainability for PyTorch, offering state-of-the-art methods for Explainable AI in computer vision. It includes a comprehensive collection of Pixel Attribution methods for various tasks like Classification, Object Detection, Semantic Segmentation, and more. The package supports high performance with full batch image support and includes metrics for evaluating and tuning explanations. Users can visualize and interpret model predictions, making it suitable for both production and model development scenarios.
AwesomeResponsibleAI
Awesome Responsible AI is a curated list of academic research, books, code of ethics, courses, data sets, frameworks, institutes, newsletters, principles, podcasts, reports, tools, regulations, and standards related to Responsible, Trustworthy, and Human-Centered AI. It covers various concepts such as Responsible AI, Trustworthy AI, Human-Centered AI, Responsible AI frameworks, AI Governance, and more. The repository provides a comprehensive collection of resources for individuals interested in ethical, transparent, and accountable AI development and deployment.
For similar jobs
weave
Weave is a toolkit for developing Generative AI applications, built by Weights & Biases. With Weave, you can log and debug language model inputs, outputs, and traces; build rigorous, apples-to-apples evaluations for language model use cases; and organize all the information generated across the LLM workflow, from experimentation to evaluations to production. Weave aims to bring rigor, best-practices, and composability to the inherently experimental process of developing Generative AI software, without introducing cognitive overhead.
LLMStack
LLMStack is a no-code platform for building generative AI agents, workflows, and chatbots. It allows users to connect their own data, internal tools, and GPT-powered models without any coding experience. LLMStack can be deployed to the cloud or on-premise and can be accessed via HTTP API or triggered from Slack or Discord.
VisionCraft
The VisionCraft API is a free API for using over 100 different AI models. From images to sound.
kaito
Kaito is an operator that automates the AI/ML inference model deployment in a Kubernetes cluster. It manages large model files using container images, avoids tuning deployment parameters to fit GPU hardware by providing preset configurations, auto-provisions GPU nodes based on model requirements, and hosts large model images in the public Microsoft Container Registry (MCR) if the license allows. Using Kaito, the workflow of onboarding large AI inference models in Kubernetes is largely simplified.
PyRIT
PyRIT is an open access automation framework designed to empower security professionals and ML engineers to red team foundation models and their applications. It automates AI Red Teaming tasks to allow operators to focus on more complicated and time-consuming tasks and can also identify security harms such as misuse (e.g., malware generation, jailbreaking), and privacy harms (e.g., identity theft). The goal is to allow researchers to have a baseline of how well their model and entire inference pipeline is doing against different harm categories and to be able to compare that baseline to future iterations of their model. This allows them to have empirical data on how well their model is doing today, and detect any degradation of performance based on future improvements.
tabby
Tabby is a self-hosted AI coding assistant, offering an open-source and on-premises alternative to GitHub Copilot. It boasts several key features: * Self-contained, with no need for a DBMS or cloud service. * OpenAPI interface, easy to integrate with existing infrastructure (e.g Cloud IDE). * Supports consumer-grade GPUs.
spear
SPEAR (Simulator for Photorealistic Embodied AI Research) is a powerful tool for training embodied agents. It features 300 unique virtual indoor environments with 2,566 unique rooms and 17,234 unique objects that can be manipulated individually. Each environment is designed by a professional artist and features detailed geometry, photorealistic materials, and a unique floor plan and object layout. SPEAR is implemented as Unreal Engine assets and provides an OpenAI Gym interface for interacting with the environments via Python.
Magick
Magick is a groundbreaking visual AIDE (Artificial Intelligence Development Environment) for no-code data pipelines and multimodal agents. Magick can connect to other services and comes with nodes and templates well-suited for intelligent agents, chatbots, complex reasoning systems and realistic characters.