L3/W5/lecture_note.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Explainable AI\n",
    "\n",
    "#### Key Concepts:\n",
    "1. **Interpretability & Explainability:**\n",
    "   - **Interpretability:** The ability to explain or understand how a model makes decisions.\n",
    "   - **Explainability:** A broader term under Responsible AI, involving techniques to ensure AI decisions are understandable and transparent.\n",
    "   - **Responsible AI:** The ethical and responsible application of AI, including concerns like fairness, privacy, security, and bias mitigation.\n",
    "\n",
    "2. **Importance of Interpretability:**\n",
    "   - **Fairness:** Ensures models treat all users equitably and identifies potential biases in the training data.\n",
    "   - **Regulatory/Legal Compliance:** Helps meet legal and regulatory standards, and reduces risk of litigation.\n",
    "   - **Model Debugging & Optimization:** Enables model performance improvements by understanding internal mechanics.\n",
    "   - **Branding & Reputation:** Maintaining trust with customers and stakeholders by ensuring transparent and fair decision-making.\n",
    "\n",
    "3. **Responsible AI Components:**\n",
    "   - **Fairness:** Preventing bias and ensuring equal treatment of all demographics.\n",
    "   - **Privacy:** Handling of Personally Identifiable Information (PII) responsibly.\n",
    "   - **Security:** Protecting against attacks that extract sensitive information or manipulate model outcomes.\n",
    "\n",
    "4. **Explainable AI (XAI):**\n",
    "   - Techniques that inherently produce explainable models, e.g., decision trees.\n",
    "   - More advanced models are being developed that maintain high performance while remaining interpretable.\n",
    "\n",
    "5. **Adversarial Attacks:**\n",
    "   - Models, especially deep neural networks, are vulnerable to adversarial examples.\n",
    "   - Small, imperceptible perturbations can lead to severe misclassification.\n",
    "   - **Example:** A panda image misclassified as a gibbon with 99.3% confidence due to adversarial noise.\n",
    "   - **Black-box attacks:** Adversaries can craft attacks without direct access to the model, demonstrating how vulnerable AI systems are.\n",
    "\n",
    "6. **Challenges of Interpretability:**\n",
    "   - Interpretability becomes more difficult as model complexity increases (e.g., deep neural networks).\n",
    "   - However, techniques for achieving interpretability are improving, helping to balance model complexity with the need for transparency.\n",
    "\n",
    "#### Short Notes:\n",
    "- **Interpretability**: Crucial for understanding model decisions, especially for fairness, legal, and optimization reasons.\n",
    "- **Responsible AI**: Encompasses fairness, privacy, and security; essential for ethical AI use.\n",
    "- **XAI**: Focuses on models that are inherently explainable; decision trees are simple examples.\n",
    "- **Adversarial Attacks**: Deep models can be tricked with small perturbations, leading to significant misclassification.\n",
    "- **Fairness**: Ensuring models do not perpetuate bias is critical for responsible AI.\n",
    "- **Privacy & Security**: Models must safeguard PII and resist attacks that exploit vulnerabilities."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Interpretation Methods\n",
    "\n",
    "#### Key Concepts:\n",
    "\n",
    "1. **Interpretability:**\n",
    "   - **Definition:** Interpretability is the extent to which a model’s operations can be understood by humans through introspection or produced explanations (Biran and Cotton).\n",
    "   - **Objective:** Understand why a model made a decision, identify relevant variables, ensure model fairness, and build trust in the model's predictions.\n",
    "\n",
    "2. **Types of Interpretability Methods:**\n",
    "   - **Intrinsic Interpretability:** Certain model architectures, like linear models, tree-based models, and lattice models, are inherently interpretable. These models provide clarity on how they generate their results.\n",
    "   - **Post-Hoc Interpretability:** Methods applied after training that treat models as black boxes. They seek to understand results but often come with uncertainty. Examples include feature importance, partial dependence plots, and counterfactual explanations.\n",
    "\n",
    "3. **Criteria for Categorizing Interpretability Methods:**\n",
    "   - **Intrinsic vs. Post-Hoc:**\n",
    "     - **Intrinsic:** Models designed to be interpretable from the start (e.g., linear models, decision trees, lattice models).\n",
    "     - **Post-Hoc:** Applied after model training to explain outputs without access to the internal workings (e.g., SHAP, LIME).\n",
    "   - **Model-Specific vs. Model-Agnostic:**\n",
    "     - **Model-Specific:** Methods that require access to internal model structures (e.g., regression weights in linear models, tree structures in decision trees).\n",
    "     - **Model-Agnostic:** Methods applicable to any model without needing internal details (e.g., SHAP, feature attribution).\n",
    "   - **Local vs. Global:**\n",
    "     - **Local:** Explains individual predictions (e.g., feature attribution for a single instance using SHAP).\n",
    "     - **Global:** Explains overall model behavior across the entire dataset (e.g., global SHAP summary plots).\n",
    "\n",
    "4. **Interpretability Methods:**\n",
    "   - **Feature Importance:** Assigns a single value to each feature, indicating its importance in the model's predictions.\n",
    "   - **Partial Dependence Plots (PDPs):** Visualization showing how a feature impacts predictions on average.\n",
    "   - **Counterfactual Explanations:** Find alternative data points that would result in a different prediction by changing some features.\n",
    "   - **Pairwise Feature Interaction Strength:** Quantifies how much two features interact in influencing predictions.\n",
    "   - **SHAP (SHapley Additive exPlanations):** A popular method for both local and global interpretations, based on Shapley values from game theory.\n",
    "\n",
    "5. **Examples:**\n",
    "   - **Local Interpretation:** SHAP values can show the contribution of features in a single prediction (e.g., predicting house prices in Boston based on individual features like the percentage of lower-status population).\n",
    "   - **Global Interpretation:** SHAP summary plots provide a global view of feature contributions across the entire dataset, showing trends (e.g., how lower-status population percentage impacts house prices).\n",
    "\n",
    "#### Short Notes:\n",
    "- **Intrinsic Interpretability:** Linear models, decision trees, lattice models—interpretable by design.\n",
    "- **Post-Hoc Methods:** Applied after training, treat models as black boxes, examples include SHAP, LIME.\n",
    "- **Model-Specific:** Interpretations tied to internal model details (e.g., regression weights, decision trees).\n",
    "- **Model-Agnostic:** Works on any model post-training, doesn't need internal details (e.g., SHAP, feature attribution).\n",
    "- **Local Methods:** Explain single predictions, e.g., SHAP values for a specific instance.\n",
    "- **Global Methods:** Explain overall model behavior, e.g., SHAP summary plots across the dataset.\n",
    "- **Key Tools:** SHAP (local/global), Partial Dependence Plots, Feature Importance, Counterfactual Explanations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intrinsically Interpretable Models\n",
    "\n",
    "#### Key Concepts:\n",
    "\n",
    "1. **Intrinsic Interpretability:**\n",
    "   - **Definition:** A model is intrinsically interpretable if its workings are transparent and intuitive enough to understand how it produces specific results by examining the model itself.\n",
    "   - **Examples:** Linear models and tree-based models are classic examples of intrinsically interpretable models.\n",
    "   - **Challenge with Neural Networks:** Complex neural network architectures tend to be opaque (\"black boxes\"), making post-hoc interpretation necessary.\n",
    "\n",
    "2. **Monotonicity:**\n",
    "   - **Definition:** A feature is monotonic if its contribution to the model’s output consistently increases, decreases, or stays constant as the feature value changes.\n",
    "   - **Importance:** Monotonicity aligns with domain knowledge and expectations, making models easier to interpret. For example, in predicting used car prices, more miles on the car should consistently decrease its value.\n",
    "\n",
    "3. **Classic Interpretable Models:**\n",
    "   - **Linear Models:** Known for their simplicity and ease of interpretation. Linear relationships are easy to understand, and feature contributions are directly related to their weights.\n",
    "   - **Tree-Based Models:** Decision trees and related methods provide visual and hierarchical interpretations of decision processes.\n",
    "\n",
    "4. **Newer Interpretable Architectures:**\n",
    "   - **RuleFit Models:** Combine linear models with decision rules, offering interpretability with the possibility of incorporating feature interactions.\n",
    "   - **TensorFlow Lattice Models:** These models apply linear interpolation between lattice points in the feature space. They allow the integration of domain knowledge (e.g., monotonicity constraints) and offer high accuracy while remaining interpretable.\n",
    "\n",
    "5. **Feature Interactions and Domain Knowledge:**\n",
    "   - **Feature Interactions:** Interactions between features, when aligned with domain knowledge, improve model interpretability. Feature engineering can be used to introduce relevant interactions.\n",
    "   - **Incorporating Domain Knowledge:** TensorFlow Lattice Models allow constraints based on domain knowledge (e.g., monotonicity, convexity) to ensure that the model produces results consistent with real-world expectations.\n",
    "\n",
    "6. **Challenges of Lattice Models:**\n",
    "   - **Dimensionality:** Lattice models struggle with high-dimensional feature spaces, as the number of parameters grows exponentially with the number of features. This \"dimensionality kryptonite\" limits their scalability to datasets with many features.\n",
    "   - **Solutions:** Ensembling techniques can help mitigate dimensionality issues, but this is beyond the scope of this discussion.\n",
    "\n",
    "7. **Feature Importance in Linear Models:**\n",
    "   - **Measure of Importance:** The absolute value of a feature’s t-statistic (learned weight scaled by its standard error) indicates its importance. A feature is more important if its weight is large and stable (low variance).\n",
    "   - **Interpretation in Linear Models:** Increases or decreases in a feature’s value lead to proportional changes in predictions, making it easy to understand how features contribute to the outcome.\n",
    "\n",
    "8. **TensorFlow Lattice Models:**\n",
    "   - **Overview:** Lattice models overlay a grid onto the feature space and use linear interpolation for predictions. They regularize the model and reduce sensitivity to examples outside the training data.\n",
    "   - **Incorporating Domain Knowledge:** Constraints (e.g., monotonicity) make the model results more interpretable and closer to real-world expectations. TensorFlow Lattice Models achieve neural network-level accuracy while being more interpretable.\n",
    "   - **Scaling Limitations:** Effective for problems with 20 or fewer features, depending on the number of vertices specified.\n",
    "\n",
    "#### Short Notes:\n",
    "- **Intrinsic Interpretability:** Transparent models like linear models and decision trees allow easy understanding of how predictions are made.\n",
    "- **Monotonicity:** Consistent feature behavior improves interpretability by aligning with domain knowledge (e.g., mileage reducing car value).\n",
    "- **TensorFlow Lattice Models:** Combine interpretability with high accuracy by using linear interpolation and domain knowledge constraints but struggle with high-dimensional data.\n",
    "- **Dimensionality:** Lattice models scale poorly with many features; ensembling can help mitigate this.\n",
    "- **Feature Importance (Linear Models):** T-statistic indicates feature importance; stable, high-weight features are more important.\n",
    "- **Newer Advances:** RuleFit and TensorFlow Lattice Models offer interpretable alternatives to opaque deep neural networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Agnostic Methods\n",
    "\n",
    "#### Key Concepts:\n",
    "\n",
    "1. **Model-Agnostic Methods:**\n",
    "   - **Definition:** These methods are not specific to any particular model architecture. They can be applied after a model has been trained, regardless of whether the model is inherently interpretable.\n",
    "   - **Applications:** Model-agnostic methods work with a wide range of models, from simple linear regressions and decision trees to complex black-box models like deep neural networks.\n",
    "\n",
    "2. **Desirable Characteristics of Model-Agnostic Methods:**\n",
    "   - **Model Flexibility:** The method should work with any type of model architecture.\n",
    "   - **Explanation Flexibility:** The method should provide various forms of explanations, such as formulas, visualizations (e.g., feature importances), or other representations that best explain the model's behavior.\n",
    "   - **Representation Flexibility:** The features used in explanations should be meaningful in the context of the model being explained. For instance, in a text classifier using word embeddings, the explanation should relate to individual words, not abstract vectors.\n",
    "\n",
    "3. **Examples of Model-Agnostic Methods:**\n",
    "   - **Feature Importance:** Methods like SHAP (Shapley Additive Explanations) provide a way to assess how much each feature contributed to a particular prediction. This can be visualized as a bar chart or other graphical representation.\n",
    "   - **LIME (Local Interpretable Model-Agnostic Explanations):** LIME approximates complex models locally with simpler models to explain individual predictions. For example, it can use linear models to explain the decision of a neural network for a specific input.\n",
    "   - **Counterfactual Explanations:** These methods generate \"what-if\" scenarios by modifying input features to see how the output changes. This helps users understand the boundaries and sensitivities of the model’s predictions.\n",
    "\n",
    "#### Short Notes:\n",
    "- **Model-Agnostic Methods:** Allow interpretation of any model, even complex black-box models like neural networks, by decoupling the explanation from the model itself.\n",
    "- **Key Traits:** Flexibility in model, explanation, and representation to adapt to different contexts and needs.\n",
    "- **Common Techniques:** SHAP, LIME, and counterfactual explanations help provide insights into feature importance and decision-making processes of various models.\n",
    "\n",
    "This overview emphasizes the adaptability and variety of model-agnostic interpretability techniques, which are crucial when dealing with complex or opaque models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Partial Dependence Plots\n",
    "\n",
    "#### Key Concepts:\n",
    "\n",
    "1. **Partial Dependence Plots (PDPs):**\n",
    "   - **Definition:** PDPs are visualizations that help understand the relationship between specific features and model predictions. They show the marginal effect of one or two features on the predicted outcome while averaging out the effects of other features.\n",
    "   - **Global Method:** PDPs assess the global relationship between features and target variables across the entire dataset, not just for individual instances.\n",
    "\n",
    "2. **How PDPs Work:**\n",
    "   - PDPs estimate the partial function, denoted as \\( f_{xs} \\), which calculates the average impact of the selected feature(s) \\( s \\) on the model's predictions while keeping other features \\( c \\) constant.\n",
    "   - **Monte Carlo Method:** PDPs use averages from the training data to estimate the partial function.\n",
    "   - **Assumption of Independence:** PDPs assume that the features being analyzed (s) are not correlated with the other features (c). If this assumption is violated, the plot might present unrealistic scenarios.\n",
    "\n",
    "3. **Example:**\n",
    "   - **Bike Rentals Dataset:** In a random forest model predicting daily bike rentals, PDPs for temperature, humidity, and wind speed show:\n",
    "     - **Temperature:** Bike rentals increase as the temperature rises to 15°C (59°F), peak, then decline after 25°C (77°F).\n",
    "     - **Humidity:** High humidity (>60) decreases bike rentals.\n",
    "     - **Seasons (categorical feature):** Spring shows slightly lower bike rentals compared to other seasons.\n",
    "\n",
    "4. **Advantages of PDPs:**\n",
    "   - **Intuitive Results:** When features are not correlated, PDPs provide intuitive insights into how changing a feature affects predictions.\n",
    "   - **Causal Interpretation:** Changes in a feature reflected in the PDP can be interpreted causally, meaning that a change in the feature should produce a consistent change in the outcome.\n",
    "   - **Easy Implementation:** Various libraries support PDPs, making them relatively easy to use in practice.\n",
    "\n",
    "5. **Disadvantages of PDPs:**\n",
    "   - **Limited to One or Two Features:** PDPs typically handle one or two features at a time due to the difficulty of visualizing higher-dimensional data.\n",
    "   - **Assumption of Independence:** The assumption that features are independent can lead to misleading interpretations if the features are actually correlated. For example, predicting walking speed based on height and weight can produce unrealistic combinations if their correlation is ignored.\n",
    "\n",
    "#### Short Notes:\n",
    "- **PDPs:** Visual tools for understanding how specific features affect model predictions by showing marginal impacts.\n",
    "- **Global Analysis:** PDPs examine the overall relationship between features and outcomes across the dataset, assuming feature independence.\n",
    "- **Advantages:** Intuitive, easy to implement, allows causal interpretation.\n",
    "- **Disadvantages:** Assumes feature independence, limited to 1-2 features for visual clarity.\n",
    "\n",
    "This summary highlights how PDPs provide valuable global insights into model behavior, while also pointing out their limitations, especially with correlated features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Permutation Feature Importance\n",
    "\n",
    "\n",
    "#### Key Concepts:\n",
    "\n",
    "1. **Permutation Feature Importance:**\n",
    "   - **Definition:** This method evaluates the importance of a feature by measuring how much the prediction error increases when that feature's values are shuffled (permuted), which breaks the relationship between the feature and the target variable.\n",
    "   - **Key Idea:** If shuffling a feature significantly increases the model's error, the feature is important because the model relied on it. If shuffling does not change the error, the feature is unimportant.\n",
    "\n",
    "2. **Algorithm:**\n",
    "   - **Inputs:** Model, features, labels (targets), and an error metric.\n",
    "   - **Steps:**\n",
    "     1. Compute the original model error using the true feature values.\n",
    "     2. For each feature, permute its values and compute the new model error.\n",
    "     3. Calculate feature importance as the difference or ratio between the original and permuted errors.\n",
    "     4. Rank features by their importance scores, with higher scores indicating more important features.\n",
    "\n",
    "3. **Example:** \n",
    "   - **Bike Rentals Dataset:** Suppose we permute the \"temperature\" feature in a model predicting bike rentals. If the model error increases significantly, it indicates that temperature is an important feature for predicting rentals.\n",
    "\n",
    "4. **Advantages of Permutation Feature Importance:**\n",
    "   - **Intuitive Interpretation:** The importance score directly reflects how much the model relies on a feature for accurate predictions.\n",
    "   - **Global Insight:** It provides a global view of feature importance, including interactions between features, since shuffling also disrupts feature interactions.\n",
    "   - **No Model Retraining Required:** Unlike methods that involve removing features and retraining the model, this method simply requires shuffling feature values, saving time and computational resources.\n",
    "\n",
    "5. **Disadvantages of Permutation Feature Importance:**\n",
    "   - **Choice of Data:** It's unclear whether to calculate permutation feature importance on training or test data. Both have trade-offs: using training data can overestimate importance, while using test data may underestimate it if the model is overfitting.\n",
    "   - **Correlated Features:** Like Partial Dependence Plots (PDPs), permutation feature importance struggles with correlated features. Shuffling one feature may inadvertently change the model’s reliance on another correlated feature, leading to misleading importance scores.\n",
    "   - **Access to Original Data:** The method requires access to the original labeled dataset. If the model is provided without the training data, this approach cannot be used.\n",
    "\n",
    "#### Short Notes:\n",
    "- **Permutation Feature Importance:** Measures feature importance by shuffling feature values and observing the impact on model error.\n",
    "- **Advantages:** Intuitive, includes feature interactions, does not require model retraining.\n",
    "- **Disadvantages:** Data choice (train vs. test), problems with correlated features, requires access to original data.\n",
    "\n",
    "Permutation feature importance is a practical and effective method for evaluating feature importance, offering a balance of interpretability and efficiency, but it has limitations with correlated features and dataset accessibility."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Shapley Values\n",
    "\n",
    "#### Key Concepts:\n",
    "\n",
    "1. **Shapley Value (Game Theory Origin):**\n",
    "   - **Background:** The Shapley value, introduced by Lloyd Shapley in 1951, originates from cooperative game theory and is used to fairly distribute the total gain among players based on their contribution to the coalition. Shapley won the Nobel Prize in Economics in 2012 for this concept.\n",
    "   - **Application to ML:** In machine learning, the \"players\" are the features of the dataset, and the Shapley value helps determine how much each feature contributes to the prediction outcome. The \"game\" is the prediction task, and the \"gain\" is the difference between the prediction for a particular instance and the average prediction for all instances.\n",
    "\n",
    "2. **Concrete Example:**\n",
    "   - **Apartment Price Prediction:** Suppose a model predicts an apartment price of €300,000, and the average prediction across all apartments is €310,000. The goal is to explain this €10,000 difference using Shapley values. Each feature (e.g., apartment size, location, presence of a park) contributes a portion of this difference, either positively or negatively.\n",
    "\n",
    "3. **Key Properties of Shapley Values:**\n",
    "   - **Efficiency:** The total gain (difference between the actual prediction and the average prediction) is fully distributed among the features.\n",
    "   - **Symmetry:** If two features contribute equally to the prediction, they receive the same Shapley value.\n",
    "   - **Dummy:** Features that do not affect the prediction get a Shapley value of zero.\n",
    "   - **Additivity:** If two models are combined, the Shapley value for each feature in the combined model is the sum of the Shapley values from the individual models.\n",
    "\n",
    "4. **Advantages of Shapley Values:**\n",
    "   - **Fair Distribution:** Shapley values fairly distribute the contributions of all features in an instance, making it a comprehensive explanation method.\n",
    "   - **Contrastive Explanations:** Shapley values can provide contrastive explanations by comparing the prediction to either the average prediction, a subset of the data, or even a single data point.\n",
    "   - **Theoretical Rigor:** Shapley values are grounded in a strong theoretical framework, making them suitable for situations that require legally sound explanations (e.g., the \"right to explanations\" in AI-driven decisions).\n",
    "\n",
    "5. **Disadvantages of Shapley Values:**\n",
    "   - **Computational Complexity:** Calculating exact Shapley values is computationally expensive, often requiring approximations in real-world applications, especially for large datasets or complex models.\n",
    "   - **Interpretation Challenges:** Shapley values can be misinterpreted as the impact of removing a feature from the model, when in reality, they represent the contribution of a feature to the difference between the prediction and the mean prediction.\n",
    "   - **Lack of Selective Explanations:** Shapley values consider all features, which may not align with the human preference for simpler, selective explanations like those provided by LIME.\n",
    "   - **Correlation Issue:** Like many interpretability methods, Shapley values do not handle correlated features well. Correlated features should ideally be removed during feature selection.\n",
    "\n",
    "6. **Key Use Cases:**\n",
    "   - **Legal and Regulatory Requirements:** Shapley values are often recommended when robust, legally compliant explanations are needed, such as in regulated industries where explainability is a requirement.\n",
    "   - **Global Interpretability:** Shapley values provide a full explanation of model predictions by distributing contributions among all features, making them suitable for global interpretability tasks.\n",
    "\n",
    "7. **SHAP (SHapley Additive exPlanations):**\n",
    "   - **Overview:** SHAP is a popular method that builds on Shapley values, but can provide more efficient and feature-selective explanations, making it more practical for real-world use.\n",
    "\n",
    "#### Short Notes:\n",
    "- **Shapley Value:** A game-theoretic approach for fairly distributing feature contributions to a model’s prediction.\n",
    "- **Advantages:** Fair, comprehensive, theoretically sound, contrastive explanations.\n",
    "- **Disadvantages:** Computationally expensive, requires all features, struggles with correlated features.\n",
    "- **SHAP:** A more practical, feature-selective method based on Shapley values.\n",
    "\n",
    "Shapley values are a powerful and theoretically grounded method for model interpretability, especially in scenarios requiring comprehensive and fair explanations, but their computational cost and complexity can be limiting in practice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Shapley Additive Explanations (SHAP)\n",
    "\n",
    "#### Key Concepts:\n",
    "\n",
    "1. **SHAP Overview:**\n",
    "   - **Definition:** SHAP (SHapley Additive exPlanations) is an open-source library that provides model-agnostic explanations based on game-theoretic principles, specifically the Shapley value.\n",
    "   - **Purpose:** SHAP assigns importance values to features for particular predictions and is designed to explain the output of any machine learning model, making it versatile and applicable across different model architectures.\n",
    "   - **Connection to Game Theory:** SHAP builds on Shapley values from cooperative game theory and connects optimal credit allocation with local explanations. It integrates recent advancements in Shapley value theory to provide high-performance explanations.\n",
    "\n",
    "2. **SHAP Extensions:**\n",
    "   - **Tree Explainer:** A high-speed, exact algorithm optimized for tree-based ensemble models (e.g., random forests, gradient boosting).\n",
    "   - **Deep Explainer:** An approximation algorithm tailored for deep learning models, providing efficient SHAP value computations.\n",
    "   - **Gradient Explainer:** Combines ideas from integrated gradients, SHAP, and SmoothGrad into a single expected value equation, suitable for models with differentiable outputs.\n",
    "   - **Kernel Explainer:** Uses locally weighted linear regression to estimate SHAP values for any model, making it broadly applicable.\n",
    "\n",
    "3. **SHAP Visualization Tools:**\n",
    "   - **Force Plots:** These visualize SHAP values as forces that push predictions away from a baseline (typically the average prediction). Features are shown as arrows, with positive contributions (increasing the prediction) in red and negative contributions (decreasing the prediction) in blue. The plot illustrates how different features contribute to the final prediction by balancing these forces.\n",
    "   - **Summary Plots:** These combine feature importance with feature effects. Each point represents a SHAP value for a specific feature and instance, with color indicating feature values (low in blue, high in red). Features are ranked by importance, providing a clear view of how features impact model predictions across the dataset.\n",
    "   - **Dependence Plots:** These show the relationship between feature values (on the x-axis) and SHAP values (on the y-axis). The plot helps identify correlations between specific features and their influence on predictions. For instance, a plot could reveal how energy prices change over the day based on time features like \"hour.\"\n",
    "\n",
    "4. **Advantages of SHAP:**\n",
    "   - **Model-Agnostic:** SHAP works across all types of machine learning models, from tree-based models to deep neural networks.\n",
    "   - **Feature Interaction Insights:** SHAP reveals how individual feature values interact and contribute to predictions, providing a detailed understanding of model behavior.\n",
    "   - **Visual Interpretability:** SHAP’s visualization tools, such as force plots and summary plots, make it easier to understand and communicate model predictions and feature contributions.\n",
    "\n",
    "5. **Use Cases:**\n",
    "   - **Feature Importance Analysis:** SHAP helps identify the most important features in a model, guiding decisions around feature engineering and model interpretation.\n",
    "   - **Compliance and Explainability:** SHAP can be used in regulated industries to provide legally compliant explanations of AI-driven decisions.\n",
    "   - **Debugging and Refinement:** SHAP can be employed to diagnose model issues by showing how individual features are influencing predictions, making it easier to spot biases or inconsistencies.\n",
    "\n",
    "6. **Examples:**\n",
    "   - **Energy Price Prediction:** A SHAP dependence plot could show how energy prices fluctuate throughout the day based on features like time (e.g., \"hour\") and temperature (e.g., \"Lok 15 temperature\"). SHAP enables users to see the relationships between features and their impact on pricing predictions at different times.\n",
    "\n",
    "#### Short Notes:\n",
    "- **SHAP:** A model-agnostic tool for explaining predictions based on Shapley values.\n",
    "- **Extensions:** Includes Tree Explainer (for tree ensembles), Deep Explainer (for deep learning), Gradient Explainer, and Kernel Explainer.\n",
    "- **Visualizations:** Force plots show feature contributions as forces, while summary and dependence plots reveal feature importance and interactions.\n",
    "- **Use Cases:** Useful for feature importance analysis, compliance, debugging, and model refinement.\n",
    "\n",
    "SHAP provides a powerful, flexible, and theoretically grounded approach to model interpretability, combining advanced explanation techniques with user-friendly visualization tools for a wide range of machine learning models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Testing Concept Activation Vectors\n",
    "\n",
    "#### Key Concepts:\n",
    "\n",
    "1. **TCAV Overview:**\n",
    "   - **Purpose:** Testing Concept Activation Vectors (TCAV) is a method for interpreting deep learning models by relating their internal representations to high-level, human-friendly concepts. This helps address the challenge of interpreting complex, opaque models, such as neural networks, that often operate on low-level features rather than intuitive, high-level concepts.\n",
    "   - **Concept Activation Vectors (CAVs):** These vectors represent a specific concept (e.g., \"curly hair\" or \"CEO\") in the internal vector space of a neural network. CAVs allow researchers to interpret the neural network's state in relation to these human-understandable concepts, offering a bridge between abstract model internals and human intuition.\n",
    "\n",
    "2. **How TCAV Works:**\n",
    "   - **Defining Concepts:** To define a concept, users provide example data (e.g., images or text) that represent that concept. These examples can come from outside the model’s training data, which allows for flexibility in concept definition.\n",
    "   - **Learning Concept Activation Vectors:** Once a concept is defined using example data, TCAV trains a linear classifier (e.g., an SVM) to distinguish between this concept and a random set of data in the neural network’s bottleneck layer (a layer that represents high-level features). The resulting vector in the model’s feature space represents the direction of the concept (the CAV).\n",
    "   - **Testing CAVs:** To measure how sensitive the model’s predictions are to the concept represented by the CAV, TCAV evaluates the model’s predictions for a set of inputs and computes how much the CAV influences the predictions. This allows researchers to quantify the importance of a concept in a model's decision-making.\n",
    "\n",
    "3. **Applications:**\n",
    "   - **Interpreting Model Predictions:** By using CAVs, you can measure the extent to which a concept affects a model’s output, providing insights into the model’s decision-making process. For example, in an image classification model, TCAV can reveal how much the concept of \"curly hair\" influences predictions for images of people.\n",
    "   - **Revealing Biases:** TCAV can help identify biases in models by analyzing the influence of specific concepts on predictions. For example, sorting images by their similarity to a concept like \"CEO\" might reveal biases in how the model associates certain visual features (e.g., pinstripes) with the concept.\n",
    "   - **Generalization Across Domains:** TCAV allows the application of concepts across different datasets and models. For instance, a concept learned from one set of images (e.g., \"CEO\") can be used to sort or interpret images in a different dataset, helping to generalize findings.\n",
    "\n",
    "4. **Example Use Cases:**\n",
    "   - **Image Sorting by Concept:** In one example, images were sorted by their similarity to two different concepts: \"CEO\" and \"model women.\" Images most similar to the \"CEO\" concept displayed pinstripes, a feature often associated with business attire. For the \"model women\" concept, images showing women in neckties were ranked higher in similarity compared to images of men in neckties, indicating a potential bias in the learned concept.\n",
    "   - **Bias Detection:** The sorting of images by CAV similarity can reveal potential biases in the training data or in the model’s internal representations. For example, if the model consistently associates \"CEO\" with male-coded attire, it may reflect an underlying bias in the data used to train the concept.\n",
    "\n",
    "5. **Advantages of TCAV:**\n",
    "   - **Human-Centric Interpretation:** TCAV provides an intuitive way to interpret deep learning models by relating their internal states to human-understandable concepts, making the interpretation process accessible to both experts and non-experts.\n",
    "   - **Flexibility in Concept Definition:** Concepts can be defined using external data, allowing researchers to explore concepts that were not explicitly present in the model’s training data.\n",
    "   - **Bias Detection and Debugging:** TCAV helps in identifying and addressing biases in models by analyzing the impact of specific concepts on predictions, making it a valuable tool for model debugging and fairness analysis.\n",
    "\n",
    "6. **Challenges:**\n",
    "   - **Concept Definition:** Defining meaningful and representative concepts requires careful selection of example data, which can be subjective and may introduce bias if not done properly.\n",
    "   - **Computation:** TCAV involves training additional classifiers (e.g., linear classifiers for CAVs), which can add computational overhead, especially when working with large models or multiple concepts.\n",
    "\n",
    "#### Short Notes:\n",
    "- **TCAV:** A method to interpret deep learning models by relating their internal states to human-friendly concepts using Concept Activation Vectors (CAVs).\n",
    "- **Concept Definition:** Concepts are defined using user-provided example data and are represented as vectors in the model’s internal feature space.\n",
    "- **Use Cases:** TCAV helps interpret model predictions, reveal biases, and generalize concepts across datasets.\n",
    "- **Advantages:** Provides human-centric model interpretations, flexibility in concept definition, and helps with bias detection.\n",
    "- **Challenges:** Requires careful concept definition and additional computation.\n",
    "\n",
    "TCAV offers a powerful approach to deep learning model interpretability by connecting complex internal representations to intuitive human concepts, making it easier to understand and diagnose model behavior."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LIME\n",
    "\n",
    "#### Key Concepts:\n",
    "\n",
    "1. **Overview of LIME:**\n",
    "   - **Purpose:** LIME is a framework for creating local explanations for the predictions of any black-box machine learning model. It is model-agnostic, meaning it can be applied to any model type, including neural networks, random forests, and SVMs.\n",
    "   - **Local Interpretability:** Unlike global interpretation methods, which attempt to explain the model as a whole, LIME focuses on explaining individual predictions, making it a tool for local interpretability.\n",
    "\n",
    "2. **How LIME Works:**\n",
    "   - **Black-Box Model Probing:** LIME treats the model as a black box that generates predictions when given input data. It does not require access to the model's internals (e.g., weights or parameters).\n",
    "   - **Generating Perturbed Samples:** LIME generates new samples by perturbing the original data (i.e., making small changes to it) and records the model’s predictions for these perturbed samples.\n",
    "   - **Training a Local Model:** Using the perturbed samples and their corresponding predictions, LIME trains a simple, interpretable model (e.g., a linear model or decision tree). This local model approximates the behavior of the black-box model in the vicinity of the data point being explained.\n",
    "   - **Local Fidelity:** The goal of the local model is to achieve high fidelity to the black-box model’s predictions for the specific data point, even if the local model does not generalize well to other regions of the input space.\n",
    "\n",
    "3. **Use Cases:**\n",
    "   - **Explaining Individual Predictions:** LIME is particularly useful for explaining why a model made a specific prediction. For example, in a medical diagnosis model, LIME can be used to explain why a particular patient was classified as having a disease.\n",
    "   - **Model Debugging:** By analyzing the explanations provided by LIME, researchers can identify potential issues in the model, such as unexpected feature importance or biases.\n",
    "\n",
    "4. **Advantages:**\n",
    "   - **Model-Agnostic:** LIME can be used with any machine learning model, as it does not rely on knowledge of the model’s architecture or parameters.\n",
    "   - **Flexibility:** LIME provides flexibility in selecting the type of interpretable model used for local explanations (e.g., linear models, decision trees).\n",
    "   - **Scalability:** LIME can scale to different types of data, such as text, images, and tabular data, making it versatile across various domains.\n",
    "\n",
    "5. **Challenges:**\n",
    "   - **Stability:** The explanations produced by LIME can sometimes vary with different perturbations, which may lead to inconsistencies.\n",
    "   - **Interpretation Quality:** While LIME provides local explanations, these explanations are dependent on the choice of interpretable model and may not always fully capture the nuances of the black-box model’s behavior.\n",
    "   - **Computational Cost:** Generating perturbed samples and training local models can be computationally expensive, especially for large datasets and complex models.\n",
    "\n",
    "#### Short Notes:\n",
    "- **LIME:** A framework for creating local explanations for predictions from any black-box machine learning model.\n",
    "- **Perturbation-Based:** LIME generates perturbed samples and trains an interpretable local model to approximate the black-box model’s behavior around a specific prediction.\n",
    "- **Local Fidelity:** The goal is to achieve high fidelity to the black-box model’s prediction in the vicinity of the data point being explained.\n",
    "- **Advantages:** Model-agnostic, flexible, and scalable across different data types.\n",
    "- **Challenges:** Stability, interpretation quality, and computational cost.\n",
    "\n",
    "LIME is a powerful tool for generating local explanations of individual predictions, offering insights into model behavior even when the model itself is complex or opaque."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AI Explanations\n",
    "\n",
    "#### Key Concepts:\n",
    "\n",
    "1. **Google's AI Explanations Service:**\n",
    "   - **Overview:** Google’s AI Explanations is a managed service that integrates with Google Cloud’s AI Platform Prediction. It provides feature attributions to help users understand model outputs for classification and regression tasks.\n",
    "   - **Feature Attributions:** This service explains how much each feature contributes to a model's prediction, enabling users to verify model behavior, detect biases, and identify areas for improvement.\n",
    "\n",
    "2. **How It Works:**\n",
    "   - **Prediction Requests:** When making a prediction request through AI Platform Prediction, users typically receive only the prediction. By requesting an explanation, users receive both the prediction and feature attribution details.\n",
    "   - **Visualizations:** The service provides various visualizations to aid interpretation:\n",
    "     - **Tabular Data:** Feature attributions are displayed as charts showing the impact of each feature on the prediction.\n",
    "     - **Image Data:** Overlays on images highlight which pixels contribute most to the prediction. \n",
    "\n",
    "3. **Methods of Feature Attribution:**\n",
    "   - **Sampled Shapley:** Based on Shapley values, provides feature attribution by sampling different feature combinations.\n",
    "   - **Integrated Gradients:** A method that calculates gradients of the prediction output with respect to the input features along a path from a baseline (e.g., blacked-out image) to the actual input. It averages these gradients and computes the attribution by multiplying the average gradients by the original input values.\n",
    "   - **XRAI (eXplanation with Ranked Area Integrals):** An extension of integrated gradients specifically for image classification. It performs pixel-level attribution and then segments the image into regions. Attributions within each region are aggregated and ranked to determine the most salient areas contributing to the prediction.\n",
    "\n",
    "4. **Advantages of Google’s AI Explanations Service:**\n",
    "   - **Integrated into Cloud Infrastructure:** Provides seamless integration with Google Cloud’s AI Platform, simplifying the process of obtaining and visualizing explanations.\n",
    "   - **Variety of Methods:** Supports multiple attribution methods (Sampled Shapley, Integrated Gradients, XRAI) to suit different types of models and data.\n",
    "\n",
    "5. **Challenges:**\n",
    "   - **Complexity in Interpretation:** Even with visualizations, understanding and correctly interpreting the attributions may still be challenging, especially for complex models and datasets.\n",
    "   - **Computational Resources:** Some methods, especially those involving deep networks or large datasets, may require significant computational resources.\n",
    "\n",
    "#### Short Notes:\n",
    "- **Google’s AI Explanations:** A managed service for feature attribution in AI Platform Prediction, providing insights into model outputs for classification and regression tasks.\n",
    "- **Feature Attribution Methods:**\n",
    "  - **Sampled Shapley:** Based on Shapley values, involves sampling feature combinations.\n",
    "  - **Integrated Gradients:** Calculates and averages gradients to determine feature importance.\n",
    "  - **XRAI:** Extends integrated gradients for image data, segmenting images and ranking regions by contribution.\n",
    "- **Advantages:** Cloud integration, multiple attribution methods, and useful visualizations.\n",
    "- **Challenges:** Interpretation complexity and computational resource demands.\n",
    "\n",
    "This summary outlines how Google’s AI Explanations service helps in understanding model predictions by providing feature attributions and visualizations, making it easier to interpret and improve machine learning models."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.12.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}