Sensitivity Analysis

Quantify how input parameters influence model outputs using variance-based SOBOL indices and the SALib library

Sensitivity analysis quantifies how changes in input parameters affect model outputs. The implementation uses variance-based SOBOL indices through the SALib library to identify which parameters most strongly influence your model predictions.

Key Concepts

First-Order (S1)

Direct contribution of each parameter to output variance. An S1 of 0.3 means that parameter alone accounts for 30% of variance.

Total-Order (ST)

Direct effects plus all interaction effects involving that parameter. ST - S1 indicates interaction strength.

Second-Order (S2)

Pairwise interactions between parameters. Computationally expensive but reveals parameter coupling.

Basic Usage

After fitting a model, call sensitivity_analysis() on your DataFit object:

import ionworkspipeline as iwp

# Set up and run your fit
parameters = {
    "param_a": iwp.Parameter("param_a", initial_value=1.0, bounds=(0.5, 2.0)),
    "param_b": iwp.Parameter("param_b", initial_value=0.5, bounds=(0.1, 1.0)),
}

data_fit = iwp.DataFit(
    objectives=objective,
    parameters=parameters,
    optimizer=iwp.optimizers.ScipyLeastSquares(),
)
results = data_fit.run(fixed_parameters)

# Run sensitivity analysis
sens_results = data_fit.sensitivity_analysis(n_samples=512)

The usage and options shown here are not exhaustive. See the API reference for full details on sensitivity analysis and configuration.

Understanding Results

The method returns a dictionary with SOBOL indices:

{
    "S1": [0.65, 0.25],           # First-order indices
    "S1_conf": [0.05, 0.04],      # 95% confidence intervals
    "ST": [0.72, 0.35],           # Total-order indices
    "ST_conf": [0.06, 0.05],      # 95% confidence intervals
}

Interpretation Example

For S1 = [0.65, 0.25] and ST = [0.72, 0.35]:

Parameter	Direct Effect (S1)	Interaction Effect (ST - S1)	Total Effect (ST)
Parameter 1	65%	7%	72%
Parameter 2	25%	10%	35%

• Unaccounted variance: 1 - sum(S1) = 10% from joint interactions
• Total effects sum > 1: Because interaction contributions involve both parameters

If ST ≈ 0 for a parameter, it has negligible influence on model outputs within the specified bounds. Consider fixing that parameter or checking if bounds are physically realistic.

Configuration Options

Sample Size

sens_results = data_fit.sensitivity_analysis(n_samples=256)

The SOBOL algorithm generates n_samples × (2 × n_params + 2) evaluations. Recommendations:

• Use powers of 2: 256, 512, 1024
• Larger values reduce confidence intervals but increase computation time
• Start with 256 for exploratory analysis

Custom Cost Function

sens_results = data_fit.sensitivity_analysis(
    n_samples=512,
    cost=iwp.costs.SumSquaredError()
)

Progress Tracking

sens_results = data_fit.sensitivity_analysis(
    n_samples=512,
    show_progress=True
)

Second-Order Indices

sens_results = data_fit.sensitivity_analysis(
    n_samples=512,
    calc_second_order=True
)

# Access second-order indices
S2 = sens_results["S2"]       # Matrix of pairwise interactions
S2_conf = sens_results["S2_conf"]

Element S2[i,j] quantifies the interaction between parameters i and j beyond their individual effects.

Practical Workflow

1. Initial Parameter Screening

   Run with moderate sample size to identify important parameters:

   data_fit.setup(params_for_pipeline)
   sens = data_fit.sensitivity_analysis(n_samples=256)
   
   param_names = list(parameters.keys())
   for name, st, st_conf in zip(param_names, sens["ST"], sens["ST_conf"]):
       print(f"{name}: ST = {st:.3f} ± {st_conf:.3f}")
   

   Parameters with ST < 0.05 are typically negligible.

2. Refine Analysis

   For important parameters, use more samples:

   sens = data_fit.sensitivity_analysis(n_samples=1024)
   
3. Investigate Interactions

   If ST - S1 is large for multiple parameters:

   sens = data_fit.sensitivity_analysis(
       n_samples=512,
       calc_second_order=True
   )
   S2 = sens["S2"]  # Large S2[i,j] indicates strong interaction
   

Handling Failed Evaluations

If model evaluations fail for some parameter combinations:

UserWarning: 15 out of 853 evaluations (1.76%) failed or returned invalid values.

If many evaluations fail (>5%):

• Check parameter bounds for physical validity
• Verify model stability across the parameter space
• Consider tightening bounds around the fitted values

Log-Transform Option

If the cost landscape has an extremely large range, use apply_log_transform:

sens_results = data_fit.sensitivity_analysis(
    n_samples=512,
    apply_log_transform=True
)

With log transform, the sensitivity analysis represents multiplicative changes in cost due to parameters, rather than additive changes.

Mathematical Background

SOBOL indices are variance-based measures that decompose the total output variance into contributions from individual parameters and their interactions. This approach treats the model as a black box and samples the entire parameter space, making it a global sensitivity method—in contrast to local methods like gradient-based sensitivity that only characterize behavior near a single point.

Variance Decomposition

Consider a model $Y = f(X_1, X_2, \ldots, X_p)$ where $X_i$ are independent input parameters. The total variance of the output can be decomposed as:

$$V(Y) = \sum_i V_i + \sum_{i<j} V_{ij} + \cdots + V_{1,2,\ldots,p}$$

where $V_i$ is the variance contribution from parameter $i$ alone, $V_{ij}$ captures the interaction between parameters $i$ and $j$, and so on for higher-order interactions.

First-Order Index

The first-order index $S_i$ measures the direct contribution of parameter $X_i$ to the output variance, excluding any interactions:

$$S_i = \frac{V_{X_i}(E_{X_{\sim i}}(Y|X_i))}{V(Y)}$$

The notation $X_{\sim i}$ means “all parameters except $X_i$”. The inner expectation $E_{X_{\sim i}}(Y|X_i)$ averages over all other parameters while holding $X_i$ fixed. The outer variance $V_{X_i}(\cdot)$ then measures how much this conditional mean varies as $X_i$ changes.

Interpretation: If $S_i = 0.3$, then 30% of the output variance is directly attributable to $X_i$ alone, without considering interactions.

Total-Order Index

The total-order index $S_{T_i}$ captures both the direct effect of $X_i$ and all interactions involving $X_i$:

$$S_{T_i} = \frac{E_{X_{\sim i}}(V_{X_i}(Y|X_{\sim i}))}{V(Y)} = 1 - \frac{V_{X_{\sim i}}(E_{X_i}(Y|X_{\sim i}))}{V(Y)}$$

The difference $S_{T_i} - S_i$ quantifies how much of parameter $i$‘s influence comes through interactions with other parameters.

Interpretation: If $S_{T_i} \approx 0$, the parameter has negligible influence on the output—even through interactions—and could potentially be fixed without loss of model fidelity.

Second-Order Index

The second-order index $S_{ij}$ isolates the interaction effect between parameters $i$ and $j$:

$$S_{ij} = \frac{V_{X_i,X_j}(E_{X_{\sim ij}}(Y|X_i,X_j))}{V(Y)} - S_i - S_j$$

This subtracts the individual first-order effects to reveal only the joint contribution that cannot be attributed to either parameter alone.

Interpretation: A large $S_{ij}$ indicates that the effect of parameter $i$ depends on the value of parameter $j$ (and vice versa). This coupling may suggest physical dependencies or identifiability issues.

Saltelli Estimators

Computing SOBOL indices requires evaluating high-dimensional integrals. The Saltelli sampling scheme provides efficient Monte Carlo estimators using two independent sample matrices $\mathbf{A}$ and $\mathbf{B}$, each of size $n \times p$.

For each parameter $i$, a hybrid matrix $\mathbf{A}_B^{(i)}$ is constructed by taking all columns from $\mathbf{A}$ except column $i$, which comes from $\mathbf{B}$.

First-order index estimator:

$$\hat{S}_i = \frac{\frac{1}{n}\sum_{j=1}^{n} f(\mathbf{B})_j(f(\mathbf{A}_B^{(i)})_j - f(\mathbf{A})_j)}{V(Y)}$$

Total-order index estimator:

$$\hat{S}_{T_i} = \frac{\frac{1}{2n}\sum_{j=1}^{n} (f(\mathbf{A})_j - f(\mathbf{A}_B^{(i)})_j)^2}{V(Y)}$$

This scheme requires $n(2p + 2)$ model evaluations for first and total-order indices, where $n$ is the base sample size and $p$ is the number of parameters. The computational cost scales linearly with the number of parameters, making it tractable for moderate-dimensional problems.

Comparison with Other Methods

Method	Purpose	What it Characterizes
Sensitivity Analysis	Which parameters matter?	Parameter importance
Linear Confidence Intervals	How precisely are parameters estimated?	Parameter uncertainty
MCMC Sampling	What are parameter distributions?	Full posterior distributions

Use sensitivity analysis before extensive MCMC sampling to identify which parameters are worth the computational investment.

References

1. Herman, J., & Usher, W. (2017). SALib: An open-source Python library for Sensitivity Analysis. Journal of Open Source Software, 2(9), 97.
2. Saltelli, A., et al. (2010). Variance based sensitivity analysis of model output. Computer Physics Communications, 181(2), 259-270.