A toolbox to iNNvestigate neural networks' predictions!

albermax, updated 🕥 2023-02-28 14:34:22

iNNvestigate neural networks!

Different explanation methods on ImageNet.

Introduction
Installation
Usage and Examples
More documentation
Contributing
Releases

Introduction

In the recent years neural networks furthered the state of the art in many domains like, e.g., object detection and speech recognition. Despite the success neural networks are typically still treated as black boxes. Their internal workings are not fully understood and the basis for their predictions is unclear. In the attempt to understand neural networks better several methods were proposed, e.g., Saliency, Deconvnet, GuidedBackprop, SmoothGrad, IntegratedGradients, LRP, PatternNet and PatternAttribution. Due to the lack of a reference implementations comparing them is a major effort. This library addresses this by providing a common interface and out-of-the-box implementation for many analysis methods. Our goal is to make analyzing neural networks' predictions easy!

If you use this code please star the repository and cite the following paper:

Alber, M., Lapuschkin, S., Seegerer, P., Hägele, M., Schütt, K. T., Montavon, G., Samek, W., Müller, K. R., Dähne, S., & Kindermans, P. J. (2019). iNNvestigate neural networks! Journal of Machine Learning Research, 20. @article{JMLR:v20:18-540, author = {Maximilian Alber and Sebastian Lapuschkin and Philipp Seegerer and Miriam H{{\"a}}gele and Kristof T. Sch{{\"u}}tt and Gr{{\'e}}goire Montavon and Wojciech Samek and Klaus-Robert M{{\"u}}ller and Sven D{{\"a}}hne and Pieter-Jan Kindermans}, title = {iNNvestigate Neural Networks!}, journal = {Journal of Machine Learning Research}, year = {2019}, volume = {20}, number = {93}, pages = {1-8}, url = {http://jmlr.org/papers/v20/18-540.html} }

Installation

iNNvestigate is based on Keras and TensorFlow 2 and can be installed with the following commands:

bash pip install innvestigate

Please note that iNNvestigate currently requires disabling TF2's eager execution.

To use the example scripts and notebooks one additionally needs to install the package matplotlib:

bash pip install matplotlib

The library's tests can be executed via pytest. The easiest way to do reproducible development on iNNvestigate is to install all dev dependencies via Poetry: ```bash git clone https://github.com/albermax/innvestigate.git cd innvestigate

poetry install poetry run pytest ```

Usage and Examples

The iNNvestigate library contains implementations for the following methods:

function:
gradient: The gradient of the output neuron with respect to the input.
smoothgrad: SmoothGrad averages the gradient over number of inputs with added noise.
signal:
deconvnet: DeConvNet applies a ReLU in the gradient computation instead of the gradient of a ReLU.
guided: Guided BackProp applies a ReLU in the gradient computation additionally to the gradient of a ReLU.
pattern.net: PatternNet estimates the input signal of the output neuron. (Note: not available in iNNvestigate 2.0)
attribution:
input_t_gradient: Input * Gradient
deep_taylor[.bounded]: DeepTaylor computes for each neuron a root point, that is close to the input, but which's output value is 0, and uses this difference to estimate the attribution of each neuron recursively.
lrp.*: LRP attributes recursively to each neuron's input relevance proportional to its contribution of the neuron output.
integrated_gradients: IntegratedGradients integrates the gradient along a path from the input to a reference.
miscellaneous:
input: Returns the input.
random: Returns random Gaussian noise.

The intention behind iNNvestigate is to make it easy to use analysis methods, but it is not to explain the underlying concepts and assumptions. Please, read the according publication(s) when using a certain method and when publishing please cite the according paper(s) (as well as the iNNvestigate paper). Thank you!

All the available methods have in common that they try to analyze the output of a specific neuron with respect to input to the neural network. Typically one analyses the neuron with the largest activation in the output layer. For example, given a Keras model, one can create a 'gradient' analyzer:

```python import tensorflow as tf import innvestigate tf.compat.v1.disable_eager_execution()

model = create_keras_model()

analyzer = innvestigate.create_analyzer("gradient", model) ```

and analyze the influence of the neural network's input on the output neuron by:

python analysis = analyzer.analyze(inputs)

To analyze a neuron with the index i, one can use the following scheme:

python analyzer = innvestigate.create_analyzer("gradient", model, neuron_selection_mode="index") analysis = analyzer.analyze(inputs, i)

Let's look at an example (code) with VGG16 and this image:

Input image.

```python import tensorflow as tf import tensorflow.keras.applications.vgg16 as vgg16 tf.compat.v1.disable_eager_execution()

import innvestigate

Get model

model, preprocess = vgg16.VGG16(), vgg16.preprocess_input

Strip softmax layer

model = innvestigate.model_wo_softmax(model)

Create analyzer

analyzer = innvestigate.create_analyzer("deep_taylor", model)

Add batch axis and preprocess

x = preprocess(image[None])

Apply analyzer w.r.t. maximum activated output-neuron

a = analyzer.analyze(x)

Aggregate along color channels and normalize to [-1, 1]

a = a.sum(axis=np.argmax(np.asarray(a.shape) == 3)) a /= np.max(np.abs(a))

Plot

plt.imshow(a[0], cmap="seismic", clim=(-1, 1)) ```

Analysis image.

Tutorials

In the directory examples one can find different examples as Python scripts and as Jupyter notebooks:

Introduction to iNNvestigate: shows how to use iNNvestigate.
Comparing methods on MNIST: shows how to train and compare analyzers on MNIST.
Comparing output neurons on MNIST: shows how to analyze the prediction of different classes on MNIST.
Comparing methods on ImageNet: shows how to compare analyzers on ImageNet.
Comparing networks on ImageNet: shows how to compare analyzes for different networks on ImageNet.
Sentiment Analysis.
Development with iNNvestigate: shows how to develop with iNNvestigate.
Perturbation Analysis.

To use the ImageNet examples please download the example images first (script).

Contributing

If you would like to contribute or add your analysis method please open an issue or submit a pull request.

Releases

Can be found here.

Acknowledgements

Adrian Hill acknowledges support by the Federal Ministry of Education and Research (BMBF) for the Berlin Institute for the Foundations of Learning and Data (BIFOLD) (01IS18037A).

Issues

Bump werkzeug from 2.2.2 to 2.2.3

opened on 2023-02-16 06:41:42 by dependabot[bot]

Bumps werkzeug from 2.2.2 to 2.2.3.

Release notes

Sourced from werkzeug's releases.

2.2.3

This is a fix release for the 2.2.x release branch.

Changes: https://werkzeug.palletsprojects.com/en/2.2.x/changes/#version-2-2-3

Milestone: https://github.com/pallets/werkzeug/milestone/26?closed=1

This release contains security fixes for:

https://github.com/pallets/werkzeug/security/advisories/GHSA-xg9f-g7g7-2323

https://github.com/pallets/werkzeug/security/advisories/GHSA-px8h-6qxv-m22q

Changelog

Sourced from werkzeug's changelog.

Version 2.2.3

Released 2023-02-14

Ensure that URL rules using path converters will redirect with strict slashes when the trailing slash is missing. :issue:2533

Type signature for get_json specifies that return type is not optional when silent=False. :issue:2508

parse_content_range_header returns None for a value like bytes */-1 where the length is invalid, instead of raising an AssertionError. :issue:2531

Address remaining ResourceWarning related to the socket used by run_simple. Remove prepare_socket, which now happens when creating the server. :issue:2421

Update pre-existing headers for multipart/form-data requests with the test client. :issue:2549

Fix handling of header extended parameters such that they are no longer quoted. :issue:2529

LimitedStream.read works correctly when wrapping a stream that may not return the requested size in one read call. :issue:2558

A cookie header that starts with = is treated as an empty key and discarded, rather than stripping the leading ==.

Specify a maximum number of multipart parts, default 1000, after which a RequestEntityTooLarge exception is raised on parsing. This mitigates a DoS attack where a larger number of form/file parts would result in disproportionate resource use.

Commits

22a254f release version 2.2.3
517cac5 Merge pull request from GHSA-xg9f-g7g7-2323
babc8d9 rewrite docs about request data limits
09449ee clean up docs
fe899d0 limit the maximum number of multipart form parts
cf275f4 Merge pull request from GHSA-px8h-6qxv-m22q
8c2b4b8 don't strip leading = when parsing cookie
7c7ce5c [pre-commit.ci] pre-commit autoupdate (#2585)
19ae03e [pre-commit.ci] auto fixes from pre-commit.com hooks
a83d3b8 [pre-commit.ci] pre-commit autoupdate
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/albermax/innvestigate/network/alerts).

Add support for all output activations

opened on 2023-02-15 07:37:41 by Rubinjo

I'm working on a binary classification problem and therefore have a sigmoid activation instead of a softmax activation function for my output layer. I have adjusted the model_wo_softmax function to accept any kind of activation by giving it as an argument to also cover binary classification problems.

See here the old vs the new call: innvestigate.model_wo_softmax(model) innvestigate.model_wo_output_activation(model, "softmax")

I have also edited all examples and documentation that cover this function, so everything should now have this new function.

Bump ipython from 8.9.0 to 8.10.0

opened on 2023-02-11 02:34:02 by dependabot[bot]

Bumps ipython from 8.9.0 to 8.10.0.

Commits

15ea1ed release 8.10.0
560ad10 DOC: Update what's new for 8.10 (#13939)
7557ade DOC: Update what's new for 8.10
385d693 Merge pull request from GHSA-29gw-9793-fvw7
e548ee2 Swallow potential exceptions from showtraceback() (#13934)
0694b08 MAINT: mock slowest test. (#13885)
8655912 MAINT: mock slowest test.
a011765 Isolate the attack tests with setUp and tearDown methods
c7a9470 Add some regression tests for this change
fd34cf5 Swallow potential exceptions from showtraceback()
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Fix `is_layer_at_idx` for LRP

opened on 2023-02-10 20:03:53 by Rubinjo

Fixes the is_layer_at_idx function by binding the loop variable i to the lambda function and looping through the model's layers to find the corresponding index.

Closes #264

Example of fix

I have used a simple CNN network to test the implementation, see below the model summary.

model_summary

I used the following LRP settings. analyzer = LRP( model, rule="Z", input_layer_rule="Flat", until_layer_idx=3, until_layer_rule="Epsilon", ) For verification, I printed the rule object and the layer it will be applied to.

layer+rule

As you can see all three rules are being applied.

Discussion

The until_layer_idx argument will now count every layer in your model, also Input, Reshape, Pooling, etc.

[Question] how to save the analyzer object for later online use.

opened on 2023-01-08 11:32:22 by ShaiBiton

Hi guys,

I love this package and started to use it again after you moved to tf2 (also used it when it was in its tf1 days). So thanks for all the hard work !)

My question is, I'm trying to save the analyzer after generated for future use but I seem to unable to do it. neither pickling/dill helped or even save the tf2 model itself. the generation can take some time and in order to use it as an online feature I have to find a workaround.

from what I've got so far, I cant pickle/dill the object because its using TF infra (which has parts written in C), and I can save the generated model (analyzer_obj._analyzer_model) because the eager execution is disabled (apparently its important..)

anything you guys can contribute from your experience?

P.S the only solution I see is to use full on TF2 and remove the line (tf.compat.v1.disable_eager_execution()) and use tf.GradientTape. I tried to do it but it has issues with the session that add difficulty to this.

Thanks !!) Shai

[BUG] `AnalyzerNetworkBase` analyzers error when using `BatchNormalization` layers

opened on 2022-10-11 14:35:07 by adrhill

On iNNvestigate v2.0.1, creating an analyzer inheriting from AnalyzerNetworkBase errors when the model contains a BatchNormalization layer, e.g.: tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'dense_2_input' with dtype float and shape [?,50]

This might be due to batch normalisation layers keeping moving averages of the mean and standard deviation of the training data, causing problems with the Keras history when reversing the computational graph in iNNvestigate's create_analyzer_model.

Minimal example reproducing the issue

```python import numpy as np import tensorflow as tf from keras.layers import BatchNormalization, Dense from keras.models import Sequential

import innvestigate

tf.compat.v1.disable_eager_execution()

input_shape = (50,) x = np.random.rand(100, *input_shape) y = np.random.rand(100, 2)

model1 = Sequential() model1.add(Dense(10, input_shape=input_shape)) model1.add(Dense(2))

model2 = Sequential() model2.add(Dense(10, input_shape=input_shape)) model2.add(BatchNormalization()) model2.add(Dense(2))

def run_analysis(model): model.compile(optimizer="adam", loss="mse") model.fit(x, y, epochs=10, verbose=0)

analyzer = innvestigate.create_analyzer("gradient", model)
analyzer.analyze(x)

print("Model without BatchNormalization:") # passes run_analysis(model1) print("Model with BatchNormalization:") # errors run_analysis(model2) ```

Full stacktrace

``` Model with BatchNormalization: /Users/funks/Library/Caches/pypoetry/virtualenvs/innvestigate-issues-W0iScZgu-py3.10/lib/python3.10/site-packages/tensorflow/python/client/session.py:1480: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. ret = tf_session.TF_SessionRunCallable(self._session._session, Traceback (most recent call last): File "/Users/funks/Developer/innvestigate-issues/open/issue_238_v3", line 35, in run_analysis(model2) File "/Users/funks/Developer/innvestigate-issues/open/issue_238_v3", line 29, in run_analysis analyzer.analyze(x) File "/Users/funks/Library/Caches/pypoetry/virtualenvs/innvestigate-issues-W0iScZgu-py3.10/lib/python3.10/site-packages/innvestigate/analyzer/network_base.py", line 250, in analyze self.create_analyzer_model() File "/Users/funks/Library/Caches/pypoetry/virtualenvs/innvestigate-issues-W0iScZgu-py3.10/lib/python3.10/site-packages/innvestigate/analyzer/network_base.py", line 196, in create_analyzer_model self._analyzer_model = kmodels.Model( File "/Users/funks/Library/Caches/pypoetry/virtualenvs/innvestigate-issues-W0iScZgu-py3.10/lib/python3.10/site-packages/tensorflow/python/training/tracking/base.py", line 629, in _method_wrapper result = method(self, *args, **kwargs) File "/Users/funks/Library/Caches/pypoetry/virtualenvs/innvestigate-issues-W0iScZgu-py3.10/lib/python3.10/site-packages/keras/engine/functional.py", line 146, in __init__ self._init_graph_network(inputs, outputs) File "/Users/funks/Library/Caches/pypoetry/virtualenvs/innvestigate-issues-W0iScZgu-py3.10/lib/python3.10/site-packages/tensorflow/python/training/tracking/base.py", line 629, in _method_wrapper result = method(self, *args, **kwargs) File "/Users/funks/Library/Caches/pypoetry/virtualenvs/innvestigate-issues-W0iScZgu-py3.10/lib/python3.10/site-packages/keras/engine/functional.py", line 181, in _init_graph_network base_layer_utils.create_keras_history(self._nested_outputs) File "/Users/funks/Library/Caches/pypoetry/virtualenvs/innvestigate-issues-W0iScZgu-py3.10/lib/python3.10/site-packages/keras/engine/base_layer_utils.py", line 175, in create_keras_history _, created_layers = _create_keras_history_helper(tensors, set(), []) File "/Users/funks/Library/Caches/pypoetry/virtualenvs/innvestigate-issues-W0iScZgu-py3.10/lib/python3.10/site-packages/keras/engine/base_layer_utils.py", line 253, in _create_keras_history_helper processed_ops, created_layers = _create_keras_history_helper( File "/Users/funks/Library/Caches/pypoetry/virtualenvs/innvestigate-issues-W0iScZgu-py3.10/lib/python3.10/site-packages/keras/engine/base_layer_utils.py", line 253, in _create_keras_history_helper processed_ops, created_layers = _create_keras_history_helper( File "/Users/funks/Library/Caches/pypoetry/virtualenvs/innvestigate-issues-W0iScZgu-py3.10/lib/python3.10/site-packages/keras/engine/base_layer_utils.py", line 253, in _create_keras_history_helper processed_ops, created_layers = _create_keras_history_helper( [Previous line repeated 3 more times] File "/Users/funks/Library/Caches/pypoetry/virtualenvs/innvestigate-issues-W0iScZgu-py3.10/lib/python3.10/site-packages/keras/engine/base_layer_utils.py", line 251, in _create_keras_history_helper constants[i] = backend.function([], op_input)([]) File "/Users/funks/Library/Caches/pypoetry/virtualenvs/innvestigate-issues-W0iScZgu-py3.10/lib/python3.10/site-packages/keras/backend.py", line 4275, in __call__ fetched = self._callable_fn(*array_vals, File "/Users/funks/Library/Caches/pypoetry/virtualenvs/innvestigate-issues-W0iScZgu-py3.10/lib/python3.10/site-packages/tensorflow/python/client/session.py", line 1480, in __call__ ret = tf_session.TF_SessionRunCallable(self._session._session, tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'dense_2_input' with dtype float and shape [?,50] [[{{node dense_2_input}}]] ```

Releases

v2.0.2 2023-01-31 17:24:37

What's Changed

Support TensorFlow 2.11 by @adrhill in https://github.com/albermax/innvestigate/pull/300
Update CI workflows and project setup by @adrhill in https://github.com/albermax/innvestigate/pull/301
Fix Perturbate on RGB images by @adrhill in https://github.com/albermax/innvestigate/pull/306
Fix documentation build

Full Changelog: https://github.com/albermax/innvestigate/compare/2.0.1...2.0.2

v2.0.1 2022-09-14 15:49:19

What's Changed

Remove analyzer.fit by @adrhill in https://github.com/albermax/innvestigate/pull/289
Fixes to README and documentation

Full Changelog: https://github.com/albermax/innvestigate/compare/2.0.0...2.0.1

v2.0.0 2022-08-01 16:00:28

iNNvestigate for TensorFlow 2

What's Changed

Format code by @adrhill in https://github.com/albermax/innvestigate/pull/247
Switch setup to Poetry by @adrhill in https://github.com/albermax/innvestigate/pull/257
Update CI by @adrhill in https://github.com/albermax/innvestigate/pull/258
Add issue templates by @adrhill in https://github.com/albermax/innvestigate/pull/259
Update backend by @adrhill in https://github.com/albermax/innvestigate/pull/263
Update analyzer init and serialization by @adrhill in https://github.com/albermax/innvestigate/pull/266
iNNvestigate 2.0 by @adrhill in https://github.com/albermax/innvestigate/pull/277
Update NumPy lower bound to 1.22, drop Python 3.7 support by @adrhill in https://github.com/albermax/innvestigate/pull/280

New Contributors

@adrhill made their first contribution in https://github.com/albermax/innvestigate/pull/247

Full Changelog: https://github.com/albermax/innvestigate/compare/1.0.8...2.0.0

Maximilian Alber

GitHub Repository

A toolbox to iNNvestigate neural networks' predictions!

iNNvestigate neural networks!

Table of contents

Introduction

If you use this code please star the repository and cite the following paper:

Installation

Usage and Examples

Get model

Strip softmax layer

Create analyzer

Add batch axis and preprocess

Apply analyzer w.r.t. maximum activated output-neuron

Aggregate along color channels and normalize to [-1, 1]

Plot

Tutorials

More documentation

Contributing

Releases

Acknowledgements

Issues

Bump werkzeug from 2.2.2 to 2.2.3

2.2.3

Version 2.2.3

Add support for all output activations

Bump ipython from 8.9.0 to 8.10.0

Fix `is_layer_at_idx` for LRP

Example of fix

Discussion

[Question] how to save the analyzer object for later online use.

[BUG] `AnalyzerNetworkBase` analyzers error when using `BatchNormalization` layers

Minimal example reproducing the issue

Full stacktrace

Releases

v2.0.2 2023-01-31 17:24:37

What's Changed

v2.0.1 2022-09-14 15:49:19

What's Changed

v2.0.0 2022-08-01 16:00:28

iNNvestigate for TensorFlow 2

What's Changed

New Contributors

Maximilian Alber