F2HF: Feed-Forward Network as a Noisy Head Filter in Vision Transformer Explanations

ICCIT 2025

I. Hossen, K. S. Alam

Khulna University of Engineering & Technology

research research vision-transformers interpretability attention-mechanism explainable-ai
Demo Google Colab Paper GitHub

Abstract

Vision Transformers have established themselves as the state-of-the-art method for computer vision tasks involving large datasets. Yet, the underlying mechanism behind their superior performance remains insufficiently understood. Several attempts have been made to understand the decision-making process of these models. Despite promising results, these methods frequently suffer from noisy outputs or exhibit limited faithfulness in their explanations. Our analysis reveals that a key limitation of existing interpretability methods lies in their inability to capture the Vision Transformer architecture completely. Recent research has revealed that the feed-forward network (FFD) acts as a token filter. Based on this finding, we designed an interpretability method incorporating a feed-forward network as an attention filter for each attention map. We named this method F2HF (Feed-Forward Head Filtering Saliency Generator). The baseline we worked on generated a single attention map by accumulating each block's attention map and multiplying that with classifier gradient heatmap. However, the addition of Feed-Forward filtering in the encoder attention map shows as much as 16.4% improvement of the IoU. Similarly, improvement was observed in precision and F1 score. In a subjective test, three human subjects blindly selected the less noisy heatmap for 500 images, and in 84% of the cases, they chose the saliency map generated by F2HF. The experiment and the method can be reproduced in a consumer-grade GPU (NVIDIA Tesla T4).


Key Contribution

When the object is small or hard to distinguish from the background, F2HF generates cleaner saliency maps than existing methods.

Problem

When a neural network mades a decision, it is difficult to understand why that decision has been made. For example, if there is an images of a tiger and a cat (which both have highly similar facial structure and texture) and a vision model has decided the image to be cat, how would we know the model only focused on the features (i.e., structure and texture) of the cat and not on the features of the tiger? Interpretabiliy methods such as GradCAM, Integrated Gradients can be a tool to explore this question. However, in our experiments, these method seemed to not do good in cases where the object itself had very minimal presence in the overall image (e.g., the total image of 224 by 224 pixels but the object itself was present in 50 by 50 pixels) or when the object itself was difficult to distinguish from the background. Visual example of these 2 problems were added below.

Figure 1: Examples of minimal object presence (left) and high foreground-background similarity (right).

See their attribution


In the above mentioned cases, existing methods are likely to attribute many background pixels comparetively highly. A possible reason is gradient shattering as most of these methods are gradient-based.

Besides, while recent research has shown the MLP to be the knowledge-base of the model, we found these methods to totally ignore MLP's contribution in modifying Attention's output. This is specially true where attention output is passed directly into LayerNorm which also takes MLP output of the last layer. The structure is shown in Figure-2.

Image of ViT Encoder
Figure 2: Transformer encoder block architecture.

In the ViT architecture, LayerNorm takes concatenated vector of attention output and previous layer's MLP output and then normalize the whole concatenated vector where mean and standard deviation were calculted from the whole concatenated vector itself. Therefore, during BackProp, attention output itself are being modified by the MLP output i.e. MLP of layer l-1 learns to filter attention output of layer l. This was ignored on all previous methods.

Solution

We use MLP output of layer l-1 to filter the attention output of layer l. This helps in 2 ways: (i) Replication of MLP's Attention filtering capability in the heatmap generation i.e., making heatmap generation process more closer to the ViT architecture itself, (ii) Eliminating the gradient shattering effect specially in the deeper layers.

Overview of F2HF
Figure 3: Saliency Generation Flowchart using F2HF. We leverage the previous block's final linear layer to filter out unnecessary attention heads.

Results

We rigorously tested F2HF against state-of-the-art baselines. The results were incredibly promising.

Qualitative comparison of generated heatmaps
Figure 4: Comparison Between Proposed method, Beyond Intuition (Token-wise), Integrated Gradient and Rollout. Red marked region indicate attribution while white is background (the darker the red color for a token, the higher the attribution). Our proposed method (f2hf) correctly attributes the object itself while rest of the methods struggle.

Attribution map of previously shown images in Figure-1

Qualitative comparison of generated heatmaps
Qualitative comparison of generated heatmaps
Figure 5: Comparison Between Proposed method, Beyond Intuition (Token-wise), Integrated Gradient and Rollout for the images shown in Figure-1. Red marked region indicate attribution while white is background (the darker the red color for a token, the higher the attribution).

We ran a subjective blind test where three human subjects reviewed 500 images and heatmap and chose the least noisy, most accurate heatmap. In 84% of the cases, humans preferred F2HF over existing methods.

Quantitatively, the addition of Feed-Forward filtering provides massive improvements. We saw up to a 16.4% improvement in Intersection over Union (IoU), along with consistent leaps in Precision and F1 scores.

Segmentation Test Comparison (Higher is better)

Architecture Method Pix. Acc (%) mIoU mAP mF1
ViT Base
 Proposed (F2HF) 81.36 0.65 0.88 0.45
Beyond Int.-T 77.47 0.60 0.87 0.44
Beyond Int.-H 78.04 0.61 0.86 0.44
Rollout 70.33 0.52 0.82 0.42
Raw Attention 65.91 0.41 0.72 0.19
Int. Gradient 67.31 0.49 0.78 0.38
GradCAM 65.91 0.41 0.72 0.19
ViT Large
 Proposed (F2HF) 77.35 0.61 0.85 0.42
Beyond Int.-T 71.46 0.52 0.82 0.39
Beyond Int.-H 77.14 0.59 0.85 0.43
Rollout 66.00 0.47 0.79 0.39
Raw Attention 67.41 0.40 0.70 0.11
Int. Gradient 65.89 0.43 0.72 0.30
GradCAM 63.49 0.40 0.70 0.10

Positive Perturbation Test Results (Lower is better)

Perturbation Top-K Proposed (F2HF) B. Int. (T) B. Int. (H) Rollout IG
10% Top-1 62.14% 65.37% 63.60% 63.55% 72.10%
Top-3 77.26% 80.33% 78.88% 78.60% 85.68%
Top-5 81.94% 84.56% 83.43% 82.95% 89.10%
20% Top-1 48.37% 53.51% 63.59% 50.76% 65.94%
Top-3 63.80% 69.03% 78.87% 66.27% 80.39%
Top-5 69.82% 74.58% 83.43% 71.84% 84.49%
30% Top-1 37.50% 43.59% 41.08% 40.96% 59.25%
Top-3 52.58% 58.74% 56.15% 55.82% 74.31%
Top-5 59.10% 64.70% 62.23% 61.76% 79.26%

Key takeaway from our reserach

By integrating Feed-Forward filtering into the attention mechanism, F2HF provides a significantly more faithful and noise-free interpretability method for Vision Transformers.

Note: Please visit this link if you need the paper PDF as the published version is locked behind paywall.

Citation

@INPROCEEDINGS{11491078,
  author={Hossen, Imran and Alam, Kazi Saeed},
  booktitle={2025 28th International Conference on Computer and Information Technology (ICCIT)}, 
  title={F2HF: Feed-Forward Network as a Noisy Head Filter in Vision Transformer Explanations}, 
  year={2025},
  volume={},
  number={},
  pages={2675-2680},
  keywords={Satellite images;Earth Observing System;Feeds;Antennas;Radio broadcasting;Frequency modulation;Filtering;Filters;Circuits and systems;HTTP;interpretability;attention;feed-forward network},
  doi={10.1109/ICCIT68739.2025.11491078}}