Skin Cancer Detection Using Preprocessed Vision Transformers: An Interpretability-Driven Approach

I. Introduction

Skin cancer is an issue affecting health all over the world, and proper diagnosis of the issue would lead to better patient outcomes. Deep learning, in particular Convolutional Neural Networks (CNNs), has brought more objective and faster screening techniques. This paper assesses the diagnostic capability of Vision Transformers (ViT) with Explainable AI (XAI) on HAM10000 data (10,015 pigmented skin lesion images).

The researchers employed a critical preprocessing pipeline that included morphology transformations to remove hair, image segmentation to identify regions of interest, and data augmentation to address imbalanced classes. Five XAI techniques were used, namely, Grad-CAM, LIME, SHAP, Integrated Gradients, and Saliency Maps, to improve diagnostic transparency.

The proposed system suggests that ViT + Grad-CAM architecture was the best performer with a total accuracy of 96.6%. The model achieved 0.92 precision, 0.90 recall, and 0.92 F1 score. Grad-CAM was especially useful in a clinical setting as it is fast to run and localizes classes accurately.

II. Literature Review

More recent and up-to-date findings indicate a current paradigm shift in favor of Vision Transformers (ViT). In contrast to CNNs, ViTs employ a self-attention mechanism that is used to model global relationships in an entire image at the same time. ViTs were able to handle the intricate spatial correlations in the HAM10000 dataset and achieve competitive accuracy, despite the inductive biases of regular convolutions [1], [2]. ViT architecture could achieve 99% accuracy in class imbalance cases, which is a better choice of early melanoma detection [3].

One of the major gaps in research that [4] has found is shortcut learning, where the models confuse hair or clinical features as a diagnostic proxy. U-Net-based methods are introduced to perform automated hair segmentation to fight this, and the methods enhanced the Dice scores of the subsequent classification stages greatly [5].

Morphological Black-Hat transformations have become a standard means of taking care of the model being centered on pathological biomarkers [6]. Preprocessing, such as CIE Lab binarization, is a requirement of the Clinical-Grade AI because it separates the Region of Interest (ROI) and avoids the bias of the model due to different skin tones or lighting conditions [7].

Grad-CAM to be the most effective tool in terms of real-time localization and has a good correspondence to the clinical ABCD rule [8]. Though SHAP provides high mathematical faithfulness through game-theoretic values, it is computationally slow enough that it must not be used in practice in real-time [9]. LIME is a terrific instance-level transparency but may be unstable with various perturbations of images [10]. Integrated Gradients are able to overcome the so-called gradient saturation issue with less complex saliency maps, which offer a more competent attribution of feature importance [11].

Late 2024 comparative studies, however, reached up to 99.3% accuracy with hybrid segmentation-ViT models, but with high computational overheads frequently. Human-AI collaboration based on visual explanations enhances dermatologists’ sensitivity to 100%, which is particularly important in real-world settings [12], [13]. The effectiveness of CNN-based local feature extraction and ViT-based global modeling (which is a hybrid model) has been proven and has since been optimized in terms of mobile clinical application by the creation of the multi-scale LMS-ViT framework [14], [15].

TABLE I. Literature Review

References	Technique Used	Advantages	Disadvantages
[1, 2, 3]	ViT	Captures long-range spatial correlation	High data dependency, computational cost
[14, 15]	Hybrid CNN-ViT / LMS-ViT	Combines local feature extraction with global reasoning	High computational overhead
[5, 6, 7]	U-Net / Black-Hat / CIE Lab	Effectively removes hair artifacts	Segmentation errors propagate to classification
[8]	Grad-CAM (XAI)	Fast, real-time localization	Heatmaps may emphasize irrelevant background pixels
[9]	SHAP (XAI)	High mathematical faithfulness	Prohibitive for real-time applications
[10]	LIME (XAI)	Provides instance-level transparency	Inconsistent results under image perturbation
[11]	Integrated Gradients (XAI)	Resolves gradient saturation	Requires baseline selection

III. Methodology of Proposed System

The methodology pipeline, as depicted in Fig. 1., for this study follows an intricate process of defining a research problem, followed by detailed analysis of preexisting studies in the field of skin cancer detection. The combination of advanced processing of images, state-of-the-art transformer architectures, and various explainability frameworks enables the framework to assure that every prediction is based on the relevant pathological features. The subsequent scenarios outline the chronological order of the system implementation, starting with the initial data refinement to the development of visual evidence that will be used in clinical validation.

Fig. 1. Methodology pipeline of the proposed system

A. Dataset Details

The Human Against Machine with 10,000 training images (HAM10000) dataset is a standard collection of 10,015 dermatoscopy images of pigmented skin lesions, which was selected to overcome the deficiency of diversity and volume in earlier dermatological datasets. HAM10000, which is accompanied by rich metadata such as patient age, sex, and the location of the anatomical lesion, is a vital resource in the training and assessment of deep learning models in automated skin cancer diagnosis. This metadata is necessary in learning with multi-modes and the transparency of the ground-truth labels. According to the demographic data, which is represented in Fig. 2. (c), the HAM10000 dataset is comparatively balanced in terms of biological sex, with males constituting 54.0% and females 45.5%, with the remaining 0.6% not identified. The age distribution is nearly normal (Fig. 2. (b)), and with most of the patients being middle-aged, it is seen that the largest concentration of the lesions is in the 40-50 age range. As represented in the graph Fig. 2. (d), anatomical evidence indicates that lesions are mostly located at the back, followed by the lower extremities and the trunk, but the neck and the feet are among the least frequent sites in which the sampled lesions are located.

Fig. 2. (a) Distribution of lesion types

Fig. 2. (b) Age distribution

Fig. 2. (c) Gender distribution

Fig. 2. (d) Top 10 lesion locations

B. Preprocessing Pipeline

The critical stage of digital image processing to remove clinical noise and normalize data used in deep learning is referred to as preprocessing. Preprocessing balances the input data and cleans the data by filtering out the input to ensure that the model learns to recognize diagnostic pathological features and is not fooled by environmental or statistical biases. Results each stage of the preprocessing techniques are depicted and visualized in Fig. 3.

1. Hair Removal / Artifact Correction: It starts with the conversion of an RGB image into a grey scale to isolate the variations in structure. This is followed by a morphological Black-Hat transformation with a rectangular structuring element of size 9 x 9 and this operation emphasizes dark and thin and linear structures (hairs) which are smaller than the kernel size in comparison with the light skin background. These artifacts are then defined using a binary mask, which is produced through thresholding. Lastly, the picture is rebuilt with the help of the Inpainting algorithm, which fills the masked pixels with the color and texture of the neighboring boundary pixels.

2. Image Segmentation: It starts with the transformation of the picture from the RGB color to increase the difference between the lesion and the skin. A Gaussian blur is used to eliminate high-frequency noise, and then we use Binarization, which automatically computes an optimal threshold to distinguish between the lesion and the background. To clean up the end result of the mask, morphological Closing and Opening are applied with an elliptical window of size 2 X 2 to seal holes in the interior and eliminate tiny background noise. The last segmented image is obtained through a bitwise-AND operation between the original image and the refined mask and it retains the original color, and texture of the lesion but eliminates the background.

3. Normalization: Since the raw dermatoscopic images are represented by 8-bit integers, each pixel is scaled to a floating-point range [0, 1] by dividing the pixel by 255.0. Photos are scaled down to a common resolution to fit the input demands of the chosen deep learning architecture, so that the shapes of tensors across the pipeline are similar.

4. Data Augmentation: This is done by geometric transformations such as random horizontal and vertical flips, and orthogonal rotations of 90, 180, and 270 degrees. Photometric adjustment is also done by random scaling of brightness by a factor of between 0.8 and 1.2.

Fig. 3. Result of preprocessing techniques: hair removal, image segmentation, normalization, and data augmentation

C. Train - Test Split

In order to assess the performance of the model on hidden data, a Hold-out validation strategy was used to split the data into training and test. The combination of 10,015 samples was divided by an 80-to-20 ratio, which yielded 8,012 images to be used as a model training sample and 2,003 images to be used in final performance analysis. An arbitrary state 42 was used so that the split could replicate consistently when repeated in an experiment. The given partitioning enables the network to acquire representative features on the training set but offers a statistically significant test set to evaluate the generalizability.

D. Image Classification using Vision Transformers (ViT)

A Vision Transformer (ViT) is a computer vision deep learning architecture that uses a Transformer architecture, originally created to process Natural Language Processing (NLP), in order to perform computer vision. ViTs view an image as a sequence of data instead of using localized filters to process the image and identify edges and textures. In this way, they will be able to record the global context and long-range relationship between various parts of an image, even at the very first layer.

ViT divides the image of a skin lesion into a grid of small fixed-size patches (e.g., 16 by 16 pixels). The patches are handled as though words in a sentence. These patches are squashed and turned into vectors. Since the transformers process all patches simultaneously, the model uses Positional Encodings to keep track of the location of each component of the lesion in the original image. The model can do a comparison on one side of a lesion (such as an irregular border) with a feature on the other side (such as color variegation) simultaneously to comprehend the global symmetry and context. The information of all patches is collected by a special token (the CLS token) by attention layers. The last representation is fed to a Multi-Layer Perceptron (MLP) to provide the final diagnosis.

E. Explainable Artificial Intelligence (XAI) Implementation

Explainable AI (XAI) is a subdivision of artificial intelligence that aims to expose human users to the internal reasoning and decision-making processes of complex machine learning models. Traditional deep learning models can be viewed as black boxes in many cases, i.e., they deliver precise results, but do not tell how they were obtained, whereas XAI will give the reason behind why a certain result was obtained. XAI achieves it with the help of techniques such as feature attribution and visualization that underline the points of data that impact the result the most.

1. Gradient-weighted Class Activation Mapping (Grad-CAM): Grad-CAM is an eminent visualization method and generates heatmaps to indicate the significance of pixels in an image regionally. It uses the gradients of a target class (e.g., Melanoma) that flows into the final convolutional layer to generate a localization map of the significant regions. When considering the HAM10000 dataset, Grad-CAM enables the researcher to look at the images to ensure that the neural network is appropriately focusing on the morphology of the lesion or whether it has moved its attention to the tissue surrounding it, which is healthy. This has been visualized in Fig. 4.

Fig. 4. ViT + Grad-CAM visualization

2. Local Interpretable Model-agnostic Explanations (LIME): LIME is a model-agnostic algorithm, which proposes a simple locally linear model to explain individual predictions by estimating the more complex global model. This algorithm works by distorting the input picture, dividing it into so-called super-pixels and randomizing the hiding of selected ones, to see the difference it produces in the prediction probability. Such a technique is especially useful in detecting so-called shortcut learning: a model may falsely conclude that a benign diagnosis is present in a given clinical state or lighting scenario and not the lesion itself.

3. Shapley Additive Explanations (SHAP): Based on cooperative game theory, SHAP is a mathematically sound way to attribute features, which computes the "Shapley value" of each pixel. These values show the average marginal contribution of a feature to the final prediction in the set of all possible feature coalitions. SHAP is frequently favored in the research of skin cancer due to its consistency and theoretical basis, providing a pixel-level granulated map that shows which particular visual textures had a positive or negative influence on the malignancy scale.

4. Integrated Gradients (IG): Integrated Gradients solves the gradient saturation issue of conventional saliency maps, in which minimal fluctuations on the input do not produce further fluctuations on the result. IG measures the sum of the gradients that are covered by a line between a baseline (usually a black image) and the original input. This makes sure that all the areas of the lesion leading to the classification are taken into consideration, and so a cleaner and more faithful representation of the internal logic of the model is obtained than using conventional backpropagation techniques.

5. Saliency Maps: The saliency maps are one of the oldest versions of XAI, and they are aimed at deriving the output score with respect to the input image. These maps illustrate the pixels that the model is the most sensitive to. Although they tend to be noisier than Grad-CAM or IG, they still give a high-resolution image of the edges and subtle-grained textures that the model believes are diagnostic. Comparing saliency maps of various architectures in a research environment can be used to understand how diverse models perceive the structural limits between various skin pathologies.

IV. Experimental Results

The outcome of the experiments strictly compares in favor of the ViT + Grad-CAM architecture, which was the most successful in terms of the overall accuracy of 96.6%. This advantage is owed to the fact that the Vision Transformer possesses a multi-scale attention mechanism, enabling the model to better capture global symmetry, which can be used as a more important malignancy signifier than conventional CNNs. Although SHAP and Integrated gradients demonstrate slightly more recall scores and F1 scores, Grad-CAM is the ultimately better option as far as real-world use is concerned because of the speed of execution and class-discriminating localization.

Saliency Maps and Integrated Gradients are susceptible to visual artifacts and noise, and occasionally emphasize meaningless background pixels that are not related to clinical features, which is called the gradient saturation problem. SHAP is also highly scalable (with its game-theoretic nature being exceedingly expensive to run on high-resolution images), and LIME itself uses image "perturbations" (hiding parts of the image), which may result in unrealistic inputs and therefore unstable or inconsistent answers. The majority of these post-hoc techniques have a similar drawback; they can give plausible visual explanations that appear correct to a human being but do not accurately mimic the actual inner workings of the model, which can give a false sense of security in a clinical environment.

In order to strictly evaluate the diagnostic strength of the model using HAM10000 data, a multi-dimensional evaluation framework is used. A number of statistical measures are obtained to give a good representation of the model performance. Accuracy, precision, recall, and the F1 score are used to measure the overall performance of the model.

$$precision = TP / (TP + FP)$$ $$recall = FP / (TP + FP)$$ $$accuracy = (TP + TN) / (TP + TN + FP + FN)$$ $$f1 score = (2 X precision X recall) / (precision + recall)$$

Where TP stands for True Positive, TN for True Negative, FP for False Positive, and FN for False Negative. Accuracy is determined to be equal to the ratio of observations that are correctly predicted to the number of observations. Precision is a measure of how much the model only gives relevant results.** Recall is the measure of the ability of the model to detect all the instances of a class. It is the capability of the model to identify a certain disease correctly, which is essential in identifying life-threatening conditions. F1-Score is used as the harmonic mean of the precision and recall, and is the main parameter used to consider the tradeoff between false negatives and false positives.

The model's accuracy of 96% showed a high percentage of accurate classifications. This is comparatively higher than other models. The model's 0.92 precision for the malignant class indicates that it is good at detecting malignant lesions while reducing false positives. The malignant class's recall was 0.90, indicating that the model effectively identifies malignant lesions while lowering false negatives. The model's strong performance was demonstrated by the F1 score, a balanced metric that considers both precision and recall, which was 0.92. The performance metrics have been enlisted in Table II and the comparison between each XAI framework along with ViT has been depicted in Fig. 5. (a).

According to the confusion matrix, Fig. 5. (b), the classification errors of the model are all in terms of false negatives (29 false negatives). The number of false positives was 5, which means that the model convict some benign lesions as malignant. These classification difficulties are explained by the various technical factors, such as shortcut learning, in which the model menaces clinical noise, such as hair or background artifact, and diagnostic features.

TABLE II. Performance Analysis

Model	Accuracy	Precision	Recall	F1 Score
ViT + Grad-CAM (proposed system)	0.96	0.92	0.90	0.92
ViT + Lime	0.92	0.90	0.88	0.90
ViT + SHAP	0.94	0.95	0.94	0.95
ViT + Saliency Maps	0.95	0.94	0.92	0.93
ViT + Integrated Gradients	0.95	0.92	0.93	0.93

Fig. 5. (a) Performance benchmarks of ViT Models by XAI validation method

Fig. 5. (b) Confusion matrix of Grad-CAM implementation

V. Conclusion

To sum up, this research confirms that Vision Transformers (ViT) combined with Explainable AI (XAI) can help improve the accuracy and transparency of skin cancer diagnosis to a greater extent. Through a highly robust pipeline of morphological hair removal and ROI segmentation, the ViT model was able to learn global features in pathology, and the level of accuracy with the Grad-CAM framework is high at 96.6%. This combination was the most effective to be used in clinical practice as it offered the most reliable second opinion using class-discriminative heatmaps.

The proposed solution has been associated with a number of critical issues that need to be resolved to guarantee the clinical viability and technical strength. The self-attention aspect of the Vision Transformer requires a lot of computational power, making it difficult to run on mobile devices with limited resources that can be used to deploy the technology to the intended real-time screening environment. The accuracy of automated preprocessing is still critical to the system performance, with failure to clean hair or segmentation may cause artifacts, and threaten diagnostic quality. Class imbalance of the dataset creates a longstanding threat to the sensitivity of the model to less common malignant lesions, whereas inconsistency of post-hoc XAI techniques may result in interpretability fatigue in clinicians.

To proceed with this research in the future, the researcher plans to move towards inherent interpretability and a wider range of skin phototypes in the dataset to promote diagnostic equity across the globe. The creation of lightweight MobileViT screening applications to be used in real-time screening in underserved areas and data integration with multimodal patient data would also increase predictive power. These interpretable models will need extensive clinical trials to assess the feasible effects of the technology on the reduction of biopsies and patient survival rate in regular dermatological practices.

References

[1]     Yang, G., Luo, S., & Greer, P. (2023). A novel vision transformer model for skin cancer classification. Neural Processing Letters, 55(7), 9335–9351.

[2]     Nikitin, V., & Shapoval, N. (2023). Vision transformer for skin cancer classification. Scientific Collection InterConf+, 33, 449–460.

[3]     Demiroğlu, U. (2025). Diagnosis of the Skin Cancer by Vision Transformers. Duzce University Journal of Science and Technology, 13(1), 588–598.

[4]     Sangwan, H. (2024). Quantifying Explainable AI Methods in Medical Diagnosis: A study in skin cancer. medRxiv, 2012–2024.

[5]     Doğan, M., & Özkan, \.Ilker Ali. (2024). Automated Hair Segmentation in Dermoscopy Images with U-Net Based Approaches. 2024 13th Mediterranean Conference on Embedded Computing (MECO), 1–4.

[6]     Akram, A., Rashid, J., Jaffar, M. A., Faheem, M., & Amin, R. ul. (2023). Segmentation and classification of skin lesions using hybrid deep learning method in the Internet of Medical Things. Skin Research and Technology, 29(11), e13524.

[7]     Alrabai, A., Echtioui, A., & Kallel, F. (2025). Exploring Pre-Trained Models for Skin Cancer Classification. Applied System Innovation, 8(2), 35.

[8]     Baabu, K. B., & Raja, S. D. M. (2025). Skin cancer detection using vision transformer with GNN and explainable AI (grad-cam). 2025 International Conference on Advances in Modern Age Technologies for Health and Engineering Science (AMATHE), 1–6.

[9]     ÖZKURT, C. (2024). Advancing Skin Cancer Diagnosis through the Comparison of SHAP and Layer-wise Relevance Propagation (LRP).

[10]   Vaghela, A. B., Patel, N. K., Panchal, R. K., Bamniya, K. H., & Patel, H. R. (2025). XAI-SKIN: An Approaches for the Diagnosis and Classification of Skin Diseases based on LIME Method. International Journal of Environmental Sciences, 11(3s), 350–363.

[11]   Sundararajan, M., Taly, A., & Yan, Q. (2017). Axiomatic attribution for deep networks. International Conference on Machine Learning, 3319–3328.

[12]   Chung, M., Won, J. B., Kim, G., Kim, Y., & Ozbulak, U. (2024). Evaluating Visual Explanations of Attention Maps for Transformer-Based Medical Imaging. International Conference on Medical Image Computing and Computer-Assisted Intervention, 110–120.

[13]   Winkler, J. K., Blum, A., Kommoss, K., Enk, A., Toberer, F., Rosenberger, A., & Haenssle, H. A. (2023). Assessment of diagnostic performance of dermatologists cooperating with a convolutional neural network in a prospective clinical study: human with machine. JAMA Dermatology, 159(6), 621–627.

[14]   Hamsalekha, H., George, G. D., & Satheesha, T. Y. (2025). A novel deep learning approach for automated melanoma classification using hybrid cnn and vision transformer model. Fusion Practice and Applications, 19(2), 92–101.

[15]   Leema, A. A., Balakrishnan, P., Gopichand, G., & Rajarajan, G. (2025). LMS-ViT: a multi-scale vision transformer approach for real-time smartphone-based skin cancer detection. Frontiers in Artificial Intelligence, 8, 1612502.