Applying vision transformer to assess multi-scale morphological features in mammography for breast cancer detection: multiscale image morphological extraction vision transformer (MIME-ViT)

Screening mammography can have limited sensitivity and specificity, especially in dense breasts, which leads to missed cancers and unnecessary follow-up procedures. Deep learning models such as CNNs and Vision Transformers (ViTs) offer a way to improve on this.

Radiologists read a mammogram at both a macro scale (the whole breast) and a micro scale (fine findings). MIME-ViT was designed to mirror this by extracting morphological features at several scales at once.

The model was trained and evaluated on TOMPEI-CMMD, which adds lesion segmentations to the public CMMD collection (mammograms of 1,775 Chinese patients examined between July 2012 and January 2016). Only MLO views were used, giving 2,601 breast images.

Images were split 7:1.5:1.5 by patient into 1,819 training, 391 validation, and 391 test images. Each image was padded with black to 2,294×2,294 and resized to 2,048×2,048.

MIME-ViT has four ViT branches that take 512×512, 256×256, 64×64, and 16×16-pixel patches, plus a CNN branch built from residual blocks with 3×3 kernels. Because of GPU memory limits, the 16×16 branch applies convolution and pooling before its ViT.

The branch outputs are concatenated and the model predicts, for each 64×64-pixel region of the 2,048×2,048 image, whether it contains a lesion. A region counts as positive when the lesion mask covers more than 20% of it. Training combined Dice and Focal losses and used no pre-training. The model has 112.9M parameters (DETR: 41.3M).

Baselines were DETR and YOLOv8x, each trained from scratch and with COCO pre-training. Metrics were mean IoU on images with lesions and specificity on lesion-free images.

MIME-ViT reached a mean IoU of 0.3342 over all lesions, 0.3797 for masses, and 0.2491 for calcifications. It beat DETR-Scratch (0.0000) but scored below DETR-Pretrained (0.3691), YOLOv8x-Scratch (0.3400), and YOLOv8x-Pretrained (0.4516, the best overall).

For calcifications, MIME-ViT (0.2491) was second only to YOLOv8x-Pretrained (0.2796) and well ahead of DETR-Pretrained (0.0280) and YOLOv8x-Scratch (0.0454). It also had the smallest IoU standard deviation in every category, and for calcifications its median IoU was higher than YOLOv8x-Pretrained's.

In ablations, removing any one ViT branch (512 / 256 / 64), switching to 5×5 kernels, or fusing by addition all lowered mean IoU to between 0.2947 and 0.3198, so each component contributed to performance.

The multiscale ViT + CNN design learned lesion localization from about 1,800 training images without any pre-training, a setting in which DETR detected nothing. It was also relatively strong and stable on calcifications, which are small findings.

Kashiwada Y, Takaya E, Hiroya M, Matsuda N, Yashima T, Kobayashi T, Tamiya G, Ueda T. 2025. Applying vision transformer to assess multi-scale morphological features in mammography for breast cancer detection: multiscale image morphological extraction vision transformer (MIME-ViT). PeerJ Computer Science 11:e3252 https://doi.org/10.7717/peerj-cs.3252

Kashiwada, Y., Takaya, E., Hiroya, M., Matsuda, N., Yashima, T., Kobayashi, T., Tamiya, G., & Ueda, T. (2025). Applying vision transformer to assess multi-scale morphological features in mammography for breast cancer detection: multiscale image morphological extraction vision transformer (MIME-ViT). PeerJ Computer Science, 11, e3252. https://doi.org/10.7717/peerj-cs.3252

@article{kashiwada2025mimevit,
  author       = {Kashiwada, Yuki and Takaya, Eichi and Hiroya, Mei and Matsuda, Nanako and Yashima, Takumi and Kobayashi, Tomoya and Tamiya, Gen and Ueda, Takuya},
  title        = {Applying vision transformer to assess multi-scale morphological features in mammography for breast cancer detection: multiscale image morphological extraction vision transformer {(MIME-ViT)}},
  journal      = {PeerJ Computer Science},
  volume       = {11},
  pages        = {e3252},
  issn         = {2376-5992},
  publisher    = {PeerJ Inc.},
  year         = {2025},
  doi          = {10.7717/peerj-cs.3252},
  url          = {https://doi.org/10.7717/peerj-cs.3252}
}

Citation files: kashiwada2025mimevit.bib (BibTeX) · kashiwada2025mimevit.ris (RIS / EndNote / Zotero)