Applying PVT to Semantic Segmentation

Here, we take MMSegmentation v0.13.0 as an example, applying PVTv2 to SemanticFPN.

For details see Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions.

If you use this code for a paper please cite:

@misc{wang2021pyramid, title={Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions}, author={Wenhai Wang and Enze Xie and Xiang Li and Deng-Ping Fan and Kaitao Song and Ding Liang and Tong Lu and Ping Luo and Ling Shao}, year={2021}, eprint={2102.12122}, archivePrefix={arXiv}, primaryClass={cs.CV} }

Usage

Install MMSegmentation.

Data preparation

First, prepare ADE20K according to the guidelines in MMSegmentation.

Then, download the weights pretrained on ImageNet at here, and put them in a folder pretrained/

Results and models

| Backbone | Iters | mIoU | Config | Download | | :-------------: | :-----: | :------: | :------------: | :----: | | PVTv2-B0 + Semantic FPN | 40K | 37.2 | config | | | PVTv2-B1 + Semantic FPN | 40K | 42.5 | config | | | PVTv2-B2 + Semantic FPN | 40K | 45.2 | config | | | PVTv2-B3 + Semantic FPN | 40K | 47.3 | config | | | PVTv2-B4 + Semantic FPN | 40K | 47.9 | config | | | PVTv2-B5 + Semantic FPN | 40K | 48.7 | config | |

Evaluation

To evaluate PVTv2-B2 + SemFPN on a single node with 8 gpus run: dist_test.sh configs/sem_fpn/PVT/fpn_pvtv2_b2_ade20k_40k.py /path/to/checkpoint_file 8 --out results.pkl --eval mIoU

Training

To train PVTv2-B2 + SemFPN on a single node with 8 gpus run:

dist_train.sh configs/sem_fpn/PVT/fpn_pvtv2_b2_ade20k_40k.py 8

License

This repository is released under the Apache 2.0 license as found in the LICENSE file.

Issues

a question of loading checkpoints

opened on 2022-11-23 07:50:29 by uuz21181522

Hello I try to run the dist_test.sh and use [pvt_v2_b2.pth as pretrained ckeckpoints] and [configs/sem_fpn/PVTv2/fpn_pvtv2_b2_ade20k_40k.py] but get an error: Traceback (most recent call last): File "./test.py", line 159, in Traceback (most recent call last): File "./test.py", line 159, in main() File "./test.py", line 127, in main main() File "./test.py", line 127, in main model.CLASSES = checkpoint['meta']['CLASSES'] KeyErrormodel.CLASSES = checkpoint['meta']['CLASSES']: 'meta' KeyError: 'meta'

And the model and loaded state dict do not match exactly too.

How many iterations do you train to get the ade20k result

opened on 2022-02-10 03:27:16 by ChengHan111

Hi, I am curious on how many iterations are you training on ade20k result. I looked for Segformer paper using 160K iters to reach the performance of mIOU 37.4(MiT_B0). But I found you are using 40k to reach a competitive result (mIOU 37.2). Is that a typo or it is reaching a competitive performance with much less iterations? Also, could you provide the pretrained model for pvt_b0 to b5?

Thanks