Group-Wise Semantic Mining for Weakly Supervised Semantic Segmentation

Lixy1997, updated 🕥 2022-05-11 10:12:15

Group-Wise Semantic Mining for Weakly Supervised Semantic Segmentation

This is a PyTorch implementation of our group-wise learning framework for weakly supervised semantic segmentation, which is accepted in AAAI-2021. The extended version has been accepted by IEEE TIP.

Group-Wise Learning for Weakly Supervised Semantic Segmentation. [Paper]

Tianfei Zhou, Liulei Li, Xueyi Li, Chunmei Feng, Jianwu Li, and Ling Shao. AAAI 2021/IEEE TIP 2022.

News

Our new paper for weakly supervised semantic segmentation has been accepted to CVPR 2022, with significantly improved performance. Please see RCA.

Prerequisites

We train the model using PyTorch 1.4.0 with four NVIDIA RTX 2080Ti GPU with 11GB memory per card.

Other minor Python modules can be installed by running

bash pip install -r requirements.txt

Training

Clone

git clone -- recursive https://github.com/Lixy1997/Group-WSSS

Prepare Dataset

In the paper, we use PASCAL VOC 2012 for training. Here are the steps to prepare the data:

  1. Download the PASCAL VOC 2012 dataset.

  2. Create soft links:

    cd data; ln -s your/path VOC2012;

Stage #1: Train the classification network for group-wise semantic mining

  1. Once the data is prepared, please run python train.py for training the classification network with our default parameters.

    After the network is finished, you can resize the maps to the original image size by

    bash cd run/pascal python vis.py 2. Move the resized maps to the data/VOCdevkit/VOC2012/ folder.

Put the saliency maps to the data/VOCdevkit/VOC2012/ folder, or you can run DSS model to generate saliency maps by yourself.

  1. Generate the pseudo labels of the training set by

    bash python gen_labels.py

Stage #2: Train the semantic segmentation network

Once the pseudo ground-truths are generated, they are employed to train the semantic segmentation network. We use Deeplab-v2 in all experiments. But most popular FCN-like segmentation networks can be used instead.

Our Results

  1. The visualization of CAMs generated by our group-wise semantic mining can be downloaded from Google Drive.

  2. The saliency maps used as pseudo labels can be downloaded from Google Drive.

  3. The pseudo ground-truths of PASCAL VOC 2012 generated by our model can be downloaded from Google Drive.

  4. The segmentation results of val and test sets of PASCAL VOC 2012 dataset can be downloadeded from Google Drive. For reproducing scores of the test set, please submit the results of test set to the official website following the instructions of the official website.

Citation

If you find this work useful for your research, please consider citing the following paper: Latex @inproceedings{li2021group, title={Group-Wise Semantic Mining for Weakly Supervised Semantic Segmentation}, author={Li, Xueyi and Zhou, Tianfei and Li, Jianwu and Zhou, Yi and Zhang, Zhaoxiang}, booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, pages={1984--1992}, year={2021} } @article{zhou2022group, title={Group-Wise Learning for Weakly Supervised Semantic Segmentation}, author={Zhou, Tianfei and Li, Liulei and Li, Xueyi and Feng, Chun-Mei and Li, Jianwu and Shao, Ling}, journal={IEEE Transactions on Image Processing}, year={2022}, volume={31}, pages={799-811}, publisher={IEEE} }

Issues

关于论文和代码，想请教几个问题

opened on 2022-04-14 02:17:49 by mymuli

您好！关于论文和代码，我想请教以下几个问题： Q1: 原文中有描述“Greedy Mini-Batch Sampling. During training, we design a heuristic, greedy strategy to sample K training images in each mini-batch ”，但我在代码里面，没有找到相关采样策略，只找到从训练集中读取数据的操作。我想问一下，这个采样数量K是否等于模型训练的参数“batch_size”? 是否直接由batch_size来确定采样数量K？ https://github.com/Lixy1997/Group-WSSS/blob/ab6898bcfab364a0e82bd63e783c9993db25c9c6/train.py#L106

Q2: 一般情况下，在一定范围内，模型训练时“batch_size”越大越好。我想问一下Group-WSSS模型(Full Model)在PASCAL VOC2012实验下，batch_size设置为多少？“sample K”与“batch_size”是否相等呢？不相等的话，其值各是多少？

Q3: 在论文中，公式(8) 消息节点mi是对“相邻节点的累加聚合+自身节点信息”。但在代码中发现对相邻节点的信息聚合是通过concat操作来实现的，另外发现没有加上自身节点信息。 WechatIMG5375 https://github.com/Lixy1997/Group-WSSS/blob/ab6898bcfab364a0e82bd63e783c9993db25c9c6/models/vgg_gnn.py#L123

where's the saliency_aug

opened on 2022-04-06 10:07:52 by fsted

Hi,there is a file path './saliency_aug' in gen_labels.py , But I hadn't found it ; Where's it?

About reproduce

opened on 2022-03-29 08:38:46 by fsted

This is a very creative paper，I want to reproduce your results in your paper，But the result is not very ideal；I want to know whether the default parameters you set can get the actual results in the paper. Maybe I should increase the number of network layers or iterations? Thanks 😄

关于多卡训练

opened on 2022-02-25 00:56:46 by 911456451

您好，我在使用多卡训练模型时，进程会随机终止，但是单卡没有问题，请问有什么解决方法吗

About gen_label

opened on 2021-11-17 01:36:03 by Hfhstrive

Hi, thank you for your code. I have a question about label generation. When I run "python gen_labels.py", it generates the folder proxy_label_1, but when I output the label images, I find that all the images are all 255 pixels. Looking forward to your reply！

The code of COCO experiments

opened on 2021-05-09 12:47:49 by zwy1996

Hi, nice work!

Thank you for your code. I found that the project did not contain the code of coco experiments. Could you please offer the code?

Thank you!

Lixy1997
GitHub Repository