This is a collection of our NAS and Vision Transformer work.

microsoft, updated 🕥 2023-03-13 08:09:33

Neural Architecture Design and Search Tweet

This is a collection of our NAS and Vision Transformer work

TinyViT (@ECCV'22): TinyViT: Fast Pretraining Distillation for Small Vision Transformers

MiniViT (@CVPR'22): MiniViT: Compressing Vision Transformers with Weight Multiplexing

CDARTS (@TPAMI'22): Cyclic Differentiable Architecture Search

AutoFormerV2 (@NeurIPS'21): Searching the Search Space of Vision Transformer

iRPE (@ICCV'21): Rethinking and Improving Relative Position Encoding for Vision Transformer

AutoFormer (@ICCV'21): AutoFormer: Searching Transformers for Visual Recognition

Cream (@NeurIPS'20): Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search

We also implemented our NAS algorithms on Microsoft NNI (Neural Network Intelligence).


  • :sunny: Hiring research interns for neural architecture search, tiny transformer design, model compression projects: [email protected]
  • :boom: Jul, 2022: Code for TinyViT is now released.
  • :boom: Apr, 2022: Code for MiniViT is now released.
  • :boom: Mar, 2022: MiniViT has been accepted by CVPR'22.
  • :boom: Feb, 2022: Code for CDARTS is now released.
  • :boom: Feb, 2022: CDARTS has been accepted by TPAMI'22.
  • :boom: Jan, 2022: Code for AutoFormerV2 is now released.
  • :boom: Oct, 2021: AutoFormerV2 has been accepted by NeurIPS'21, code will be released soon.
  • :boom: Aug, 2021: Code for AutoFormer is now released.
  • :boom: July, 2021: iRPE code (with CUDA Acceleration) is now released. Paper is here.
  • :boom: July, 2021: iRPE has been accepted by ICCV'21.
  • :boom: July, 2021: AutoFormer has been accepted by ICCV'21.
  • :boom: July, 2021: AutoFormer is now available on arXiv.
  • :boom: Oct, 2020: Code for Cream is now released.
  • :boom: Oct, 2020: Cream was accepted to NeurIPS'20



TinyViT is a new family of tiny and efficient vision transformers pretrained on large-scale datasets with our proposed fast distillation framework. The central idea is to transfer knowledge from large pretrained models to small ones. The logits of large teacher models are sparsified and stored in disk in advance to save the memory cost and computation overheads.

TinyViT overview


MiniViT is a new compression framework that achieves parameter reduction in vision transformers while retaining the same performance. The central idea of MiniViT is to multiplex the weights of consecutive transformer blocks. Specifically, we make the weights shared across layers, while imposing a transformation on the weights to increase diversity. Weight distillation over self-attention is also applied to transfer knowledge from large-scale ViT models to weight-multiplexed compact models.

MiniViT overview


In this work, we propose new joint optimization objectives and a novel Cyclic Differentiable ARchiTecture Search framework, dubbed CDARTS. Considering the structure difference, CDARTS builds a cyclic feedback mechanism between the search and evaluation networks with introspective distillation.

CDARTS overview


In this work, instead of searching the architecture in a predefined search space, with the help of AutoFormer, we proposed to search the search space to automatically find a great search space first. After that we search the architectures in the searched space. In addition, we provide insightful observations and guidelines for general vision transformer design.

AutoFormerV2 overview


AutoFormer is new one-shot architecture search framework dedicated to vision transformer search. It entangles the weights of different vision transformer blocks in the same layers during supernet training. Benefiting from the strategy, the trained supernet allows thousands of subnets to be very well-trained. Specifically, the performance of these subnets with weights inherited from the supernet is comparable to those retrained from scratch.

AutoFormer overview


Image RPE (iRPE for short) methods are new relative position encoding methods dedicated to 2D images, considering directional relative distance modeling as well as the interactions between queries and relative position embeddings in self-attention mechanism. The proposed iRPE methods are simple and lightweight, being easily plugged into transformer blocks. Experiments demonstrate that solely due to the proposed encoding methods, DeiT and DETR obtain up to 1.5% (top-1 Acc) and 1.3% (mAP) stable improvements over their original versions on ImageNet and COCO respectively, without tuning any extra hyperparamters such as learning rate and weight decay. Our ablation and analysis also yield interesting findings, some of which run counter to previous understanding.

iRPE overview


[Paper] [Models-Google Drive][Models-Baidu Disk (password: wqw6)] [Slides] [BibTex]

In this work, we present a simple yet effective architecture distillation method. The central idea is that subnetworks can learn collaboratively and teach each other throughout the training process, aiming to boost the convergence of individual models. We introduce the concept of prioritized path, which refers to the architecture candidates exhibiting superior performance during training. Distilling knowledge from the prioritized paths is able to boost the training of subnetworks. Since the prioritized paths are changed on the fly depending on their performance and complexity, the final obtained paths are the cream of the crop.


```bibtex @InProceedings{tiny_vit, title={TinyViT: Fast Pretraining Distillation for Small Vision Transformers}, author={Wu, Kan and Zhang, Jinnian and Peng, Houwen and Liu, Mengchen and Xiao, Bin and Fu, Jianlong and Yuan, Lu}, booktitle={European conference on computer vision (ECCV)}, year={2022} }

@InProceedings{MiniViT, title = {MiniViT: Compressing Vision Transformers With Weight Multiplexing}, author = {Zhang, Jinnian and Peng, Houwen and Wu, Kan and Liu, Mengchen and Xiao, Bin and Fu, Jianlong and Yuan, Lu}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2022}, pages = {12145-12154} }

@article{CDARTS, title={Cyclic Differentiable Architecture Search}, author={Yu, Hongyuan and Peng, Houwen and Huang, Yan and Fu, Jianlong and Du, Hao and Wang, Liang and Ling, Haibin}, journal={IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)}, year={2022} }

@article{S3, title={Searching the Search Space of Vision Transformer}, author={Minghao, Chen and Kan, Wu and Bolin, Ni and Houwen, Peng and Bei, Liu and Jianlong, Fu and Hongyang, Chao and Haibin, Ling}, booktitle={Conference and Workshop on Neural Information Processing Systems (NeurIPS)}, year={2021} }

@InProceedings{iRPE, title = {Rethinking and Improving Relative Position Encoding for Vision Transformer}, author = {Wu, Kan and Peng, Houwen and Chen, Minghao and Fu, Jianlong and Chao, Hongyang}, booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}, month = {October}, year = {2021}, pages = {10033-10041} }

@InProceedings{AutoFormer, title = {AutoFormer: Searching Transformers for Visual Recognition}, author = {Chen, Minghao and Peng, Houwen and Fu, Jianlong and Ling, Haibin}, booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}, month = {October}, year = {2021}, pages = {12270-12280} }

@article{Cream, title={Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search}, author={Peng, Houwen and Du, Hao and Yu, Hongyuan and Li, Qi and Liao, Jing and Fu, Jianlong}, journal={Advances in Neural Information Processing Systems}, volume={33}, year={2020} } ```


License under an MIT license.


UserWarning: [WARNING] The module `rpe_ops` is not built. For better training performance, please build `rpe_ops`.

opened on 2023-03-25 13:46:23 by nanle771

Why did I execute cd rpe_ ops/ python install --user And displays successful installation; But there were still errors during training UserWarning: [WARNING] The module rpe_ ops is not built. For better training performance, please build rpe_ ops.

The training detail of AutoFormer

opened on 2023-02-27 07:47:33 by mxjecho

Hi, I just tested the acc of the largest model from supernet-T, and I found that the largest model acc is 67.188. it is unreasonable since the acc of the largest model is usually the highest. I trained a supernet according to the script you provided, and the largest model accuracy is 76.3.

I'm confused as to why the supernet you provided has this phenomenon. How to determine if the training has converged?


Does #param gamma matter?

opened on 2023-02-21 07:33:30 by WisonZ None

Search space of AutoFormer

opened on 2023-01-31 17:36:26 by HamdiAbed

What is the correct way to change embedding sizes in Tiny or Small supernet? what is the criteria of selecting the embedding sizes. I got this error almost every time I try to set EMBED_DIM in the supernet-T.yaml

qkv = self.qkv(x).reshape([B, N, 3, self.sample_num_heads, -1]).permute(2, 0, 3, 1, 4) RuntimeError: shape '[16, 32, 3, 3, -1]' is invalid for input of size 61440

search space of AutoFormer

opened on 2023-01-12 02:26:45 by mxjecho

Hi, I was wondering if 'patch_size/token_num' can be added in the search space of AutoFormer?

Rethinking and Improving Relative Position Encoding for Vision Transformer with memory optimized attentions

opened on 2022-12-29 10:23:37 by jakubMitura14

Hello I was wondering whether your relative positional encoding schemes would work with approximate attention mechanisms for example like presented in flash attention


Static Files 2021-08-04 03:42:12

move checkpoints and logs into this release version


Open source projects and samples from Microsoft

GitHub Repository

nas automl vision-transformer rpe vit-compression efficiency knowledge-distillation