All Publications
-
Efficient Training of Large Vision Models via Advanced Automated Progressive Learning
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
2026; 48 (8): 8798-8812
Abstract
The rapid advancements in Large Vision Models (LVMs), such as Vision Transformers (ViTs), diffusion models, and visual autoregressive models, have led to an increasing demand for computational resources, resulting in substantial financial and environmental costs. This growing challenge highlights the necessity of developing efficient training methods for LVMs. Progressive learning, a training strategy in which model capacity gradually increases during training, has shown promise in addressing these challenges. In this paper, we take a practical step toward the efficient training of LVMs by automating progressive learning. We focus first on the pre-training of LVMs, using ViTs as a case study. We propose AutoProg-One, an automated progressive learning scheme featuring momentum growth (MoGrow) and the one-shot growth schedule search. Additionally, we extend our approach beyond pre-training to address the transfer learning and fine-tuning of LVMs. We also expand the scope of AutoProg to encompass a wider range of LVMs, including diffusion models and visual autoregressive model. First, we introduce AutoProg-Zero, by enhancing the AutoProg framework with a novel zero-shot automated progressive learning method, eliminating the need for one-shot supernet training. Second, we introduce a novel Unique Stage Identifier (SID) scheme to bridge the gap during network growth. These innovations, integrated with the core principles of AutoProg, offer a comprehensive solution for efficient training across various LVM scenarios. Extensive experiments show that AutoProg accelerates ViT pre-training by up to 1.85 ×on ImageNet and accelerates the fine-tuning of diffusion models, and visual autoregressive model by up to 2.86 × and 1.89 ×, with comparable or even better performance. This work provides a robust and scalable approach to efficient training of LVMs, with potential applications in a wide range of vision tasks.
View details for DOI 10.1109/TPAMI.2026.3673336
View details for Web of Science ID 001815311000042
View details for PubMedID 41817969
-
BossNAS Family: Block-Wisely Self-Supervised Neural Architecture Search
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
2025; 47 (5): 3500-3514
Abstract
Recent advances in hand-crafted neural architectures for visual recognition underscore the pressing need to explore architecture designs comprising diverse building blocks. Concurrently, neural architecture search (NAS) methods have gained traction as a means to alleviate human efforts. Nevertheless, the question of whether NAS methods can efficiently and effectively manage diversified search spaces featuring disparate candidates, such as Convolutional Neural Networks (CNNs) and transformers, remains an open question. In this work, we introduce a novel unsupervised NAS approach called BossNAS (Block-wisely Self-supervised Neural Architecture Search), which aims to address the problem of inaccurate predictive architecture ranking caused by a large weight-sharing space while mitigating potential ranking issue caused by biased supervision. To achieve this, we factorize the search space into blocks and introduce a novel self-supervised training scheme called Ensemble Bootstrapping, to train each block separately in an unsupervised manner. In the search phase, we propose an unsupervised Population-Centric Search, optimizing the candidate architecture towards the population center. Additionally, we enhance our NAS method by integrating masked image modeling and present BossNAS++ to overcome the lack of dense supervision in our block-wise self-supervised NAS. In BossNAS++, we introduce the training technique named Masked Ensemble Bootstrapping for block-wise supernet, accompanied by a Masked Population-Centric Search scheme to promote fairer architecture selection. Our family of models, discovered through BossNAS and BossNAS++, delivers impressive results across various search spaces and datasets. Our transformer model discovered by BossNAS++ attains a remarkable accuracy of 83.2% on ImageNet with only 10.5B MAdds, surpassing DeiT-B by 1.4% while maintaining a lower computation cost. Moreover, our approach excels in architecture rating accuracy, achieving Spearman correlations of 0.78 and 0.76 on the canonical MBConv search space with ImageNet and the NATS-Bench size search space with CIFAR-100, respectively, outperforming state-of-the-art NAS methods.
View details for DOI 10.1109/TPAMI.2025.3529517
View details for Web of Science ID 001465416300004
View details for PubMedID 40031006
-
DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Vision Transformers.
IEEE transactions on pattern analysis and machine intelligence
2023; 45 (4): 4430-4446
Abstract
Dynamic networks have shown their promising capability in reducing theoretical computation complexity by adapting their architectures to the input during inference. However, their practical runtime usually lags behind the theoretical acceleration due to inefficient sparsity. In this paper, we explore a hardware-efficient dynamic inference regime, named dynamic weight slicing, that can generalized well on multiple dimensions in both CNNs and transformers (e.g. kernel size, embedding dimension, number of heads, etc.). Instead of adaptively selecting important weight elements in a sparse way, we pre-define dense weight slices with different importance level by nested residual learning. During inference, weights are progressively sliced beginning with the most important elements to less important ones to achieve different model capacity for inputs with diverse difficulty levels. Based on this conception, we present DS-CNN++ and DS-ViT++, by carefully designing the double headed dynamic gate and the overall network architecture. We further propose dynamic idle slicing to address the drastic reduction of embedding dimension in DS-ViT++. To ensure sub-network generality and routing fairness, we propose a disentangled two-stage optimization scheme. In Stage I, in-place bootstrapping (IB) and multi-view consistency (MvCo) are proposed to stablize and improve the training of DS-CNN++ and DS-ViT++ supernet, respectively. In Stage II, sandwich gate sparsification (SGS) is proposed to assist the gate training. Extensive experiments on 4 datasets and 3 different network architectures demonstrate our methods consistently outperform the state-of-the-art static and dynamic model compression methods by a large margin (up to 6.6%). Typically, we achieves 2-4× computation reduction and up to 61.5% real-world acceleration on MobileNet, ResNet-50 and Vision Transformer, with minimal accuracy drops on ImageNet. Code release: https://github.com/changlin31/DS-Net.
View details for DOI 10.1109/TPAMI.2022.3194044
View details for PubMedID 35895643
-
SELongVLM: Empowering Long Video Language Models With Self-Corrective Clip Selection
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
2026; 48 (7): 8694-8709
Abstract
Recent advances in multimodal large language models (MLLMs) have enabled impressive progress in visual-language reasoning, yet long-video understanding remains a formidable challenge due to the need for coherent reasoning over ultra-long spatiotemporal dependencies. Existing methods struggle with the vast candidate space for relevant information in long videos, often failing to distinguish meaningful events from redundant content. We identify two critical and previously under-explored issues: absolute redundancy, where static visual content inflates token counts without adding narrative value, and relative redundancy, where task-irrelevant segments introduce noise that impairs reasoning. Compounding these issues is the weak spatiotemporal modeling in current MLLMs, which limits their ability to capture complex event dynamics. To address these multifaceted challenges, we introduce SELongVLM, a dynamically lenient-to-stringent selection long video language model. SELongVLM integrates two coordinated branches: a Residual Token Pruner (RTP) that removes repetitive background tokens via inter-frame residual modeling thus mitigating absolute redundancy while preserving motion cues, and a Semantic-aware Self-Correction Selector (SCSelector) that progressively refines query-relevant clip selection without frame-level annotations to reduce relative redundancy, guided by a stringent-to-lenient self-correcting mechanism during optimization. To ensure causal continuity and bolster spatiotemporal reasoning across disjoint clips, the framework further incorporates an action-aware operation for intra-clip dynamics and a temporal memory for cross-clip context, enabling robust spatiotemporal inference on long videos. Extensive experiments across eight benchmarks demonstrate that SELongVLM markedly outperforms existing models on both general and specialized long-video tasks. Specifically, it achieves 65.5% on VideoMME and 69.8% on MLVU for general benchmarks, and delivers strong performance on four specialized benchmarks - for example, 39.2% on TOMATO for fine-grained temporal reasoning and 69.2% on EventBench for event-level understanding.
View details for DOI 10.1109/TPAMI.2026.3673141
View details for Web of Science ID 001786034400048
View details for PubMedID 41817966
-
Mitigating Data Redundancy to Revitalize Transformer-Based Long-Term Time Series Forecasting System
ACM TRANSACTIONS ON INTELLIGENT SYSTEMS AND TECHNOLOGY
2026; 17 (3)
View details for DOI 10.1145/3735651
View details for Web of Science ID 001825950800001
-
Let LLM Tell What to Prune and How Much to Prune
edited by Singh, A., Fazel, M., Hsu, D., Lacoste-Julien, S., Berkenkamp, F., Maharaj, T., Wagstaff, K., Zhu, J.
JMLR-JOURNAL MACHINE LEARNING RESEARCH. 2025: 70833-70849
View details for Web of Science ID 001693170600132
-
DNA Family: Boosting Weight-Sharing NAS With Block-Wise Supervisions.
IEEE transactions on pattern analysis and machine intelligence
2024; 46 (5): 2722-2740
Abstract
Neural Architecture Search (NAS), aiming at automatically designing neural architectures by machines, has been considered a key step toward automatic machine learning. One notable NAS branch is the weight-sharing NAS, which significantly improves search efficiency and allows NAS algorithms to run on ordinary computers. Despite receiving high expectations, this category of methods suffers from low search effectiveness. By employing a generalization boundedness tool, we demonstrate that the devil behind this drawback is the untrustworthy architecture rating with the oversized search space of the possible architectures. Addressing this problem, we modularize a large search space into blocks with small search spaces and develop a family of models with the distilling neural architecture (DNA) techniques. These proposed models, namely a DNA family, are capable of resolving multiple dilemmas of the weight-sharing NAS, such as scalability, efficiency, and multi-modal compatibility. Our proposed DNA models can rate all architecture candidates, as opposed to previous works that can only access a sub- search space using heuristic algorithms. Moreover, under a certain computational complexity constraint, our method can seek architectures with different depths and widths. Extensive experimental evaluations show that our models achieve state-of-the-art top-1 accuracy of 78.9% and 83.6% on ImageNet for a mobile convolutional network and a small vision transformer, respectively. Additionally, we provide in-depth empirical analysis and insights into neural architecture ratings.
View details for DOI 10.1109/TPAMI.2023.3335261
View details for PubMedID 37988208
-
No Token Left Behind: Efficient Vision Transformer via Dynamic Token Idling
edited by Liu, T., Yue, L., Webb, G., Wang, D.
SPRINGER-VERLAG SINGAPORE PTE LTD. 2024: 28-41
View details for DOI 10.1007/978-981-99-8388-9_3
View details for Web of Science ID 001148047100003
-
Automated Progressive Learning for Efficient Training of Vision Transformers
IEEE COMPUTER SOC. 2022: 12476-12486
View details for DOI 10.1109/CVPR52688.2022.01216
View details for Web of Science ID 000870759105055
-
Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shift
IEEE. 2021: 12334-12344
View details for DOI 10.1109/ICCV48922.2021.01213
View details for Web of Science ID 000798743202051
-
BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search
IEEE. 2021: 12261-12271
View details for DOI 10.1109/ICCV48922.2021.01206
View details for Web of Science ID 000798743202044
-
Dynamic Slimmable Network
IEEE COMPUTER SOC. 2021: 8603-8613
View details for DOI 10.1109/CVPR46437.2021.00850
View details for Web of Science ID 000739917308082
-
Block-wisely Supervised Neural Architecture Search with Knowledge Distillation
IEEE COMPUTER SOC. 2020: 1986-1995
View details for DOI 10.1109/CVPR42600.2020.00206
View details for Web of Science ID 000620679502025
-
Knowledge driven temporal activity localization
JOURNAL OF VISUAL COMMUNICATION AND IMAGE REPRESENTATION
2019; 64
View details for DOI 10.1016/j.jvcir.2019.102628
View details for Web of Science ID 000492798600035