Publications
✉ Corresponding Author · † Project Lead · * Equal Contribution
2026
- Mitigating Dimensional Collapse in Parameter-Efficient Multi-Task LearningIn Proceedings of the 34th ACM International Conference on Multimedia, Nov 2026ACM MM Bib
@inproceedings{sun2026mitigating, title = {Mitigating Dimensional Collapse in Parameter-Efficient Multi-Task Learning}, author = {Sun, Hao and Zhang, Xu and Lu, Ming and Ma, Zhan}, booktitle = {Proceedings of the 34th ACM International Conference on Multimedia}, year = {2026}, month = nov, } - arXiv preprint arXiv:2601.03955, Jan 2026
Existing 1D visual tokenizers for autoregressive (AR) generation largely follow the design principles of language modeling, as they are built directly upon transformers whose priors originate in language, yielding single-hierarchy latent tokens and treating visual data as flat sequential token streams. However, this language-like formulation overlooks key properties of vision, particularly the hierarchical and residual network designs that have long been essential for convergence and efficiency in visual models. To bring "vision" back to vision, we propose the Residual Tokenizer (ResTok), a 1D visual tokenizer that builds hierarchical residuals for both image tokens and latent tokens. The hierarchical representations obtained through progressively merging enable cross-level feature fusion at each layer, substantially enhancing representational capacity. Meanwhile, the semantic residuals between hierarchies prevent information overlap, yielding more concentrated latent distributions that are easier for AR modeling. Cross-level bindings consequently emerge without any explicit constraints. To accelerate the generation process, we further introduce a hierarchical AR generator that substantially reduces sampling steps by predicting an entire level of latent tokens at once rather than generating them strictly token-by-token. Extensive experiments demonstrate that restoring hierarchical residual priors in visual tokenization significantly improves AR image generation, achieving a gFID of 2.34 on ImageNet-256 with only 9 sampling steps.
@article{zhang2026restok, title = {{R}es{T}ok: Learning Hierarchical Residuals in 1D Visual Tokenizers for Autoregressive Image Generation}, author = {Zhang, Xu and Da, Cheng and Yang, Huan and Gai, Kun and Lu, Ming and Ma, Zhan}, journal = {arXiv preprint arXiv:2601.03955}, year = {2026}, month = jan, }
2025
- In Proceedings of the 42nd International Conference on Machine Learning, Jul 2025
Existing multimodal large model-based image compression frameworks often rely on a fragmented integration of semantic retrieval, latent compression, and generative models, resulting in suboptimal performance in both reconstruction fidelity and coding efficiency. To address these challenges, we propose a residual-guided ultra lowrate image compression named ResULIC, which incorporates residual signals into both semantic retrieval and the diffusion-based generation process. Specifically, we introduce Semantic Residual Coding (SRC) to capture the semantic disparity between the original image and its compressed latent representation. A perceptual fidelity optimizer is further applied for superior reconstruction quality. Additionally, we present the Compression-aware Diffusion Model (CDM), which establishes an optimal alignment between bitrates and diffusion time steps, improving compression-reconstruction synergy. Extensive experiments demonstrate the effectiveness of ResULIC, achieving superior objective and subjective performance compared to state-of-the-art diffusion-based methods with -80.7%, -66.3% BD-rate saving in terms of LPIPS and FID.
@inproceedings{ke2025resulic, title = {Ultra Lowrate Image Compression with Semantic Residual Coding and Compression-aware Diffusion}, author = {Ke, Anle and Zhang, Xu and Chen, Tong and Lu, Ming and Zhou, Chao and Gu, Jiawen and Ma, Zhan}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {29626--29650}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = jul, publisher = {PMLR}, url = {https://proceedings.mlr.press/v267/ke25c.html}, } - In 2025 IEEE International Conference on Multimedia and Expo (ICME), Jun 2025
Selected for oral presentation at IEEE ICME 2025.
In recent years, compressed domain semantic inference has primarily relied on learned image coding models optimized for mean squared error (MSE). However, MSE-oriented optimization tends to yield latent spaces with limited semantic richness, which hinders effective semantic inference in downstream tasks. Moreover, achieving high performance with these models often requires fine-tuning the entire vision model, which is computationally intensive, especially for large models. To address these problems, we introduce Perception-Oriented Latent Coding (POLC), an approach that enriches the semantic content of latent features for high-performance compressed domain semantic inference. With the semantically rich latent space, POLC requires only a plug-and-play adapter for fine-tuning, significantly reducing the parameter count compared to previous MSE-oriented methods. Experimental results demonstrate that POLC achieves rate-perception performance comparable to state-of-the-art generative image coding methods while markedly enhancing performance in vision tasks, with minimal fine-tuning overhead.
@inproceedings{zhang2025polc, title = {Perception-Oriented Latent Coding for High-Performance Compressed Domain Semantic Inference}, author = {Zhang, Xu and Lu, Ming and Chen, Yan and Ma, Zhan}, booktitle = {2025 IEEE International Conference on Multimedia and Expo (ICME)}, year = {2025}, month = jun, pages = {1-6}, keywords = {Deep learning;Bridges;Adaptation models;Image coding;Codes;Computational modeling;Semantics;Encoding;Optimization;learned image coding;compressed domain semantic inference;perception-oriented optimization;compressed representation;deep learning}, doi = {10.1109/ICME59968.2025.11209906}, }
2024
- In Advances in Neural Information Processing Systems, Dec 2024
Image coding for multi-task applications, catering to both human perception and machine vision, has been extensively investigated. Existing methods often rely on multiple task-specific encoder-decoder pairs, leading to high overhead of parameter and bitrate usage, or face challenges in multi-objective optimization under a unified representation, failing to achieve both performance and efficiency. To this end, we propose Multi-Path Aggregation (MPA) integrated into existing coding models for joint human-machine vision, unifying the feature representation with an all-in-one architecture. MPA employs a predictor to allocate latent features among task-specific paths based on feature importance varied across tasks, maximizing the utility of shared features while preserving task-specific features for subsequent refinement. Leveraging feature correlations, we develop a two-stage optimization strategy to alleviate multi-task performance degradation. Upon the reuse of shared features, as low as 1.89% parameters are further augmented and fine-tuned for a specific task, which completely avoids extensive optimization of the entire model. Experimental results show that MPA achieves performance comparable to state-of-the-art methods in both task-specific and multi-objective optimization across human viewing and machine analysis tasks. Moreover, our all-in-one design supports seamless transitions between human- and machine-oriented reconstruction, enabling task-controllable interpretation without altering the unified model.
@inproceedings{zhang2024mpa, title = {All-in-One Image Coding for Joint Human-Machine Vision with Multi-Path Aggregation}, author = {Zhang, Xu and Guo, Peiyao and Lu, Ming and Ma, Zhan}, booktitle = {Advances in Neural Information Processing Systems}, month = dec, editor = {Globerson, A. and Mackey, L. and Belgrave, D. and Fan, A. and Paquet, U. and Tomczak, J. and Zhang, C.}, pages = {71465--71503}, publisher = {Curran Associates, Inc.}, url = {https://proceedings.neurips.cc/paper_files/paper/2024/file/8395fdf356059eaa92afd39e3952a677-Paper-Conference.pdf}, volume = {37}, year = {2024}, }