Comparison with current models on panoptic grounded captioning. We compare our proposed PANORAMA model with prior grounding models on an in-domain scene and on challenging images from domains not represented in PanoCaps. Prior methods identify few entities and may miss key elements of the scene, segment regions coarsely, and fall back on wrong or generic grounded phrases. PANORAMA produces denser and more complete descriptions, grounding each mentioned entity with an instance-level mask.

Abstract

Intelligent systems that act in the world require image understanding that is both comprehensive and spatially grounded. Current vision-language models (VLMs) can generate fluent and detailed image captions, but reliably associating them with image pixels remains challenging. Existing methods that combine dense captioning with pixel-level grounding often produce either incomplete descriptions or inaccurate segmentation masks. We study this problem through panoptic grounded captioning, a task that requires a VLM to describe both foreground objects and background regions while grounding each referring phrase with pixel-level masks. We make three contributions. First, we introduce PanoCaps, a human-annotated benchmark constructed from panoptic segmentation datasets. It provides dense captions with near-complete pixel coverage and image-text alignments at the entity level, supporting both training and evaluation. We further propose a phrase-mask matching protocol and a generalized Panoptic Quality (gPQ) metric that jointly evaluates textual and mask agreement. Second, we formulate phrase grounding as selection from a phrase-conditioned pool of mask proposals and introduce PANORAMA, a VLM that conditions a pretrained segmenter on contextualized phrase representations to obtain candidate masks and learns to select those corresponding to each phrase. Training this interface jointly with caption generation enables PANORAMA to produce high-quality masks while allowing each phrase to refer to a single region or multiple instances. Third, PANORAMA achieves the best overall grounding on PanoCaps and matches or exceeds specialized models across several pixel-level grounding tasks. Experiments show that our method produces precise entity-level segmentations while maintaining detailed, mask-consistent captions.

PanoCaps Dataset

PanoCaps is our human-annotated dataset for panoptic grounded captioning: detailed scene-level captions paired with pixel-level panoptic masks for every mentioned entity. It contains about 3.5K images, 34K panoptic masks, and 17.9K unique noun phrases, with referenced regions covering ≈99% of image pixels, so both foreground objects and background “stuff” are described. Captions and phrase–mask alignments are written and verified by human annotators, giving tightly aligned image–text–mask supervision with open-vocabulary mentions. We additionally release two training resources built for PANORAMA: COCONut-PanCap-Recaptioned, regenerated grounded captions for the 118K COCONut-PanCap training images, and MRSeg-Referring-Expressions, 101K referring expressions adapted from the multi-granularity MR-Seg data of SegLLM.

Annotated examples by source. One image each from COCONut, ADE20K, and VIPSeg, with its panoptic masks and human-written caption; every grounded phrase is shown in the color of its mask.

Method

PANORAMA generates panoptic grounded captions on an image by generating a scene description where every mentioned entity is linked to its corresponding pixel-level segmentation masks. The vision-language model writes the caption and emits a [SEG] token for each referring phrase. A concept bridge projects the hidden state of that token into a concept vector, which prompts a proposal model to produce a pool of candidate masks specific to the phrase. A match scorer then compares each candidate with the concept vector and selects the relevant ones. This “condition–propose–select” design lets a phrase ground to a single region, to several instances, or to none at all.

Qualitative Results

Results on grounding tasks. The top row shows PANORAMA on PanoCaps. The bottom row presents results on referring expression segmentation (RES), generalized referring expression segmentation (GRES), GroundingSuite (GSEval), and grounded conversation generation (GCG).

BibTeX

@article{pieri2026panorama,
  title   = {PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection},
  author  = {Pieri, Sara and Kazakos, Evangelos and Chen, Shizhe and Sivic, Josef and Schmid, Cordelia},
  journal = {arXiv preprint arXiv:2609.19143},
  year    = {2026}
}

Acknowledgements

This work was performed using HPC resources from GENCI-IDRIS (Grant 2025-AD011015795 and AD011015795R1). It was funded in part by the French government under management of Agence Nationale de la Recherche as part of the “France 2030" program, reference ANR-23-IACL-0008 (PR[AI]RIE-PSAI project), the ANR project VideoPredict ANR-21-FAI1-0002- 01. Cordelia Schmid would like to acknowledge the support by the Körber European Science Prize.

Copyright

The documents contained in these directories are included by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a non-commercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright.