Abstract:In this work, we present SeqFormer for video instance segmentation. SeqFormer follows the principle of vision transformer that models instance relationships among video frames. Nevertheless, we observe that a stand-alone instance query suffices for capturing a time sequence of instances in a video, but attention mechanisms shall be done with each frame independently. To achieve this, SeqFormer locates an instance in each frame and aggregates temporal information to learn a powerful representation of a video-level instance, which is used to predict the mask sequences on each frame dynamically. Instance tracking is achieved naturally without tracking branches or post-processing. On YouTube-VIS, SeqFormer achieves 47.4 AP with a ResNet-50 backbone and 49.0 AP with a ResNet-101 backbone without bells and whistles. Such achievement significantly exceeds the previous state-of-the-art performance by 4.6 and 4.4, respectively. In addition, integrated with the recently-proposed Swin transformer, SeqFormer achieves a much higher AP of 59.3. We hope SeqFormer could be a strong baseline that fosters future research in video instance segmentation, and in the meantime, advances this field with a more robust, accurate, neat model. The code is available at this https URL.
| Comments: | ECCV 2022, Oral |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2112.08275 [cs.CV] |
| (or arXiv:2112.08275v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2112.08275 arXiv-issued DOI via DataCite |
Submission history
From: Junfeng Wu [view email]
[v1]
Wed, 15 Dec 2021 17:09:18 UTC (4,682 KB)
[v2]
Thu, 21 Jul 2022 17:28:33 UTC (10,624 KB)