Human-Object Interaction (HOI) detection aims to localize humansand objects in images and classify their interactions. Zero-shot HOIfocuses on recognizing interactions that are not observed duringtraining, requiring models to generalize beyond seen verb-objectcompositions. Recent approaches leverage Vision-Language Models(VLMs), benefiting from rich semantic representations. However,they often rely on global or detector-centric features that compress in-teraction cues and hinder fine-grained spatial reasoning. To overcomethis limitation, we propose FineHOI, a zero-shot HOI frameworkthat explicitly models interactions from dense patch-level features.Our approach is motivated by the observation that human-objectinteractions are defined by localized spatial relationships, which arenot preserved by global and detector-centric representations. To thisend, we introduce an Adaptive Part-Level Attention module thatdecomposes humans and objects into semantically coherent parts viaunsupervised clustering, and re-weights them based on their interac-tion relevance. These representations are then integrated through aRegion-Aware Interaction Transformer that integrates part-aware andglobal features and produces the final HOI embedding. Extensiveexperiments demonstrate that FineHOI consistently outperformsexisting zero-shot HOI methods, achieving particularly strong gainson unseen interactions. Code is available here.
FineHOI: Part-Aware Dense Representations for Zero-Shot Human-Object Interaction Detection
Francesco Tonini
;Cigdem BeyanSupervision
2026-01-01
Abstract
Human-Object Interaction (HOI) detection aims to localize humansand objects in images and classify their interactions. Zero-shot HOIfocuses on recognizing interactions that are not observed duringtraining, requiring models to generalize beyond seen verb-objectcompositions. Recent approaches leverage Vision-Language Models(VLMs), benefiting from rich semantic representations. However,they often rely on global or detector-centric features that compress in-teraction cues and hinder fine-grained spatial reasoning. To overcomethis limitation, we propose FineHOI, a zero-shot HOI frameworkthat explicitly models interactions from dense patch-level features.Our approach is motivated by the observation that human-objectinteractions are defined by localized spatial relationships, which arenot preserved by global and detector-centric representations. To thisend, we introduce an Adaptive Part-Level Attention module thatdecomposes humans and objects into semantically coherent parts viaunsupervised clustering, and re-weights them based on their interac-tion relevance. These representations are then integrated through aRegion-Aware Interaction Transformer that integrates part-aware andglobal features and produces the final HOI embedding. Extensiveexperiments demonstrate that FineHOI consistently outperformsexisting zero-shot HOI methods, achieving particularly strong gainson unseen interactions. Code is available here.| File | Dimensione | Formato | |
|---|---|---|---|
|
3767308.3836016.pdf
accesso aperto
Tipologia:
Documento in Post-print
Licenza:
Creative commons
Dimensione
11.51 MB
Formato
Adobe PDF
|
11.51 MB | Adobe PDF | Visualizza/Apri |
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.



