Human-Object Interaction (HOI) detection aims to localize humansand objects in images and classify their interactions. Zero-shot HOIfocuses on recognizing interactions that are not observed duringtraining, requiring models to generalize beyond seen verb-objectcompositions. Recent approaches leverage Vision-Language Models(VLMs), benefiting from rich semantic representations. However,they often rely on global or detector-centric features that compress in-teraction cues and hinder fine-grained spatial reasoning. To overcomethis limitation, we propose FineHOI, a zero-shot HOI frameworkthat explicitly models interactions from dense patch-level features.Our approach is motivated by the observation that human-objectinteractions are defined by localized spatial relationships, which arenot preserved by global and detector-centric representations. To thisend, we introduce an Adaptive Part-Level Attention module thatdecomposes humans and objects into semantically coherent parts viaunsupervised clustering, and re-weights them based on their interac-tion relevance. These representations are then integrated through aRegion-Aware Interaction Transformer that integrates part-aware andglobal features and produces the final HOI embedding. Extensiveexperiments demonstrate that FineHOI consistently outperformsexisting zero-shot HOI methods, achieving particularly strong gainson unseen interactions. Code is available here.

FineHOI: Part-Aware Dense Representations for Zero-Shot Human-Object Interaction Detection

Francesco Tonini
;
Cigdem Beyan
Supervision
2026-01-01

Abstract

Human-Object Interaction (HOI) detection aims to localize humansand objects in images and classify their interactions. Zero-shot HOIfocuses on recognizing interactions that are not observed duringtraining, requiring models to generalize beyond seen verb-objectcompositions. Recent approaches leverage Vision-Language Models(VLMs), benefiting from rich semantic representations. However,they often rely on global or detector-centric features that compress in-teraction cues and hinder fine-grained spatial reasoning. To overcomethis limitation, we propose FineHOI, a zero-shot HOI frameworkthat explicitly models interactions from dense patch-level features.Our approach is motivated by the observation that human-objectinteractions are defined by localized spatial relationships, which arenot preserved by global and detector-centric representations. To thisend, we introduce an Adaptive Part-Level Attention module thatdecomposes humans and objects into semantically coherent parts viaunsupervised clustering, and re-weights them based on their interac-tion relevance. These representations are then integrated through aRegion-Aware Interaction Transformer that integrates part-aware andglobal features and produces the final HOI embedding. Extensiveexperiments demonstrate that FineHOI consistently outperformsexisting zero-shot HOI methods, achieving particularly strong gainson unseen interactions. Code is available here.
2026
Human-object interaction, vision-language models
File in questo prodotto:
File Dimensione Formato  
3767308.3836016.pdf

accesso aperto

Tipologia: Documento in Post-print
Licenza: Creative commons
Dimensione 11.51 MB
Formato Adobe PDF
11.51 MB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11562/1209309
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? ND
social impact