LYH卢英豪Yinghao Lu

Review · Human-Object Interaction

HOI 人物交互检测综述：从组合关系到开放词汇A Review of Human-Object Interaction Detection

HOI 检测的核心不是“看见人和物”，而是要理解人、物、动作三者构成的组合关系。HOI detection is not only about seeing humans and objects; it is about recognizing their compositional interactions.

Topic note·HOI Detection Review·
参考综述Survey paper → ← 返回笔记Back to notes

1. 任务定义

Human-Object Interaction Detection 通常要输出一个三元组：人框、物体框和交互类别，例如 person ride bicycle、person hold cup。它比普通目标检测更难，因为模型不仅要定位实体，还要判断两者之间是否存在动作关系。

实体定位找出图像中的人和物体，通常依赖目标检测器或端到端 Transformer 检测框架。
交互分类判断人—物对之间的动作语义，需要人体姿态、空间布局、物体类别和上下文共同参与。
组合泛化同一个动作可以对应多个物体，同一个物体也能参与多个动作，类别天然长尾且组合空间巨大。
开放词汇真实场景不会只出现训练集中定义好的动作，模型需要借助语言知识泛化到未见交互。

2. 经典研究路线

早期 HOI 方法多采用两阶段流程：先检测人和物，再枚举人—物 pair 并分类交互。这条路线直观但 pair 数量多，匹配和推理成本高。随后 DETR 风格方法开始把 HOI 看成集合预测任务，用 query 直接学习人、物和交互关系，简化了人工后处理。

综述里可以把方法大致分成三条线：第一类强调空间与姿态几何关系；第二类强调上下文和图结构建模；第三类引入 CLIP、LLM 等视觉语言知识，走向零样本和开放词汇。

3. 为什么开放词汇 HOI 很重要

HOI 类别有明显长尾：常见动作容易学，罕见动作样本少，而未见动作组合在真实场景中又很常见。开放词汇方法的关键，是把交互类别从固定分类器变成可由语言描述扩展的语义空间。

我的理解：开放词汇 HOI 的难点不只是“把类别名喂给 CLIP”，而是要让视觉区域、人—物空间关系和语言描述在同一个语义粒度上对齐。很多方法的改进点，本质都在解决“视觉证据太局部、语言标签太粗糙、动作语义太相近”这三个问题。

4. 读论文时可以抓的主线

  1. 检测结构：它是两阶段 pair 分类，还是端到端 query 预测？人框、物框、交互分数如何产生？
  2. 语义来源：只用类别 one-hot，还是引入 CLIP 文本编码、LLM 描述、多语言描述或 prompt learning？
  3. 泛化设置：是 rare/non-rare、unseen object、unseen verb，还是更严格的 open-vocabulary？
  4. 对齐粒度：对齐的是整图、局部人—物 union 区域、对象区域，还是动作描述中的细粒度属性？

5. 和后面几篇论文的关系

GEN-VLKT 代表把 CLIP 视觉语言知识引入端到端 HOI 的早期强基线；CMMP 进一步强调条件化多模态 prompt；EZ-HOI 关注如何高效适配 VLM；SGC-Net 则把大模型生成的细粒度交互描述用于开放词汇比较。它们连起来，基本就是 HOI 从“封闭集检测”走向“开放语义理解”的一条路线。

HOI DetectionDETR-style QueriesCLIPPrompt LearningOpen Vocabulary

参考入口

1. Task definition

Human-Object Interaction detection predicts a triplet: human box, object box, and interaction label, such as person ride bicycle or person hold cup. The challenge is to localize entities and reason about their action relation.

Entity localizationDetect humans and objects with object detectors or end-to-end Transformer frameworks.
Interaction classificationRecognize action semantics from pose, spatial layout, object category, and context.
Compositional generalizationActions and objects form a large long-tailed compositional space.
Open vocabularyReal-world interactions go beyond fixed training labels, so language knowledge becomes important.

2. Main research lines

Early methods often detect humans and objects first, then enumerate human-object pairs for interaction classification. DETR-style methods recast HOI as set prediction, making the pipeline more compact.

3. Why open-vocabulary HOI matters

HOI labels are long-tailed and compositional. Open-vocabulary methods replace a fixed classifier with an expandable language-conditioned semantic space.

My take: open-vocabulary HOI is less about feeding class names to CLIP and more about aligning visual evidence, spatial relations, and language descriptions at the right granularity.

4. A useful reading checklist

  1. What detection structure is used: two-stage pair classification or end-to-end queries?
  2. Where does semantic knowledge come from: labels, CLIP text embeddings, LLM descriptions, or prompt learning?
  3. What generalization setting is evaluated?
  4. What granularity is aligned: image, union region, object region, or detailed action description?

5. Connection to the selected papers

GEN-VLKT introduces vision-language knowledge into end-to-end HOI; CMMP develops conditional multimodal prompts; EZ-HOI focuses on efficient VLM adaptation; SGC-Net uses granular LLM descriptions for open-vocabulary comparison.

HOI DetectionDETR-style QueriesCLIPPrompt LearningOpen Vocabulary