本文为加拿大多伦多大学（作者：Shikhar Sharma）的硕士论文，共30页。

针对视频动作识别和生成视频自然语言描述的任务，提出了基于软注意的模型。我们使用多层递归神经网络（RNN）与长短期记忆（LSTM）单元相结合的空间和时间模型。我们的模型学习有选择地集中在视频帧部分，并在注视几次后对视频进行分类；还能够通过时空扫描生成描述视频的句子。该模型从本质上了解了视频帧中哪些部分与当前的任务相关，并对相关帧给予了更高度的重视。我们评估了UCF-11（YouTube动作）、HMDB-51和Hollywood2数据集上的动作识别模型，并分析了该模型如何根据场景和正在执行的动作集中注意力。我们评估了YouTube2Text数据集上的描述生成模型，并在生成单词时可视化模型的注意特性。

We propose soft attention based models forthe tasks of action recognition in videos and generating natural languagedescriptions of videos. We use multi-layered Recurrent Neural Networks (RNNs)with Long Short-Term Memory (LSTM) units which are deep both spatially andtemporally. Our model learns to focus selectively on parts of the video framesand classifies videos after taking a few glimpses. It is also able to generatesentences describing the videos using spatio-temporal glimpses across them. Themodel essentially learns which parts in the frames are relevant for the task athand and attaches higher importance to them. We evaluate the action recognitionmodel on UCF-11 (YouTube Action), HMDB-51 and Hollywood2 datasets and analyzehow the model focuses its attention depending on the scene and the action beingperformed. We evaluate the description generation model on YouTube2Text datasetand visualize the model’s attention as it generates words.

1 引言与相关工作

2 基于视觉注意力的动作识别

3 基于视觉注意力的视频描述

4 结论与未来工作展望

更多精彩文章请关注公众号：