本文为瑞士苏黎世大学（作者：DANIELNEIL）的博士论文，共154页。

基于事件的传感器以生物灵感进行构建，与传统的传感器类型大不相同。标准视觉传感器使用像素阵列在每次对传感器进行采样时产生包含每个像素处的光强度帧，标准音频传感器随时间产生声音振幅的波形。基于事件的传感器输出通常比较稀疏，产生的输出事件发生在场景中的信息变化之后，通常具有较低的延迟和准确的时间，并且是数据驱动的，而不是采样的。这些新型传感器产生的输出与传统传感器截然不同。不幸的是，这些差异使得将标准数据分析技术应用于基于事件的数据变得困难，尽管当前的图像理解和声学处理计算技术处于非常先进的状态。

近年来，机器学习在场景理解方面，特别是在深度学习方面，取得了长足的进步。本文的目的是研究如何利用这些新型传感器在保持基于事件传感器优势的同时，从机器学习技术的现状中吸取经验。本文认为基于帧的传统数据限制了新型机器学习算法的发现范围。虽然机器学习算法已经取得了巨大的成功，但与生物推理相比，它们的成就微不足道，这可能是由于除了如何处理之外，对处理内容的基本假设造成的。也就是说，通过放宽对将要处理的数据类型的期望，也许可以发现更好的算法不仅适用于基于事件的生物传感器，而且优于传统的机器学习算法。

本文是在多个抽象层次上进行研究的。第二章介绍了定制的硬件平台，对现有的硬件机器学习算法进行了原型设计。这项工作旨在确保最先进的机器学习和新型传感器类型的优势都保持在最基本的硬件级别，并更好地理解算法的局限性。事实上，这揭示了与传统机器学习算法相比，两者相结合时最显著的瓶颈是精度损失，从而激发了第3章中的工作，显著提高了事件驱动神经网络对固定不变场景（如图像分析，这可能是深度学习研究得最充分的问题）的分析准确性。第4章讨论了这一主要局限性，探讨了传统深度学习不具备的优势，但可用于事件驱动的深度网络。第5章通过引入一种新的算法（阶段LSTM）来形成本论文的主要贡献，该算法与事件驱动的传感器一起观察动态变化的场景。事实上，正如上面假设的那样，阶段LSTM在事件驱动输入和标准的基于帧的输入方面都比传统的深度神经网络具有显著优势。第6章研究了这些优势的来源，以确定模型是否足够简单和有利。最后，在阶段LSTM发展过程中所做的观察也促使在计算中检验基于事件的传感原理，在第7章中进行了探讨，并证明了当传感器原理应用于计算时，会产生显著的计算加速。总的来说，本文介绍了相关硬件实现及算法，这些硬件实现和算法利用了深度学习的灵感以及基于事件的传感器优势，为平台添加了智能元素，以实现新一代低功耗、更快响应和更精确的系统。

Event-based sensors, built with biologicalinspiration, differ greatly from traditional sensor types. A standard visionsensor uses a pixel array to produce a frame containing the light intensity atevery pixel whenever the sensor is sampled; a standard audio sensor produces awaveform of sound amplitude over time. Event-based sensors, on the other hand,are typically substantially sparser in their output, producing output eventsthat occur upon informative changes in the scene, usually with low latency andaccurate timing, and are data-driven rather than sampled. The outputs producedby these novel sensor types differ radically from traditional sensors.Unfortunately, these differences make it hard to apply standard data analysistechniques to event-based data, despite the advanced state of computationaltechniques for image understanding and acoustic processing. Machine learningespecially has made great strides in recent years towards scene understanding,and particularly in the area of deep learning. The goal of this thesis is tostudy how to make use of these novel sensors to draw from the stateof-the-artin machine learning while maintaining the advantages of event-based sensors.This thesis takes the view that frame-based, traditional data has limited thescope of discovery for new kinds of machine learning algorithms. While machinelearning algorithms have reached great success, their achievements pale in comparisonto biological reasoning, and perhaps this arises from the fundamentalassumptions about what is processed in addition to how. That is, by relaxingexpectations on the kinds of data that will be processed, perhaps even betteralgorithms can be discovered that not only work with biologically-inspiredevent-based sensors but also outperform traditional machine learningalgorithms. This thesis is studied at multiple levels of abstraction. InChapter 2, custom hardware platforms are introduced that prototype an existingmachine learning algorithm in hardware. That work aims to ensure that theadvantages of both state-of-the-art machine learning and the novel sensor typesare maintained at the most fundamental hardware level and to understand thelimitations of the algorithms better. Indeed, this revealed that the mostsignificant bottleneck when combining both is the accuracy loss compared totraditional machine learning algorithms, and motivates the work in Chapter 3that dramatically increases the accuracy of event-driven neural networks forfixed, unchanging scenes (e.g., image analysis, perhaps the most well-studiedproblem in deep learning currently). With that primary limitation addressed,Chapter 4 explores advantages that are unavailable to traditional deep learningbut are available to event-driven deep networks. Chapter 5 forms perhaps thekey contribution of this thesis by introducing a novel algorithm, Phased LSTM,that natively works with event-driven sensors observing dynamic and changingscenes. Indeed, as hypothesized above, Phased LSTM offers significantadvantages over traditional deep neural networks, both for event-driven inputsand for standard frame-based inputs. Chapter 6 investigates the source of theseadvantages to identify if the model is sufficiently simple and advantageous.Finally, an observation made in the development of Phased LSTM motivatesexamining a principle of event-based sensing within computation as well,explored in Chapter 7, and demonstrates the significant computational speedupsthat can result when sensor principles are also applied to computation.Overall, this thesis introduces hardware implementations and algorithms thatuse inspiration from deep learning and the advantages of event-based sensors toadd intelligence to platforms to achieve a new generation of lower-power,faster-response, and more accurate systems.

1 基于事件的传感器与机器学习简介

2 用于深度网络的基于事件的硬件系统

3 从深度学习中发现最先进的研究方法

4 基于事件的深度网络的独特优化方法

5 设计一种直接从基于事件数据中学习的模型

6 确定一种新架构的效率

7 将基于事件传感器的理论扩展到实际计算

8 结论与基于事件机器学习的未来研究展望

附录 源代码与具体实现的细节

更多精彩文章请关注公众号：