记忆增强神经网络 (Memory-Augmented Neural Networks)

Memory-Augmented Neural Networks (MANNs) were introduced in 2014 by two concurrent research efforts: Neural Turing Machines and Memory Networks. Since then, they expanded into a broader topic that spans beyond these original implementations — so I decided to dedicate it an entire series. For episode #1, I will stick to a high-level intuitive overview. This article is meant to distill the last 7 years of research into a 7-minute read, removing paper-specific terminology and implementation details that didn’t pass the test of time. Throughout the MANN series, I will dive deeper into the specifics of various proposed designs for neural memories, from their inception in 2014 to the latest updates in 2020.

：M埃默里增加了的神经网络(曼斯)是由两个并行的研究工作，在2014年推出了神经图灵机和记忆网络 。 从那时起，他们扩展到了涵盖这些原始实现之外的更广泛的主题-因此，我决定将其全过程投入使用。 对于第1集，我将坚持高层的直观概述。 本文旨在将最近7年的研究提炼成7分钟的阅读内容，删​​除未经过时间考验的纸张专用术语和实施细节。 在整个MANN系列中，从2014年问世到2020年最新更新，我都会更深入地探讨各种拟议的神经记忆设计的细节。

Memory-Augmented Neural Networks (MANNs) are differentiable versions of the von Neumann architecture. The neural memory is separate from the rest of the model parameters and, similarly to the RAM, stores long-term information.

记忆增强神经网络(MANN)是von Neumann体系结构的可区分版本。 神经存储器与其余模型参数是分开的，并且类似于RAM，它存储长期信息。

The traditional von Neumann architecture differentiates between a CPU (Central Processing Unit) and three levels of memory: registers — very fast, but with storage capability limited to a few values; main memory (e.g. RAM)— faster, with enough storage to accommodate for instructions and data to run a program, and external memory (e.g. hard drive) — slow, but with room for virtually all data used by a computer.

传统的冯·诺依曼 ( von Neumann)架构将CPU(中央处理单元)和三个内存级别区分开： 寄存器 -非常快，但是存储能力限制在几个值之内； 主内存 (例如RAM)速度较慢，具有足够的存储空间来容纳运行程序的指令和数据，而外部存储器 (例如硬盘驱动器)则较慢，但实际上可以容纳计算机使用的所有数据。

Memory-Augmented Neural Networks (MANNs) are differentiable versions of the von Neumann architecture (more on this in the next section). The bulk of the neural network can be thought of as the CPU. Certain architectures like RNNs (Recurrent Neural Networks) have built-in memory that is analogous to the registers, storing short-term information. The neural memory is separate from the rest of the model parameters and, similarly to the RAM, stores long-term information. It consists of an array of memory slots (i.e., a matrix) and, most commonly, stores continuous representations of information (text, images, etc.)

记忆增强神经网络(MANN)是冯·诺依曼(von Neumann)架构的可区分版本(下一节将对此进行详细介绍)。 可以将神经网络的大部分视为CPU。 诸如RNN(递归神经网络)之类的某些体系结构具有类似于寄存器的内置存储器，可存储短期信息。 神经存储器与其余模型参数分开，并且类似于RAM，它存储长期信息。 它由一个内存插槽阵列(即一个矩阵)组成，最常见的是存储信息的连续表示(文本，图像等)。

The component that directly interacts with the neural memory via read and write operations is called a controller. In early work, the controller coincided with the rest of the model (i.e. all the parameters outside the memory), so it acted as the interface between the memory and the “external world”. It was often implemented as a recurrent neural network. More recently, with the advent of massive Transformer-based architectures, the controller is only a small subset of the model, and mediates the communication between the memory and the rest of the network.

通过读写操作与神经记忆直接交互的组件称为控制器 。 在早期的工作中，控制器与模型的其余部分(即存储器外部的所有参数)一致，因此它充当存储器与“外部世界”之间的接口。 它通常被实现为递归神经网络。 最近，随着大规模基于Transformer的体系结构的出现，控制器仅是模型的一小部分，并介导了存储器与网络其余部分之间的通信。

什么是可区分的架构？ (What is a differentiable architecture?)

MANNs are differentiable, von Neumann architectures are not — but what exactly does this mean? You might recall the following definitions:

MANN是可区分的，而von Neumann体系结构则不是—但是这到底意味着什么？ 您可能会想起以下定义：

A differentiable function of a real variable is a function whose derivative exists at each point in its domain — Wikipedia.

实变量的可微函数是其导数存在于其域中每个点Wikipedia上的函数。

The derivative of a function of a real variable measures the sensitivity of the function value (output value) with respect to a change in its argument (input value) — Wikipedia.

实变量的函数的导数用于衡量函数值(输出值)对其自变量(输入值)的变化的敏感性-Wikipedia 。

The von Neumann architecture performs non-differentiable operations. For instance, consider a read operation: when the CPU fetches its next instruction from RAM, it specifies an address (the input) and receives back an instruction (the output). The input domain is thus unsigned integers, so the operation is not defined over a real variable. According to the definition above, differentiability is out of the question.

冯·诺依曼架构执行不可微操作。 例如，考虑一个读操作：当CPU从RAM中获取下一条指令时，它指定一个地址(输入)并接收回一条指令(输出)。 因此，输入域是无符号整数，因此未在实变量上定义操作。 根据上面的定义，可微性是不可能的。

使操作与众不同： 软读写 (Making operations differentiable: soft reads and writes)

The core reason why RAM reads are not differentiable is that they operate over a discrete space of addresses. Neural memories propose an adjustment:

RAM读取不可区分的核心原因是它们在地址的离散空间上运行。 神经记忆建议调整：

Instead of reading from a single entry, perform a weighted read from all entries.

而不是从单个条目读取，而是从所有条目执行加权读取。

For each memory slot i, the controller specifies a real-valued weight wᵢ such that all weights sum up to 1. This changes the input of the read operation from a single integer value (the address) to a vector of real values (the per-slot weights), which is the first requirement for differentiability. Note that this revised operation is strictly more general: when a single weight is set to 1.0 and all others to 0.0, we are effectively reading from a single entry. The same reasoning applies to writes: instead of writing a value x to a single memory slot, we update each entry i by a weighted value wᵢ * x.

对于每个内存插槽i ，控制器指定一个实数权重w 1 ，以使所有权重之和为1。这会将读取操作的输入从单个整数值(地址)更改为实数值的向量(每个-slot权重)，这是可微性的首要要求。 请注意，这种修改后的操作严格来说更为笼统：将单个权重设置为1.0并将所有其他权重设置为0.0时，我们实际上是在从单个条目中读取内容。 同样的道理也适用于写操作：不是将值x写入单个内存插槽，而是通过加权值wᵢ* x更新每个条目i 。

These operations are called soft reads and writes, due to the continuous nature of the weights wᵢ. This contrasts with the hard reads and writes to RAM, where the controller makes a hard decision regarding the memory slot to operate on.

由于权重wᵢ的连续性质，这些操作称为软读写 。 这与对RAM的硬性读写形成对比，在RAM中，控制器对要操作的内存插槽做出硬性决定。

计算软权重：基于内容和基于位置的寻址 (Computing the soft weights: content- vs location-based addressing)

RAM is accessed based on location — read operations specify the exact address to read from. Neural memories are typically accessed based on content — queries specify what to read, not where to read from.

根据位置访问RAM-读取操作指定要读取的确切地址。 神经存储器通常根据内容进行访问-查询指定要读取的内容，而不是从何处读取的内容。

How does the controller compute the per-slot weights wᵢ?

控制器如何计算每个插槽的权重wᵢ ？

First, a note on terminology: the mechanism to compute weights wᵢ is often referred to as memory addressing, since it determines which memory slots are addressed, and how much attention is paid to each. Addressing neural memories can be done based on content or location.

首先，关于术语的注释：计算权重wᵢ的机制通常称为内存寻址，因为它确定要寻址的内存插槽以及对每个内存插槽的关注程度。 可以根据内容或位置完成对神经记忆的寻址。

With content-based addressing, the weights wᵢ reflect how relevant the content of slot i is in resolving an incoming query. For instance, for a question answering task, the memory query could be an embedding of the actual question. The controller must then upweight the memory slots that are good answer candidates. Most commonly, wᵢ is the dot product or cosine similarity between the embeddings of the content in slot i and the query. Finally, all weights are normalized via softmax so that they sum up to 1.

使用基于内容的寻址 ，权重将反映插槽i的内容在解决传入查询中的相关性。 例如，对于问题解答任务，内存查询可以是实际问题的嵌入。 然后，控制器必须增加作为良好答案候选对象的内存插槽的重量。 最常见的是， wᵢ是插槽i中内容的嵌入与查询之间的点积或余弦相似度。 最后，所有权重均通过softmax进行归一化，以使其总和为1。

With location-based addressing, the weights wᵢ reflect how much attention to be paid to location i, irrespective of its content. Although less common, neural memories are designed to support location-based addressing when the task they have to solve involves simple logical or arithmetic operations like adding two variables x and y — in this case, the controller should be able to retrieve the operands from memory regardless of their exact value. In the following episodes, I will dive deeper into the specifics of computing location-based weights wᵢ.

对于基于位置的寻址，权重w 1反映了对位置i的重视程度，而与其内容无关 。 尽管不那么常见，但神经存储器被设计为在其必须解决的任务涉及简单的逻辑或算术运算(例如添加两个变量x和y)时支持基于位置的寻址-在这种情况下，控制器应该能够从存储器中检索操作数不管它们的确切价值。 在以下几集中，我将更深入地研究计算基于位置的权重wᵢ的细节。

降低计算成本：稀疏读写 (Reducing the computational cost: sparse reads and writes)

Making reads and writes differentiable comes with a computational cost. Each query is now resolved in linear time O(N), where N is the number of memory slots (in contrast, hard reads and writes take constant O(1) time). When the input to a network is a sequence (e.g. a text document) of length L, it is common to make one query for each element in the sequence — which brings the cost up to O(N*L). During training, soft reads and writes are also memory-inefficient; computing gradients for the entire memory requires making a copy of it.

使读写差异化会带来计算成本。 现在，每个查询都以线性时间O(N)进行解析，其中N是内存插槽的数量(相比之下，硬读写需要固定的O(1)时间)。 当网络的输入是长度为L的序列(例如文本文档)时，通常会对序列中的每个元素进行一次查询，这会使成本增加到O(N * L)。 在训练过程中，软读取和写入的内存效率也很低。 计算整个内存的梯度需要对其进行复制。

Follow-up research focused on reducing the O(N) cost to either O(log N) (Rae et al. [2]) or O(sqrt N) (Lample et al. [3]). While the two approaches are quite different, their common ground is to operate on a subset of the memory, as opposed to all entries. In other words, they limit the number of non-zero weights wᵢ to a small constant K (somewhere between 2 and 8), and apply gradient descent only on the slots with non-zero weight.

后续研究集中于将O(N)成本降低到O(log N) (Rae等人[2])或O(sqrt N) (Lample等人[3])。 尽管这两种方法截然不同，但它们的共同基础是在存储器的一个子集上进行操作，而不是在所有条目上进行操作。 换句话说，它们将非零权重w 1的数量限制为一个小的常数K(在2到8之间)，并且仅对权重非零的时隙应用梯度下降。

现状 (Current State of Affairs)

Memory-Augmented Neural Networks have shown promising results in artificial tasks (e.g. they learn to copy a sequence a given number of times), some natural language tasks (question answering, machine translation) and some computer vision tasks (character recognition). However, they are yet to become mainstream. There are interesting research opportunities in multiple directions: reducing their computational cost, speeding up training, understanding under what circumstances they are most useful, and integrating them with the state-of-the-art Transformers.

记忆增强神经网络在人工任务(例如，他们学习复制给定次数的次数)，一些自然语言任务(问题回答，机器翻译)和一些计算机视觉任务(字符识别)方面显示出令人鼓舞的结果。 但是，它们尚未成为主流。 在多个方向上都有有趣的研究机会：降低它们的计算成本，加快培训速度，了解它们在什么情况下最有用以及将它们与最新的变形金刚集成在一起。

The rest of this series will reflect upon what’s been done and where to go next. If you would like me to cover a particular paper, feel free to leave a comment below.

本系列的其余部分将反思已完成的工作以及下一步要去哪里。 如果您希望我介绍特定的论文，请在下面发表评论。