Convolutional Neural Networks (CNN) have been the backbone of the image classification and object detection developments seen over the past years. To achieve improving error rates on the famous ImageNet classification contest, CNN architectures became deeper and deeper and introduced skip layers (residual layers) between near layers and layers “deeper into” the model.

卷积神经网络(CNN)已成为过去几年中图像分类和对象检测发展的骨干力量。 为了在著名的ImageNet分类竞赛中提高错误率，CNN架构变得越来越深，并在近层和“更深入”模型的层之间引入了跳过层(残留层)。

CNN的局限性 (Limitations of CNNs)

However, besides all successes CNNs have major drawbacks. On a high level these limitations are:1. Focus on object existence, not localization2. Overlapping objects / Segmentation is a hard task3. Low 3D viewpoint variation

但是，除了所有成功之外，CNN都有很多缺点。 从总体上讲，这些限制是：1。 关注对象的存在，而不是本地化2。 重叠对象/细分是一项艰巨的任务3。 低3D视点变化

These limitations can be explained by the overall architecture used in CNNs. In each convolutional layer so called Feature Maps try to learn image features (e.g. lines, corners) based on image gradients. Before the output of one convolutional layer is fed into the next convolutional layer, a MaxPooling layer typically shrinks the feature resolution by factor 4 by only keeping the maximum value across a 2x2 pooling window. While this progressing feature map shrinkage allows the CNN to learn higher-level concepts in deeper layers and reduces computational complexity, we achieve viewpoint invariance, but we also lose the pose and location information of the object the detected by the CNN. Especially, we lose the relative spatial relationships between features. So far object detection/localization algorithms (SSD, YOLOv3) circumvented this limitation by performing image classification in parallel in multiple pre-selected frame within the original image. [1][2]

这些限制可以通过CNN中使用的整体体系结构来解释。 在每个卷积层中，所谓的特征贴图尝试根据图像梯度学习图像特征(例如，线，角)。 在将一个卷积层的输出馈送到下一个卷积层之前，MaxPooling层通常仅通过在2x2合并窗口中保持最大值将特征分辨率缩小4倍。 尽管这种不断发展的特征图收缩使CNN可以在更深的层中学习更高级的概念并降低计算复杂度，但我们实现了视点不变性，但同时也丢失了CNN检测到的物体的姿态和位置信息。 特别是，我们失去了要素之间的相对空间关系。 到目前为止，对象检测/定位算法(SSD，YOLOv3)通过在原始图像内的多个预选帧中并行执行图像分类来规避了此限制。 [1] [2]

胶囊网络 (Capsule Networks)

To provide a solution Sara Sabour, Nicholas Frosst and Geoffrey E Hinton proposed the use of Capsules in their seminal paper “Dynamic Routing Between Capsules” [3]. In the most high-level notion, Capsule Networks do not forward individual neuron activations from one layer to the next layer, but instead each capsule represents a small nested neural network which outputs a whole vector. The total length of a capsule’s output vector encodes the probability that a certain feature has been detected. The direction of the vector lengths helps representing the state of the detected feature (e.g. location, pose, scale). When a feature moves across the image, the length of the vector should stay the same (as the feature will til be detected), but the vector’s direction will change. This behavior was called activity equivariance by Hinton.

为了提供解决方案，Sara Sabour，Nicholas Frosst和Geoffrey E Hinton在其开创性论文“胶囊之间的动态路由”中提出了使用胶囊的方法[3]。 在最高级的概念中，胶囊网络不会将单个神经元激活从一层转发到下一层，而是每个胶囊代表一个小的嵌套神经网络，该神经网络输出整个矢量。 胶囊输出矢量的总长度编码检测到某个特征的概率。 向量长度的方向有助于表示检测到的特征的状态(例如，位置，姿势，比例)。 当特征在图像上移动时，矢量的长度应保持不变(因为将检测到该特征)，但是矢量的方向将改变。 这种行为被欣顿称为活动等方差。

A capsule sj then does not forward its output vector vj blindly to every capsule in the next layer. Instead, a capsule predicts the output of all capsules in the next layer given its own output vector vj and the respective coupling coefficient cij and forwards its output only to that capsule whose predicted output results in the largest vector. By this “next-layer output prediction” capsulesj ensures that it selects the most appropriate capsule for a given higher-level feature. Depending on the resulting next-layer output vector, the coupling coefficient cij be updated.

这样，胶囊s j不会盲目地将其输出向量v j转发给下一层中的每个胶囊。 取而代之的是，胶囊在给定其自身的输出矢量v j和相应的耦合系数c ij的情况下，预测下一层中所有胶囊的输出，并将其输出仅转发到其胶囊的预测输出导致最大矢量。 通过这种“下一层输出预测”，胶囊j确保为给定的较高级特征选择最合适的胶囊。 取决于所得的下一层输出矢量，耦合系数c ij被更新。

Each capsule’s state sj is calculated as the weighted sum of the matrix multiplication of output/prediction vectors of the capsules from the lower layer with the coupling coefficient cij between sj and the respective lower-level capsule si.

每个胶囊的状态s j被计算为来自下层的胶囊的输出/预测矢量的矩阵乘积的加权和， 其中 ，s j与相应的下层胶囊si之间具有耦合系数c ij 。

Of course, capsules in the first capsule layer of a capsule network calculate their activation based on the input from the previous convolution layer. In this case, no coupling coefficient cij exists.

当然，胶囊网络的第一胶囊层中的胶囊基于来自先前卷积层的输入来计算其激活。 在这种情况下，不存在耦合系数c ij 。

As the capsule’s output vector indicates the probability of having detected a certain feature, capsule sj’s output vector vj is “squashed”, so that long vectors sum up to 1 max and short vectors are close to zero.

由于胶囊的输出向量指示检测到某个特征的可能性，因此胶囊s j的输出向量v j被“压缩”，因此长向量的总和为1个最大值，而短向量的总和接近零。

The coupling coefficients cij defines the “activation routing” between a capsule and all potential parent capsules in the next layer and sum to 1. The softmax-like calculation ensure that the most likely “parent” capsule gets the “most” of capsule sj’s output.

耦合系数c ij定义了胶囊与下一层中所有潜在父胶囊之间的“激活路线”，总和为1。类似softmax的计算可确保最可能的“父”胶囊获得s的“最大”。 j的输出。

By following the presented calculations, the routing preferences between capsules and the prediction of next layer activations, Capsule Network claim to address the CNN limitations listed above, especially modelling stronger feature relationships then CNN could represent which is a very strong tool to boost image segmentation.

通过遵循提出的计算，胶囊之间的路由选择以及对下一层激活的预测，胶囊网络声称可以解决上面列出的CNN局限性，特别是建模更强的特征关系，然后CNN可以代表这是一个非常强大的工具，可以促进图像分割。

胶囊网络用例 (Capsule Networks Use Cases)

The CapsNet architecture represent a baseline implementation of Capsule Networks for the MNIST dataset. CapsNet not only achieves state-of-the-art CNN performance in digit classification, but it especially excels in the MultiMNIST dataset where each image includes 2 overlapping digits.

CapsNet体系结构表示MNIST数据集的Capsule Networks的基线实现。 CapsNet不仅在数字分类方面达到了最先进的CNN性能，而且在每个图像包含2个重叠数字的MultiMNIST数据集中尤其出色。

The CapsNet was designed with an initial classical convolutional layer, followed by 2 capsule layers.

CapsNet的设计具有初始经典卷积层，然后是2个胶囊层。

At the CVPR 2019 conference several capsule use cases were presented. The left image below demonstrates how CapsNet is able to correctly detect overlapping digits due to the feature bonding between capsules. The right image demonstrates how SegCaps, a segmentation network with capsules, achieves more accurate results in segmenting granular retinal vessels.

在CVPR 2019会议上，提出了几种胶囊用例。 下图左图展示了CapsNet如何通过胶囊之间的特征键正确检测重叠的数字。 右图显示了带有胶囊的分割网络SegCaps如何在分割视网膜颗粒血管中获得更准确的结果。