Local Response Normalization (LRN)

• Layer type: LRN
• Doxygen Documentation
• Header: ./include/caffe/layers/lrn_layer.hpp
• CPU Implementation: ./src/caffe/layers/lrn_layer.cpp
• CUDA GPU Implementation: ./src/caffe/layers/lrn_layer.cu
• Parameters (LRNParameter lrn_param)
  • Optional
    • local_size [default 5]: the number of channels to sum over (for cross channel LRN) or the side length of the square region to sum over (for within channel LRN)
    • alpha [default 1]: the scaling parameter (see below)
    • beta [default 5]: the exponent (see below)
    • norm_region [default ACROSS_CHANNELS]: whether to sum over adjacent channels (ACROSS_CHANNELS) or nearby spatial locations (WITHIN_CHANNEL)

The local response normalization layer performs a kind of “lateral inhibition” by normalizing over local input regions. In ACROSS_CHANNELS mode, the local regions extend across nearby channels, but have no spatial extent (i.e., they have shape local_size x 1 x 1). In WITHIN_CHANNEL mode, the local regions extend spatially, but are in separate channels (i.e., they have shape 1 x local_size x local_size). Each input value is divided by $(1 + (\alpha/n) \sum_i x_i^2)^\beta$, where $n$ is the size of each local region, and the sum is taken over the region centered at that value (zero padding is added where necessary).

Parameters

• Parameters (LRNParameter lrn_param)
• From ./src/caffe/proto/caffe.proto:

// Message that stores parameters used by LRNLayer
message LRNParameter {
  optional uint32 local_size = 1 [default = 5];
  optional float alpha = 2 [default = 1.];
  optional float beta = 3 [default = 0.75];
  enum NormRegion {
    ACROSS_CHANNELS = 0;
    WITHIN_CHANNEL = 1;
  }
  optional NormRegion norm_region = 4 [default = ACROSS_CHANNELS];
  optional float k = 5 [default = 1.];
  enum Engine {
    DEFAULT = 0;
    CAFFE = 1;
    CUDNN = 2;
  }
  optional Engine engine = 6 [default = DEFAULT];
}