Overview

I will introduce the implementation of a new activation function called FReLU in keras.

For more information about FReLU + implementation in pytorch, please read the following article. It is explained very carefully. Reference: Birth & explanation of new activation function "FReLU"!

This article is about implementing FReLU in tf.keras. With Depthwise Conv2D and Lambda, you can implement it quickly.

Definition

y= \max(x,\mathbb{T}(x))

$ \ mathbb {T} (\ cdot) $ is DepthwiseConv2D.

Premise

tensorflow 2.3.0

Implementation

`FReLU.py`


from tensorflow.keras.layers import DepthwiseConv2D,BatchNormalization
import tensorflow as tf
from tensorflow.keras.layers import Lambda
from tensorflow.keras import backend as K

#Mathematical symbol max()Definition of
def max_unit(args):
    inputs , depthconv_output = args
    return tf.maximum(inputs, depthconv_output)

def FReLU(inputs, kernel_size = 3):
    #T(x)Part of
    x = DepthwiseConv2D(kernel_size, strides=(1, 1), padding='same')(inputs)
    x = BatchNormalization()(x)
    #Calculate tensor shape for Lambda
    x_shape = K.int_shape(x)
    #max(x, T(x))Part of
    x = Lambda(max_unit, output_shape=(x_shape[1], x_shape[2], x_shape[3]))([inputs, x])
    return x

At the end

The implementation itself is easy, isn't it? I will try to implement the Layer version when I have time. It's nice because the name will appear when that person summarizes. If you have any questions or concerns, please leave a comment.

Implement FReLU with tf.keras