memo: Calc | Inverse Function of Activations

Table of contents

Inverse of Sigmoid

1
2
def inverse_sigmoid(x):
    return torch.log(x/(1-x))

Inverse of Softmax

Determining Existence

Only monotonic function has its inverse function. ¹

The softmax function is not invertible?

Any constant C will work. Or C=(1−log(x⋅y⋅z))/3)

Code

Implement a layer to invert the softmax by tensorflow: ²

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
import tensorflow as tf

def inv_softmax(x, C):
   return tf.math.log(x) + C

import math
input = tf.keras.layers.Input(shape=(1,10))
x = tf.keras.layers.Lambda(lambda x : inv_softmax(x, math.log(10.)),name='inv_softmax')(input)
model = tf.keras.Model(inputs=input, outputs=x)

a = tf.zeros([1, 1, 10])
a = tf.nn.softmax(a)
a = model(a)
print(a.numpy())

Ref

What is an intuitive interpretation for the softmax transformation?


(2023-02-23)

Inverse of Sum

Is the summation operation invertible?

(DDG search: “inverse summation” or “invert a summation”)

Built with Hugo
Theme Stack designed by Jimmy