RAdam implemented in Keras & TensorFlow

CyberZHG, updated πŸ•₯ 2022-01-22 10:40:58

Keras RAdam

Version License


Unofficial implementation of RAdam in Keras.


bash pip install keras-rectified-adam

External Link


```python from tensorflow import keras import numpy as np from keras_radam import RAdam

Build toy model with RAdam optimizer

model = keras.models.Sequential() model.add(keras.layers.Dense(input_shape=(17,), units=3)) model.compile(RAdam(), loss='mse')

Generate toy data

x = np.random.standard_normal((4096 * 30, 17)) w = np.random.standard_normal((17, 3)) y = np.dot(x, w)


model.fit(x, y, epochs=5) ```

Use Warmup

```python from keras_radam import RAdam

RAdam(total_steps=10000, warmup_proportion=0.1, min_lr=1e-5) ```

Zhao HG

Knowledge is bacon. Please don't send emails.

GitHub Repository Homepage

keras optimizers adam radam rectified-adam tensorflow