# Self Normalizing Networks

> Self Normalizing Networks is an Agent Skill for development published in the bioinf-jku/SNNs repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Use this skill when working in the Self-Normalizing Networks tutorial repository, especially for TensorFlow, PyTorch, Conda environment, or SELU/AlphaDropout implementation tasks.

- Skill name: `self-normalizing-networks`
- Category: [Development](https://www.openparable.com/skill/development/1)
- Source: https://github.com/bioinf-jku/SNNs/blob/HEAD/SKILL.md
- Repository: bioinf-jku/SNNs (1,588 GitHub stars when collected)

## SKILL.md

# Self-Normalizing Networks

This repository contains tutorial implementations for Self-Normalizing Networks
(SNNs) based on Klambauer et al. Preserve the teaching purpose of the examples:
show how SELU networks are constructed and compared across TensorFlow and
PyTorch implementations.

## Minimal Tabular SNN Example

Below is a minimal prototype (see TF_2_x/TABULAR-MLP-SELU.py) for showing SNNs 
on tabular data: load a small scikit-learn dataset, standardize the features, 
and train a compact Keras MLP with SELU, LeCun initialization, and AlphaDropout.

```python
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from tensorflow import keras
from tensorflow.keras import layers

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0, stratify=y
)

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

model = keras.Sequential(
    [
        keras.Input(shape=(X_train.shape[1],)),
        layers.Dense(64, activation="selu", kernel_initializer="lecun_normal"),
        layers.AlphaDropout(0.05),
        layers.Dense(32, activation="selu", kernel_initializer="lecun_normal"),
        layers.Dense(1, activation="sigmoid"),
    ]
)
model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=1e-3),
    loss="binary_crossentropy",
    metrics=["accuracy"],
)
model.fit(X_train, y_train, epochs=20, batch_size=32, validation_split=0.2)
print(model.evaluate(X_test, y_test, verbose=0))
```

## Core SNN Rules

- Pair SELU activations with LeCun normal initialization.
- Use AlphaDropout when dropout is used in SELU networks.
- In PyTorch, use `nn.init.kaiming_normal_(..., mode="fan_in", nonlinearity="linear")`
  for SELU layers; this matches LeCun normal initialization.
- Keep comparisons between ReLU/dropout and SELU/AlphaDropout conceptually clear.
- Do not add BatchNorm to SNN examples unless explicitly requested; it changes the
  comparison.

## Repository Structure

- `TF_2_x/` contains current TensorFlow/Keras scripts.
- `Pytorch/` contains PyTorch notebooks.
- The root `environment.yml` is the main environment for current TF2 and PyTorch
  examples.

---

Source: https://www.openparable.com/skill/development/self-normalizing-networks (OpenParable). Cite the canonical URL above.
