ML Cheatsheet

https://ml-cheatsheet.readthedocs.io/en/latest/
Admin
ML Cheatsheet is a visual guide to machine learning concepts featuring diagrams, code examples, and...
Tokens:45,601
Snippets:357
Trust Score:8.3
Update:9 months ago
Show doc for...
Tokens:
Raw
### NumPy Random Normal Distribution Example (Python)

Source: https://ml-cheatsheet.readthedocs.io/en/latest/forwardpropagation

Demonstrates how to generate a NumPy array with random numbers drawn from a standard normal distribution and how to check its shape.

```python
arr = np.random.randn(1, 2)

print(arr)
print(arr.shape)

```

--------------------------------

### MLP Training and Evaluation Example in PyTorch

Source: https://ml-cheatsheet.readthedocs.io/en/latest/architectures

Demonstrates a complete training and evaluation process for an MLP on the FashionMNIST dataset. It includes data loading, model initialization, training for a specified number of epochs, saving the trained model, and calculating the accuracy on a test set.

```python
def main():
    train_set = torchvision.datasets.FashionMNIST(
        root = './FMNIST',
        train = True,
        download = False,
        transform = transforms.Compose([
            transforms.ToTensor()
        ])
    )
    mlp = MLP()
    loader = torch.utils.data.DataLoader(train_set, batch_size = 1000)
    optimizer = optim.Adam(mlp.parameters(), lr=0.01)
    loss_func=nn.CrossEntropyLoss()
    for i in range(0,15):
        train(mlp,loader,loss_func,optimizer)
    print("Finished Training")
    torch.save(mlp.state_dict(), "./mlpmodel.pt")
    test_set = torchvision.datasets.FashionMNIST(
        root = './FMNIST',
        train = False,
        download = False,
        transform = transforms.Compose([
            transforms.ToTensor()
        ])
    )
    testloader = torch.utils.data.DataLoader(test_set, batch_size=4)
    correct = 0
    total = 0
    with torch.no_grad():
        for data in testloader:
            images, labels = data
            outputs = mlp(images)
            _, predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()
    print('Accuracy of the network on the 10000 test images: %d %%' % (
        100 * correct / total))


```

--------------------------------

### Scikit-Learn Logistic Regression Setup (Python)

Source: https://ml-cheatsheet.readthedocs.io/en/latest/logistic_regression

Prepares data for scikit-learn's LogisticRegression model. This includes normalizing features to a specified range and splitting the data into training and testing sets.

```python
import sklearn
from sklearn.linear_model import LogisticRegression
from sklearn.cross_validation import train_test_split

# Normalize grades to values between 0 and 1 for more efficient computation
normalized_range = sklearn.preprocessing.MinMaxScaler(feature_range=(-1,1))

# Extract Features + Labels
labels.shape =  (100,) #scikit expects this
features = normalized_range.fit_transform(features)

# Create Test/Train
features_train,features_test,labels_train,labels_test = train_test_split(features,labels,test_size=0.4)
```

--------------------------------

### RNN Class Implementation in Python

Source: https://ml-cheatsheet.readthedocs.io/en/latest/layers

Defines the RNN class with methods for initialization, parameter setup, and a forward pass. It takes input dimensions, hidden layer dimensions, output dimensions, and batch size as parameters. The forward method processes input sequences and returns output sequences.

```python
import numpy as np
from typing import List

# Assuming softmax is defined elsewhere or imported
def softmax(x):
    return np.exp(x) / np.sum(np.exp(x), axis=0)

class RNN:
    def __init__(self, input_dim: int, hidden_dim: int, output_dim: int, batch_size=1) -> None:
        self.input_dim = input_dim
        self.hidden_dim = hidden_dim
        self.out_dim = output_dim
        self.batch_size = batch_size
        # initialization
        self.params = self._init_params()
        self.hidden_state = self._init_hidden_state()

    def _init_params(self) -> List[np.array]:
        scale = 0.01
        Waa = np.random.normal(scale=scale, size=[self.hidden_dim, self.hidden_dim])
        Wax = np.random.normal(scale=scale, size=[self.hidden_dim, self.input_dim])
        Wy = np.random.normal(scale=scale, size=[self.out_dim, self.hidden_dim])
        ba = np.zeros(shape=[self.hidden_dim, 1])
        by = np.zeros(shape=[self.out_dim, 1])
        return [Waa, Wax, Wy, ba, by]

    def _init_hidden_state(self) -> np.array:
        return np.zeros(shape=[self.hidden_dim, self.batch_size])

    def forward(self, input_vector: np.array) -> np.array:
        """
        input_vector:
            dimension: [num_steps, self.input_dim, self.batch_size]
        out_vector:
            dimension: [num_steps, self.output_dim, self.batch_size]
        """
        Waa, Wax, Wy, ba, by = self.params
        output_vector = []
        for vector in input_vector:
            self.hidden_state = np.tanh(
                np.dot(Waa, self.hidden_state) + np.dot(Wax, vector) + ba
            )
            y = softmax(
                np.dot(Wy, self.hidden_state) + by
            )
            output_vector.append(y)
        return np.array(output_vector)

if __name__ == "__main__":
    input_data = np.array([
        [
            [1, 3]
            , [2, 4]
            , [3, 6]
        ]
        , [
            [4, 3]
            , [3, 4]
            , [1, 5]
        ]
    ])
    batch_size = 2
    input_dim = 3
    output_dim = 4
    hidden_dim = 5
    time_step = 2
    rnn = RNN(input_dim=input_dim, batch_size=batch_size, output_dim=output_dim, hidden_dim=hidden_dim)
    output_vector = rnn.forward(input_vector=input_data)
    print("RNN:")
    print(f"Input data dimensions: {input_data.shape}")
    print(f"Output data dimensions {output_vector.shape}")
    ## We will get the following output:
    ##  RNN:
    ## Input data dimensions: (2, 3, 2)
    ## Output data dimensions (2, 4, 2)
```

--------------------------------

### Prediction Function with Sigmoid in Python

Source: https://ml-cheatsheet.readthedocs.io/en/latest/logistic_regression

This function calculates the probability of a class label being 1 using the sigmoid activation. It takes feature vectors and weights as input, computes the dot product to get 'z', and then applies the sigmoid function. The output is a 1D array of probabilities.

```python
def predict(features, weights):
  '''
  Returns 1D array of probabilities
  that the class label == 1
  '''
  z = np.dot(features, weights)
  return sigmoid(z)
```

--------------------------------

### Neural Network Initialization in Python

Source: https://ml-cheatsheet.readthedocs.io/en/latest/forwardpropagation

Initializes the weights and biases for a neural network. `init_weights` creates weight matrices (Wh, Wo) with random values scaled appropriately for the layer sizes, using a common initialization strategy. `init_bias` creates bias matrices (Bh, Bo) with small positive values.

```python
INPUT_LAYER_SIZE = 1
HIDDEN_LAYER_SIZE = 2
OUTPUT_LAYER_SIZE = 2

def init_weights():
    Wh = np.random.randn(INPUT_LAYER_SIZE, HIDDEN_LAYER_SIZE) * \
                np.sqrt(2.0/INPUT_LAYER_SIZE)
    Wo = np.random.randn(HIDDEN_LAYER_SIZE, OUTPUT_LAYER_SIZE) * \
                np.sqrt(2.0/HIDDEN_LAYER_SIZE)


def init_bias():
    Bh = np.full((1, HIDDEN_LAYER_SIZE), 0.1)
    Bo = np.full((1, OUTPUT_LAYER_SIZE), 0.1)
    return Bh, Bo

```

--------------------------------

### Weight Initialization for Neural Network Layers (Python)

Source: https://ml-cheatsheet.readthedocs.io/en/latest/forwardpropagation

Initializes the weight matrices for the hidden and output layers of a neural network using NumPy. Weights are drawn from a normal distribution and scaled to help with training.

```python
import numpy as np

def init_weights():
    Wh = np.random.randn(INPUT_LAYER_SIZE, HIDDEN_LAYER_SIZE) * \
                np.sqrt(2.0/INPUT_LAYER_SIZE)
    Wo = np.random.randn(HIDDEN_LAYER_SIZE, OUTPUT_LAYER_SIZE) * \
                np.sqrt(2.0/HIDDEN_LAYER_SIZE)
    return Wh, Wo

```

--------------------------------

### PyTorch Autoencoder Training Function

Source: https://ml-cheatsheet.readthedocs.io/en/latest/architectures

A basic training function for an Autoencoder in PyTorch. It iterates through a data loader, performs a forward pass to get the output, calculates the loss between the output and the original inputs, and then backpropagates the error to update the model's weights using the provided optimizer.

```python
def train(net, loader, loss_func, optimizer):
    net.train()
    for inputs, _ in loader:
        inputs = Variable(inputs)

        output = net(inputs)
        loss = loss_func(output, inputs)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

```

--------------------------------

### Backpropagation Implementation in Python

Source: https://ml-cheatsheet.readthedocs.io/en/latest/backpropagation

A Python function demonstrating the backpropagation algorithm. It calculates layer errors (Eo, Eh) and the derivatives of cost with respect to weights (dWo, dWh), then updates the weights (Wh, Wo) using a learning rate (lr). Dependencies include helper functions like feed_forward, relu_prime, cost, and cost_prime, as well as variables like Zo, Zh, H, yHat, y, and x.

```python
def relu_prime(z):
    if z > 0:
        return 1
    return 0

def cost(yHat, y):
    return 0.5 * (yHat - y)**2

def cost_prime(yHat, y):
    return yHat - y

def backprop(x, y, Wh, Wo, lr):
    yHat = feed_forward(x, Wh, Wo)

    # Layer Error
    Eo = (yHat - y) * relu_prime(Zo)
    Eh = Eo * Wo * relu_prime(Zh)

    # Cost derivative for weights
    dWo = Eo * H
    dWh = Eh * x

    # Update weights
    Wh -= lr * dWh
    Wo -= lr * dWo

```

--------------------------------

### K-Nearest Neighbor Implementation using NumPy

Source: https://ml-cheatsheet.readthedocs.io/en/latest/classification_algos

Implements the K-Nearest Neighbor (KNN) algorithm. It calculates Euclidean distances between a target point and training data, identifies the k nearest neighbors, and predicts the target label by averaging for regression or taking the majority vote for classification.

```python
def KNN(training_data, target, k, func):
    """
    training_data: all training data point
    target: new point
    k: user-defined constant, number of closest training data
    func: functions used to get the the target label
    """
    # Step one: calculate the Euclidean distance between the new point and all training data
    neighbors= []
    for index, data in enumerate(training_data):
        # distance between the target data and the current example from the data.
        distance = euclidean_distance(data[:-1], target)
        neighbors.append((distance, index))

    # Step two: pick the top-K closest training data
    sorted_neighbors = sorted(neighbors)
    k_nearest = sorted_neighbors[:k]
    k_nearest_labels = [training_data[i][1] for distance, i in k_nearest]

    # Step three: For regression problem, take the average of the labels as the result;
    #             for classification problem, take the most common label of these labels as the result.
    return k_nearest, func(k_nearest_labels)
```

--------------------------------

### Dropout Implementation for Neural Networks (Python)

Source: https://ml-cheatsheet.readthedocs.io/en/latest/regularization

Demonstrates a Python implementation of the dropout technique for neural networks using NumPy. It randomly sets a fraction of input units to 0 at each update during training time, which helps prevent overfitting.

```python
import numpy as np

A = np.arange(20).reshape((5,4))

print("Given input: ")
print(A)

def dropout(X, drop_probability):
    keep_probability = 1 - drop_probability
    mask = np.random.uniform(0, 1.0, X.shape) < keep_probability
    if keep_probability > 0.0:
        scale = (1/keep_probability)
    else:
        scale = 0.0
    return mask * X * scale

print("\n After Dropout: ")
print(dropout(A,0.5))
```

--------------------------------

### Bias Term Initialization for Neural Network Layers (Python)

Source: https://ml-cheatsheet.readthedocs.io/en/latest/forwardpropagation

Initializes the bias vectors for the hidden and output layers of a neural network using NumPy. Bias terms are initialized to a default value and help shift activation functions.

```python
import numpy as np

def init_bias():
    Bh = np.full((1, HIDDEN_LAYER_SIZE), 0.1)
    Bo = np.full((1, OUTPUT_LAYER_SIZE), 0.1)
    return Bh, Bo

```

--------------------------------

### Broadcasting in Numpy

Source: https://ml-cheatsheet.readthedocs.io/en/latest/linear_algebra

Demonstrates Numpy's broadcasting mechanism, which relaxes dimension requirements for element-wise operations. Matrices are compatible if dimensions are equal or one of the dimensions is size 1. This allows operations between arrays of different shapes.

```python
a = np.array([
 [1],
 [2]
])
b = np.array([
 [3,4],
 [5,6]
])
c = np.array([
 [1,2]
])

# Same no. of rows
# Different no. of columns
# but a has one column so this works
a * b

# Same no. of columns
# Different no. of rows
# but c has one row so this works
b * c

# Different no. of columns
# Different no. of rows
# but both a and c meet the
# size 1 requirement rule
a + c
```

--------------------------------

### VAE Training Loop in PyTorch

Source: https://ml-cheatsheet.readthedocs.io/en/latest/architectures

A basic training loop for a Variational Autoencoder (VAE) model in PyTorch. It iterates through a data loader, performs a forward pass, calculates the VAE loss, and updates model weights using backpropagation and an optimizer. Requires a PyTorch model, data loader, loss function, and optimizer.

```python
def train(model, loader, loss_func, optimizer):
    model.train()
    for inputs, _ in loader:
        inputs = Variable(inputs)

        output, mean, logvar = model(inputs)
        loss = vae_loss(output, inputs, mean, logvar, loss_func)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

```

--------------------------------

### Feed Forward Calculation with Matrices in Python

Source: https://ml-cheatsheet.readthedocs.io/en/latest/forwardpropagation

Implements the feed-forward pass for a neural network, accepting a matrix of inputs (X). It calculates the weighted inputs and activations for both the hidden and output layers using matrix operations and the ReLU activation function. Dependencies include NumPy for numerical operations.

```python
def feed_forward(X):
    '''
    X    - input matrix
    Zh   - hidden layer weighted input
    Zo   - output layer weighted input
    H    - hidden layer activation
    y    - output layer
    yHat - output layer predictions
    '''

    # Hidden layer
    Zh = np.dot(X, Wh) + Bh
    H = relu(Zh)

    # Output layer
    Zo = np.dot(H, Wo) + Bo
    yHat = relu(Zo)
    return yHat

```

--------------------------------

### Early Stopping Implementation in Python

Source: https://ml-cheatsheet.readthedocs.io/en/latest/regularization

Implements the Early Stopping meta-algorithm to determine the optimal number of training epochs. It monitors validation error to prevent overfitting and returns the best network, its iteration count, and validation error. This function requires a 'Network' object with 'clone', 'train', and 'error' methods, and assumes numpy is imported as 'np'.

```python
import numpy as np

def early_stopping(theta0, (x_train, y_train), (x_valid, y_valid), n = 1, p = 100):
  """ The early stopping meta-algorithm for determining the best amount of time to train.
      REF: Algorithm 7.1 in deep learning book.

      Parameters:
      n: int; Number of steps between evaluations.
      p: int; "patience", the number of evaluations to observe worsening validataion set.
      theta0: Network; initial network.
      x_train: iterable; The training input set.
      y_train: iterable; The training output set.
      x_valid: iterable; The validation input set.
      y_valid: iterable; The validation output set.

      Returns:
      theta_prime: Network object; The output network.
      i_prime: int; The number of iterations for the output network.
      v: float; The validation error for the output network.
  """
  # Initialize variables
  theta = theta0.clone()       # The active network
  i = 0                        # The number of training steps taken
  j = 0                        # The number of evaluations steps since last update of theta_prime
  v = np.inf                   # The best evaluation error observed thusfar
  theta_prime = theta.clone()  # The best network found thusfar
  i_prime = i                  # The index of theta_prime

  while j < p:
      # Update theta by running the training algorithm for n steps
      for _ in range(n):
          theta.train(x_train, y_train)

      # Update Values
      i += n
      v_new = theta.error(x_valid, y_valid)

      # If better validation error, then reset waiting time, save the network, and update the best error value
      if v_new < v:
          j = 0
          theta_prime = theta.clone()
          i_prime = i
          v = v_new

      # Otherwise, update the waiting time
      else:
          j += 1

  return theta_prime, i_prime, v

```

--------------------------------

### Initialize Weights for Multivariable Regression (Python)

Source: https://ml-cheatsheet.readthedocs.io/en/latest/linear_regression

Initializes the weights (coefficients) for a multivariable regression model. Typically, weights are initialized to zero or small random values. This snippet demonstrates initializing three weights (W1, W2, W3) to 0.0 and then organizing them into a numpy array suitable for matrix operations.

```Python
import numpy as np

W1 = 0.0
W2 = 0.0
W3 = 0.0
weights = np.array([
    [W1],
    [W2],
    [W3]
])
```

--------------------------------

### DecisionTree Base Class for Training

Source: https://ml-cheatsheet.readthedocs.io/en/latest/classification_algos

Provides the fundamental structure and `fit` method for a Decision Tree. It manages the tree's data, labels, and the process of building the tree by iteratively splitting nodes.

```python
class DecisionTree() :
    def fit(self, X, y):
        """
        X: train data, dimensition [num_sample, num_feature]
        y: label, dimension [num_sample, ]
        """
        self.data = X
        self.labels = y
        num_sample, num_feature = X.shape
        self.feature_num = num_feature
        data_idx = list(range(num_sample))
        # Set the root of the tree
        self.root = TreeNode(data_idx=data_idx, depth=0, child_lst=[])
        queue = [self.root]
        while queue:
            node = queue.pop(0)
            # Check if the terminate criterion has been met
            if node.depth>self.max_depth or len(node.data_idx)==1:
                # Set the label for the leaf node
                self.set_label(node)
            else:
                # Split the node
                child_nodes = self.split_node(node)
                if not child_nodes:
                    self.set_label(node)
                else:
                    queue.extend(child_nodes)

```

--------------------------------

### ELU Activation Function and its Derivative (Python)

Source: https://ml-cheatsheet.readthedocs.io/en/latest/activation_functions

Implements the Exponential Linear Unit (ELU) activation function and its derivative. ELU handles negative inputs by applying an exponential function, aiming to converge costs faster and provide more accurate results than ReLU. It uses an 'alpha' constant for negative values.

```python
def elu(z,alpha):
	return z if z >= 0 else alpha*(e^z -1)
```

```python
def elu_prime(z,alpha):
	return 1 if z > 0 else alpha*np.exp(z)
```

--------------------------------

### Simple Network Feedforward Calculation (Python)

Source: https://ml-cheatsheet.readthedocs.io/en/latest/forwardpropagation

Calculates the output prediction of a simple neural network with one hidden layer. It takes input data, hidden layer weights, and output layer weights as arguments. It uses a ReLU activation function.

```python
def relu(z):
    return max(0,z)

def feed_forward(x, Wh, Wo):
    # Hidden layer
    Zh = x * Wh
    H = relu(Zh)

    # Output layer
    Zo = H * Wo
    output = relu(Zo)
    return output

```

--------------------------------

### PyTorch GAN Training Loop

Source: https://ml-cheatsheet.readthedocs.io/en/latest/architectures

Provides a PyTorch training function for a GAN. This function iterates through epochs and data batches, updating both the discriminator and generator networks. It calculates losses for real and fake data and performs backpropagation.

```python
def train(netD, netG, loader, loss_func, optimizerD, optimizerG, num_epochs):
    netD.train()
    netG.train()
    device = "cuda:0" if torch.cuda.is_available() else "cpu"
    for epoch in range(num_epochs):
        for i, data in enumerate(loader, 0):
            netD.zero_grad()
            realtens = data[0].to(device)
            b_size = realtens.size(0)
            label = torch.full((b_size,), 1, dtype=torch.float, device=device) # gen labels
            output = netD(realtens)
            errD_real = loss_func(output, label)
            errD_real.backward() # backprop discriminator fake and real based on label
            noise = torch.randn(b_size, 200, 1, 1, device=device)
            fake = netG(noise)
            label.fill_(0)
            output = netD(fake.detach()).view(-1)
            errD_fake = loss_func(output, label)
            errD_fake.backward() # backprop discriminator fake and real based on label
            errD = errD_real + errD_fake # discriminator error
            optimizerD.step()
            netG.zero_grad()
            label.fill_(1)  
            output = netD(fake).view(-1)
            errG = loss_func(output, label) # generator error
            errG.backward()
            optimizerG.step()

```

--------------------------------

### Compute Definite Integral Approximation - Python

Source: https://ml-cheatsheet.readthedocs.io/en/latest/calculus

Approximates the definite integral of a function `func` between limits `a` and `b` using a Riemann sum with small rectangular strips. It requires a function to integrate and the lower and upper bounds as input. The accuracy depends on the width `h` of the strips.

```python
def get_integral(func, a, b):
    """Compute the area under `func` between x=a and x=b."""
    h = 0.0001               # width of small rectangle
    x = a                    # start at x=a
    total = 0
    while x <= b:            # continue until x=b
        total += h*func(x)   # area of rect is base*height
        x += h
    return total

def f(x): return x**2                    # some test function f(x)=x^2
computed = get_integral(f, 1, 3)
def actualF(x): return 1.0/3.0*x**3
actual = actualF(3) - actualF(1)
computed, actual    # = 8.6662, 8.6666   # pretty close if you ask me...

```

--------------------------------

### Neural Network Layer Sizes Definition (Python)

Source: https://ml-cheatsheet.readthedocs.io/en/latest/forwardpropagation

Defines the size of the input, hidden, and output layers for a neural network. These constants are used to determine the dimensions of weight matrices and bias vectors.

```python
INPUT_LAYER_SIZE = 1
HIDDEN_LAYER_SIZE = 2
OUTPUT_LAYER_SIZE = 2

```

--------------------------------

### Weighted Input Calculation for Matrices in Python

Source: https://ml-cheatsheet.readthedocs.io/en/latest/forwardpropagation

Calculates the weighted input (Zh) for the hidden layer by performing a dot product between the input matrix (X) and the weight matrix (Wh), followed by adding the bias vector (Bh). This operation is designed to handle matrix inputs efficiently using NumPy.

```python
Zh = np.dot(X, Wh) + Bh

```

--------------------------------

### PyTorch Implementation of a Convolutional VAE Model

Source: https://ml-cheatsheet.readthedocs.io/en/latest/architectures

Defines a Convolutional Variational Autoencoder (VAE) model using PyTorch. It includes an encoder to map input images to a latent space mean and variance, a decoder to reconstruct images from latent vectors, and a forward pass method that integrates encoding, sampling, and decoding. Dependencies include `torch`, `nn.Module`, `nn.Linear`, `nn.Conv2d`, `nn.ConvTranspose2d`, `nn.BatchNorm2d`, `nn.LeakyReLU`, `nn.Sigmoid`, and `Variable`.

```python
class VAE(nn.Module):
    def __init__(self, in_shape, n_latent):
        super().__init__()
        self.in_shape = in_shape
        self.n_latent = n_latent
        c,h,w = in_shape
        self.z_dim = h//2**2 # receptive field downsampled 2 times
        self.encoder = nn.Sequential(
            nn.BatchNorm2d(c),
            nn.Conv2d(c, 32, kernel_size=4, stride=2, padding=1),  # 32, 16, 16
            nn.BatchNorm2d(32),
            nn.LeakyReLU(),
            nn.Conv2d(32, 64, kernel_size=4, stride=2, padding=1),  # 32, 8, 8
            nn.BatchNorm2d(64),
            nn.LeakyReLU(),
        )
        self.z_mean = nn.Linear(64 * self.z_dim**2, n_latent)
        self.z_var = nn.Linear(64 * self.z_dim**2, n_latent)
        self.z_develop = nn.Linear(n_latent, 64 * self.z_dim**2)
        self.decoder = nn.Sequential(
            nn.ConvTranspose2d(64, 32, kernel_size=3, stride=2, padding=0),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.ConvTranspose2d(32, 1, kernel_size=3, stride=2, padding=1),
            CenterCrop(h,w),
            nn.Sigmoid()
        )

    def sample_z(self, mean, logvar):
        stddev = torch.exp(0.5 * logvar)
        noise = Variable(torch.randn(stddev.size()))
        return (noise * stddev) + mean

    def encode(self, x):
        x = self.encoder(x)
        x = x.view(x.size(0), -1)
        mean = self.z_mean(x)
        var = self.z_var(x)
        return mean, var

    def decode(self, z):
        out = self.z_develop(z)
        out = out.view(z.size(0), 64, self.z_dim, self.z_dim)
        out = self.decoder(out)
        return out

    def forward(self, x):
        mean, logvar = self.encode(x)
        z = self.sample_z(mean, logvar)
        out = self.decode(z)
        return out, mean, logvar

```

--------------------------------

### Calculate Gradient Descent Update for Weights and Bias

Source: https://ml-cheatsheet.readthedocs.io/en/latest/linear_regression

This function calculates the partial derivatives of the cost function with respect to weight and bias, then updates these parameters using gradient descent. It iterates through the dataset to compute the average gradient and applies the learning rate to adjust the weight and bias, aiming to minimize the cost.

```python
def update_weights(radio, sales, weight, bias, learning_rate):
    weight_deriv = 0
    bias_deriv = 0
    companies = len(radio)

    for i in range(companies):
        # Calculate partial derivatives
        # -2x(y - (mx + b))
        weight_deriv += -2*radio[i] * (sales[i] - (weight*radio[i] + bias))

        # -2(y - (mx + b))
        bias_deriv += -2*(sales[i] - (weight*radio[i] + bias))

    # We subtract because the derivatives point in direction of steepest ascent
    weight -= (weight_deriv / companies) * learning_rate
    bias -= (bias_deriv / companies) * learning_rate

    return weight, bias
```

--------------------------------

### Numerical Derivative Calculation (Python)

Source: https://ml-cheatsheet.readthedocs.io/en/latest/calculus

Computes the numerical derivative of a given function at a specific point using a small step size 'h'. This method approximates the slope of the tangent line. It requires a function and the point of evaluation as input.

```python
def get_derivative(func, x):
    """Compute the derivative of `func` at the location `x`."""
    h = 0.0001                          # step size
    return (func(x+h) - func(x)) / h    # rise-over-run

def f(x): return x**2                   # some test function f(x)=x^2
x = 3                                   # the location of interest
computed = get_derivative(f, x)
actual = 2*x

computed, actual   # = 6.0001, 6        # pretty close if you ask me...

```

--------------------------------

### Train Model using Gradient Descent Iterations

Source: https://ml-cheatsheet.readthedocs.io/en/latest/linear_regression

This function trains a model by iteratively updating weights and biases using the `update_weights` function. It calculates the cost after each iteration and logs the progress, including weight, bias, and cost values. The training loop continues for a specified number of iterations, storing the cost history for evaluation.

```python
def train(radio, sales, weight, bias, learning_rate, iters):
    cost_history = []

    for i in range(iters):
        weight,bias = update_weights(radio, sales, weight, bias, learning_rate)

        #Calculate cost for auditing purposes
        cost = cost_function(radio, sales, weight, bias)
        cost_history.append(cost)

        # Log Progress
        if i % 10 == 0:
            print "iter={:d}    weight={:.2f}    bias={:.4f}    cost={:.2}".format(i, weight, bias, cost)

    return weight, bias, cost_history
```
Tokens:
Raw