Skip to content

Latest commit

 

History

History
53 lines (42 loc) · 2.38 KB

README.md

File metadata and controls

53 lines (42 loc) · 2.38 KB
Raw

ScratchNN_DigitMNIST

NN for Digit Recognizer (MNIST). Created using only Numpy and Math

Built a simple neural network from scratch using only NumPy, without relying on any high-level libraries like TensorFlow, PyTorch, or Keras. The task was to classify digits from the MNIST dataset, which contains images of handwritten digits.

What I learned

  • Get to know MNIST: Basic dataset for learning computer vision, and one of the MNIST is the digit recognizer
  • Numpy syntax and how to use it
  • Visualizing math formula for forward/backward pass including each matrix/vector size
  • Converting math formula/equation into syntax
  • Creating basic network architecture without high-level libraries
  • Making function of weight initialization, forward pass, gradient descent, backward pass, update parameter, etc in python
  • Softmax and ReLU function and its derivative

Update

I just found out that torch and numpy have similar syntax, so I try to create another version with torch and with some support to GPU compute (Which overkill lol)

Similarities:

  • np.array ; torch.tensor
  • np.random.randn ; torch.randn
  • np.maximum ; torch.max
  • np.exp ; (tensor variable).exp()
  • np.dot ; using @ syntax to do matrix mul
  • np.sum ; (tensor var).sum() -> the keepdim parameter is important, and the axis too
  • np.argmax ; torch.argmax

Add on notes:

  • Torch has integrated one hot encoding function (torch.nn.Functional.one_hot)

  • Shuffling tensor is not straight forward like using np.shuffle

  • Always check the shape of the tensor

  • Tried weight init using normal distribution numbers (randn) and it caused Vanishing/Exploding Gradient

  • Better Initialization Techniques (Fixing Vanishing/Exploding Gradient Problem)

     Xavier/Glorot Initialization:

 For tanh activation: W = np.random.randn(shape) * np.sqrt(1 / n_in)
 For sigmoid activation: W = np.random.randn(shape) * np.sqrt(2 / (n_in + n_out))


 He Initialization (good for ReLU activations):
 W = np.random.randn(shape) * np.sqrt(2 / n_in)
 Scaled Uniform Initialization:
 W = np.random.rand(shape) * 2 - 1) * scale_factor

 Where:

 n_in is the number of input neurons
 n_out is the number of output neurons
 scale_factor is a small number (e.g., 0.1)

Coded and Created by Han Summer 2024

Part of The 20th Summer Project