https://github.com/willyfh/batch-size-vs-iteration

A small experiment to investigate the relationship between batch size and number of iterations
https://github.com/willyfh/batch-size-vs-iteration

Last synced: 5 days ago
JSON representation

A small experiment to investigate the relationship between batch size and number of iterations

• Host: GitHub
• URL: https://github.com/willyfh/batch-size-vs-iteration
• Owner: willyfh
• Created: 2023-08-07T23:32:05.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2023-08-08T00:01:09.000Z (over 1 year ago)
• Last Synced: 2023-08-08T00:52:37.639Z (over 1 year ago)
• Size: 2.93 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# The Relationship of Batch Size and Number of Iterations

In order to achieve a similar performance when using a different batch size, gradient accumulation is often applied. However, there is also an assumption that adjusting the number of epochs (to maintain the number of iterations accross different batch sizes) can also help to achieve a similar performance.

This small experiment attempts to address the following questions:

1) Do we need to increase the number of epochs, i.e., fix the number of iterations, as we increase the batch size?

2) Or do we need to fix the number of epochs, i.e., decrease the number of iterations, as we increase the batch size?

Note: While other hyperparameters, such as the learning rate, may also impact the results, I maintain consistent values for these parameters across all experiments.

## Dataset

## Architecture

## Results