diff --git a/README.md b/README.md index 4df529d..e9b6378 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,21 @@ DropGrad is a regularization method for neural networks that works by randomly (and independently) setting gradient values to zero before an optimization step. Similarly to Dropout, it has a single parameter, `drop_rate`, the probability of setting each parameter gradient to zero. In order to de-bias the remaining gradient values, they are divided by `1.0 - drop_rate`. +## Table of Contents + + * [Features](#features) + * [What's New in Version 0.3.5?](#whats-new-in-version-035) + * [Directory Structure](#Directory-Structure) + * [Installation](#installation) + * [Usage](#usage) + * [Examples](#examples) + * [Testing](#testing) + * [Analysis](#analysis) + * [Windows CUDA Setup](#windows-cuda-setup) + * [Contributing](#contributing) + * [License](#license) + * [Star History](#star-history) + ## Features - Simple and easy-to-use gradient regularization technique @@ -15,7 +30,7 @@ DropGrad is a regularization method for neural networks that works by randomly ( - Utilizes mixed-precision training for improved performance and memory efficiency (CUDA devices only) - Cross-platform compatibility: Works seamlessly on macOS, Windows, and Linux -## Updates in Version 0.3.5 +## What's New in Version 0.3.5? - Added support for the Lion optimizer in the ViT experiments - Implemented gradient clipping to prevent gradient explosion and improve training stability @@ -24,44 +39,72 @@ DropGrad is a regularization method for neural networks that works by randomly ( - Updated test suite to cover various aspects of DropGrad, including initialization, optimization step, drop rate scheduling, and saving of loss values - Code refactoring and documentation enhancements for better readability and maintainability -## Code Structure - -``` -dropgrad/ -│ -├── docs/ -│ ├── analysis.md -│ └── windows_cuda_setup.md -│ -├── dropgrad/ -│ ├── __init__.py -│ ├── dropgrad_opt.py -│ └── dropgrad_scheduler.py -│ -├── examples/ -│ ├── basic_usage.py -│ ├── lr_scheduler_integration.py -│ ├── full_update_drop.py -│ └── vit_experiments/ -│ ├── vit_model.py -│ ├── train.py -│ ├── visualize.py -│ ├── mathematical_analysis.py -│ ├── benchmark_visualizations.py -│ └── *.pth -│ -├── tests/ -│ ├── __init__.py -│ ├── test_dropgrad.py -│ ├── test_dropgrad_optimizer.py -│ └── test_dropgrad_scheduler.py -│ -├── .gitignore -├── LICENSE -├── pyproject.toml -├── README.md -└── requirements.txt -``` +## Directory Structure + + + + + + + + + + + + + + + + + + + + + +
Description Quick Access

Getting Started

+The examples directory contains sample code demonstrating various use cases of DropGrad, including basic usage, integration with learning rate schedulers, applying full update drop, and training a Vision Transformer (ViT) on the CIFAR-10 dataset under different regularization scenarios. +
+└── examples
+    ├── basic_usage.py
+    ├── lr_scheduler_integration.py
+    ├── full_update_drop.py
+    └── vit_experiments
+        ├── vit_model.py
+        ├── train.py
+        ├── visualize.py
+        ├── mathematical_analysis.py
+        ├── benchmark_visualizations.py
+        └── *.pth
+

Documentation

+The docs directory contains detailed documentation and analysis of the DropGrad method, as well as instructions for setting up CUDA on Windows for PyTorch and DropGrad. +
+└── docs
+    ├── analysis.md
+    └── windows_cuda_setup.md
+

Core DropGrad Implementation

+The dropgrad directory contains the core implementation of the DropGrad optimizer and drop rate schedulers. +
+└── dropgrad
+    ├── __init__.py
+    ├── dropgrad_opt.py
+    └── dropgrad_scheduler.py
+

Testing

+The tests directory contains the test suite for DropGrad, ensuring the correctness of the implementation. The tests cover the functionality of the DropGrad optimizer and the drop rate schedulers. +
+└── tests
+    ├── __init__.py
+    ├── test_dropgrad.py
+    ├── test_dropgrad_optimizer.py
+    └── test_dropgrad_scheduler.py
+

Configuration and Setup

+This section highlights the key files related to project configuration, requirements, and licensing. +
+├── .gitignore
+├── LICENSE
+├── pyproject.toml
+├── README.md
+└── requirements.txt
+
## Installation