Intro

READ THE REPORT HERE

The starting points for this notebook that you should look at are this README.md, the end_to_end.ipynb file where we detail everything, and train.py. We DO NOT expect you to train the model yourself. Instead we explain how we perform the training with train.py and simply refer to the saved model weights in the notebook. You can additionally run a demo (see the disclaimer about the origin of the code in demo.py at the bottom of README.md) with python demo.py.

Setup Instructions

NOTE: While I would love to provide a docker container, it is impossible to do with a HPC due to security concerns with root priviledge (read more about it here). Instead, you will have to set up your own environment. Also note that I can't come up with a containerized set-up approach to setting things up as the requirements change based on your hardware capabilities.

1. Clone the repository.
2. Visit this Kaggle page, download the dataset, and extract the data into a folder you must name /data/. WARNING: please ensure that you clear enough space (about 9 GBs). The dataset will be quite large. It should look like this when you are done:

root_directory
   |- ...
   |- data
      |- asl_alphabet_test
         |- data...
      |- asl_alphabet_train
         |- data...
   |- ...

3. Have Anaconda and create a conda environment named ASLenv: conda create --name ASLenv python=3.10 and conda activate ASLenv

4. Install dependencies:

   Iff using Nvidia GPU w/ Windows - tested on Windows:

   • Iff GPU: Install CudaNN 8.6 and cuda-toolkit v11.8
   • Iff GPU: Update your Nvidia driver to at least version 5xx.
   • Visit pypi.org to download tensorflow-gpu v2.10 .whl file
   • Run: pip install tensorflow_gpu-2.10.0-cp310-cp310-win_amd64.whl from whatever directory you downloaded to.
   • pip install -r ./dependencies/GPUrequirements.txt

   Elif using MacOS:

   • Navigate to home directory and exit from any current conda environment
   • Download miniforge to install the package for a workaround for conda on M1 Macs.
   • Install miniforge: sh ~/Downloads/Miniforge3-MacOSX-arm64.sh
   • Activate miniforge: source ~/miniforge3/bin/activate
   • Install apple-specific TF dependencies: conda install -c apple tensorflow-deps==2.9.0
   • Create a new conda environment: conda create --name ASLenv python=3.10
   • Acitvate the environment: conda activate ASLenv
   • Install additional dependencies:
     • python -m pip install tensorflow-macos==2.9.0
     • python -m pip install tensorflow-metal==0.5.0
   • Enter project directory in ASLenv and run: pip install -r ./dependencies/CPUrequirements.txt
   • Visit pypi.org to download tensorflow-macos v2.10 .whl file. Download either x86 or ARM64 depending on your device.
   • Run:
     • For M1/apple silicon: pip install tensorflow_macos-2.10.0-cp310-cp310-macosx_12_0_arm64.whl
     • For intel silicon: pip install tensorflow_macos-2.10.0-cp310-cp310-macosx_12_0_x86_64.whl
   • Finally, install jupyter notebook deps: python -m pip install ipykernel
   • Additional Mac Tensorflow install troubleshooting resources:
     • Apple Forum
     • Medium Article
     • YouTube Video

   Else:

   • pip install -r ./dependencies/CPUrequirements.txt

Training

Note that to perform training, you will need to:

1. Have access to decent hardware. I strongly encourage you to work with a device that has a GPU. [If running on HPC]: SLURM for managing and scheduling Linux clusters + CudaToolKit and cudnn availability with HPC resources. You will need to adapt the .sbatch file to your specific needs.
2. Visit comet-ml and create an account for free. This will allow you to easily track training progress on HPCs or on your own device from a web interface! No need to check log outputs.
3. For safety reasons, get your API key from comet-ml and in your own environment create your environment variable named "COMET_API_KEY". You may need a restart for this to take effect.
4. Replace my workspace "joeshmoe03" with yours in train.py.

Arguments

- Number of epochs:       "-nepoch 3"
- Batch size:             "-batchSize 32"
- Learning rate:          "-lr 0.001"
- Resampling data split:  "-resample 3"
- Weight decay:           "-wd 0"
- Momentum:               "-mo 1.00"
- Model:                  "-model vgg", "-model resnet" (or any model you implement in ./model/ and incorporate into the ModelFactory class in ./util/model.py)
- Optimizer:              "-optim adam", "-optim sgd"
- Loss function:          "-loss sparse_categorical_cross_entropy", "-loss categorical_cross_entropy", anything you implement...
- Stopping after no imp.  "-stopping 10"
- Val perc.               "-val 0.2"
- Data directory          "-data_dir ./some/path/to/data/" (No reason to do this unless you want to recycle or completely modify repo)
- Pretraining             "-pretrain ./path/to/weights" (model weights should be stored under ./model/weights/X.weights.h5 where X is model name like "vgg" or "resnet", etc...)
- metric                  "-metric accuracy precision f1_score recall" or any combination of them. Does not work with sparse_categorical_cross_entropy loss.
- color                   "-color rgb" or "-color grayscale"
- logits                  "-logits True" or "-logits False" Adds softmax.
- img_size                "-img_size 200" ...

Examples:

python train.py -nepoch 1 -batchSize 64 -lr 0.001 -metric accuracy precision recall f1_score

python train.py -nepoch 1 -batchSize 32 -lr 0.001 -metric accuracy precision recall f1_score -model convnet4

Repo Structure

• train.sbatch: batch file for submitting training and other high compute jobs to HPC Slurm scheduler.
• train.py: a script version that is runnable in our batch file for high performance computer training.
• end_to_end.ipynb: the end-to-end jupyter notebook for our project. This should be the FIRST thing you look at.
• exploratory_analysis.ipynb: the DRAFT working file for all visualizations.
• model_experimentation.ipynb: the DRAFT working file for testing and making models.
• util: a directory containing important utility that will be used to efficiently and effectively adapt our code.
• model: a directory storing tf models and their weights if interested in pretraining. Pretrain weights should be saved to ./model/weights/VGGweights.h5 and models as ./model/VGG.py for example.
• figures: containing visualizations made from notebooks or with comet_ml.
• dependencies: folder containing requirements.txt for setting up environment.
• temp: all saved models and their associated tracked metric performances.
• demo.py: PLEASE NOTE THAT THIS IS LARGLEY NOT OUR OWN DEMO FILE. WE ADAPT IT FROM HERE AND PLUG IN OUR OWN CLASSIFIER. THE CORRESPONDING YOLO WEIGHTS IN ./temp/ ARE NOT OUR OWN. CREDIT TO Florian Bruggisser, djthegr8 DJ, and roachsinai RoachZhao. To run: python demo.py

Helpful Notes

For GPU users, here is the video that finally got my tensorflow-gpu working

References

See the bottom of end_to_end.ipynb