A lightweight implementation of LipNet, a deep-learning model for lip-reading from video.
This version focuses on simplicity by using less speaker than original paper while demonstrating core concepts:
video preprocessing, spatiotemporal feature extraction, and sequence prediction.
- Input the data
- Convert it to grayscale
- Crop the mouth features
- Normalization of data
This is a frame-frame grayscaled image of the mouth-features
Input Shape: (75,46,140,1)
Kernel Size: 3x3x3
Regularization: Dropout (0.5) and Batch Normalization
Everything needed to preprocess data, train the lip-reader, and launch the demo lives under Code/. From the project root:
cd Code
#create virtual env
python3 -m venv .venv
source .venv/bin/activate
# install dependencies listed by the team
pip install -r requirements.txt
# preprocess raw videos into mouth-only frame tensors + transcripts
python3 data-preprocessing.py # edit paths inside as needed
# train the LipNet-style model (expects processed_data/*_frames.npy)
python3 Train_LipReader.py
# run the interactive demo app once a checkpoint exists in models/
cd app
streamlit run app.py