Tracking video shown on social media

[ChrisDotWDot]

Hi @fkryan,

thank you all for this interesting project :)

I have gone through all the demos, e.g., the Colab Demo Notebook; got the AUC, Avg L2 and Min L2 for the GazeFollow dataset: (AUC = 0.9584406567323296; Avg L2: 0.0984195497739822; Min L2: 0.04108351483186137), but I am wondering how you generated the video of the three actors in 'The office' - i.e., the video clip (not the .png).

Thanks

[wangduhe]

我也想问一下这个问题，

[fkryan]

Hi,

I didn't actually perform tracking to generate the demo video, although you certainly could run a tracker if that's important for your use case. I generated the video by running a Yolov5m model trained on CrowdHuman to detect the heads (https://github.com/MahenderAutonomo/yolov5-crowdhuman), and the colors of the bounding box are simply ordered left to right.

The only difference from the Colab demo notebook is that I used OpenCV instead of matplotlib to draw the visualization of the bounding boxes and gaze points onto each frame. I saved the video locally, then used OpenCV to read it frame by frame. For each frame, I detected the heads, ran Gaze-LLE inference with our gazelle_large_vitl14_inout model, then wrote the visualization to an output mp4.

Here's the visualization code I ran on each frame:

colors = [(0, 255, 0), (255, 0, 0), (0, 0, 255)]
p = 0
for bbox, gaze_point in zip(bboxes, gaze_points):
  x1, y1, x2, y2 = bbox
  cv2.rectangle(frame, (int(x1), int(y1)), (int(x2), int(y2)), colors[p%3], 2)
  if gaze_point is not None:
    gaze_x, gaze_y = gaze_point
    cv2.circle(frame, (int(gaze_x), int(gaze_y)), 10, colors[p%3], -1)
    cv2.line(frame, (int(gaze_x), int(gaze_y)), ((int(x1) + int(x2)) // 2, (int(y1) + int(y2)) // 2), colors[p%3], 2)
  p += 1

If you would like, I can push the full visualization script to this repository, but I'll need to clean up the code and replace local paths and stuff first.

[ChrisDotWDot]

Hi @fkryan,

If you would like, I can push the full visualization script to this repository....

That would be terrific!
My use case is getting a handle on where people look in conversations - and how this might change under different listening conditions/settings. Your visualization is such a neat way of demonstrating the essence of this- and showing the functionality of Gaze-LLE.

All the best and thanks for the wonderfully prompt reply 👍

[ChrisDotWDot]

Hi again @fkryan,

based on your response, I had a go at modifying the Colab demo - I used another clip from 'The office' that had three interlocutors - seemed to work - code below....

Many thanks
: )

import` torch
import cv2
import numpy as np
from retinaface import RetinaFace
from PIL import Image

# Device setup
device = 'cuda' if torch.cuda.is_available() else 'cpu'

# Load model
model, transform = torch.hub.load('fkryan/gazelle', 'gazelle_dinov2_vitl14_inout')
model.eval()
model.to(device)

def process_frame(frame, model, transform):
    """Process a single frame through the gaze detection pipeline"""
    # Convert frame to PIL Image for transform
    pil_frame = Image.fromarray(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))
    height, width = frame.shape[:2]
    
    # Detect faces
    resp = RetinaFace.detect_faces(frame)
    if not isinstance(resp, dict):
        return frame  # Return original frame if no faces detected
    
    bboxes = [resp[key]['facial_area'] for key in resp.keys()]
    
    # Prepare input for the model
    img_tensor = transform(pil_frame).unsqueeze(0).to(device)
    norm_bboxes = [[np.array(bbox) / np.array([width, height, width, height]) for bbox in bboxes]]
    
    input_data = {
        "images": img_tensor,
        "bboxes": norm_bboxes
    }
    
    # Get model predictions
    with torch.no_grad():
        output = model(input_data)
    
    # Draw visualizations
    colors = [(0, 255, 0), (255, 0, 0), (0, 0, 255)]  # BGR format for OpenCV
    
    for p, (bbox, heatmap) in enumerate(zip(bboxes, output['heatmap'][0])):
        # Draw bounding box
        x1, y1, x2, y2 = bbox
        cv2.rectangle(frame, (int(x1), int(y1)), (int(x2), int(y2)), colors[p % 3], 2)
        
        # Get gaze point from heatmap
        if output['inout'] is not None and output['inout'][0][p].item() > 0.5:
            heatmap_np = heatmap.detach().cpu().numpy()
            max_index = np.unravel_index(np.argmax(heatmap_np), heatmap_np.shape)
            
            # Calculate gaze target coordinates
            gaze_x = int(max_index[1] / heatmap_np.shape[1] * width)
            gaze_y = int(max_index[0] / heatmap_np.shape[0] * height)
            
            # Calculate face center
            face_center_x = (int(x1) + int(x2)) // 2
            face_center_y = (int(y1) + int(y2)) // 2
            
            # Draw gaze point and line
            cv2.circle(frame, (gaze_x, gaze_y), 10, colors[p % 3], -1)
            cv2.line(frame, (gaze_x, gaze_y), (face_center_x, face_center_y), colors[p % 3], 2)
            
            # Add confidence score
            conf_text = f"conf: {output['inout'][0][p].item():.2f}"
            cv2.putText(frame, conf_text, (int(x1), int(y1) - 10),
                       cv2.FONT_HERSHEY_SIMPLEX, 0.5, colors[p % 3], 2)
    
    return frame

def process_video(input_path, output_path):
    """Process entire video and save visualization"""
    # Open video file
    cap = cv2.VideoCapture(input_path)
    if not cap.isOpened():
        print(f"Error: Could not open video file {input_path}")
        return
    
    # Get video properties
    width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
    height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
    fps = int(cap.get(cv2.CAP_PROP_FPS))
    total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
    
    # Create video writer
    fourcc = cv2.VideoWriter_fourcc(*'mp4v')
    out = cv2.VideoWriter(output_path, fourcc, fps, (width, height))
    
    frame_count = 0
    
    while cap.isOpened():
        ret, frame = cap.read()
        if not ret:
            break
        
        # Process frame
        processed_frame = process_frame(frame, model, transform)
        
        # Write frame
        out.write(processed_frame)
        
        # Display progress
        frame_count += 1
        if frame_count % 10 == 0:  # Update progress every 10 frames
            progress = (frame_count / total_frames) * 100
            print(f"\rProcessing: {progress:.1f}% complete", end="")
    
    print("\nProcessing complete!")
    
    # Release resources
    cap.release()
    out.release()

def main():
    input_video = "output.mp4"
    output_video = "output_viz.mp4"
    
    print(f"Processing video: {input_video}")
    print(f"Output will be saved to: {output_video}")
    
    process_video(input_video, output_video)

if __name__ == "__main__":
    `main()

[msabers]

Hi again @fkryan,

based on your response, I had a go at modifying the Colab demo - I used another clip from 'The office' that had three interlocutors - seemed to work - code below....

Many thanks : )

import` torch
import cv2
import numpy as np
from retinaface import RetinaFace
from PIL import Image

# Device setup
device = 'cuda' if torch.cuda.is_available() else 'cpu'

# Load model
model, transform = torch.hub.load('fkryan/gazelle', 'gazelle_dinov2_vitl14_inout')
model.eval()
model.to(device)

def process_frame(frame, model, transform):
    """Process a single frame through the gaze detection pipeline"""
    # Convert frame to PIL Image for transform
    pil_frame = Image.fromarray(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))
    height, width = frame.shape[:2]
    
    # Detect faces
    resp = RetinaFace.detect_faces(frame)
    if not isinstance(resp, dict):
        return frame  # Return original frame if no faces detected
    
    bboxes = [resp[key]['facial_area'] for key in resp.keys()]
    
    # Prepare input for the model
    img_tensor = transform(pil_frame).unsqueeze(0).to(device)
    norm_bboxes = [[np.array(bbox) / np.array([width, height, width, height]) for bbox in bboxes]]
    
    input_data = {
        "images": img_tensor,
        "bboxes": norm_bboxes
    }
    
    # Get model predictions
    with torch.no_grad():
        output = model(input_data)
    
    # Draw visualizations
    colors = [(0, 255, 0), (255, 0, 0), (0, 0, 255)]  # BGR format for OpenCV
    
    for p, (bbox, heatmap) in enumerate(zip(bboxes, output['heatmap'][0])):
        # Draw bounding box
        x1, y1, x2, y2 = bbox
        cv2.rectangle(frame, (int(x1), int(y1)), (int(x2), int(y2)), colors[p % 3], 2)
        
        # Get gaze point from heatmap
        if output['inout'] is not None and output['inout'][0][p].item() > 0.5:
            heatmap_np = heatmap.detach().cpu().numpy()
            max_index = np.unravel_index(np.argmax(heatmap_np), heatmap_np.shape)
            
            # Calculate gaze target coordinates
            gaze_x = int(max_index[1] / heatmap_np.shape[1] * width)
            gaze_y = int(max_index[0] / heatmap_np.shape[0] * height)
            
            # Calculate face center
            face_center_x = (int(x1) + int(x2)) // 2
            face_center_y = (int(y1) + int(y2)) // 2
            
            # Draw gaze point and line
            cv2.circle(frame, (gaze_x, gaze_y), 10, colors[p % 3], -1)
            cv2.line(frame, (gaze_x, gaze_y), (face_center_x, face_center_y), colors[p % 3], 2)
            
            # Add confidence score
            conf_text = f"conf: {output['inout'][0][p].item():.2f}"
            cv2.putText(frame, conf_text, (int(x1), int(y1) - 10),
                       cv2.FONT_HERSHEY_SIMPLEX, 0.5, colors[p % 3], 2)
    
    return frame

def process_video(input_path, output_path):
    """Process entire video and save visualization"""
    # Open video file
    cap = cv2.VideoCapture(input_path)
    if not cap.isOpened():
        print(f"Error: Could not open video file {input_path}")
        return
    
    # Get video properties
    width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
    height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
    fps = int(cap.get(cv2.CAP_PROP_FPS))
    total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
    
    # Create video writer
    fourcc = cv2.VideoWriter_fourcc(*'mp4v')
    out = cv2.VideoWriter(output_path, fourcc, fps, (width, height))
    
    frame_count = 0
    
    while cap.isOpened():
        ret, frame = cap.read()
        if not ret:
            break
        
        # Process frame
        processed_frame = process_frame(frame, model, transform)
        
        # Write frame
        out.write(processed_frame)
        
        # Display progress
        frame_count += 1
        if frame_count % 10 == 0:  # Update progress every 10 frames
            progress = (frame_count / total_frames) * 100
            print(f"\rProcessing: {progress:.1f}% complete", end="")
    
    print("\nProcessing complete!")
    
    # Release resources
    cap.release()
    out.release()

def main():
    input_video = "output.mp4"
    output_video = "output_viz.mp4"
    
    print(f"Processing video: {input_video}")
    print(f"Output will be saved to: {output_video}")
    
    process_video(input_video, output_video)

if __name__ == "__main__":
    `main()

Thanks for sharing. Would it be possible for you to share the version of libraries you used? I have been trying to run it, but each time I receive a new conflicting issue.

[ChrisDotWDot]

..Would it be possible for you to share the version of libraries you used?

1. Create base environment with Python 3.9 - I didn't use the .yml

conda create -n gazelle python=3.9 -y

2. Activated it

conda activate gazelle

3. Installed PyTorch

conda install pytorch torchvision torchaudio cpuonly -c pytorch -y

4. Installed the core 'scientific' packages

conda install numpy pandas matplotlib scikit-learn scipy pillow -y

5. Installed the other conda packages

conda install -c conda-forge pyyaml requests tqdm joblib fsspec -y

6. Installed all pip packages

pip install tensorflow==2.18.0 keras==3.7.0 opencv-python==4.10.0.84 retina-face==0.0.17 gdown==5.2.0 timm huggingface-hub beautifulsoup4 rich safetensors

#needed tf-keras for TensorFlow 2.18+ to work with older Keras-based packages like retinaface.
pip install tf-keras