fmwork Tooling Enhancements: Monitoring, Benchmarking, and Analysis #24
Description
Background
We need to enhance several utilities and features in our fmwork toolset to improve experiment capabilities, monitoring, and input options. These enhancements will make experiments more comprehensive, provide better insights, and support more diverse testing scenarios.
Overview of Required Enhancements
Monitoring Tools
- Memory Footprint: Add utilities to track memory usage during experiments
- Power Readouts: Implement power consumption monitoring
Experiment Workflow Improvements
- Benchmarking Loop: Add randomization options for sequences per repetition
- Input Options: Improve handling of synthetic and real datasets
Model-Specific Analytics
- MoE Model Statistics: Track expert triggering distribution for Mixture of Experts models