[WIP] Upgrades

[Erotemic]

I'm working on improving the repo so it can be used seemlessly with arbitrary COCO manifests (currently using the kwcoco package, but that can be factored out).

I'm also making other changes / fixing issues / style as I go along. I'm attempting to keep the git commit messages clean so pieces of this can be broken off if they want to be merged upstream. But my end goal is to just make this usable as a command line tool that doesn't requires messing with config files and putting them in the right place.

[henrytsui000]

Hi,

Apologies for the delay—I’m currently serving my country’s mandatory military duty, so code reviews will be slower than usual. I’ll gradually review the code over the coming days. I can still check my email daily, so feel free to reach out if anything is urgent.

Best regards,
Henry Tsui

[Erotemic]

Of course! Most of my work on this is as a hobby project, so I'm not available all the time either.

That being said, this PR has gotten very big and should be broken down into smaller more manageable pieces for review. In my most recent work, I've been less disciplined about keeping things non-experimental, so some commits are changing hard-coded defaults. Those experimental sections should not be merged.

To move forward it would be best to prioritize a list of features added in this branch to separate into a standalone branch. Reviewing what I have done so far could help build that list. In the meantime here is a list of what I think some of the more important features added here are:

• Support for kwcoco, which makes it much easier to train on a custom dataset. The new file train_kwcoco_demo.sh provides an end-to-end example that generates a dataset, trains on it, predicts the new model, and then evaluates it.

• Exposing all options for lightning's Trainer. I did this in a more recent commit, and it makes a big difference. Giving the user access to accumulate_batches and how to handle gradient clipping makes this repo applicable to many more problems.

• Remove the dependency on opencv-python. For reasons I've discussed in other locations: Add cv2 graphics / headless extras to setup.py open-mmlab/mmcv#2775

• Readd HorizontalFlip and VerticalFlip augmentations, but default them to zero.

• Write the training config yaml to the training directory (answers the question: what model do these weights correspond to?)

• Fix an issue with loading pretrained weights, and handle more cases where the model has changed slightly (By including torch-liberator we can do even better than this!)

• Adding more logging statements

• Improve the output of forward functions to be a dictionary instead of a tuple (which makes them much easier to change in the future)

• Fix issues with logging (at least when wandb is disabled)

• Visualize training and validation batches on disk (needs a little cleanup, but is getting there, and very useful for debugging)

• Visualize tensorboard graphs on disk

• Support for non-SGD optimizers (I have AdamW working, but the custom scheduler code seems not to work right yet).

• Customize Trainer to leverage CUDA cores on 3090 GPUs.

Let me know which of these you want to prioritize and I'll split them into separate PRs. Feel free to give any general reviews here.

I do have one minimal open PR here: #160

[salvaba94]

Hi @Erotemic , did you manage to reproduce the results of the original YOLOv9 with this set of upgrades?

[Erotemic]

I haven't tried, but I also didn't do anything to modify algorithm behavior or defaults. Just quality of life improvements.