An offline, configuration-driven PyTorch project for the competition 面向城市场景的视觉多模态目标检测. It implements RGB, infrared, and uint16 depth data auditing; leakage-aware group splits; three single-model FCOS variants; independent official-style evaluation; robustness ablations; inference; and strict flat-ZIP submission validation.
No model weight is downloaded automatically. No code path trains on test data. TTA, checkpoint soup, checkpoint averaging, multi-model voting, multi-model WBF, and test-set self-training are absent or fail closed.
Each sample is an exactly basename-aligned tuple:
- RGB: 3-channel uint8 PNG in
[0,255]. - Infrared: normally 3-channel uint8 PNG with nearly identical channels; loaded unchanged and reduced to one channel according to YAML.
- Depth: single-channel uint16 PNG in millimetres. It is always read unchanged. Values outside the configurable
[300,20000]mm range are invalid. - Training label: every non-empty line must contain exactly
class_id center_x center_y width heightin normalized coordinates. A four-field line is rejected with filename and line number; it is never repaired.
The 12 classes are person, boat, animal, seat, sign, bicycle, car, ball, light, garbage can, uav, tricycle, IDs 0-11.
Each prediction line is class_id center_x center_y width height confidence. Every expected test basename receives a TXT, including an empty file when there is no prediction. There are at most 100 boxes per image; all numbers are finite and in [0,1]; ZIP members are TXT files at archive root.
The model-selection metric is the independent implementation of COCO-style mAP@50-95: IoU 0.50:0.05:0.95, global confidence sorting per class, one-to-one same-image/same-class matching, precision envelope, and 101 recall points. Classes with no GT are reported as None and ignored rather than silently treated as zero. Small/medium/large APs are diagnostic COCO area bins; the official competition score is only map50_95.
The current ML environment was verified with Python 3.13.14, PyTorch 2.12.0, torchvision 0.27.0, and OpenCV 5.0.0. CUDA is unavailable; MPS is available in a normal host process and has completed real-data forward/backward tests at both 64 and 768 px. A restricted sandbox can incorrectly report mps_available=False, so verify with an actual MPS tensor allocation. The implementation targets Python 3.8+ and PyTorch 2.x.
The active remote CUDA runner was separately verified with Python 3.12.3, PyTorch 2.8.0+cu128, torchvision 0.23.0+cu128, OpenCV 5.0.0, and CUDA 12.8. Local and remote test evidence is recorded separately in runs/auto_loop_state.json; neither environment's result is presented as evidence for the other.
The detector is torchvision's mature anchor-free FCOS with its decoupled classification/regression/centerness head, FPN, focal classification loss, IoU-family regression loss, and class-aware NMS. It is wrapped by DetectorAdapter, so backend inputs remain a dictionary of separate modalities until the adapter.
A pinned optional RGB backend, YOLO11l through ultralytics==8.4.115, is
available only for the pre-registered E9 backend screen. It is anchor-free,
uses a decoupled DFL head, accepts only a local checkpoint, and remains
isolated from the tri-modal adapter until two group folds prove a material
gain. It is not silently substituted for the configured FCOS models.
Model variants:
rgb_fcos: RGB ResNet-FPN FCOS baseline.early_fusion_6ch_fcos:[RGB3, IR1, log_depth1, valid_mask1]; the ResNet stem accepts six channels. Local RGB stem weights are copied exactly and new channels start at zero.trimodal_rgf_p2_fcos/ TriModal-RGF-P2: one pretrained-capable RGB ResNet C2-C5 main path, lightweight GroupNorm IR/depth pyramids, C2-C5 reliability-gated residual fusion, torchvision FPN, and FCOS P2-P6 at strides 4/8/16/32/64. Disabling P2 produces the conventional P3-P7 pyramid; P2 never replaces P6.
RGF keeps an exact RGB identity residual. IR/depth contributions are multiplied by explicit availability after sigmoid gating; depth is additionally multiplied by the downsampled validity mask. Learnable residual strengths start at 0.05. Gate spatial/channel tensors are available for visualization. There are no DCNv2/custom CUDA extensions and global cross-attention is disabled.
With the formal ResNet50 configs, measured parameter counts are 34,136,505 for RGB and 35,977,369 for strong, a ratio of 1.054 (<1.8). On the active RTX 5090 runner, the hook estimates are 80.42 GMAC for RGB at its configured 640 px and 322.11 GMAC for P2-P6 strong at 768 px. Measured batch-1 FP32 forward latency was 12.98 ms and 23.29 ms respectively over three timed iterations after warmup; these short microbenchmarks are device-specific, not end-to-end dataloader throughput. The labelled parameter/AdamW-plus-activation lower bounds were about 1.38 GB and 2.98 GB, while measured forward allocator peaks were 267 MB and 792 MB. Real training peak still depends on batch size, AMP, optimizer and dataloader memory.
Use an environment that already contains a compatible PyTorch/torchvision pair. The project never upgrades them implicitly:
cd /path/to/aic
python -m pip install -e .
python -m pip install pytest tensorboardFor the verified local environment:
/opt/homebrew/Caskroom/miniconda/base/envs/ml/bin/python -m pip install -e .The optional YOLO runtime must be installed from the retained pinned wheels,
then used with YOLO_OFFLINE=true. Exact wheel provenance and the no-index
install command are in
docs/ULTRALYTICS_OFFLINE_RUNTIME.md.
Copy configs/data.example.yaml and set absolute paths. Directory auto-discovery is only a convenience; if more than one candidate exists it aborts and requires YAML.
train:
rgb_dir: /data/train/RGB
ir_dir: /data/train/Infrared
depth_dir: /data/train/Depth
label_dir: /data/train/labels
manifest: /work/audit_train/manifest.csv
val:
manifest: /work/splits/fold_0_val.csv
test:
rgb_dir: /data/test/RGB
ir_dir: /data/test/Infrared
depth_dir: /data/test/Depth
manifest: /work/audit_test/manifest.csv
depth: {min_mm: 300, max_mm: 20000, resize_mode: nearest, invalid_format_policy: error}
ir: {inconsistent_policy: first, max_channel_difference_warning: 2}
loader: {batch_size: 4, workers: 4, pin_memory: true, runtime_warning_limit: 10}Paths are never hard-coded in Python. RGB stays float [0,1] until the detector transform applies ImageNet normalization. IR is one-channel float [0,1]. Depth becomes [log_depth, valid_mask]; invalid normalized depth is exactly zero. RGB/IR use bilinear geometry, while depth and validity use nearest-neighbour geometry. All geometry parameters are shared by every modality and boxes.
The extracted official archives on this machine use a local configuration copied from configs/data.example.yaml. The audit found:
- Training and Phase-1 test are fully basename-paired: 2000 labelled training tuples and 1000 unlabelled test tuples, with no missing, extra, duplicate, corrupt, or size-mismatched modality.
- Training contains 1851 PNG tuples plus 149 JPEG tuples; test contains 845 PNG tuples plus 155 JPEG tuples. JPEG depth is uint8 three-channel, not recoverable metric uint16 depth.
- The loader never invents millimetres from those JPEGs. With the explicitly configured
depth.invalid_format_policy: unavailable, it supplies zero depth channels andavailability[2]=0, making the depth RGF contribution exactly zero. - Four illegal normalized boxes occur in three label files:
000050.txtlines 27 and 37,003107.txtline 2, and003817.txtline 3. Original labels are not edited.manifest_usable.csvexcludes those three samples. - The training audit therefore has 2000 paired, 1848 strictly compliant, and 1997 usable samples. Class 11 has only 25 valid instances. The Phase-1 test manifest contains all 1000 sample IDs, including all 155 JPEG tuples.
- With IR channel-difference tolerance 2, 1012 training and 426 test images warn. The local YAML deliberately uses
channel_mode: mean; it does not assume the three stored channels are identical.
The complete training-audit, test-audit, and split reports are local generated artifacts under artifacts/; they are excluded from Git and may be regenerated with the commands below.
Audit labelled training data:
python tools/audit_data.py \
--config /absolute/path/data.yaml --split train \
--output-dir /absolute/path/audit_trainAudit unlabelled test data:
python tools/audit_data.py \
--config /absolute/path/data.yaml --split test --no-labels \
--output-dir /absolute/path/audit_testThe audit writes audit_report.json, audit_samples.csv, the complete manifest.csv, fully compliant manifest_strict.csv, and manifest_usable.csv for the explicit missing-depth fallback. It checks exact basename pairing, missing/duplicate/extra files, decode failures, aligned dimensions, RGB/IR/depth dtype and channels, IR channel differences, depth percentiles/valid ratio, every invalid label line, distributions, empty labels, box areas, filename groups, pHash near-duplicate clusters, and group IDs. Any audit error gives a non-zero exit code.
Create three group folds and a fast group holdout:
python tools/make_splits.py \
--manifest /absolute/path/audit_train/manifest_usable.csv \
--output-dir /absolute/path/splits --folds 3 --seed 42 \
--holdout-fraction 0.15Filename sequence and pHash relationships are joined into leakage-safe components. Folds use an approximate stratified group assignment; reports list class/group distributions and missing validation classes. A near-duplicate component cannot cross train/validation. Do not use ordinary frame-wise random splitting for model selection.
The checked competition folds were generated with split seed 42; model training uses seed 3407. Do not regenerate the checked folds with the training seed. The locked local CSV SHA-256 values are 76333093c31846a75174a576ef1afe2ea33cfc2ee715bffbafb38abe6364889f (fold_0_train.csv), 728bf7e0906c5edab49043bdf86db4fac887a97cd35913fa501482c6e99bb6f1 (fold_0_val.csv), c7761c110dee7b8a369639329fa99700129d63eb6c9b968f014babde136dd418 (fold_1_train.csv), and cf55332b76c93ec57b280b3356393a2e219871f9f13c1eaa7487d2243c6fa31b (fold_1_val.csv). Remote copies can have different raw hashes only because their absolute path prefixes are rewritten for the server; sample IDs, groups, labels, and row assignment must stay identical.
All examples below use one model and one checkpoint. Local logging writes CSV, JSON, TensorBoard, environment, model summary, per-class AP, checkpoints, raw validation predictions, and prediction overlays.
RGB baseline:
python tools/train.py \
--config configs/baseline_rgb.yaml --data /absolute/path/data.yaml \
--train-manifest /absolute/path/splits/fold_0_train.csv \
--val-manifest /absolute/path/splits/fold_0_val.csv \
--set model.pretrained=/absolute/path/fcos_resnet50_fpn_coco.pt --device cudaSix-channel baseline:
python tools/train.py \
--config configs/baseline_early_fusion_6ch.yaml --data /absolute/path/data.yaml \
--train-manifest /absolute/path/splits/fold_0_train.csv \
--val-manifest /absolute/path/splits/fold_0_val.csv \
--set model.pretrained=/absolute/path/fcos_resnet50_fpn_coco.pt --device cudaTriModal-RGF-P2:
python tools/train.py \
--config configs/strong_trimodal_rgf_p2.yaml --data /absolute/path/data.yaml \
--train-manifest /absolute/path/splits/fold_0_train.csv \
--val-manifest /absolute/path/splits/fold_0_val.csv \
--set model.pretrained=/absolute/path/fcos_resnet50_fpn_coco.pt --device cudaResume the same run, including optimizer/scheduler/EMA state:
python tools/train.py \
--config configs/strong_trimodal_rgf_p2.yaml --data /absolute/path/data.yaml \
--train-manifest /absolute/path/splits/fold_0_train.csv \
--val-manifest /absolute/path/splits/fold_0_val.csv \
--resume runs/E4_rgf_moddrop_p2/last.pt --device cudaDDP uses PyTorch's launcher; train and validation are sharded by group-manifest rows, validation records are gathered before the independent evaluator, and only rank 0 writes artifacts:
python -m torch.distributed.run --standalone --nproc_per_node=2 tools/train.py \
--config configs/strong_trimodal_rgf_p2.yaml --data /absolute/path/data.yaml \
--train-manifest /absolute/path/splits/fold_0_train.csv \
--val-manifest /absolute/path/splits/fold_0_val.csvThe repeat-factor sampler is also rank-sharded and deterministically reseeded per epoch.
Use a local public pretrained checkpoint without download:
python tools/train.py \
--config configs/strong_trimodal_rgf_p2.yaml --data /absolute/path/data.yaml \
--train-manifest /absolute/path/splits/fold_0_train.csv \
--val-manifest /absolute/path/splits/fold_0_val.csv \
--set model.pretrained=/absolute/path/local_fcos_or_resnet.pt --device cudaThe FCOS load report lists every loaded, adapted, skipped-shape, missing, and
unexpected key. In FCOS YAML, boolean model.pretrained: true is rejected
because it would imply an online download; provide an absolute checkpoint path.
The isolated YOLO runner separately proves that its already-loaded local model
was transferred into the 12-class training graph.
For the optional RGB backend screen, first create a symlink-only view from the already locked group split; this reads no IR/depth data and refuses sample, group, or physical-RGB leakage:
python tools/make_yolo_rgb_view.py \
--train-manifest /absolute/path/splits/fold_0_train.csv \
--val-manifest /absolute/path/splits/fold_0_val.csv \
--output-dir /absolute/path/yolo_rgb_fold0Then train from the one explicit local COCO checkpoint. Formal screens must
use --fraction 1.0; smaller fractions are reserved for labelled smoke runs.
All material hyperparameters are explicit and every epoch is retained so the
independent evaluator, rather than backend fitness, selects the checkpoint:
YOLO_OFFLINE=true python tools/ultralytics_rgb.py train \
--weights /absolute/path/yolo11l.pt \
--weights-sha256 9ebd0e09d59811db4b1d61e2bc6730649608b1ac47f8dd01e2da6bca7c20023f \
--data /absolute/path/yolo_rgb_fold0/data.yaml \
--font /usr/share/fonts/truetype/dejavu/DejaVuSans.ttf \
--run-dir /absolute/path/runs/E9_yolo11l_rgb_fold0 \
--epochs 60 --imgsz 640 --batch 32 --device 0 --workers 8 \
--seed 3407 --optimizer AdamW --lr0 0.001 --lrf 0.01 \
--weight-decay 0.0005 --warmup-epochs 3 --nbs 32 \
--mosaic 0.2 --mixup 0 --close-mosaic 10 \
--degrees 5 --translate 0.1 --scale 0.15 \
--fliplr 0.5 --flipud 0 --perspective 0 --multi-scale 0 \
--save-period 1 --fraction 1.0The local font is mandatory. During model.train, the adapter blocks all
IPv4/IPv6 connections, replaces Ultralytics' reference-model AMP check (which
would otherwise fetch yolo26n.pt), and routes every plot font request to the
explicit local file. Actual AMP remains enabled and is accepted only when
offline_runtime_report.json records finite train batches, optimizer steps,
and zero blocked network/download attempts.
Select exactly one epoch checkpoint with the independent 101-point evaluator:
YOLO_OFFLINE=true python tools/select_yolo_checkpoint.py \
--run-dir /absolute/path/runs/E9_yolo11l_rgb_fold0 \
--manifest /absolute/path/splits/fold_0_val.csv \
--output-dir /absolute/path/runs/E9_yolo11l_rgb_fold0_official_selection \
--expected-checkpoints 60 --imgsz 640 --batch 8 --device 0On macOS, an ImageNet ResNet-50 checkpoint may be placed at ${HOME}/.cache/torch/hub/checkpoints/resnet50-11ad3fa6.pth. MPS training should explicitly enable CPU fallback for any unsupported operator:
PYTORCH_ENABLE_MPS_FALLBACK=1 PYTHONPATH=src \
/opt/homebrew/Caskroom/miniconda/base/envs/ml/bin/python tools/train.py \
--config configs/strong_trimodal_rgf_p2.yaml \
--data configs/data.aic2026.local.yaml \
--train-manifest artifacts/splits_real/fold_0_train.csv \
--val-manifest artifacts/splits_real/fold_0_val.csv \
--set model.pretrained=${HOME}/.cache/torch/hub/checkpoints/resnet50-11ad3fa6.pth \
--run-dir runs/E4_fold0_seed3407_mps --device mpsHigh-resolution fine-tune starts a new optimizer from one prior checkpoint; it does not average checkpoints:
python tools/train.py \
--config configs/final_train.yaml --stage high_resolution_finetune \
--data /absolute/path/data.yaml \
--train-manifest /absolute/path/splits/fold_0_train.csv \
--val-manifest /absolute/path/splits/fold_0_val.csv \
--init-checkpoint /absolute/path/E4_best.pt \
--run-dir runs/E5_fold0 --device cudaAfter E0-E6 group-fold ablations, keep only components improving at least two folds. Choose a fixed final epoch from the median/stable fold-best region, then train one final model on all official training data:
python tools/train.py \
--config configs/final_train.yaml --data /absolute/path/data.yaml \
--train-manifest /absolute/path/audit_train/manifest_usable.csv \
--set model.pretrained=/absolute/path/fcos_resnet50_fpn_coco.pt \
--epochs REPLACE_WITH_FIXED_VALIDATED_EPOCH \
--run-dir runs/final_single_model --device cudaThe submission checkpoint is runs/final_single_model/final_ema.pt. It contains one model and its EMA, not a cross-checkpoint average.
final_train.yaml keeps the E6 repeat-factor sampler disabled until two group folds demonstrate an improvement. Enable it with --set train.repeat_factor_sampler=true only after that evidence exists; the audit's rare class count alone is not validation evidence.
Normal independent evaluation:
python tools/validate.py \
--config configs/strong_trimodal_rgf_p2.yaml --data /absolute/path/data.yaml \
--manifest /absolute/path/splits/fold_0_val.csv \
--checkpoint /absolute/path/best.pt --device cuda \
--modes normal --output runs/validation_fold0.jsonAll robustness modes (normal,rgb_only,ir_only,depth_only,no_ir,no_depth,no_rgb,degraded_rgb,degraded_ir,degraded_depth):
python tools/validate.py \
--config configs/strong_trimodal_rgf_p2.yaml --data /absolute/path/data.yaml \
--manifest /absolute/path/splits/fold_0_val.csv \
--checkpoint /absolute/path/best.pt --device cuda \
--modes all --output runs/robustness_fold0.jsonValidation-only score/NMS search:
python tools/validate.py \
--config configs/strong_trimodal_rgf_p2.yaml --data /absolute/path/data.yaml \
--manifest /absolute/path/splits/fold_0_val.csv \
--checkpoint /absolute/path/best.pt --device cuda \
--modes normal --tune-postprocess \
--score-grid 0.001,0.005,0.01,0.03 \
--nms-grid 0.45,0.55,0.60,0.65,0.70 \
--output runs/postprocess_fold0.jsonNever run postprocessing search against a test manifest.
Run exactly one checkpoint, one configured scale, one forward pass, no TTA:
python tools/infer.py \
--config configs/final_train.yaml --data /absolute/path/data.yaml \
--manifest /absolute/path/audit_test/manifest.csv \
--checkpoint /absolute/path/final_ema.pt \
--output runs/final_predictions.pt --device cudaCreate all TXT files, force validation, then make a flat ZIP:
python tools/make_submission.py \
--predictions runs/final_predictions.pt \
--output-dir runs/submission_txt \
--zip runs/submission.zip \
--report runs/submission_report.jsonRe-check independently before upload:
python tools/check_submission.py \
--prediction-dir runs/submission_txt \
--manifest /absolute/path/audit_test/manifest.csv \
--zip runs/submission.zip \
--report runs/submission_check_report.jsoncheck_submission.py fails on missing/extra TXT, nested or corrupt ZIP members, malformed fields, non-finite/out-of-range values, non-positive sizes, illegal classes, or more than 100 boxes.
Store optional gate maps during inference and render failure groups:
python tools/infer.py \
--config configs/final_train.yaml --data /absolute/path/data.yaml \
--manifest /absolute/path/audit_test/manifest.csv \
--checkpoint /absolute/path/final_ema.pt \
--output runs/predictions_with_gates.pt --device cuda --save-gates
python tools/visualize_predictions.py \
--manifest /absolute/path/audit_test/manifest.csv \
--predictions runs/predictions_with_gates.pt \
--output-dir runs/visualizations --score-threshold 0.05The tool produces RGB prediction/GT overlays alongside IR, depth, and optional C2 IR/depth gate heatmaps, plus directories for false positives, false negatives, localization failures below IoU 0.75, and small-target failures.
Measure parameters, hook-estimated MACs, latency, throughput, and CUDA peak memory when applicable:
python tools/benchmark.py --config configs/strong_trimodal_rgf_p2.yaml \
--pretrained /absolute/path/fcos_resnet50_fpn_coco.pt --device cuda --size 768 \
--output runs/benchmark_strong.jsonWhen official data is unavailable, synthetic files are used only to verify the code path:
python tools/create_synthetic_data.py --output artifacts/synthetic
/opt/homebrew/Caskroom/miniconda/base/envs/ml/bin/python -m pytest -qUse the interpreter that owns PyTorch; the macOS base interpreter is not a supported test environment for this checkout.
The verified end-to-end smoke commands use configs/smoke.yaml, audit the synthetic files, group-split them, train two batches, validate, infer four test samples, produce every TXT, and validate the ZIP. A synthetic mAP is not evidence of real-data quality and must never be entered as an official score.
- Depth becomes 8-bit: never use default grayscale/RGB loading.
load_depth_uint16usescv2.IMREAD_UNCHANGEDand rejects non-uint16 input. - Modal dimensions differ: audit stops and names the sample and each shape. Do not independently resize modalities.
- Four-field label: the PDF contains one typo, but its formal label description and evaluation section require
norm_h. The parser reportsfile:line, expected 5, and refuses the row. - Missing empty TXT: submission writing iterates the manifest, not only detections, so no-prediction samples get a zero-byte TXT.
- Test basename mismatch: audit reports missing, extra, and duplicate counterparts. Multiple discovery candidates are never guessed.
- Automatic weight download: all torchvision constructors use
weights=None. Supply a local path throughmodel.pretrained; booleantrueis rejected. - Out-of-memory: lower loader batch size and raise
train.accumulation_steps; preserve the intended effective batch size. Benchmark before changing the formal input size. - MPS falsely reported unavailable: restricted execution can hide the host OS version from PyTorch. In a normal terminal, verify
torch.backends.mps.is_available()andtorch.ones(1, device="mps"); usePYTORCH_ENABLE_MPS_FALLBACK=1for training.
The open papers fetched for design review are recorded with source, stderr, version, and SHA-256 in docs/papers/FETCH_LOG.md; paper facts and project inferences are separated in docs/papers/DESIGN_NOTES.md. CSAA supports cheap multi-stage channel/spatial fusion; TFDet supports a future training-only foreground auxiliary ablation; FPN supports retaining P2 as part of the full pyramid. The project does not copy their dual full backbones, hard channel switching, deformable CUDA operators, correlation-max default loss, late detector fusion, or their dataset scores.
Real competition data was audited and used for CPU and MPS two-batch functional smoke tests plus a full 1000-sample Phase-1 inference/submission-format check. A 768 px ResNet-50 strong-model MPS training step also completed with finite loss. No full official-data training was started because the requested schedule is 160-220 epochs and the project instructions explicitly limit automatic execution to smoke tests. The smoke checkpoint is randomly initialized and is not a competition candidate; its mAP=0.0 is not a claimed competition score. Full training time, final epoch, and score remain unmeasured until the group-fold runs complete.