Scale Issues in Aerial Reconstruction Using Input Poses

[dengyangyan]

We ran several experiments on large-scale scene datasets using COLMAP’s camera intrinsics and poses. However, we found a clear scale mismatch between the point cloud generated by MapAnything and the original COLMAP point cloud. In the figure below, the full sparse COLMAP point cloud is shown, with the pink box indicating MapAnything’s output.

Additionally, the estimated poses differ from those produced by COLMAP.

[cr333]

The scale of point maps is fundamentally ambiguous unless one has metric or ground-truth camera poses because of the scale ambiguity in structure-from-motion. Similarly, camera poses recovered from structure-from-motion are ambiguous up to rigid transformation, i.e. an arbitrary translation and rotation.

For these reasons, it's not expected that point maps or camera poses match between COLMAP and MapAnything (or any other structure-from-motion approach).

As for why the spatial extent of the reconstructions is different, it looks like different input images were used for COLMAP vs MapAnything.

Does this address your concern?

If not, please clarify and provide more details about your problem setting how you use COLMAP and MapAnything.

[Nik-V9]

Hi, thanks for trying out MapAnything!

Are the COLMAP poses here metric? Do you also input the poses to MapAnything? If the COLMAP poses are non-metric, I presume you are using ignore_pose_scale_inputs=True?

We have observed that the metric scale estimation of MapAnything for aerial images is bad and that could be the reason why you are seeing a small scale for the predictions.

[dengyangyan]

Thank you very much for your patient reply. In our case, the COLMAP poses were obtained through bundle adjustment using the real poses (from DJI Terra). At the same time, COLMAP also outputs a point cloud that roughly aligns with the LiDAR ground truth, so the COLMAP poses here are indeed metric.

Regarding the images, due to limited GPU memory, the inputs to MapAnything—
$${ I_m, K_m, R_m, t_m | m=1,\ldots,M }$$ — are a subset of the full image set $${ I_i, K_i, R_i, t_i| i=1,\ldots,N }$$.

Under these conditions, it seems reasonable to expect the reconstructed 3D point cloud to be consistent with the corresponding COLMAP sub-region. However, we observed clear discrepancies in both scale and pose when compared with our input parameters.

We are curious about the reason behind this discrepancy. Could it be related to biases or limitations in the training data?

Once again, thank you very much for your excellent work.

[Nik-V9]

Thanks for the clarification!

Yes, you are correct in expecting the model output to be consistent with the COLMAP scale when the input poses are metric.

As I mentioned before, I think this is a limitation in the training data. The only aerial data we have is BlendedMVS which is up to scale. We found other metric depth estimators such as MoGe-2, Metric3Dv2 or even the new Depth Anything 3 to be suboptimal for metric scale estimation of aerial images. This is mainly due to lack of good training data.

I believe that this issue can be solved by finetuning MapAnything on Aerial MegaDepth: #36