cuDSS fails on some system

[lyehe]

cuDSS worked on 5090 (WSL), but failed on a L40s (Ubuntu) with same data/config.
The failure was quite sneaky. While it reported successful ba steps(a few out of 20), values are minimally optimized.

[JingruiYu]

Thank you @lyehe for your feedback and for providing detailed information about the issue.

We will arrange to reproduce and test on an L40s machine. If you could share some log details—such as error messages, optimization progress, or any other environment information—that would be greatly appreciated and would help us better diagnose and resolve the problem.

Thank you again!

[lyehe]

There was actually no error message. The ceres BA runs and returns somewhat valid result, but not actually optimizing anything. The output pose is completely off.

Using the other systems or use CPU will have a much higher cost reduction with the same data.

I20251104 23:16:03.979126 1061536 bundle_adjustment_solver.cc:128] [BA COST] initial total_cost = 1061.76
I20251104 23:16:03.979626 1061536 bundle_adjustment_solver.cc:137] [BA COST] initial group 'relative_extrinsic' cost = 0 (1120 blocks)
I20251104 23:16:03.979990 1061536 bundle_adjustment_solver.cc:137] [BA COST] initial group 'absolute_extrinsic' cost = 0 (8 blocks)
I20251104 23:16:03.980363 1061536 bundle_adjustment_solver.cc:137] [BA COST] initial group 'relative_pose_vehicle' cost = 0 (114 blocks)
I20251104 23:16:03.980723 1061536 bundle_adjustment_solver.cc:137] [BA COST] initial group 'reprojection_constant_frame' cost = 9.25507 (20 blocks)
I20251104 23:16:03.985553 1061536 bundle_adjustment_solver.cc:137] [BA COST] initial group 'reprojection_vehicle_frame' cost = 1052.51 (14603 blocks)
I20251104 23:16:03.985563 1061536 bundle_adjustment_solver.cc:153] [BA COST] initial percent_by_group: relative_extrinsic=0.00%, absolute_extrinsic=0.00%, relative_pose_vehicle=0.00%,
reprojection_constant_frame=0.87%, reprojection_vehicle_frame=99.13%
I20251104 23:16:03.985567 1061536 bundle_adjustment_solver.cc:52] Using: CUDSS_SPARSE + SPARSE_NORMAL_CHOLESKY
iter      cost      cost_change  |gradient|   |step|    tr_ratio  tr_radius  ls_iter  iter_time  total_time
   0  1.061764e+03    0.00e+00    1.62e+04   0.00e+00   0.00e+00  1.00e+04        0    8.58e-03    2.30e-02
   1  1.061764e+03    0.00e+00    1.62e+04   0.00e+00   0.00e+00  5.00e+03        1    5.46e-02    7.76e-02
   2  1.061764e+03    0.00e+00    1.62e+04   0.00e+00   0.00e+00  1.25e+03        1    5.55e-03    8.31e-02
   3  1.061764e+03    0.00e+00    1.62e+04   0.00e+00   0.00e+00  1.56e+02        1    5.54e-03    8.87e-02
   4  1.061764e+03    0.00e+00    1.62e+04   0.00e+00   0.00e+00  9.77e+00        1    6.44e-03    9.51e-02
   5  1.061764e+03    0.00e+00    1.62e+04   0.00e+00   0.00e+00  3.05e-01        1    6.91e-03    1.02e-01
   6  1.061764e+03    0.00e+00    1.62e+04   0.00e+00   0.00e+00  4.77e-03        1    5.73e-03    1.08e-01
   7  1.061764e+03    0.00e+00    1.62e+04   0.00e+00   0.00e+00  3.73e-05        1    5.59e-03    1.13e-01
   8  1.061763e+03    1.08e-03    1.62e+04   0.00e+00   2.82e+00  1.12e-04        1    1.63e-02    1.30e-01
   9  1.061763e+03    0.00e+00    1.62e+04   0.00e+00   0.00e+00  5.59e-05        1    6.56e-03    1.36e-01
  10  1.061763e+03    0.00e+00    1.62e+04   0.00e+00   0.00e+00  1.40e-05        1    5.51e-03    1.42e-01
  11  1.061763e+03    0.00e+00    1.62e+04   0.00e+00   0.00e+00  1.75e-06        1    5.47e-03    1.47e-01
  12  1.061763e+03    0.00e+00    1.62e+04   0.00e+00   0.00e+00  1.09e-07        1    5.41e-03    1.53e-01
  13  1.061763e+03    0.00e+00    1.62e+04   0.00e+00   0.00e+00  3.41e-09        1    5.40e-03    1.58e-01
  14  1.061763e+03    0.00e+00    1.62e+04   0.00e+00   0.00e+00  5.33e-11        1    5.41e-03    1.64e-01
  15  1.061763e+03    0.00e+00    1.62e+04   0.00e+00   0.00e+00  4.16e-13        1    5.26e-03    1.69e-01
  16  1.061763e+03    0.00e+00    1.62e+04   0.00e+00   0.00e+00  1.63e-15        1    5.47e-03    1.74e-01
  17  1.061763e+03    0.00e+00    1.62e+04   0.00e+00   0.00e+00  3.18e-18        1    5.43e-03    1.80e-01
  18  1.061763e+03    0.00e+00    1.62e+04   0.00e+00   0.00e+00  3.10e-21        1    5.46e-03    1.85e-01
  19  1.061763e+03    0.00e+00    1.62e+04   0.00e+00   0.00e+00  1.51e-24        1    5.40e-03    1.91e-01
  20  1.061763e+03    0.00e+00    1.62e+04   0.00e+00   0.00e+00  3.70e-28        1    5.36e-03    1.96e-01
  21  1.061763e+03    0.00e+00    1.62e+04   0.00e+00   0.00e+00  4.51e-32        1    5.33e-03    2.01e-01
  22  1.061763e+03    0.00e+00    1.62e+04   0.00e+00   0.00e+00  2.76e-36        1    5.37e-03    2.07e-01
I20251104 23:16:04.198035 1061536 bundle_adjustment_solver.cc:73] Ceres summary:
I20251104 23:16:04.198071 1061536 bundle_adjustment_solver.cc:74]
Solver Summary (v 2.2.0-eigen-(3.4.0)-no_lapack-eigensparse-cuda-(12060)-cudss-(0.6.0))

                                     Original                  Reduced
Parameter blocks                         4558                     4556
Parameters                              13722                    13715
Effective parameters                    13674                    13668
Residual blocks                         15865                    15865
Residuals                               36698                    36698

Minimizer                        TRUST_REGION
Trust region strategy     LEVENBERG_MARQUARDT
Sparse linear algebra library     CUDA_SPARSE

                                        Given                     Used
Linear solver          SPARSE_NORMAL_CHOLESKY   SPARSE_NORMAL_CHOLESKY
Threads                                     8                        8
Linear solver ordering              AUTOMATIC                     4556

Cost:
Initial                          1.061764e+03
Final                            1.061763e+03
Change                           1.078601e-03

Minimizer iterations                       23
Successful steps                            2
Unsuccessful steps                         21

Time (in seconds):
Preprocessor                         0.014392

  Residual only evaluation           0.001385 (1)
  Jacobian & residual evaluation     0.015829 (2)
  Linear solver                      0.165955 (22)
Minimizer                            0.193200

Postprocessor                        0.000738
Total                                0.208330

Termination:                      CONVERGENCE (Minimum trust region radius reached. Trust region radius: 2.755065e-36 <= 1.000000e-32)

[lyehe]

Someone told me they ran into similar problem with GPU colmap build before.
This should be related to the ceres + cudss build.

[willdzeng]

@lyehe would you mind trying the latest release with cuda13 see if you still have this error on L40?