Thanks! LGTM. One minor question: what s the justification of switching the order of transformation and 3D point in most cost functions?

Ceres pre-compiles Schur / partitioned matrix view templates in the following pattern. Unfortunately, 7 parameters are not included in this list, but it's generally better to keep small parameters in the front to increase the chance that a static matrix template can be applied, or maybe I misunderstood what is happening in ceres-solver/internal/ceres/schur_eliminator.h. We should maybe request to add a 2_3_7 version.

schur_eliminator_2_2_2.cc.o
schur_eliminator_2_2_2.cc.o.d
schur_eliminator_2_2_3.cc.o
schur_eliminator_2_2_3.cc.o.d
schur_eliminator_2_2_4.cc.o
schur_eliminator_2_2_4.cc.o.d
schur_eliminator_2_2_d.cc.o
schur_eliminator_2_2_d.cc.o.d
schur_eliminator_2_3_3.cc.o
schur_eliminator_2_3_3.cc.o.d
schur_eliminator_2_3_4.cc.o
schur_eliminator_2_3_4.cc.o.d
schur_eliminator_2_3_6.cc.o
schur_eliminator_2_3_6.cc.o.d
schur_eliminator_2_3_9.cc.o
schur_eliminator_2_3_9.cc.o.d
schur_eliminator_2_3_d.cc.o
schur_eliminator_2_3_d.cc.o.d
schur_eliminator_2_4_3.cc.o
schur_eliminator_2_4_3.cc.o.d
schur_eliminator_2_4_4.cc.o
schur_eliminator_2_4_4.cc.o.d
schur_eliminator_2_4_6.cc.o
schur_eliminator_2_4_6.cc.o.d
schur_eliminator_2_4_8.cc.o
schur_eliminator_2_4_8.cc.o.d
schur_eliminator_2_4_9.cc.o
schur_eliminator_2_4_9.cc.o.d
schur_eliminator_2_4_d.cc.o
schur_eliminator_2_4_d.cc.o.d
schur_eliminator_2_d_d.cc.o
schur_eliminator_2_d_d.cc.o.d
schur_eliminator_3_3_3.cc.o
schur_eliminator_3_3_3.cc.o.d
schur_eliminator_4_4_2.cc.o
schur_eliminator_4_4_2.cc.o.d
schur_eliminator_4_4_3.cc.o
schur_eliminator_4_4_3.cc.o.d
schur_eliminator_4_4_4.cc.o
schur_eliminator_4_4_4.cc.o.d
schur_eliminator_4_4_d.cc.o
schur_eliminator_4_4_d.cc.o.d
schur_eliminator_d_d_d.cc.o
schur_eliminator_d_d_d.cc.o.d

From what I know, the dimensions of E and F (the unknowns) in Ceres correspond to Point3D (3D) and Rigid3d (6D, tangent space; not the manifold itself) in the BundleAdjuster.
When camera intrinsics are fixed for the BundleAdjuster, Ceres invokes schur_eliminator_2_3_6.cc; when optimizing the three parameter sets of Point3D (3D), Rigid3d (6D) and Camera (k), it calls schur_eliminator_2_3_n.cc instead.