Alexander Veicht · Paul-Edouard Sarlin · Philipp Lindenberger · Marc Pollefeys

GeoCalib accurately estimates the camera intrinsics and gravity direction from a single image
by combining geometric optimization with deep learning.

GeoCalib is an algorithm for single-image calibration: it estimates the camera intrinsics and gravity direction from a single image only. By combining geometric optimization with deep learning, GeoCalib provides a more flexible and accurate calibration compared to previous approaches. This repository hosts the inference, evaluation, and training code for GeoCalib and instructions to download our training set OpenPano.

We provide a small inference package geocalib that requires only minimal dependencies and Python >= 3.9. First clone the repository and install the dependencies:

git clone https://github.com/cvg/GeoCalib.git && cd GeoCalib
python -m pip install -e .
# OR
python -m pip install -e "git+https://github.com/cvg/GeoCalib#egg=geocalib"

Here is a minimal usage example:

from geocalib import GeoCalib

device = "cuda" if torch.cuda.is_available() else "cpu"
model = GeoCalib().to(device)

# load image as tensor in range [0, 1] with shape [C, H, W]
image = model.load_image("path/to/image.jpg").to(device)
result = model.calibrate(image)

print("camera:", result["camera"])
print("gravity:", result["gravity"])

Check out our demo notebook for a full working example.

python -m geocalib.interactive_demo --camera_id 0

model = torch.hub.load("cvg/GeoCalib", "GeoCalib", trust_repo=True)

GeoCalib currently supports the following camera models via the camera_model parameter:

1. pinhole (default) models only the focal lengths fx and fy but no lens distortion.
2. simple_radial models weak distortions with a single polynomial distortion parameter k1.
3. simple_divisional models strong fisheye distortions with a single distortion parameter k1, as proposed by Fitzgibbon in Simultaneous linear estimation of multiple view geometry and lens distortion (CVPR 2001).

The default model is optimized for pinhole images. To handle lens distortion, use the following:

model = GeoCalib(weights="distorted")  # default is "pinhole"
result = model.calibrate(image, camera_model="simple_radial")  # or pinhole, simple_divisional

The principal point is assumed to be at the center of the image and is not optimized. Additional models can be implemented by extending the Camera object.

When either the intrinsics or the gravity are already known, they can be provided as follows:

# known intrinsics:
result = model.calibrate(image, priors={"focal": focal_length_tensor})

# known gravity:
result = model.calibrate(image, priors={"gravity": gravity_direction_tensor})

To calibrate multiple images captured by the same camera, pass a list of images to GeoCalib:

# batch is a list of tensors, each with shape [C, H, W]
result = model.calibrate(batch, shared_intrinsics=True)

The full evaluation and training code is provided in the single-image calibration library siclib, which can be installed as:

python -m pip install -e siclib

Running the evaluation commands will write the results to outputs/results/.

Running the evaluation commands will download the dataset to data/lamar2k which will take around 400 MB of disk space.

python -m siclib.eval.lamar2k --conf geocalib-pinhole --tag geocalib --overwrite

python -m siclib.eval.lamar2k --conf deepcalib --tag deepcalib --overwrite

python -m siclib.eval.lamar2k --conf uvp --tag uvp --overwrite data.preprocessing.edge_divisible_by=null

python -m siclib.eval.lamar2k --checkpoint <experiment name> --tag <eval name> --overwrite

Running the evaluation commands will download the dataset to data/megadepth2k or data/memegadepth2k-radial which will take around 2.1 GB and 1.47 GB of disk space respectively.

python -m siclib.eval.megadepth2k --conf geocalib-pinhole --tag geocalib --overwrite

python -m siclib.eval.megadepth2k_radial --conf geocalib-pinhole --tag geocalib --overwrite model.camera_model=simple_radial

python -m siclib.eval.megadepth2k --conf deepcalib --tag deepcalib --overwrite

python -m siclib.eval.megadepth2k --conf uvp --tag uvp --overwrite data.preprocessing.edge_divisible_by=null

python -m siclib.eval.megadepth2k --checkpoint <experiment name> --tag <eval name> --overwrite

Running the evaluation commands will download the dataset to data/tartanair which will take around 1.85 GB of disk space.

python -m siclib.eval.tartanair --conf geocalib-pinhole --tag geocalib --overwrite

python -m siclib.eval.tartanair --conf deepcalib --tag deepcalib --overwrite

python -m siclib.eval.tartanair --conf uvp --tag uvp --overwrite data.preprocessing.edge_divisible_by=null

python -m siclib.eval.tartanair --checkpoint <experiment name> --tag <eval name> --overwrite

Before downloading and running the evaluation, you will need to agree to the terms of use for the Stanford2D3D dataset. Running the evaluation commands will download the dataset to data/stanford2d3d which will take around 885 MB of disk space.

python -m siclib.eval.stanford2d3d --conf geocalib-pinhole --tag geocalib --overwrite

python -m siclib.eval.stanford2d3d --conf deepcalib --tag deepcalib --overwrite

python -m siclib.eval.stanford2d3d --conf uvp --tag uvp --overwrite data.preprocessing.edge_divisible_by=null

python -m siclib.eval.stanford2d3d --checkpoint <experiment name> --tag <eval name> --overwrite

If you want to provide priors during the evaluation, you can add one or multiple of the following flags:

python -m siclib.eval.<benchmark> --conf <config> \
    --tag <tag> \
    data.use_prior_focal=true \
    data.use_prior_gravity=true \
    data.use_prior_k1=true

python -m siclib.eval.inspect <benchmark> <one or multiple tags>

python -m siclib.eval.inspect lamar2k geocalib

The OpenPano dataset is a new dataset for single-image calibration which contains about 2.8k panoramas from various sources, namely HDRMAPS, PolyHaven, and the Laval Indoor HDR dataset. While this dataset is smaller than previous ones, it is publicly available and it provides a better balance between indoor and outdoor scenes.

python -m siclib.datasets.utils.tonemapping --hdr_dir data/indoorDatasetCalibrated --out_dir data/laval-tonemap

python -m siclib.datasets.utils.download_openpano --name openpano --laval_dir data/laval-tonemap

python -m siclib.datasets.create_dataset_from_pano --config-name openpano

python -m siclib.datasets.create_dataset_from_pano --config-name openpano n_workers=10 device=cuda

python -m siclib.datasets.create_dataset_from_pano --config-name openpano_radial

As for the evaluation, the training code is provided in the single-image calibration library siclib, which can be installed by:

python -m pip install -e siclib

Once the OpenPano Dataset has been downloaded and prepared, we can train GeoCalib with it:

First download the pre-trained weights for the MSCAN-B backbone:

mkdir weights
wget "https://cloud.tsinghua.edu.cn/d/c15b25a6745946618462/files/?p=%2Fmscan_b.pth&dl=1" -O weights/mscan_b.pth

Then, start the training with the following command:

python -m siclib.train geocalib-pinhole-openpano --conf geocalib --distributed

Feel free to use any other experiment name. By default, the checkpoints will be written to outputs/training/. The default batch size is 24 which requires 2x 4090 GPUs with 24GB of VRAM each. Configurations are managed by Hydra and can be overwritten from the command line. For example, to train GeoCalib on a single GPU with a batch size of 5, run:

python -m siclib.train geocalib-pinhole-openpano \
    --conf geocalib \
    data.train_batch_size=5 # for 1x 2080 GPU

Be aware that this can impact the overall performance. You might need to adjust the learning rate and number of training steps accordingly.

If you want to log the training progress to tensorboard or wandb, you can set the train.writer option:

python -m siclib.train geocalib-pinhole-openpano \
    --conf geocalib \
    --distributed \
    train.writer=tensorboard

The model can then be evaluated using its experiment name:

python -m siclib.eval.<benchmark> --checkpoint geocalib-pinhole-openpano \
    --tag geocalib-retrained

python -m siclib.train deepcalib-openpano --conf deepcalib --distributed

If you use any ideas from the paper or code from this repo, please consider citing:

@inproceedings{veicht2024geocalib,
  author    = {Alexander Veicht and
               Paul-Edouard Sarlin and
               Philipp Lindenberger and
               Marc Pollefeys},
  title     = {{GeoCalib: Single-image Calibration with Geometric Optimization}},
  booktitle = {ECCV},
  year      = {2024}
}

The code is provided under the Apache-2.0 License while the weights of the trained model are provided under the Creative Commons Attribution 4.0 International Public License. Thanks to the authors of the Laval Indoor HDR dataset for allowing this.