Spec-NeRF jointly optimizes the degradation parameters and achieves high-quality multi-spectral image reconstruction results at novel views, which only requires a low-cost camera (like a phone camera but in RAW mode) and several off-the-shelf color filters. We also provide real scenarios and synthetic datasets for related studies. Code is available at https://github.com/CPREgroup/SpecNeRF-v2

This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.

Practical full projector compensation allows the projection display to quickly adapt to textured projection surfaces and unexpected movement without interrupting the display procedure or requiring unacceptable waiting time. To achieve this by a projector and a RGB camera without adding any extra devices, a possible solution is correcting both color and geometry by directly capturing and analyzing the projected natural images content. In this work, we cast the full projector compensation as a non-negative constrained optimization problem, and present the first optimization-based framework that can handle both geometric calibration and radiometric compensation for a Projector-camera system(Procams) using only a few sampling images. The framework not only takes advantage of the ability of the deep neural network to approximate the channel-dependent response function of the projector, but also provides a flexible solution to adapt to the change of geometry and reflectance caused by movement. Benefit from the optimization-based scheme, our method can guarantee both accurate color calculation and efficient movement and reflectance estimation in a continuous projection display procedure. The experimental results show our method outperforms other state-of-the-art end-to-end full projector compensation methods with better-displayed image quality, less computational time, smaller memory consumption, more geometric accuracy, and far more compact network architecture.

We propose an unsupervised blind fusion network that operates on a single HSI and RGB image pair and requires neither known degradation models nor any training data. Our method takes full advantage of an unrolling network and coordinate encoding to provide a state-of-the-art HSI reconstruction. It can also estimate the degradation parameters relatively accurately through the neural representation and implicit regularization of the degradation model.