JNMR Joint Non-Linear Motion Regression for Video Frame Interpolation

Meiqin Liu1, Chenming Xu1, Chao Yao2, Chunyu Lin1, Yao Zhao1

1 Beijing Jiaotong University, Beijing, China

2 University of Science and Technology Beijng, Beijing, China

Abstract

Video frame interpolation (VFI) aims to generate predictive frames by motion-warping from bidirectional refer- ences. Most examples of VFI utilize spatiotemporal semantic information to realize motion estimation and interpolation. However, due to variable acceleration, irregular movement trajectories, and camera movement in real-world cases, they can not be sufficient to deal with non-linear middle frame estimation. In this paper, we present a reformulation of the VFI as a joint non-linear motion regression (JNMR) strategy to model the complicated inter-frame motions. Specifically, the motion trajectory between the target frame and multiple reference frames is regressed by a temporal concatenation of multi-stage quadratic models. Then, a comprehensive joint distribution is constructed to connect all temporal motions. Moreover, to reserve more contextual details for joint regression, the feature learning network is devised to explore clarified feature expressions with dense skip- connection. Later, a coarse-to-fine synthesis enhancement module is utilized to learn visual dynamics at different resolutions with multi-scale textures. The experimental VFI results show the effectiveness and significant improvement of joint motion regression over the state-of-the-art methods.

Pipeline

Illustration of JNMR structure details. Here, we incorporate temporal-aware motion sequences to model different quadratic regressions for the middle motions. Thus, the entire model is regressed by optimizing each individual sub-non-linear model.

Highlights

  1. We propose a joint multi-variable non-linear motion regression strategy for motion compensation of VFI. The refined interpolation modeling can reconstruct the actual kinematic characteristics among large and complicated motions.

  2. We leverage ConvLSTM to realize the joint multi-stage quadratic model for JNMR, which enhances the temporal consistency in inter-frame prediction.

  3. We design the regression-driven feature learning module and coarse-to-fine enhancement module, separately opti- mizing the feature extraction with competitive parameters and pleasing visual qualities on different resolutions.

Results

Materials

paper Code

Citation

@article{liu2023JNMR,
  title={JNMR: Joint Non-Linear Motion Regression for Video Frame Interpolation},
  author={Liu, Meiqin and Xu, Chenming and Yao, Chao and Lin, Chunyu and Zhao, Yao},
  journal={IEEE Transactions on Image Processing},
  volume={32},
  number={},
  pages={5283--5295},
  year={2023},
  publisher={IEEE}
}

Contact

If you have any questions, please contact Chao Yao at yaochao@ustb.edu.cn