Seeing More Clearly: A Video Stabilizer using CNN with Speeded-up Robust features

Student: Junting Wang
Table: COMP3
Experimentation location: Home
Regulated Research (Form 1c): No
Project continuation (Form 7): No

Display board image not available

Abstract:

Bibliography/Citations:

Zhang, Fang Lue, et al. “Coherent Video Generation for Multiple Hand-Held Cameras with Dynamic Foreground.” Computational Visual Media, vol. 6, no. 3, Tsinghua University Press, Sept. 2020, pp. 291–306, doi:10.1007/s41095-020-0187-3.

Xu, Yufei, et al. DUT: Learning Video Stabilization by Simply Watching Unstable Videos. https://github.com/Annbless/DUTCode.

Liu, Yu-Lun, et al. Hybrid Neural Fusion for Full-Frame Video Stabilization. https://alex04072000.github.io/FuSta/.

Wu, Rouwan, et al. “Robust Global Motion Estimation for Video Stabilization Based on Improved K-Means Clustering and Superpixel.” Sensors, vol. 21, no. 7, MDPI AG, Apr. 2021, doi:10.3390/s21072505.

2019 IEEE Visual Communications and Image Processing (VCIP). IEEE, 2019.

Institute of Electrical and Electronics Engineers, and IEEE Signal Processing Society. 2019 IEEE International Conference on Image Processing (ICIP) : Proceedings : September 22-25, 2019, Taipei International Convention Center (TICC), Taipei, Taiwan.

Choi, Jinsoo, and In So Kweon. “DIFRINT: Deep Iterative Frame Interpolation for Full-Frame Video Stabilization.” Proceedings - 2019 International Conference on Computer Vision Workshop, ICCVW 2019, Institute of Electrical and Electronics Engineers Inc., 2019, pp. 3732–36, doi:10.1109/ICCVW.2019.00463.

Su, Shuochen, et al. Deep Video Deblurring for Hand-Held Cameras. https://www.cs.ubc.ca/labs/imager/tr/2017/DeepVideoDeblurring.

Kim, Byeonghwi, et al. “Automated White Blood Cell Counting in Nailfold Capillary Using Deep Learning Segmentation and Video Stabilization.” Sensors (Switzerland), vol. 20, no. 24, MDPI AG, Dec. 2020, pp. 1–22, doi:10.3390/s20247101.

Ma, Tiezheng, et al. “Effective Video Stabilization via Joint Trajectory Smoothing and Frame Warping.” IEEE Transactions on Visualization and Computer Graphics, vol. 26, no. 11, IEEE Computer Society, Nov. 2020, pp. 3163–76, doi:10.1109/TVCG.2019.2923196.

Ali, Muhammad Kashif, et al. Learning Deep Video Stabilization without Optical Flow.

Huang, Chia-Hung, et al. StableNet: Semi-Online, Multi-Scale Deep Video Stabilization.

Niskanen, Matti, et al. “Video Stabilization Performance Assessment.” 2006 IEEE International Conference on Multimedia and Expo, ICME 2006 - Proceedings, vol. 2006, 2006, pp. 405–08, doi:10.1109/ICME.2006.262522.

Yu, Jiyang, and Ravi Ramamoorthi. Robust Video Stabilization by Optimization in CNN Weight Space.

Zhao, Minda, and Qiang Ling. “Adaptively Meshed Video Stabilization.” IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, no. 9, Institute of Electrical and Electronics Engineers Inc., Sept. 2021, pp. 3504–17, doi:10.1109/TCSVT.2020.3040753.

Wang, Miao, et al. Deep Online Video Stabilization.

Guilluy, Wilko, et al. “Video Stabilization: Overview, Challenges and Perspectives.” Signal Processing: Image Communication, vol. 90, Elsevier B.V., Jan. 2021, doi:10.1016/j.image.2020.116015.

Shi, Zhenmei, et al. Deep Online Fused Video Stabilization.

Xu, Sen Zhe, et al. “Deep Video Stabilization Using Adversarial Networks.” Computer Graphics Forum, vol. 37, no. 7, Blackwell Publishing Ltd, Oct. 2018, pp. 267–76, doi:10.1111/cgf.13566.

Wang, Miao, et al. “Deep Online Video Stabilization with Multi-Grid Warping Transformation Learning.” IEEE Transactions on Image Processing, vol. 28, no. 5, Institute of Electrical and Electronics Engineers Inc., May 2019, pp. 2283–92, doi:10.1109/TIP.2018.2884280.

Background Scene Enlargenment in Selfie Video through Deep Neural Network: A Survey. doi:10.1089/cyber.2009.

Institute of Electrical and Electronics Engineers, et al. 2020 IEEE International Conference on Multimedia and Expo Workshops (ICMEW) : 06-10 July 2020, London, UK.

Rosebrock, Adrian. “Blur Detection with Opencv.” PyImageSearch, PyImageSearch, 17 Apr. 2021, https://pyimagesearch.com/2015/09/07/blur-detection-with-opencv/.

 


Additional Project Information

Project website: -- No project website --
Presentation files:
Additional Resources: -- No resources provided --
Project files:
Project files
 

Research Plan:

Currently, there is no commercialized product that stabilizes videos through deep learning methods. This project aims to create a solution to devise a method for addressing this void in the domain of video stabilization.

My research timeline contains 5 parts:

  1. researching existing deep learning algorithms and choosing one algorithm (Apr 2021-Jun 2021)
  2. perfecting/twisting this algorithm to achieve better results or real time stabilization (Jul 2021-Aug 2021)
  3. incorporating this algorithm into a user-friendly application (Sep 2021-Oct 2021)
  4. evaluate this algorithm through comparison with current solutions, specifically Adobe premiere pro (Oct 2021-Dec 2021)
  5. organize data & finish project paper (Jan 2022-Feb 2022)

I plan to finish a large portion of the technical stages of this project during summer break so that I have more time to work on this project.

 

Questions and Answers

1. What was the major objective of your project and what was your plan to achieve it? 

 

The major objective of my project was to develop a solution that stabilizes videos using deep learning algorithms and to achieve the capability to stabilize videos in real-time. I planned to find existing methods and change it to achieve a better stabilization result and real-time stabilization.

 

       a. Was that goal the result of any specific situation, experience, or problem you encountered?  

 

Yes. Videos are a part of our life because they enable us to document what has occurred. It, like photographs, records moments in time; however, videos are animated, which allows for much more possibilities. Currently, the most significant drawback of videos is that they may easily become shaky and unstable unless they are utilized with expensive and bulky stabilizers. Personally, I enjoy using my mobile phone to film significant events and frequently encounter unsteady videos, which is why I decided to dive more into the topic of video stabilization.

 

       b. Were you trying to solve a problem, answer a question, or test a hypothesis?

Yes

2. What were the major tasks you had to perform in order to complete your project?

       a. For teams, describe what each member worked on.

 

I had to perform a variety of tasks, beginning with preparatory research and concluding with paper writing. I had to learn about the current video stabilization methods, research, and the underlying principles, including what sort of neural networking is employed and how it is used. I also studied the fundamentals of building a convolutional neural network. All of this information is intended to assist me in improving the algorithm and developing a real-time stabilizing approach, which I was unable to accomplish due to device regulations and other factors. In addition, I integrated this into a user-friendly program and conducted a thorough comparison with the stabilization service included in Adobe Premiere Pro.

 

3. What is new or novel about your project?

       a. Is there some aspect of your project's objective, or how you achieved it that you haven't done before?

       b. Is your project's objective, or the way you implemented it, different from anything you have seen?

       c. If you believe your work to be unique in some way, what research have you done to confirm that it is?

 

My project is an innovative approach to video stabilization. Deep learning-based video stabilization is a relatively new concept in this field without much research. All existing programs, such as Adobe Premiere Pro, employ traditional video stabilizing techniques. If my product is commercialized, it will be the first to use deep learning to achieve this function. Furthermore, almost everything related to this project is brand-new for me. Because this is my first venture into deep learning, I had to learn everything from the scratch, including initially training the MNIST database. My work, I believe, is unique because it incorporates the deep learning specialization method into a computer application. During the research portion of this project, I closely read over 30 articles in detail and looked through many more internet sources; none of them performed the same thing.

 

4. What was the most challenging part of completing your project?

      a. What problems did you encounter, and how did you overcome them?

      b. What did you learn from overcoming these problems?

 

The most challenging aspect of finishing this project is meeting the configuration criteria of running my program. The deep learning algorithm requires a cuda core GPU which is not available on my laptops since I use mac. I first attempted to update the program's source code to force it to process using the CPU rather than the GPU to support my device, but I was unable to do so even after multiple tries. I still want to work on this project, so I thought of renting a computer, which sounded ridiculous given that I had never heard of computer rentals before. Although there are no physical computer rentals, I was able to locate online computer rentals that were specifically designed for deep learning. I could rent and share an online computer from which I could manage some of the features and run my code remotely, which addressed my problem. Although this seems like a cheap fix, it solved my problem and allowed me to complete a more in-depth project. Cheap fixes are sometimes beneficial for the overall project.

 

5. If you were going to do this project again, are there any things you would you do differently the next time?

 

If I were going to undertake this project again, I would conduct a considerable study to determine the viability of fulfilling certain functionalities before beginning to work on the project. In the given instance, I only discovered that my computing power was insufficient to train the model when I had already started to make changes to the network. If I was aware of this issue ahead of time, I would have adjusted my research plan earlier and potentially found alternative research directions to improve my product.

 

6. Did working on this project give you any ideas for other projects? 

 

Yes, when examining the different ways to assess a video's quality, particularly in relation to its stability, I only discovered a few studies in this field. Most current video quality ratings are more concerned with the illumination and sharpness of the video than with its stability. I might do another project that utilizes deep learning to assess the stability and other measures of a video.

 

7. How did COVID-19 affect the completion of your project?

 

It didn’t affect the completion of my project since my project is entirely software-based and does not require a lab to conduct.