Deformation @ TU
1 like719 views
This document summarizes a research project that used a 3D morphable face model to improve stereo reconstruction of faces and visualize the reconstructed model on a smartphone. The research methodology involved fitting a morphable model to single images to estimate shape, extracting texture, using the model to aid stereo reconstruction, and deforming the model to match. Evaluation showed improved reconstruction accuracy when incorporating the morphable model. The reconstructed face model was then visualized on a smartphone using virtual reality hardware.
Deformation @ TU
- 1. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India Using Morphable Face Model to Improve Stereo Reconstruction and Visualising the Model on a Smartphone HARDIK JAIN Under the Guidance of Prof. Olaf HELLWICH Prof. RS ANAND
- 2. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India PRESENTATION OUTLINE Introduction Motivation Research Methodology Results and Evaluation Visualisation Conclusion and Scope for Future Work Further Reading 1 of 25
- 3. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India INTRODUCTION Face Reconstruction Multiple Image Reconstruction Visual structure from motion Stereo Reconstruction Single Image Reconstruction 3D Morphable Model 2 of 25
- 4. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India 3D Morphable Model Statistical model of face meshes which are in dense correspondence. Principal component analysis (PCA) is performed on these set of meshes. Mean shape 勍, 1 Principal Components and , 2 eigen values Shape parameter vector cs = 1, , 1 iisi M i model SsS 2 , 1 1= = 鰹￥ INTRODUCTI ON Single Image Reconstruction Morphable Model 3 of 25
- 5. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India MOTIVATION Stereo Reconstruction Stereo Model Stereo Model High Quality Face Scan Stereo Model and High Quality Scan Cloud Compare 4 of 25
- 6. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India Face Model MOTIVATION Single Image Reconstruction Face Image Face Model Face Model Cloud Compare 5 of 25
- 7. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India MOTIVATION Single Image Reconstruc tion Stereo Model Deformed Face Model Shape Information Texture & Smoothness Deformation 6 of 25
- 8. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India RESEARCH METHODOLOGY Morphable Model Stereo Pair Image Deformed Face Model Single Image Landmarking Pose Estimation Shape Fitting Texture Extraction Global& Local Deformation Stereo Reconstruction Face Model 7 of 25 Method Overview
- 9. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India Landmarking Annotation To find 犒 = 1 1, , 2 Obtain Face Bounding Box Cascade based Regression Method Initial estimate is centred to the Bounding box Regressor rt , 犒F(t) 犒F(t+1) = 犒F(t) + rt , 犒F(t) RESEARCH METHODOLOGY Single Image Reconstruction Landmark Annotated Face Image 8 of 25
- 10. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India Pose Estimation Camera Orientation Matrix, = , where 2 and 3 Affine Camera Model Gold Standard Algorithm of Hartley & Zisserman RESEARCH METHODOLOGY Single Image Reconstruction 9 of 25 2D Landmark Points 3D Points on MM
- 11. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India Shape Fitting Estimation of shape Parameter s Probabilistic Approach Minimise = =1 3 2, 2 2 2, 2 + s 2 Face Image Face Shape Model RESEARCH METHODOLOGY Single Image Reconstruction iisi M i model SsS 2 , 1 1= = 鰹￥ 10 of 25
- 12. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India Texture Extraction Obtained from Face Image Isomap Algorithm Retaining Geodesic Distance Texture Map RESEARCH METHODOLOGY Single Image Reconstruction Face Image 11 of 25
- 13. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India STEREO RECONSTRUCTION Obtain stereo pair images Camera Calibration Keypoint Detection Triangulation Dense Reconstruction RESEARCH METHODOLOGY Stereo Model 12 of 25
- 14. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India DEFORMATION Motivation Surface Registration Approach Two-step deformation Global and Local transformation 犒ヰ = 陸 陸 ( ) RESEARCH METHODOLOGY 13 of 25
- 15. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India Global Deformation Using few control points Radial Basis Function , = Weighted Combination of RBF = 犒 =1 Gaussian Kernel , = 2 22 RESEARCH METHODOLOGY Deformation 14 of 25
- 16. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India = 1 applied on Sourcepartial ,1 , = 1,1 1, ,1 , 1 ,1 , , for r = (x,y,z) Face unspecific results for N=25 Few 100 milliseconds on intel quad core computer RESEARCH METHODOLOGY Deformation Global Deformation Weight Coefficients ( ) 15 of 25
- 17. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India Local Deformation Smoothen the overshoots of RBF Influence of nearby neighbors Non-rigid transformation by k nearest neighbouring vertices Procrustes Analysis Affine transformation ( and ) RESEARCH METHODOLOGY Deformation 16 of 25
- 18. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India 陸local = w0 0 + 0 + i=1 wi i + i wi = 1 d d , = i=1 ｐ w0() = 1 = 12 Requires Few seconds RESEARCH METHODOLOGY DEFORMATION Local Deformation 1 5 4 2 3 6 17 of 25
- 19. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India DEFORMED FACE MODEL RESULTS AND EVALUATION Face Image Face Model Deformed Face Model 18 of 25
- 20. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India EVALUATION RESULTS AND EVALUATION 19 of 25 3.5532 3.3073 2.8738
- 21. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India EVALUATION RESULTS AND EVALUATION 20 of 25 4.2734 2.9985 2.0133
- 22. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India EVALUATION RESULTS AND EVALUATION 21 of 25 2.9982 2.5212 2.174
- 23. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India VISUALISATION Virtual Reality Using Smartphone and Cardboard Viewer Android phones with support of OpenGL ES 3.1 Implemented on SDK provided by Google 22 of 25
- 24. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India Smartphone View VISUALISATION 23 of 25
- 25. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India CONCLUSION AND SCOPE FOR FUTURE WORK Improved reconstruction after information fusion Technique Could be used for various other objects Cheap alternative visualisation platform Smartphone visualisation Environment Improvement 24 of 25
- 26. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India FURTHER READING V. Blanz and T. Vetter, A morphable model for the synthesis of 3D faces, in Proceedings of the 26th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 1999, pp. 187194. P. Huber, G. Hu, R. Tena, P. Mortazavian, W. P. Koppen, W. Christmas, M. Ratsch, and J. Kittler, A multiresolution 3D morphable face model and fitting framework, in 11th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, February 2016. P. Huber, Z.-H. Feng, W. Christmas, J. Kittler, and M. R辰tsch, Fitting 3D morphable models using local features, arXiv preprint arXiv:1503.02330, 2015. V. Kazemi and J. Sullivan, One millisecond face alignment with an ensemble of regression trees, in Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on. IEEE, 2014, pp. 1867 1874. R. W. Sumner, J. Schmid, and M. Pauly, Embedded deformation for shape manipulation, ACM Transactions on Graphics (TOG), vol. 26, no. 3, p. 80, 2007. Cardboard. https://developers.google.com/cardboard/overview, 2016. 25 of 25
- 27. Computer Vision and Remote Sensing Technical University Berlin, Germany Department of Electrical Engineering IIT Roorkee, India FURTHER READING Thank You