ºÝºÝߣ

ºÝºÝߣShare a Scribd company logo

(Paper note) Real time rgb-d camera relocalization via randomized ferns for keyframe encoding

?Download as PPTX, PDF?
?
e8xu
e8xu

Paper note I take a note on relocalization module of ElasticFusion, and try to address why ElasticFusion failed (loss track) on certain image sequences. Providing some improvement direction for futhure research. Glocker, Ben, et al. "Real-time rgb-d camera relocalization via randomized ferns for keyframe encoding." IEEE transactions on visualization and computer graphics 21.5 (2015): 571-583.

Convert to study materialsBETA

Transform any presentation into ready-made study material¡ªselect from outputs like summaries, definitions, and practice questions.

1 of 22
Downloaded 11 times
Paper Note Real-time rgb-d camera relocalization via randomized ferns for keyframe encoding Organized by: Jacky Liu jjkka132.slide (at) gmail.com 03 Feb 2017 National Chung Cheng University RVL (Robot Vision Laboratory) Paper Note, Jacky Liu, 03 Feb 2017
Why to read? ? To improve the robustness of ElasticFusion[1], one needs to study why relocalization module failed from recover after taking ¡°white wall¡± scene. ? This paper is the relocalization module of ElasticFusion. 03 Feb 2017 2 [1] Whelan, Thomas, et al. "ElasticFusion: Dense SLAM Without A Pose Graph." Robotics: science and systems. Vol. 11. 2015.
Paper information ? MLA reference Glocker, Ben, et al. "Real-time rgb-d camera relocalization via randomized ferns for keyframe encoding." IEEE transactions on visualization and computer graphics 21.5 (2015): 571-583. ? Citation until 03 Feb 2017: 9 times ? Key words 1. Camera relocalization 2. Tracking recovery 3. Dense tracking and mapping 4. Marker-free augmented reality 03 Feb 2017 3
Outline 1. Real-time method 2. Compact representation of randomized fern 3. No spatial sampling heuristics 03 Feb 2017 4
Keyframe-based relocalization via randomized ferns for frame encoding 1. Frame encoding 2. Frame dissimilarity via Hamming Distance 3. Precision/Recall of BlockHD 4. Harvesting Keyframes 5. Tracking Recovery via pose retrieval 6. Fern Construction 03 Feb 2017 5
Thinking behind topics A. Frame encoding ¡ú Efficient way to represent frames, instead of using whole image. B. Frame dissimilarity via Hamming Distance ¡ú How to compare two frames? C. Precision/Recall of BlockHD ¡ú BlockHD encoding property D. Harvesting Keyframes ¡ú How to determine ¡°how densely we record the keyframe?¡± E. Tracking Recovery via pose retrieval ¡ú How to use encoded datastructure? F. Fern Construction ¡ú Detail of randomize fern method 03 Feb 2017 6
A - Frame encoding 03 Feb 2017 7 ¡­ m blocks (fern) n node Frame code: ?????1 ?????2 ¡­ ????? ? = 0010 1101 ¡­ 0100 Frame id in order: 1 2 3 ¡­ k Code table for frame id lookup
B - Frame dissimilarity via Hamming Distance 03 Feb 2017 8 ??????? ? ? ? , ? ? ? = 1 ? ?=1 ? ? ? ? ? ¡Ô ? ? ? ? Frame code: ?????? = 0010 1101 ¡­ 0100 Frame code: ?????? = 0110 1001 ¡­ 0100 ?1 ?2 ? ? Ex. Frame_I has 3 blocks Hamming distance between frame I and J is 2 ??????? ??????, ?????? = 1 ? ?=1 ? ?????? ¡Ô ?????? = 1 3 ? 2 = 0.66
C - Precision/Recall of BlockHD code length ¨J change that two frame with low BlockHD ¨K precision ¨J recall ¨K vise versa 03 Feb 2017 9 Frame code: ?????? = 0010 1101 ¡­ 0100 Frame code: ?????? = 0110 1001 ¡­ 0100
D - Harvesting Keyframes ? How to determine ¡°how densely we record the keyframe?¡± ? Avoid to use spatial sampling heuristics => non optimal scene coverage ? Minimum BlockHD K_I ?? = min ?? ??????? ? ? ? , ? ? ? = min ?? ( ? ? ??? ? ) 03 Feb 2017 10 If ?? > ?, which t is predefined threshold add current frame as new keyframe else discard current frame
E - Tracking Recovery via pose retrieval ? When tracking fails for an incoming frame, use most similar keyframe to retrieve pose proposals for tracking recovery. ? If reinitialization is unsuccessful for all proposed poses => discard current frame, repeat recovery until tracking is recovered ? Possible enhancement Actively hint user to move the camera into areas of good keyframe coverage. 03 Feb 2017 11
F - Fern Construction ? Randomized parameters ? Sampling pixel position x (uniform distribution) ? Number of ferns m ? Keyframe acceptance threshold t ? Number of neighbors k 03 Feb 2017 12
Experiment overview ? System based on KinectFusion (model based ICP) ? Two distinct evaluation set ? Tracked frame for simulating harvesting of keyframes ? Performance evaluation ? Challenges of dataset 1. Ambiguties (repeated steps in ¡®Stairs¡¯) 2. Specularities (reflections in ¡®RedKitchen¡¯) 3. Motion blur 4. Lighting conditions 5. Flat surfaces 6. Sensor noise 03 Feb 2017 13
Experiment ¨C determine loss ? Plausibility check on the magnitude of camera motion and on the ICP residual error. ? If the relative motion or residual is too large, ICP reports tracking loss and relocalization is initiated 03 Feb 2017 14
Performance ? Percentage of frames for which the pose was successfully recovered. ? Definition of ¡°successful recovery¡± Compare estimated pose with ground truth ? Translation error < 2cm ? Rotation error < 2 degrees ? Averaging three run to count ¡°randomize¡± part of the algorithm 03 Feb 2017 15
Pose proposal strategies ? Using which keyframe pose to do ICP? ? NN (nearest neighbor) ¨C keyframe with smallest BlockHD ? kNN (k nearest neighbor) ¨C Exam each k keyframe with smallest BlockHD, and pick the one with smallest ICP residual ? WAP (weighted average pose) 03 Feb 2017 16
Result ? Percentages correspond to the number of successfully recovered frames 03 Feb 2017 17
Offline v.s. Real-time ? Offline training methods outperform proposed real- time method. However, offline method is limited to revisited scene. 03 Feb 2017 18
Scalability to larger scenes ? Sufficient for representing large scenes of more than 30m^3 and thousands of frames. 03 Feb 2017 21
Marker-free augmented reality ? AR become more applicable when robustness of KinectFusion been improved by relocalization algorithm. 03 Feb 2017 22
Conclusion ? Solve the main cause of tracking failure in KinectFusion ? Compact frame encoding ? Lower performance compare to offline method Future ? frame-to-frame tracking could give additional information about the relative pose trajectory. ? Loop closure 03 Feb 2017 23
Recall the goal of reading this paper ? To improve the robustness of ElasticFusion[1], one needs to study why relocalization module failed from recover after taking ¡°white wall¡± scene. ? ElasticFusion loose track (https://youtu.be/mOmqd82EfDY) ? After meet the door, ICP report tracking loss. ? Relocalization module perform recovery. ? No loop closure to fix the tracking drift. ? To improve the robustness of ElasticFusion => Apply loop closure. Define ¡°uncertainty¡± of frame sequence, and use it to relocate path. 03 Feb 2017 24

More Related Content

(Paper note) Real time rgb-d camera relocalization via randomized ferns for keyframe encoding

  • 1. Paper Note Real-time rgb-d camera relocalization via randomized ferns for keyframe encoding Organized by: Jacky Liu jjkka132.slide (at) gmail.com 03 Feb 2017 National Chung Cheng University RVL (Robot Vision Laboratory) Paper Note, Jacky Liu, 03 Feb 2017
  • 2. Why to read? ? To improve the robustness of ElasticFusion[1], one needs to study why relocalization module failed from recover after taking ¡°white wall¡± scene. ? This paper is the relocalization module of ElasticFusion. 03 Feb 2017 2 [1] Whelan, Thomas, et al. "ElasticFusion: Dense SLAM Without A Pose Graph." Robotics: science and systems. Vol. 11. 2015.
  • 3. Paper information ? MLA reference Glocker, Ben, et al. "Real-time rgb-d camera relocalization via randomized ferns for keyframe encoding." IEEE transactions on visualization and computer graphics 21.5 (2015): 571-583. ? Citation until 03 Feb 2017: 9 times ? Key words 1. Camera relocalization 2. Tracking recovery 3. Dense tracking and mapping 4. Marker-free augmented reality 03 Feb 2017 3
  • 4. Outline 1. Real-time method 2. Compact representation of randomized fern 3. No spatial sampling heuristics 03 Feb 2017 4
  • 5. Keyframe-based relocalization via randomized ferns for frame encoding 1. Frame encoding 2. Frame dissimilarity via Hamming Distance 3. Precision/Recall of BlockHD 4. Harvesting Keyframes 5. Tracking Recovery via pose retrieval 6. Fern Construction 03 Feb 2017 5
  • 6. Thinking behind topics A. Frame encoding ¡ú Efficient way to represent frames, instead of using whole image. B. Frame dissimilarity via Hamming Distance ¡ú How to compare two frames? C. Precision/Recall of BlockHD ¡ú BlockHD encoding property D. Harvesting Keyframes ¡ú How to determine ¡°how densely we record the keyframe?¡± E. Tracking Recovery via pose retrieval ¡ú How to use encoded datastructure? F. Fern Construction ¡ú Detail of randomize fern method 03 Feb 2017 6
  • 7. A - Frame encoding 03 Feb 2017 7 ¡­ m blocks (fern) n node Frame code: ?????1 ?????2 ¡­ ????? ? = 0010 1101 ¡­ 0100 Frame id in order: 1 2 3 ¡­ k Code table for frame id lookup
  • 8. B - Frame dissimilarity via Hamming Distance 03 Feb 2017 8 ??????? ? ? ? , ? ? ? = 1 ? ?=1 ? ? ? ? ? ¡Ô ? ? ? ? Frame code: ?????? = 0010 1101 ¡­ 0100 Frame code: ?????? = 0110 1001 ¡­ 0100 ?1 ?2 ? ? Ex. Frame_I has 3 blocks Hamming distance between frame I and J is 2 ??????? ??????, ?????? = 1 ? ?=1 ? ?????? ¡Ô ?????? = 1 3 ? 2 = 0.66
  • 9. C - Precision/Recall of BlockHD code length ¨J change that two frame with low BlockHD ¨K precision ¨J recall ¨K vise versa 03 Feb 2017 9 Frame code: ?????? = 0010 1101 ¡­ 0100 Frame code: ?????? = 0110 1001 ¡­ 0100
  • 10. D - Harvesting Keyframes ? How to determine ¡°how densely we record the keyframe?¡± ? Avoid to use spatial sampling heuristics => non optimal scene coverage ? Minimum BlockHD K_I ?? = min ?? ??????? ? ? ? , ? ? ? = min ?? ( ? ? ??? ? ) 03 Feb 2017 10 If ?? > ?, which t is predefined threshold add current frame as new keyframe else discard current frame
  • 11. E - Tracking Recovery via pose retrieval ? When tracking fails for an incoming frame, use most similar keyframe to retrieve pose proposals for tracking recovery. ? If reinitialization is unsuccessful for all proposed poses => discard current frame, repeat recovery until tracking is recovered ? Possible enhancement Actively hint user to move the camera into areas of good keyframe coverage. 03 Feb 2017 11
  • 12. F - Fern Construction ? Randomized parameters ? Sampling pixel position x (uniform distribution) ? Number of ferns m ? Keyframe acceptance threshold t ? Number of neighbors k 03 Feb 2017 12
  • 13. Experiment overview ? System based on KinectFusion (model based ICP) ? Two distinct evaluation set ? Tracked frame for simulating harvesting of keyframes ? Performance evaluation ? Challenges of dataset 1. Ambiguties (repeated steps in ¡®Stairs¡¯) 2. Specularities (reflections in ¡®RedKitchen¡¯) 3. Motion blur 4. Lighting conditions 5. Flat surfaces 6. Sensor noise 03 Feb 2017 13
  • 14. Experiment ¨C determine loss ? Plausibility check on the magnitude of camera motion and on the ICP residual error. ? If the relative motion or residual is too large, ICP reports tracking loss and relocalization is initiated 03 Feb 2017 14
  • 15. Performance ? Percentage of frames for which the pose was successfully recovered. ? Definition of ¡°successful recovery¡± Compare estimated pose with ground truth ? Translation error < 2cm ? Rotation error < 2 degrees ? Averaging three run to count ¡°randomize¡± part of the algorithm 03 Feb 2017 15
  • 16. Pose proposal strategies ? Using which keyframe pose to do ICP? ? NN (nearest neighbor) ¨C keyframe with smallest BlockHD ? kNN (k nearest neighbor) ¨C Exam each k keyframe with smallest BlockHD, and pick the one with smallest ICP residual ? WAP (weighted average pose) 03 Feb 2017 16
  • 17. Result ? Percentages correspond to the number of successfully recovered frames 03 Feb 2017 17
  • 18. Offline v.s. Real-time ? Offline training methods outperform proposed real- time method. However, offline method is limited to revisited scene. 03 Feb 2017 18
  • 19. Scalability to larger scenes ? Sufficient for representing large scenes of more than 30m^3 and thousands of frames. 03 Feb 2017 21
  • 20. Marker-free augmented reality ? AR become more applicable when robustness of KinectFusion been improved by relocalization algorithm. 03 Feb 2017 22
  • 21. Conclusion ? Solve the main cause of tracking failure in KinectFusion ? Compact frame encoding ? Lower performance compare to offline method Future ? frame-to-frame tracking could give additional information about the relative pose trajectory. ? Loop closure 03 Feb 2017 23
  • 22. Recall the goal of reading this paper ? To improve the robustness of ElasticFusion[1], one needs to study why relocalization module failed from recover after taking ¡°white wall¡± scene. ? ElasticFusion loose track (https://youtu.be/mOmqd82EfDY) ? After meet the door, ICP report tracking loss. ? Relocalization module perform recovery. ? No loop closure to fix the tracking drift. ? To improve the robustness of ElasticFusion => Apply loop closure. Define ¡°uncertainty¡± of frame sequence, and use it to relocate path. 03 Feb 2017 24