Browsing by Author "Pelser, Stefan"

Now showing 1 - 1 of 1
  • Thumbnail Image
    Item
    Mapping real-world objects into virtual reality to facilitate interaction using 6dof pose estimation
    (Stellenbosch : Stellenbosch University, 2024-03) Pelser, Stefan; Theart, Rensu; Stellenbosch University. Faculty of Engineering. Dept. of Electrical and Electronic Engineering.
    ENGLISH ABSTRACT: Virtual reality (VR) has been around for decades, but its surge into the mainstream spotlight has only recently taken hold. With this renewed focus, various avenues are being explored to refine VR experiences. One of these avenues, and the focus of this thesis, is the mapping of the real world into the virtual space. With the rise of convolutional neural networks (CNNs) in computer vision, they have become the cornerstone for addressing these vision challenges that remain difficult to solve with traditional algorithms, serving as one of the primary solutions to explore for mapping real-world entities into VR. This thesis delves into the intricate challenges and methodologies associated with achieving this mapping. A significant portion of the research is dedicated to the creation of diverse, synthetically generated datasets. These datasets, meticulously crafted, conform to the same format as the LineMOD dataset and encompass a range of scenarios and objects, ensuring a comprehensive foundation for model training. The study places particular emphasis on the EfficientPose model, a state-of-the-art model designed for 6DoF pose estimation of an object. The research methodology involved leveraging the headset’s built-in tracking and spatial mapping to capture the real world in the virtual domain. Subsequently, EfficientPose tracks real-world objects using standard RGB Logitech 270 webcams connected to the computer. The output data from EfficientPose is then relayed to the VR experience, mapping the object in near real-time, enhancing immersion and interaction. An implementation of Aruco marker object tracking was also implemented and served as a well-established baseline against which the deep learning approach was compared. While the EfficientPose model demonstrates potential in tracking various objects, especially in diverse VR interactions, the research sheds light on its limitations and areas for enhancement. Specific objects were used to evaluate the model’s efficacy: the duck from the LineMOD dataset, a gun, a knife, and a cube. These objects were chosen to test diverse object shapes. The duck and the cube were further tested in both textured and textureless versions to evaluate the impact of texture on pose estimation. Additionally, two distinct pipelines were established for the object integration: one where objects underwent a 3D reconstruction process, and another where the objects were taken directly from 3D printing models. The findings reveal that certain objects, due to their unique shape or symmetry, pose inherent challenges for pose estimation. Symmetric objects, in particular, present difficulties when the dataset is not tailored to account for their characteristics. Incorporating user interactions, particularly occlusions introduced by users’ hands, the study evaluates the model’s robustness and response. The results highlight the model’s commendable ability to handle such challenges, though with room for improvement to achieve seamless VR experiences. The findings reveal that while the model’s versatility is a significant asset, its accuracy, especially in real-world VR deployments, requires further refinement. The study also touches upon the importance of a multi-camera implementation, especially in scenarios with heavy occlusions. Such an implementation not only improves the robustness of the system but is identified as a crucial next step for refinement.