• SLAM-based dense surface reconstruction in monocular Minimally Invasive Surgery and its application to Augmented Reality

      Chen, Long; Tang, Wen; John, Nigel W.; Wan, Tao R.; Zhang, Jian Jun; Bournemouth University; University of Chester; University of Bradford (Elsevier, 2018-02-08)
      Background and Objective While Minimally Invasive Surgery (MIS) offers considerable benefits to patients, it also imposes big challenges on a surgeon's performance due to well-known issues and restrictions associated with the field of view (FOV), hand-eye misalignment and disorientation, as well as the lack of stereoscopic depth perception in monocular endoscopy. Augmented Reality (AR) technology can help to overcome these limitations by augmenting the real scene with annotations, labels, tumour measurements or even a 3D reconstruction of anatomy structures at the target surgical locations. However, previous research attempts of using AR technology in monocular MIS surgical scenes have been mainly focused on the information overlay without addressing correct spatial calibrations, which could lead to incorrect localization of annotations and labels, and inaccurate depth cues and tumour measurements. In this paper, we present a novel intra-operative dense surface reconstruction framework that is capable of providing geometry information from only monocular MIS videos for geometry-aware AR applications such as site measurements and depth cues. We address a number of compelling issues in augmenting a scene for a monocular MIS environment, such as drifting and inaccurate planar mapping. Methods A state-of-the-art Simultaneous Localization And Mapping (SLAM) algorithm used in robotics has been extended to deal with monocular MIS surgical scenes for reliable endoscopic camera tracking and salient point mapping. A robust global 3D surface reconstruction framework has been developed for building a dense surface using only unorganized sparse point clouds extracted from the SLAM. The 3D surface reconstruction framework employs the Moving Least Squares (MLS) smoothing algorithm and the Poisson surface reconstruction framework for real time processing of the point clouds data set. Finally, the 3D geometric information of the surgical scene allows better understanding and accurate placement AR augmentations based on a robust 3D calibration. Results We demonstrate the clinical relevance of our proposed system through two examples: a) measurement of the surface; b) depth cues in monocular endoscopy. The performance and accuracy evaluations of the proposed framework consist of two steps. First, we have created a computer-generated endoscopy simulation video to quantify the accuracy of the camera tracking by comparing the results of the video camera tracking with the recorded ground-truth camera trajectories. The accuracy of the surface reconstruction is assessed by evaluating the Root Mean Square Distance (RMSD) of surface vertices of the reconstructed mesh with that of the ground truth 3D models. An error of 1.24mm for the camera trajectories has been obtained and the RMSD for surface reconstruction is 2.54mm, which compare favourably with previous approaches. Second, in vivo laparoscopic videos are used to examine the quality of accurate AR based annotation and measurement, and the creation of depth cues. These results show the potential promise of our geometry-aware AR technology to be used in MIS surgical scenes. Conclusions The results show that the new framework is robust and accurate in dealing with challenging situations such as the rapid endoscopy camera movements in monocular MIS scenes. Both camera tracking and surface reconstruction based on a sparse point cloud are eff active and operated in real-time. This demonstrates the potential of our algorithm for accurate AR localization and depth augmentation with geometric cues and correct surface measurements in MIS with monocular endoscopes.
    • A Tablet-based Virtual Environment for Neurosurgery Training

      John, Nigel W.; Phillips, Nicholas I.; ap Cenydd, Llyr; Coope, David; Carleton-Bland, Nick; Kamaly-Asl, Ian; Grey, William P.; University of Chester, Leeds General Infirmary, Bangor University, University of Manchester, Cardiff University (MIT Press, 2015-10-15)
      The requirement for training surgical procedures without exposing the patient to additional risk is well accepted and is part of a national drive in the UK and internationally. Computer-based simulations are important in this context, including neurosurgical resident training. The objective of this study is to evaluate the effectiveness of a custom built virtual environment in assisting training of a ventriculostomy procedure. The training tool (called VCath) has been developed as an app for a tablet platform to provide easy access and availability to trainees. The study was conducted at the first boot camp organized for all year one trainees in neurosurgery in the UK. The attendees were randomly distributed between the VCath training group and the Control group. Efficacy of performing ventriculostomy for both groups was assessed at the beginning and end of the study using a simulated insertion task. Statistically significant changes in performance of selecting the burr hole entry point, the trajectory length and duration metrics for the VCath group, together with a good indicator of improved normalized jerk (representing the speed and smoothness of arm motion), all suggest that there has been a higher level cognitive benefit to using VCath. The app is successful as it is focused on the cognitive task of ventriculostomy, encouraging the trainee to rehearse the entry point and use anatomical landmarks to create a trajectory to the target. In straight-line trajectory procedures such as ventriculostomy, cognitive task based education is a useful adjunct to traditional methods and may reduce the learning curve and ultimately improve patient safety.
    • Training Powered Wheelchair Manoeuvres in Mixed Reality

      Day, Thomas W.; John, Nigel W.; University of Chester (IEEE, 2019-09)
      We describe a mixed reality environment that has been designed as an aid for training driving skills for a powered wheelchair. Our motivation is to provide an improvement on a previous virtual reality wheelchair driving simulator, with a particular aim to remove any cybersickness effects. The results of a validation test are presented that involved 35 able bodied volunteers divided into three groups: mixed reality trained, virtual reality trained, and a control group. No significant differences in improvement was found between the groups but there is a notable trend that both the mixed reality and virtual reality groups improved more than the control group. Whereas the virtual reality group experienced discomfort (as measured using a simulator sickness questionnaire), the mixed reality group experienced no side effects.
    • The Use of Stereoscopy in a Neurosurgery Training Virtual Environment

      John, Nigel W.; Phillips, Nicholas I.; ap Cenydd, Llyr; Pop, Serban R.; Coope, David; Kamaly-Asl, Ian; de Souza, Christopher; Watt, Simon J.; University of Chester, Leeds General Infirmary, Bangor University, University of Manchester, Salford Royal NHS Foundation Trust, Cardiff University (MIT Press, 2017-03-15)
      We have previously investigated the effectiveness of a custom built virtual environment in assisting training of a ventriculostomy procedure, which is a commonly performed procedure by a neurosurgeon and a core task for trainee surgeons. The training tool (called VCath) was initially developed as a low fidelity app for a tablet platform to provide easy access and availability to trainees. Subsequently we have developed a high fidelity version of VCath that uses a stereoscopic display to immerse the trainee in the virtual environment. This paper reports on two studies that have been carried out to compare the low and high fidelity versions of VCath, particularly to assess the value of stereoscopy. Study 1 was conducted at the second annual boot camp organized for all year one trainees in neurosurgery in the UK. Study 2 was performed on lay people, with no surgical experience. Our hypothesis was that using stereoscopy in the training task would be beneficial. Results from Study 1 demonstrated that performance improved for both the control group and the group trained with the tablet version of VCath. The group trained on the high fidelity version of VCath with a stereoscopic display showed no performance improvement. The indication is that our hypothesis is false. In Study 2, six different conditions were investigated that covered the use of training with VCath on a tablet, a mono display at two different sizes, a stereo display at two different sizes, and a control group who received no training. Results from this study with lay people show that stereoscopy can make a significant improvement to the accuracy of needle placement. The possible reasons for these results and the apparent contradiction between the two studies are discussed.
    • Using and Validating Airborne Ultrasound as a Tactile Interface within Medical Training Simulators

      Hung, Gary M. Y.; John, Nigel W.; Hancock, Chris; Hoshi, Takayuki; University of Chester (Springer International Publishing, 2014-10)
      We have developed a system called UltraSendo that creates a force field in space using an array of ultrasonic transducers cooperatively emitting ultrasonic waves to a focal point. UltraSendo is the first application of this technology in the context of medical training simulators. A face validation study was carried out at a Catheter Laboratory in a major regional hospital.
    • Using Virtual Reality to Experience Different Powered Wheelchair Configurations

      Day, Thomas W.; Headleand, Christopher J.; Pop, Serban R.; John, Nigel W.; Dobson, William; University of Chester, University of Lincoln (2017-09-31)
      This paper presents recent additions to our Wheelchair-VR application, in particular the use of different drive configurations. We have previously shown that Wheelchair-VR can be used to improve driving skills. Here we consider the utility of the application in allowing users who are in the process of purchasing or upgrading a wheelchair to experience different configurations and options in a cost-effective virtual environment. A preliminary study is presented, which suggests that this approach can be effective.
    • Virtual Reality Environment for the Cognitive Rehabilitation of Stroke Patients

      John, Nigel W.; Day, Thomas W.; Pop, Serban R.; Chatterjee, Kausik; Cottrell, Katy; Buchanan, Alastair; Roberts, Jonathan; University of Chester; Countess of Chester Hospital NHS Foundation Trust; Cadscan Ltd (IEEE, 2019-10-14)
      We present ongoing work to develop a virtual reality environment for the cognitive rehabilitation of patients as a part of their recovery from a stroke. A stroke causes damage to the brain and problem solving, memory and task sequencing are commonly affected. The brain can recover to some extent, however, and stroke patients have to relearn to carry out activities of daily learning. We have created an application called VIRTUE to enable such activities to be practiced using immersive virtual reality. Gamification techniques enhance the motivation of patients such as by making the level of difficulty of a task increase over time. The design and implementation of VIRTUE is presented together with the results of a small acceptability study.
    • Virtual reality training in cardiopulmonary resuscitation in schools

      Rees, Nigel; John, Nigel W.; Beever, Lee; Vaughan, Neil; Powell, C; Fletcher, A; Welsh Ambulance Services NHS Trust; Swansea University; University of Chester; University of Exeter; British Heart Foundation; London School of Hygiene & Tropical Medicine (Mark Allen Healthcare, 2021-09-11)
      UK average survival from Out of Hospital Cardiac Arrest (OHCA) survival is around 8.6%, which is significantly lower than other high performing countries with survival rates of over 20%. A cardiac arrest victim is 2–4 times more likely to survive OHCA with bystander CPR provision. Mandatory Teaching CPR to children in school is acknowledged to be the most effective way to reach the entire population and improving the bystander CPR rate and is endorsed by the World Health Organization (WHO) “Kids Save Lives” statement. Despite this, Wales is yet to follow other home nations by including CPR training as a mandatory within the school’s curriculum. In this paper, we explore the role of teaching CPR to schoolchildren and report on the development by Computer scientists at the University of Chester and the Welsh Ambulance Services NHS Trust (WAST) of VCPR, a virtual environment to help teach the procedure. VCPR was developed in three stages: identifying requirements and specifications; development of a prototype; and management—development of software, further funding and exploring opportunities for commercialisation. We describe the opportunities in Wales to skill up the whole population over time in CPR and present our Virtual reality (VR) technology is emerging as a powerful for teaching CPR in schools.
    • VRIA - A Framework for Immersive Analytics on the Web

      Butcher, Peter; John, Nigel W.; Ritsos, Panagiotis D.; University of Chester and Bangor University (ACM, 2019-05)
      We report on the design, implementation and evaluation of <VRIA>, a framework for building immersive analytics (IA) solutions inWeb-based Virtual Reality (VR), built upon WebVR, A-Frame, React and D3. The recent emergence of affordable VR interfaces have reignited the interest of researchers and developers in exploring new, immersive ways to visualize data. In particular, the use of open-standards web-based technologies for implementing VR in a browser facilitates the ubiquitous and platform-independent adoption of IA systems. Moreover, such technologies work in synergy with established visualization libraries, through the HTML document object model (DOM). We discuss high-level features of <VRIA> and present a preliminary user experience evaluation of one of our use-cases.