The overall goal of the system is twofold. On hand the local scene at the constructions site is continuously scanned with a 3D scanning device. These 3D data are accumulated and used to three-dimensionally reconstruct the local environment. This 3D scene is virtually shared with the remote user. So, on the other hand, the remote user is able to independently navigate within the virtually shared scenes and access additional information, such as construction plans etc. In this way he can assist the local worker by giving advice or instructions.
We developed a prototype system which is based on a NVIDA Jetson platform. It is battery driven and performs the 3D scanning operation. It is attached to the belt of a construction worker. A cable connects the JETSON device with a stereoscopic camera. This camera is integrated to the helmet of the construction worker. In this way a continuous 3D scanning of the environment of the construction worker is possible. A first data preprocessing is performed on the JETSON platform. The JETSON is connected wireless with a mobile Edge server on construction site (in our case a power laptop). This machine generates SLAM based 3D models of the local environment of the construction worker. Dependent on the viewing directions and positions of the construction worker the overall environment is accumulated to a consistent 3D point cloud. Further processing in our modules allows artifact removal and semantic understanding of the scene. This 3D scene is transmitted in a second step to a remote location. Conventional mobile networks are used for this purpose.
A second application at the remote site allows the remote user to virtually display the local 3D environment. Based on head mounted displays (HTC Vive) the remote expert can freely walk and move in the virtual scene. The big advantage compared to direct stereoscopic transmissions is that the remote expert can experience the 3D scene independent of the individual head movements of the local worker. Further on, the virtual world offers a large set of additional support options, such as overlay of further information (construction plans) etc. Based on a strong, wearable battery we tested it in on-site scanning situations with up to 1.5 hours of continuous scanning, with a total of up to 5 hours (and more) of battery duration. The system raised a high level of interest among our customers in construction and industry.
