Cybernetica and Estonian Air Navigation Services are jointly developing and offering a remote tower system since 2016. The offering constitutes a comprehensive solution, including both the system itself and a full scale of consulting services on transition management to remote tower operations. The main aim of the remote tower system is to increase efficiency by reducing costs and optimising resources, as well as improving situational awareness, flight safety, and quality of work.
The remote tower is a technological solution enabling the provision of Air Traffic Control from remote locations. Conventional and remote tower operations differ mainly in the way visual observation of the aerodrome is carried out. To conduct their daily tasks, the Air Traffic Control Officers (ATCOs) look at monitors that show the visual presentation of the remote aerodrome instead of looking straight out of the window. Thus, visual observation of the remote aerodrome and its vicinity is achieved via a visual surveillance system and not by direct out-of-the window (OTW) view from the Air Traffic Control tower. The technology behind the whole remote tower solution needs to be secure and reliable to grant ATCOs situational awareness in accordance with ICAO regulations.
Means of Recreating a Seamless 360-degree View in Real Time
One of the main challenges for remote tower is the provision of real-time 360-degree panoramic view of the aerodrome for remote facilities. In simple terms, this means recreating the OTW view by cameras, monitors and dedicated software. To achieve near real time presentation, the image of the remote aerodrome has to be captured by cameras and displayed on the monitors of ATCOs with a latency under 1 second.
In a typical setup, Cybernetica's solution is comprised of 16 circularly installed cameras for image capturing, an on-site video wall generator for creating the panoramic view and slicing it up into smaller streams to be transported over the network. A remotely located video wall display on the receiving end receives the streams, presents overlays, and manages the video. And finally, video wall monitors display the associated panoramic subviews. The main components for recreating the out-of-the window view are brought out in the below figure. The outer edges are COTS products for availability and cost-effectiveness reasons.
Panorama Image Stitching Methods Explained
The main problem lies in how to compose a seamless panoramic view from individual camera streams without image manipulation. This is difficult even with a physical setup of cameras installed perfectly side by side. Each camera has its own picture, but the task is to merge these pictures into a panoramic view without a gap. In order to achieve this, the subviews of each camera need to be stitched together to show the result at a different location on monitors. Achieving the results, quality requirements, and safety aspects, including low latency, good image quality, redundancy, low loss, etc. cannot be compromised.
Panorama stitching is not just about simple positioning to merge the video streams of adjacent cameras. Image processing tools featuring sophisticated algorithms are needed instead. The process of image stitching can be divided into two main parts: registration and compositing.
Registration is about finding the alignment between the images and some fine-tuning; this only needs to be done once and repeated if the camera views have changed. Registration workflow includes the following main steps:
• compensating for any distortions
• resizing to medium resolution
• finding (unique) features and matching the features
• finding transformation matrix telling how to put the images together
• fine-tuning
Once the transformation matrixes defining how the pixels in the original image coordinate system should be projected to the stitched image coordinate system are determined, they are handed over to the compositing part. Compositing consists of functions that process full resolution images to merge them into a panoramic view, this has to be done for every frame. Compositing workflow includes the following main steps:
• compensating for any distortions
• warping image to another perspective (perspective reprojection)
• estimating and compensating for exposure errors (as each camera can be with different lighting)
• finding seam masks (against ghost effects if the object moved in the overlapping area)
• blending images to smooth the final result, especially in the overlapping areas
Additional challenges for panorama stitching are parallax and lens distortion. Parallax, because the functions that find alignment between the images look for feature points that are assumed to be on the same surface or plane. For panoramic views, objects that are far enough can be approximated as being on the same surface. Lens distortions are non-linear effects that have to be removed or compensated beforehand, otherwise stitching will most likely result in very poor results. A full panorama example is shown on the picture below.
Technical Excellence as Safety Goal
Safety is paramount in the aviation industry and thus remote tower solution as a mission-critical system is operated in a demanding environment. Panoramic out-of-the window view of the highest quality is absolutely essential for the success of the remote tower system. The road to achieving the desired goal of seamless panoramic 360-degree view was time-consuming and strenuous. The world of video is a very specific domain with a steep learning curve and lots of small details wherein the difficulties lie. But the greater the effort, the greater the reward, and the expertise that naturally follows. The seamless panoramic view that comes within near real-time latency is a sine qua non technical excellence that enhances situational awareness and allows to meet consistent high performance and safety regulations necessary for remote tower operations.