Teamed With an Eye in the Sky

Teamed With an Eye in the Sky

Vision-Based Collaborative Navigation for UAV-UGV-Dismounted Units in GPS-Challenged Environments

A collaborative navigation state of affairs between manned and unmanned methods operates in a GPS-challenged or -denied floor atmosphere, pushed by a UAV flying above with entry to GPS alerts. The structure supplies relative place, velocity and perspective to the bottom items, manned and unmanned. During GPS outages, the UAV’s imaginative and prescient sensor detects and tracks options, and inertial drift is bounded by exterior measurements.

by Shahram Moafipoor, Lydia Bock, Jeffrey A. Fayman, Eoin Conroy and Bob Stadel, Geodetics Inc.

Today’s mission-critical protection purposes on land, air and at sea rely on correct, assured PNT (A-PNT) or resilient PNT. The massive variety of platforms in up to date protection methods demand distinctive and tailor-made options that deliver collectively sensors and rising applied sciences in novel methods. As present and rising threats can deny or degrade GPS entry, there’s an pressing have to develop sturdy autonomous navigation theories, architectures, algorithms, sensors and, finally, methods that present assured GPS-level PNT efficiency in all environments, impartial of GPS.

Currently, there is no such thing as a “silver-bullet” system that may change the capabilities offered by GPS. Alternatives are platform/software particular of their resolution.

This article presents a collaborative navigation state of affairs between manned and unmanned methods: a number of UGVs and/or dismounted troopers working in a GPS-challenged or -denied atmosphere and UAVs flying above the atmosphere with entry to GPS alerts, as proven in Figure 1. This allows the ground-based platforms to navigate with out GPS by making the most of the overflying UAV’s entry to GPS and the relative place, velocity and perspective between the UAV and floor items.


Each impartial platform has its personal complementary navigation module, however, as a group, they share their widespread assets and cooperate to deal with widespread navigation targets. The collaborative navigation depends on knowledge flowing from the UAV to the person UGVs and dismounted troopers, that are built-in in a relative prolonged Kalman Filter (EKF) the place the UAV is denoted “secondary” and the bottom items are denoted “major.”

The system can help a single secondary, which covers a predefined space, and a number of major items related through datalink to the secondary unit. Processing is carried out within the major unit for every major/secondary pair individually. The secondary unit (UAV) used on this examine was instrumented with a payload consisting of GPS/IMU/tactical LiDAR/RGB digicam and a datalink for communications with the first items. The major items (UGVs and dismounted troopers) had been all instrumented with a processing unit and GPS/IMU sensing module.

The core of the system is a relative EKF, which makes use of IMU measurements of the first and secondary items to determine the relative inertial navigation states because the prediction mannequin. The relative remark mannequin is used to replace the relative navigation resolution and calibrate the first IMU. Typically, the relative remark is offered by differential GPS processing between major/secondary pair, however on this examine the relative place was generated by the imaginative and prescient sensor on the secondary unit fairly than GPS in an effort to simulate GPS-denied circumstances for floor items.

In this method, the imaginative and prescient sensors present relative measurements between the secondary (UAV) and first (UGV/dismounted troopers). By utilizing the relative EKF, the relative remark between the first and secondary can also be used to remotely calibrate the first’s IMU (see Figure 2).

As the first items are both autos (UGVs), and/or folks in (e.g. dismounted troopers), in an effort to measure the relative distance between the secondary and first from the imaginative and prescient sensors, these options have to be detected and tracked from the secondary unit in a model-free method. Several approaches may be thought-about together with:

  •  LiDAR-only.
  •  LiDAR/digicam (level & pixel) within the type of colorized LiDAR level clouds.
  •  Mono-RGB or thermal supported by a LiDAR-based digital terrain mannequin (DTM).
  •  Dual-camera stereo configuration.

To deal with the massive quantity of LiDAR and picture knowledge related to these approaches in an inexpensive processing time, enhancements had been made to a number of features of beforehand present algorithms. These had been executed to beat dimension limitations of LiDAR and digicam knowledge, to enhance the effectivity of the automated knowledge segmentation and to offer a sturdy resolution for feature-of-interest detection.

The RGB camera-only method, regardless of many benefits, has the shortcoming of helpful operational time (day-night restrictions) and is subsequently not addressed. Thermal imaging is an attention-grabbing different as it may be applied equally effectively day and night time and since it’s delicate to things’ thermal signatures. However, the primary restriction with utilizing these cameras is the small sensor dimension (e.g. 640×512 pixels), which inhibits use over in depth areas. One option to overcome this restriction is to mount the system on a gimbal; nevertheless, in our multi-sensor integration, this method will not be possible because the LiDAR/digicam are straight georeferenced in actual time, implementing a strong geometry between elements.

For these causes, right here we give attention to the primary two approaches for characteristic detection and monitoring: LiDAR-only and LiDAR-RGB digicam (level & pixel).


LiDAR sensors may be labeled into three classes: multi-beam tactical, solid-state LiDAR and single-beam aerial scanners. Here we use multi-beam tactical grade LiDAR sensors, as the opposite two classes are redundant for our software on account of a slender subject of view (FOV). From use in autonomous driving methods, wealthy algorithms have been developed for fast characteristic detection and monitoring. However, we discovered that the signature of options from UAV-based LiDAR sensors wanting down from the platform are solely totally different from typical autonomous driving methods, wherein the LiDAR is oriented for horizontal scanning and the atmosphere is scanned in full circle (360°).

The different problem is point-cloud decision. A multi-beam tactical LiDAR sensor sometimes scans an space with 360° horizontal FOV and 20-40° vertical FOV, as proven in Figure 3. While the horizontal angular decision may be as excessive as 0.1°, the vertical FOV is normally bigger. A tactical-LiDAR sensor generally has 8–64 channels (beams) with a vertical angular decision of 1–2°. One can estimate the decision by dividing the variety of beams over the span of the vertical FOV.

In new scanners, the laser channels are usually not emitted in a symmetric sample. Rather, they’re concentrated extra alongside the middle to optimize vertical angular decision. The projection of the horizontal angular decision to the bottom can decide the along-track decision, and the across-track decision is derived by projecting the vertical angular decision to the bottom. These definitions are important, as we have to affirm having sufficient along-track and across-track decision to detect options of curiosity.

A mix of along-track and across-track decision is outlined because the footprint dimension of options (or, merely, decision). For UAV-based methods, the signature of options is primarily influenced by the across-track decision, which is a operate of flight altitude and pace. Figure 3 exhibits this signature for a automobile captured with a tactical-level UAV-LiDAR, acquired over a full cycle of m-beam scanning. This angular decision, denoted FOVV in Figure 3, can mission as much as 0.5 m across-track decision at a low flight altitude. This across-track decision could also be massive sufficient to detect automobile options, as proven in Figure 3 for one epoch of LiDAR scanning knowledge.

Unlike UGV-signatures, human physique detection from the air may be difficult with tactical grade LiDAR sensors, because the footprint dimension of LiDAR level clouds is usually decrease at excessive flight altitudes. As an instance, in Figure 4 (left), by flying at a 20 m altitude at a pace of three m/s, a tactical LiDAR can present level clouds as much as 7 cm along-track decision, which is satisfactory for human physique detection; this quantity will likely be doubled for a 40 m flight altitude, as proven in Figure 4 (proper), making it tougher to detect human our bodies even with supervisory strategies. This low decision may be important, as folks, in contrast to autos, sometimes transfer in a decent group, separated by minimal distance.

In the LiDAR-only method, our algorithm for characteristic automobile/human identification begins with extracting elevated factors from particular person scanlines and clustering them. However, this single criterion doesn’t facilitate profitable characteristic extraction in complicated environments. Therefore, further standards had been utilized to kind an computerized characteristic extraction system.

The modifications had been impressed by algorithms developed in machine-learning fields. With this method, automobile monitoring is predicated on detecting areas of curiosity, eradicating the bottom floor, clustering, and monitoring a bounding field round every characteristic of curiosity. These steps had been utilized in a distinct order in our analog method. Further testing utilizing machine studying strategies confirmed that operators figuring out options search for objects which can be:

  •  Elevated with respect to the grid floor.
  •  Rectangular in form in various dimension for various characteristic sorts.
  •  Longitudinally and vertically positioned for various characteristic sorts.
  •  Do not intervene with different options.
  •  Have applicable dimension.

Figure 5 summarizes the applied algorithm. It contains two phases, detection and monitoring. The algorithm begins with sturdy detection of options. To pace up scanning, the elevated factors had been merely reduce out by subtracting a low-order airplane, match to the grid floor in every phase. Then, clustering of peak factors is carried out primarily based on standards comparable to distance and angle. The course of is adopted by an in depth evaluation of candidate targets to lastly establish the factors as a characteristic (automobile/individual).

The final feature-identification step is detection and removing of non-feature objects from extracted characteristic candidates. In addition to object attributes comparable to size and quantity, non-feature objects could be detectable by exploiting depth knowledge.


This algorithm have to be absolutely applied from the primary body. Once the options are recognized, the algorithm is simplified to guidelines for monitoring. The monitoring filter makes use of fixed velocity, fixed elevation and fixed flip fashions concurrently to generate a monitoring resolution. In the case of multi-features, this algorithm runs individually for every seed object. In this case, a brand new unit is faraway from the answer and/or re-entered, and the algorithm begins over.

The key distinction within the applied algorithm and present deep-learning algorithm is within the scale of notion of atmosphere. In autonomous driving, the LiDAR scans the encompassing atmosphere in a big space, from which it’s simple to develop a prediction mannequin for particular person options that final till they’re previous the autonomous automobile.

However, with a UAV, options seen on the bottom exist inside a slender subject of view, which requires extension of the structure to a swam of UAVs.

Most of the challenges mentioned are related to UGVs, because the dismounted items journey over a smaller scale. In addition, there are extra challenges with human detection when working from greater altitudes. In future work, a thermal digicam is taken into account to enhance the sensor configuration for improved detection and monitoring of people.


In the Point & Pixel method, the digicam is used for colorization of the LiDAR level clouds. This colorization provides an extra dimension to the purpose clouds, which facilitates faster detection of options utilizing RGB-color segmentation.

Figure 6 exhibits the uncooked LiDAR level clouds (left) and the colorized level clouds (proper), wherein the options are extra seen for detection and classification. The standard method for colorizing LiDAR level clouds is to merge two layers of data: geo-referenced level clouds and a Geo-TIFF orthomosaic picture. To generate an orthomosaic, the scan space have to be lined by a survey grid such that the photographs are captured with excessive overlap and sidelap to offer robust geometry to be used within the aerotriangulation course of.

The most important problem in colorizing LiDAR level clouds is in producing the orthomosaic topic to parameters that might not be accessible for a lot of mapping circumstances, together with real-time mapping.

Knowing the correct place and perspective of captured photos, one logical method is to rectify photos and use the orthogonal picture for colorizing photos. In implementation, one should fastidiously compensate for the digicam’s inside calibration parameters, digicam boresight and any doable time-tag latency.

Once these parameters have been accounted for, we nonetheless observe random colorization errors. Further examine discovered that the digicam is extra delicate to uncovered vibrations of the UAV throughout knowledge acquisition than the LiDAR sensor. Several {hardware} and software program modifications had been made.

Starting from {hardware}, every UAV has a novel vibration signature, which requires damper designs ample to denoise surprising vibrations. Unaccounted-for vibrations trigger many points, from mechanical fatigue, which might put on out fragile components and electrical elements prematurely, to IMU sensor studying outputs, blurring the captured imagery and fanning results displayed on LiDAR sensor measurements and subsequent level clouds.

From the software program perspective, the primary challenges had been in creating an area-based patching algorithm for picture stitching, and in using key level options between the LiDAR level clouds and the photographs in every patch for management registration. This course of is adopted by detection/rejection of out-of-order photos in every patch and at last color-balancing throughout the photographs. Figure 7 exhibits the ultimate resolution for single picture and over an space lined with a number of photos.

To use the geo-referenced colorized level cloud, we experimented with interpolating/extrapolating the irregular colorized level clouds to an everyday grid and treating it as a 2D vary picture (raster primarily based). In this case, every level carries a depth worth, laser depth and RGB pixel values. One benefit of representing the purpose clouds within the raster kind is that they are often handled as photos. Thus, a spread of photogrammetry/computer-vision image-processing instruments and strategies may be utilized to detect options of curiosity from them.


We evaluated the efficiency of the collaborative navigation system on real-world knowledge. A UAV instrumented with a payload consisting of GPS, MEMS IMU, LiDAR (tactical 8-channel) and an RGB digicam (Sony 6000) was flown over an space the place a number of autos/folks had been working.

All sensors had been exactly synchronized to the GPS/IMU navigation resolution. The LiDAR knowledge was time-stamped and recorded internally, and the digicam photos had been exactly time-tagged and geo-tagged throughout the course of. For UGV detection, a UAV-UGV pair was operated over an open space; for the UAV-dismounted consumer, an individual was strolling within the space.

For each checks, the UAV was flown at 30 m flight altitude (AGL) and three m/s velocity. To make sure the UAV and first items are seen from the imaginative and prescient sensors on the UAV, exact flight-planning contemplating the requisite overlap space is required. We discovered this overlap space is a operate of the relative pace between the UAV and UGV/ people and the efficient swath width on the bottom. Table 1 summarizes efficient swath protection for a transferring UAV to cowl the realm with curiosity secondary items.

By flying at 3 m/s, the swath is elevated to ~18 m when utilizing the Tactical 8 and ~25 m for Tactical 16 LiDAR configurations. These issues permit exact flight planning for the UAV with respect to the UGV. Figure 8 exhibits the UAV and UGV trajectory. The GPS outage was simulated after the primary loop.


Figure 9 exhibits a pattern of the detected and tracked options, utilizing the described algorithm.

Each characteristic of curiosity was thought-about a major unit, and the relative place derived from the secondary unit was used within the relative navigation filter for every pair. Figure 10 exhibits the relative navigation resolution derived from the collaboration of the UAV-UGV case.

Once the relative navigation resolution is offered, absolutely the major navigation resolution may be decided by integrating the relative resolution to absolutely the navigation resolution of the secondary. Results in Figure 11 present improved navigation efficiency when using the relative observations throughout GPS outages. Specifically, the drift of the INS resolution is bounded by the exterior measurements offered by the relative EKF and the monitoring filter when GPS is unavailable, sustaining the specified efficiency in GPS hostile circumstances.

The relative EKF can calm down the requirement for frequent updates of the system. If the structure is just primarily based on retrieving the first items by including the relative place from the first to the secondary, the system is susceptible when no place replace is out there. However, by remotely calibrating the IMU of the first unit, the first relative navigation resolution can nonetheless be estimated, which might coast independently because the navigation resolution. This functionality additionally permits discount of the replace charge to lower than 1 Hz.

Test outcomes present that the collaborative navigation resolution between UAV and UGV-dismounted customers can enhance navigation and environmental monitoring in difficult environments.


The algorithms that allow collaborative navigation methods function throughout a number of platforms, together with UAV/UGV/dismounted warfighters. The collaborative navigation permits major items, UGV/unmanned troopers, to leverage data from their onboard sensors along with shared knowledge from the secondary UAV to attain extremely correct navigation efficiency in all circumstances, even in areas the place GPS data is unavailable.

The collaborative navigation is executed in a three-layer modular method for knowledge fusion: UGV/dismounted characteristic extraction from vision-sensors in LiDAR solely, or level (LiDAR) & pixel (digicam) mixture; characteristic monitoring utilizing a dynamic mannequin; and relative prolonged Kalman Filter used to retrieve the first navigation resolution (place, velocity and perspective) primarily based on the secondary unit.

The problem lies in managing the massive quantity of LiDAR-camera knowledge to kind by the information to extract options of curiosity and repeatedly observe them to measure the relative place between them. By sharing relative positioning data, geo-referenced LiDAR level clouds, geo-registered imagery and different navigation knowledge on particular person platforms, the collaborative navigation system will enhance general navigational accuracy.

The collaborative state of affairs can prolong to different purposes, comparable to UAV-based safety and autonomous driving methods.

A extra sensible method for future research could be in creating a community (swarm) of secondary items. In this case, the collaborative structure will flip to a 4D drawback. Each UAV node covers a selected space, and when a major node makes an inquiry, it should first understand the atmosphere. The inquiry is responded to by the neighbored secondary nodes. The inquiry may be associated to contingency occasions (its place, goal dedication, goal monitoring, and many others.). Developing a collaborative algorithm over a community requires a distributed knowledge fusion algorithm, which ought to nonetheless meet real-time or near-real-time computing constraints, impartial of the community propagation delay.


An earlier model of this text appeared as a paper at ION GNSS+ 2020 Virtual and is out there at Geodetics Inc. is an AEVEX Aerospace Company.


Shahram Moafipoor is a senior navigation scientist and director of analysis and growth at Geodetics, Inc. He holds a Ph.D. in geodetic science from The Ohio State University.

Lydia Bock is the president, co-founder and chief govt officer of Geodetics Inc. She has 35+ years of trade expertise, together with positions at SAIC and Raytheon. She holds a Ph.D. in engineering from Massachusetts Institute of Technology and has gained the Raytheon’s Micciolli Scholar award and The Ohio State University Distinguished Alumni Award.

Jeffrey A. Fayman is vice chairman of enterprise & product growth at Geodetics. He holds a Ph.D. in pc science from the Technion-Israel Institute of Technology.

Eoin Conroy accomplished his M.Sc. in geographical data methods and distant sensing at Maynooth University in Ireland. He focuses on Geodetics’ LiDAR and photogrammetric mapping initiatives.

Bob Stadel serves as vice chairman for AEVEX Aerospace. He is a graduate of the Naval Nuclear Power Program and holds an MBA from the University of Utah.