Examining the Path Toward a Driverless Tomorrow

By Jeff Jurgens, Director of Product Stewardship, Association of Equipment Manufacturers—

Autonomous equipment offers significant benefits with respect to efficiency, cost savings, and safety. These machines are poised to have a profound impact in the off-road environment, upsetting and defying current models in relation to the size of the equipment and the role of dealers and OEMs, as well as impacting the individual operator.

Although the equipment sectors that the Association of Equipment Manufacturers represents have historically been hesitant to adopt innovative technologies, they are set for a significant amount of change as autonomous vehicles dramatically alter the nature of work. Future off-road autonomous equipment will handle the cycle of operation and cyclical tasks, while teleguided machines are maneuvered by operators, sometimes in simulators thousands of miles away.

History of autonomous equipment

The first autonomous testing began in Japan in 1990. However, the process actually began with the release of field management software for mines in the late 1980s, and it became the foundation for Komatsu’s autonomous platform. The first commercial deployment was in Chile in 2007. In 2008, another system was launched at an Australian iron ore mining operation. From a remote operations center in Perth, Western Australia, workers operate autonomous mining vehicles at mines more than 1,200 kilometers (745 miles) away. Finally, in 2013, Komatsu introduced the world’s first dozer with fully automatic blade control.

The lessons learned from those applications eventually led to the first North American deployment in 2016. By 2018, Komatsu had moved over 2 billion tons with 130 fully autonomous mining trucks running across seven operational sites (six of which are fully autonomous sites). As of June 2020, those numbers have jumped to 3 billion tons moved by 251 trucks, and the fleet still currently maintains a perfect safety record with no incidents.

In another example of the exponential rate this industry is developing, Caterpillar has doubled the amount hauled in the 16 months since reaching 1 billion tons hauled in November 2018. To date, Caterpillar has 276 autonomous trucks in operation that have driven over 42 million miles without a lost-time injury.

Current status

Autonomous vehicles (AVs) currently hold a market share of less than 1% of total construction equipment sales worldwide. However, given that global equipment sales are projected to grow 2.5% each year until 2022, coupled with exponential growth in the adoption of AVs, the market share is expected to double in the next two years.

Automated shuttle buses are in use in many countries. As mentioned above, autonomous heavy haulers are already in use in mining in several countries, with automated agricultural and forestry equipment currently being developed as well. While a blend of GPS and other location tracking sensors, image sensors, and telematics assist John Deere vehicles to navigate fields today, the company still can’t truly replicate everything a human would see and feel sitting in the tractor cab. The determining factors here are farmer perception and added value. This will remain the case until more tasks, and not just movement, become automated.

Meanwhile, the forestry sector has seen limited progress with automation, largely due to working conditions being considerably larger than those experienced by other sectors. Work sites are often in remote areas, on steep slopes, and in very rugged and variable terrain. In these remote areas, there are generally no cell towers or Wi-Fi, and communication is currently expensive with limited bandwidth. The next generation of large fleets of low earth orbit satellites, however, is expected to address this problem.

Designed in partnership with Swedish engineering company Sandvik, the Syama underground gold mine became the world’s first fully autonomous mine operation. It operates with fully automated trucks, loaders, and drills. The fully autonomous operation means that the mine can operate 24 hours a day, with all operations overseen from a remote operation center.

Some companies are applying self-driving construction machinery to do repetitive tasks such as pouring concrete, bricklaying, welding, and even demolition. Excavation and other prep work is also completed by autonomous or semi-autonomous bulldozers, which can ready a jobsite to exact specifications with the help of a human programmer. A company working in heavy civil construction and grading used Built Robotics-equipped dozers to finish housing pads in a residential development.

“Built’s autonomous dozer is the real deal, and we’ve had it finish-grade hundreds of pads so far,” said Eric McCosker, project manager at Independent Construction. “Given the shortage of skilled hands in our area, that’s added production we’ve needed.”

Nuts and bolts

The challenges for autonomous off-road equipment vary significantly from those of passenger vehicles. In a farm field or construction site, lane markers or curbs are not present to assist with vision, nor is there a LiDAR system to sustain the vehicle on a required track. Moreover, automobiles do not have projections like booms and buckets, nor can most rotate on their undercarriage like a shovel. Systems for an off-road machine need to know exactly where they are, their bearing, and where their attachments are located.

Precision excavation is just what the name implies. First, the machine needs to know where it is and its orientation. In North America, we are most familiar with global positioning system (GPS) technology. As you have seen firsthand if you’ve used a smartphone navigation app, regular GPS is only accurate to about 6 meters to 10 meters (20 feet to 33 feet), and it cannot provide heading information unless the vehicle is in motion. These limitations are addressed using either differential GPS (DGPS) or real-time kinematic (RTK) GPS. DGPS and RTK both require a local stationary GPS receiver, and its location must be precisely known. The stationary receiver compares its known location to its location as calculated by the GPS receiver.

From this information, it can generate corrections, which it transmits to the other mobile GPS receivers located on the vehicle, resulting in a much more accurate solution of around 10 cm to 30 cm (4 inches to 12 inches). The heading limitation of GPS is overcome by using two GPS receivers on the machine, with antennas separated in space by some reasonable distance (1 meter to 2 meters [3 feet to 6 feet]). The absolute position of receiver 1 to receiver 2 antennae is used to calculate the absolute heading. To supplement GPS and RTK, an inertial measurement unit is installed in the vehicle, usually centrally located to the vehicle. An inertial measurement unit measures three dimensions of linear acceleration, three dimensions of rotational rate, and sometimes also a 3D magnetic field. In highly controlled work zones, the primary decision-making software and algorithms will reside in a centralized control facility and will probably include real-time human monitoring and oversight of site operation. The autonomous trucks operate with such repeatable precision that the engineers are now incorporating “vectoring,” in which an algorithm shifts the haulage route laterally a meter or so to keep the trucks from wearing ruts in the haul road.

In comparison to the applications in mining, agricultural robots are often small or mid-sized and are designed to autonomously navigate and automatically take some precise plant-specific action. Machine vision technology is often a core competency of these robots, enabling the robots to see, identify, localize, and take some intelligent site-specific action on individual plants. The machine vision increasingly uses deep learning algorithms often trained on expert-annotated image datasets, allowing the technology to far exceed the performance of conventional algorithms, as well as to match or even exceed that of expert agronomists.

For instance, Continental is developing concepts for robots that can be fitted modularly for agricultural purposes, operate autonomously, and used with the appropriate attachments to perform tasks such as monitoring the growth process of plants, precise weeding, or systematically sowing seeds and delivering nutrients. Camera-based, multispectral, or thermographic sensors are particularly important for monitoring plant growth or soil conditions.

Many robots have already grown in capability, offering faster speeds, higher frames per second, more ruggedized designs, higher onboard energy for longer operation time and a heavier load, and so on. Simple robotic implements utilizing basic row-following vision technology are already mature and not uncommon in organic farms. Advances in vision technology are transforming tractor-pulled implements, upgrading them into intelligent computerized tools able to take plant-specific precise action.

This approach does not focus on autonomy, although the tractor itself can readily be made autonomous to render the entire system automatic if necessary. This system is designed to become competitive in large farms that demand high productivity, which in turn is linked to technology parameters such as frames per second, false positives, sprayer controller speed, and so on. In the future, the system costs will likely fall, particularly if lighter versions of the algorithms on the inference side become available to render GPU processors unessential without a major performance sacrifice.

Although a rich assortment of technologies is available to AV designers – radar, GPS, GNSS, LiDAR, ultrasonic, infrared, and video – a confounding problem that inhibits off-road equipment from an easy path to autonomy is dust, dirt, and other weather conditions. Not only does weather change how the machine should act in its environment, it also reduces the accuracy of the sensors. One solution might come in redundancy – more sensors that could be used as backup.

Addressing workforce needs

Moving toward autonomous solutions may help ease the operator shortage.

“Finding experienced talent in application areas is often a challenge,” said Fred Rio, construction digital and technology product manager, Caterpillar. “Autonomous applications can provide employment opportunities to a new workforce for those with physical disabilities and wounded veterans, creating a value-added job for someone who wants to work but may face challenges in physically operating a machine. Technology and automation make the role of equipment operator more widely accessible.

“All this technology, bit data, and analytics will actually increase the number of jobs in more analytical work. It is a change from trade jobs to more analytical ones.”

Carla Boehl, senior lecturer with Curtin University’s School of the Mines, said students are being proactive in learning about the new technology.

“The students themselves are interested,” she said. “They want to do their theses in this field and learn more about automation. At a post-graduate level, we are starting to do more work regarding maintenance automation. We do a lot of big data understanding [of] what is big data [and] what algorithms can be used to support systems.”

Productivity benefits

“Using the machine autonomously improves productivity,” said Erol Ahmed, director of communications for Built Robotics. “The machine uses less fuel and moves more efficiently, which prolongs the life of the machine, reduces maintenance, and prevents unnecessary wear and tear. On average, a robotic piece of equipment from Built runs at the same speed and responsiveness as one being used manually by a person.”

By automating some tasks, skilled workers can work on more complex tasks or move to areas where they are needed most. Equipment automation also allows jobsites to run beyond normal operating hours (e.g., at night) and perform tasks in parallel so they can be done more quickly.

Autonomous machines can alter the economics of machine design, facilitating the increase of smaller machines. Indeed, this elimination of the driver overhead per vehicle is the basis of the swarm concept. There is clearly a large productivity gap today between current large and high-power vehicles and those composed of fleets of small robots. This productivity gap, however, can narrow, as the latter has substantial room for improvement.

“When we develop robots, people always say they should increase yields or lower costs, but that’s not always the case,” said Eldert van Henten, a professor at Wageningen University’s Farm Technology Group. “Robots that offer the farmer time to devote attention to business and management also have economic value.”

Reduced-cost benefits

Construction and mining technology investor Foundamental reports an average increase in mine productivity of 30% with AVs via:

• Longer production hours
• Reduced load and unit cost of 15%
• Improved tire life of 40%, since optimized controls reduce sudden acceleration and abrupt steering

While autonomous hauling system implementation would cost on average $13 million at a site with 10 trucks and 15 ancillary vehicles, the total benefits are calculated to be $19 million. This indicates that investment into autonomous technology ultimately pays off.

The first major agriculture target market is weeding. Precision action reduces consumption of agrochemicals by 90% compared to untargeted application. It also improves yield (e.g., by 5% to 10%) because collateral damage of the crops by untargeted chemical application can be minimized. This technology can further enable farmers to tackle herbicide-resistant weeds, which are a growing problem. Finally, the robots leave behind no unusable compacted soil.

Safety benefits

As noted above, in 13 years of autonomous equipment operation between Komatsu and Caterpillar, they have moved over 5 billion tons of material, and Caterpillar alone has logged over 42 million miles with zero hauler-related incidents or lost-time injuries. In contrast, a recent Department of Mines study analyzed the deaths of 52 miners over the past 12 years, finding worker fatigue and inexperience with mining risks to be the biggest cause of accidents. A report that CAVCOE coauthored predicted that with full deployment of autonomous and connected vehicles, 80% of traffic collisions, fatalities, and injuries could be eliminated. Autonomous machines can perform remote or difficult work, which helps keep people out of harm’s way.

To date, most large autonomous machines are operated in restricted facilities where proximity to the automated equipment is controlled and extremely limited. Sensor technology and processing capabilities need to advance further to identify potential obstacles that must be avoided. Safety of autonomous systems is of utmost importance.

Possible futures

The market is certainly there. The productivity gains are meaningful. Cost savings are realized. So, one might wonder, is there a catch? Traditionally, construction has had a complicated relationship with technology, but the shortage of skilled labor coupled with the surging demand for building projects has forced the industry to re-evaluate that relationship to make up the shortfall.

In the next five years we can expect the trend toward automation and electrification to accelerate from low levels to noticeable levels. Within 10 to 15 years, based on exponential trends, it is estimated that more than 80% of heavy-equipment machinery will be powered by electric powertrains and have systems centric to autonomous operation. A few heavy-equipment end users have already expressed a desire for fully autonomous work sites and electric equipment, as this helps them to lower overall costs and meet efficiency and environmental targets. The indications are that it is a no-brainer for heavy-equipment operations to transition to fully autonomous and electric operations at the earliest commercially viable opportunity.

For ag, the key to commercial success lies in the development of robust robotic and associated artificial intelligence (AI) platforms that can be utilized across the harvesting seasons of various crops. This approach is already reflected in the latest prototypes or products on the market.

Autonomous haulage systems will have a pronounced impact on future mining operations. Driverless trucks run continuously and more efficiently than their manned counterparts. Autonomous haulage systems will improve everything associated with the mining operation, from safety to operations to maintenance. They will also change the approach to mine planning and scheduling. The eight-hour production shift, for example, was based on miner fatigue levels. Production shifts in the future might be based on the refueling needs of the excavator. Or, in fact, they might disappear entirely.

In the next five years, given the current state of AV technology in general, it is difficult to envision any significant changes in this time frame. A construction project, for example, frequently involves operation close to existing structures, workers on the ground, or an environment where there may be material delivery and site visits from “civilians” who do not have the proper training to safely navigate a site shared with autonomous equipment. The current and near future of AV technology is not really designed for these use cases.

Looking five to 10 years ahead, AV technology will start to be introduced for certain tasks, such as automated snowplows on long lanes in agriculture, and in the other sectors for moving a large number of items (e.g., timber, pallets, bales) from point A to point B, especially those that travel a significant distance.

By about 2030, it is expected that AI will have reached a point where its capability exceeds that of a human for many tasks, including operating off-road equipment. The technology will allow most, if not all, of the applications to be automated. This will likely be achieved by a significant redesign of machines (cabless), rather than by a humanoid robot operating the current generation of vehicles.

Since AVs are highly suited to electrical propulsion, the other major trend to be aware of is the move away from traditional fossil fuel propulsion to battery electric vehicles. Although fuel cells and hydrogen power are a possible alternative.

The bottom line

The world is progressing in the direction of automation in its many different configurations. The innovations in sensors, AI, and connectivity, coupled with the productivity increases and other benefits, will transform the equipment industry. The outcome will not be an abrupt influx of automated heavy equipment in all sectors of the market. As an alternative, there will be incremental advances, such as driver-assistance systems, and certain applications within each market will be automated first simply because they are easier to automate.

The total deployed number of units is minor compared to the conceivable market. However, the technical viability is long proven. Notably, there is abundant room to increase productivity and to expand applicability by making constant gains within established technologies. As such, it is more a question of timing and not of technical viability.

Compared to the first, the second generation of products have already matured in terms of design, size, AI, speed, and so on. The iterations will continue. Now the task at hand is to be open to new technologies and, perhaps more importantly, see them as an added value, not a competition.

For more information, visit www.aem.org/.