Off-Highway Safety: Surround Sensing for Mobile Machines

By Enrique Busquets, Director, Product Area Mobile Electronics and Electrification, Bosch Rexroth.

Worksite safety – protecting operators and equipment from collision and damage – has always been a high priority for the off-highway industry. Whether at construction sites, agricultural fields, or road-building locations, keeping personnel safe from dangerous accidents and reducing machine damage from collisions has been the focus of many different kinds of efforts and investments.

Worker safety training and well-established, properly followed worksite safety procedures will always have an important role in these efforts. However, there is a significant opportunity for mobile machine builders to design features and capabilities into their machines that can enhance their safe operation. 

To satisfy these goals, mobile machine builders are beginning to incorporate a range of new surround sensor technologies designed specifically to improve collision avoidance capabilities and enhance operator focus and situation awareness. This technology is being made available in modular, integrated platforms that include radar, ultrasonic, and camera components that can be software-integrated, making it easier for machine builders to integrate within their existing machine platforms.

Status of surround sensing

Until recently, there had been limited investment by machine builders in the use of available surround sensing technology to improve the level of “situational awareness” crucial to improve jobsite safety. In agricultural applications, machine builders have been adding backup cameras to allow them to reverse a tractor more easily or combine and hitch to a piece of equipment.

In general the kinds of sensing technology now widely used in passenger and commercial on-road vehicles to provide driver alerts and collision avoidance capabilities have not been extensively exploited. This is, in part, due to the need to “ruggedize” these electronics for use in machines operating in more demanding environments and in equipment that typically generates higher levels of vibration that can rapidly degrade electronics.

The value of surround sensing is providing operators with a new array of sensory information about the risks and obstacles, both fixed and moving, that they could encounter operating their equipment. One of the challenges associated with introducing vision systems, radar, and ultrasonics into mobile machines is the greater complexity of their operating environments, relative to commercial and passenger vehicles. 

Compared to straightforward roads, driveways, and parking lots, risk conditions on a construction site or vegetable field can pop up randomly, which makes seeing the next risk more difficult.

The new surround sensing technologies need to be carefully selected and integrated into a given machine, based on that machine’s functions and the characteristics of its operating environment. So while each off-highway application is unique, there are three common principles that can be used to guide which types of sensory systems to use and how to integrate them into the machine.

Collision avoidance. Collision avoidance is the most easily recognized surround sensing capability. To varying degrees, cameras, ultrasonic sensors, and radar can all contribute to warning an operator of the imminent risk of collision and either alert them quickly enough to stop or change course, or automatically halt the vehicle. 

Accomplishing some of these goals, including automatic stopping, will require integrating the surround sensing inputs into the overall vehicle operation; it may also call for more sophisticated controller hardware and software, as well as making sure that an off-highway vehicle’s CANbus network can support the added data from the sensors.

Focus on the job. Experienced equipment operators are typically well-trained in situational awareness. But it does demand, to a certain extent, that their attention be somewhat divided. A forklift operator pulling skids of material from shelves in a large warehouse has to simultaneously manage the skid removal task and constantly double-check that the backup space is safe and unimpeded.

Self-explanatory operator assistance. One of the advantages of this new generation of mobile machine sensors is that, combined with the more advanced controllers and software now routinely used on off-highway equipment, operators can be supplied with information that is literally self-explanatory. For example, video combined with ultrasonic sensors can be utilized with specialized operator cab displays to indicate which obstacles in the path of travel have a lower or higher risk of collision.

Leading technology suppliers like Bosch Rexroth are now offering these sensor portfolios in modular packages. They can also supply key insights into the right sensors, or combination of sensors, and will provide the greatest enhancement to operator safety and reduction of worksite accidents and collisions. 

Each mobile machine, and the operating environment it is engineered for, presents unique working conditions and application requirements that can help determine the appropriate surround sensing technologies for a given application. 

Multicamera vision systems

As noted earlier, camera systems are not yet widespread options for off-highway equipment. There is a new range of camera systems now available that have been adapted from passenger and commercial vehicles, and improved to provide long-term performance in the more demanding off-highway applications.

There are now multicamera systems that provide high-quality imaging with other features that make them a powerful tool to keep operators fully aware of surrounding conditions. After all, our strongest and most fast-acting sense is our vision. 

These multicamera systems have high-resolution imaging of 1200 x 800 pixels. They also come with a specialized controller/CPU that provides best-in-class stitching together of the images, as well as zoom functions and the ability to add graphical overlays. 

These more advanced features make multicamera systems, with their ability to provide a 360-degree view of conditions, a more effective tool for improving safety. The new systems take into account operating characteristics such as vehicle speed, steering angle, and drive direction to continuously stitch together a coherent image. They also feature detailed guidance on where to position the cameras on each vehicle’s exterior for the most effective coverage.

Ultrasonic sensors for mobile machines

Ultrasonics can provide highly valuable short-range operating conditions. These sensors have been adapted from the latest generation of automotive sensors, with configurations that can range from one to 12 sensors, depending on the size of the machine and its typical operating environment.

The software has been created to achieve measurement rates of up to 120 milliseconds. In the kinds of sensors now available, ultrasonics provide an accurate description of static or moving objects and obstacles as close as 15 centimeters to 5 meters away, depending on the signal resolution.

Applications in which ultrasonics makes the most sense are those with tighter operating environments and many static and moving obstacles, like skid steer loaders on construction sites. A key aspect to assess of these ultrasonic platforms is how they generate and process signals from multiple sensors. 

Some systems will actuate all the ultrasonic sensors at the same time. But this runs the risk of generating false positives or negatives and other noise that can impact the quality of the sensing. More advanced systems trigger the sensors in precisely controlled sequences and process the ultrasonic echoes rapidly to capture a much more accurate portrait of the surrounding system.

In some more advanced applications, the ultrasonic data can be overlaid on the 360-degree video inputs, with colored graphics indicating which obstacles are closer and of higher risk.

Radar for fast-moving machines

While ultrasonics can provide obstacle data from close-at-hand objects, there is a significant role that radar can supply for situational awareness and surround sensing. Radar is most useful for detecting objects farther off than ultrasonics, and it can detect risks while a vehicle is in rapid movement.

There is a wide range of construction and agricultural equipment that also has to travel significant distances on roads at speeds up to 40 kilometers per hour. Radar-based collision avoidance systems offer the highest degree of safety for these vehicles when they need to operate on the roads. 

The radar systems being offered by leading suppliers feature both near- and far-range integrated antennas and ground-detection filters, and can detect and track up to 40 objects simultaneously. 

Since each mobile machine has unique operating characteristics, there may not be a need to have all three kinds of surround sensing technology – cameras, ultrasonics, and radar – incorporated into one piece of equipment. However, many widely used machines could benefit from all three. 

Wheel loaders used on construction sites often drive at high speeds from one part of a site to another or to another. At the same time, a wheel loader’s work often involves them in tight situations where ultrasonics play a more valuable role.

And in these wheel loaders, the operator cab is typically mounted high on the machine because the operator needs a 360-degree view of the surroundings. Additional multicamera displays augment the view. Combined with the potential for graphic overlays of the ultrasonic sensors, their ability to focus on the job and have self-explanatory data on risks can improve safety and reduce the risk of accident.

New generation of sensors

To make the best use of this technology, mobile machine builders who consider the potential advantages of the new generation of surround sensors will need to assess how to modify the design and engineering of their systems.

First they’ll need an intelligent assessment of which of sensor packages make the most effective contribution to safety, based on a machine’s dimensions, operating conditions, and demands on the operator.

Machine designs may need to be modified with expert guidance. Mounting brackets and wiring harnesses must be carefully selected, with particular care about exactly where cameras or ultrasonic sensors are placed to provide the most effective system performance. 

Modifications to HMI displays may also be needed so video can be easily integrated. There will also need to be an assessment of controller capabilities and communications backbone. In the case of high-quality video, Ethernet-based connectivity may be needed on the machine.

The right high-performance surround sensing technology offers the potential to significantly enhance safety on the jobsite and out in the field. It can reduce operator stress with assistance systems that make it easier to keep them situationally aware of risks around them. 

The systems have now been optimized and ruggedized for the off-highway machine environment, offering a new suite of tools to improve safety, reduce the risk of accidents and equipment damage and ultimately help off-highway operations run more smoothly.