Mitsubishi Electric’s recent announcement of Project Jabiru marks another step in the development of next-generation automotive in-cabin systems. Spearheaded by Filament Labs, the company’s advanced development group, the system is geared towards safety, sustainability, connectivity, convenience, and a human-centric design. It targets a 2–5-year release of features, either individually or in combination.
In my recent interaction with the product development team behind Project Jabiru, I was given a first-hand demonstration of the system, its working, key features, and perceived benefits. At the core of any effective automotive in-cabin system is the seamless integration of user and vehicle with the objective of enhancing the user experience (UX) from multiple perspectives: safety, connectivity, convenience, and performance. Jabiru achieves through a high-end sensor suite, including radars, cameras, thermal cameras, and directional microphone arrays. This is further reinforced by a customizable cluster and center display that can be changed to match the driver/occupant’s preference.
The Challenge of Reducing Driver Distraction
Driver distraction and drowsiness have been identified as among the leading causes of road accidents worldwide. The emergence of semi-autonomous vehicles with Level 2 and Level 2+ (hands-off) and Level 3 (eyes-off) autonomous driving (AD) features has further emphasized the need for cutting-edge vehicle occupant monitoring systems (OMS) that ensure driver/occupant safety and wellbeing.
Today, almost a quarter of all automotive problems stem from the infotainment system. Accordingly, Jabiru has been designed to be simple and intuitive to use, with a unique personality that reduces driver distraction and cognitive load, while offering drivers and passengers an enjoyable and productive interaction with the system. Moreover, traditional human-machine interface (HMI) systems have tended to be quite complicated. In the case of Jabiru, the use of best practices of web & mobile development means that the UX with the system is more responsive, straightforward, and seamless.
For instance, to improve UX, the screen adjusts automatically to accommodate different sizes or portrait vs landscape display. Moreover, the “two clicks” rule means that all information – temperature /navigation /media widgets, HVAC controls, volume, phone, passenger side temperature, and parking options, among others – are accessible in two clicks or less. This declutters the screen and reduces the burden of scrolling through multiple information sources.
Safety, Connectivity and Convenience
In-cabin monitoring systems have a critical role to play in ensuring vehicle occupant safety, health, and wellness. For this, it is necessary to track and analyze multiple parameters related to the vehicle’s occupants – seatbelt usage, whether the driver has a phone in their hand, is smoking, has their hand on the steering wheel, posture collapse, and drowsiness. The Jabiru system monitors all these factors. Moreover, the use of in-cabin cameras in the system reduces the cost of putting in sensors to monitor seatbelt use, prevents people from cheating sensors, and allows for detection in the back row. Thermal cameras perform facial tracking, determine whether a seat is occupied by a human, and automatically detect zone temperatures to adjust microclimates.
The highly accurate in-cabin micro-movement radar technology detects movement in the cabin. It bridges the gap left by cameras that cannot look through a seat. The radar can detect a heartbeat, breaths, up and down movement of a chest, and even a fly in the vehicle. It can “see” someone/something within the cabin that is not in camera view such as, for instance, a rodent or snake in the footwell.
Using sensor fusion, it can differentiate whether the car is occupied by an adult, baby, or pet and provide a phone notification if the baby or pet is left in the vehicle. Then, working in tandem with thermal cameras, it can automatically turn on the HVAC and keep the baby or animal comfortable. In time, further refinement would include being able to detect perspiration, goosebumps, shivering, or even emotions like crying and combining these multiple inputs to support informed decisions.
Gaze detection – calibrated to detect just eye movement or a combination of head and eye movement – through the IR camera at the steering wheel allows the system to detect exactly where the driver is looking. For instance, whether he/she is continuously checking their side or rear-view mirrors. This aids in safe driving.
Such capabilities open multiple use case scenarios such as, for instance, to gauge whether or not Uber or Lyft, truck or school bus drivers are focused on the road. Furthermore, it can be used to provide driver scores to help drivers continuously improve their driving skills. It can also be used to monitor and help new teen drivers get better.
One important feature of the system is posture collapse detection. Here, depending on the autonomy of the vehicle, it can slow down, park itself on the side, use automatic cruise control and, ultimately, come to a stop. In such an event, the infotainment screen automatically comes up, alerting people outside to the emergency. The system can also be equipped to call 911 or an emergency contact on its own. Importantly, the system can identify the driver and, based on previously input medical information, provide vital health information to first responders.
In terms of navigation, the company is working with TomTom. Leveraging its navigation SDK across the central & cluster display, it provides a variety of map styles like light mode, satellite mode, dark mode, and classic mode that can be scaled and personalized across multiple screens. For instance, a cluster map might provide directions on a certain route, but a passenger might want something else, like eating at a particular restaurant. The passengers are free to interact with this map as much as they want without affecting the directions and navigation on the cluster map. They can personalize all features of navigation – change the look or feel of the map and turn on and off different layers.
Mitsubishi Electric is also working with Swedish-based company, Klimator, to develop a system to determine weather-related road conditions, including identifying snow, ice, black ice, and wet and dry rain. Thermal lasers with IR detection combine data from front-facing cameras with augmented reality (AR) overlay on that stream to show slip and grip levels. This can be done 25 m in front of the vehicle with risk warning beyond the 25 m mark coming from other connected vehicles.
In addition to weather conditions, the system has also developed an algorithm to detect road hazards, including pedestrians, traffic cones, barrels, potholes, traffic signs, stop signs, one-way signs, and wrong-way signs. The real-time information conveyed to drivers allows them to make informed decisions and stay safe on the road.
Conclusion
The Jabiru system is a smart system that learns over time. Currently, in the process of testing and validation, the system is still another 2-5 years from production.
While the system has a range of interesting features, I like the SOS feature which is similar to GM’s OnStar. In this case, the system can configure which driver is involved in an emergency and generate the appropriate medical profile for emergency responders. This is because the camera can detect which driver is sitting in the driver’s seat.
Another plus for me was that the system has no restrictions on the number of drivers who can be profiled. OEMs can therefore leverage the flexibility of monetizing any additional driver profiles as a subscription feature.
I also like ‘focus mode’. The distraction meter feature related to facial detection runs in the background to check and gauge the level of driver distraction. The aim here is to help drivers tackle their distractions. So, for instance, if the system detects tiredness or sleep, it can recommend nearby coffee stops, play music to wake up the driver, or adjust the ambient lighting. For general distraction, the “focus mode” restricts access to other apps and only makes the three primary features –HVAC, media, and navigation – accessible.
Finally, another standout for me is the parking solution which leverages a high-definition locator. It automatically identifies parking structures in the vicinity and space availability in them. Besides, it supports cashless, in-vehicle payments, thereby making the parking experience more convenient.
With inputs from Amrita Shetty – Senior Manager, Communications & Content – Mobility
