Alerting the Driver with AI Hazard Detection on Android

• Download source - 53.8 MB

This is the fourth entry in a series of articles on building a real-time hazard detector using Android. In the previous article, we looked at how to highlight objects on the screen. We also noted that not every object detected is something we want to alert the user to. Other objects that the detector recognizes include birds and other vehicles, neither of which is something for which we want to alert the user. The detector does recognize people and various animals. We would want alerts for these. If the car is not moving then it may be unnecessary to alert the user at all; people walking across a street while a car waits is one scenario where a recognized entity enters the phone’s field-of-view, but a scenario in which we would want the system to not generate an alert.

To avoid unnecessarily alerting a user, we will make an opt-in list of objects that should generate alerts. Alongside the TensorFlow Lite model and the labels file, I have added a new file named concerns.txt. Each line of concerns.txt contains a line copied from labels.txt that is a label for something for which the detector will issue alerts. The detector loads these into a list in the same way that it loads the labels.

if(concernList.size() == 0) {
   InputStream concernStream = context.getAssets().open("concerns.txt");
   BufferedReader br = new BufferedReader(new InputStreamReader(concernStream));
   String line;
   while(null != (line = br.readLine())) {
       Log.i(TAG, String.format("Concern: %s", line));
       concernList.add(line);
   }
}

Sound is one option for alerting the user. If a device has stereo speakers, we can take advantage of them by generating sounds that play through the left speaker, right speaker, or both speakers to quickly indicate a general direction of the hazard. To accomplish this, I’ve made three sound files and have added them to the project’s resources. These sound files are placed in the folder app/src/res/raw. The sound files are added to the resources that we can refer to in code. I defined a few constants that hold the IDs of the sound files to reference them more easily. A variable to refer to a MediaPlayer is also needed. The MediaPlayer will be responsible for playing sound.

val ALARM_LEFT: Int = R.raw.left_sound
val ALARM_RIGHT: Int = R.raw.right_sound
val ALARM_CENTER: Int = R.raw.center_sound
var soundPlayer: MediaPlayer? = null

Creating a MediaPlayer object with one of the sound values gives us an object that can play our alert sound.

Preventing Continuous Alerts

To prevent alerts from being sent when the driver is not moving, there is a minimum speed that the user must exceed. If the user is below this speed, no sounds alerts are generated. Android’s location services return speed in meters per second. When thinking of driving speeds, I tend to think more in kilometers per hour. To set the minimum speed, I have defined one constant that sets the minimum kilometers per hour, and another constant uses this first constant to express the same speed in meters per second.

val MIN_ALERT_KMPH:Float = 10.0f
val  MIN_ALERT_MPS:Float = MIN_ALERT_KMPH*1000.0f/(60.0f*60.0f)

The detector will receive updates on the current speed of the vehicle. For the test interface, which is made for debugging and testing, the speed will be set in code and not dependent on the actual movement of the vehicle.

Rather than alert a user continuously about a hazard, there is a cooldown time before another alert is issued. When we attempt to sound an alarm, we first check whether there has been an alarm that hasn’t expired yet. If the expiration time has not passed then the alarm will not sound. Attempting to sound the alarm will also reset the cooldown time.

var coolDownExpiry:Long = 0
val COOL_DOWN_TIME = 10000

fun resetCooldown() {
   coolDownExpiry = Date().time + COOL_DOWN_TIME
}
fun hasCooldownExpired():Boolean {
   val now = Date().time
   return now > coolDownExpiry
}



fun alert(direction: Int) {
   // using the when statement to filter out invalid
   // values should they be passed to this function
   if(hasCooldownExpired() && currentSpeedMPS>MIN_ALERT_MPS) {
       when (direction) {
           ALARM_CENTER, ALARM_RIGHT, ALARM_LEFT -> {
               soundPlayer = MediaPlayer.create(context, direction)
               soundPlayer!!.start()
           }
       }
   }
   resetCooldown()
}

Most of the needed functionality for the application is now in place. The last few things to add are crash detection with emergency alerts and having the application use the live camera stream instead of user-selected images. In the next entry of this series, we will look at crash detection and notifications.