Lightning Detection with Deep Learning and Tensorflow on Android: Setting Up a TFLite Model

• Download Converted TFLite Quantized - 555.9 KB
• Download Detect Lightning - 3.2 MB
• App Debug - 19 MB
• Lightning Image - 68 MB

Introduction

In the previous article, we walked through the basic setup of the model-based app in the Android environment.

The MainActivity.java File

This is a very important file - the one that facilitates the user interaction with our Android app. We’ll walk through this file if not line by line, chunk by chunk.

This file is located in the com.ruturaj.detectlightning package. The com.ruturaj.detectlightning.mlkit contains all the class files required for object detection.

Let’s start with the packages:

package com.ruturaj.detectlightning;

import android.content.Context;
import android.content.pm.PackageInfo;
import android.content.pm.PackageManager;
import android.os.Bundle;
import android.util.Log;
import android.widget.Toast;

import androidx.annotation.NonNull;
import androidx.appcompat.app.AppCompatActivity;
import androidx.core.app.ActivityCompat;
import androidx.core.content.ContextCompat;

import com.google.mlkit.common.model.LocalModel;
import com.google.mlkit.vision.objects.custom.CustomObjectDetectorOptions;
import com.ruturaj.detectlightning.mlkit.CameraSource;
import com.ruturaj.detectlightning.mlkit.CameraSourcePreview;
import com.ruturaj.detectlightning.mlkit.GraphicOverlay;
import com.ruturaj.detectlightning.mlkit.ObjectDetectorProcessor;
import com.ruturaj.detectlightning.mlkit.PreferenceUtils;

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

public class MainActivity extends AppCompatActivity implements ActivityCompat.OnRequestPermissionsResultCallback {

Some important variables and strings are defined in class-globally:

private static final String OBJECT_DETECTION_CUSTOM = "Lightning";

private static final String TAG = "MainActivity";
private static final int PERMISSION_REQUESTS = 1;

private CameraSource cameraSource = null;
private CameraSourcePreview preview;
private GraphicOverlay graphicOverlay;
private String selectedModel = OBJECT_DETECTION_CUSTOM;

If you are familiar with the Android lifecycle, I bet you know the following method!

@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);

Setting up preview for our live stream, from the back camera.

preview = findViewById(R.id.preview_view);
if (preview == null) {
    Log.e(OBJECT_DETECTION_CUSTOM, "Preview is null");
}

Next, we draw a graphic overlay (in this case, a bounding box) on the portion of the screen where we detect lightning while in camera mode.

graphicOverlay = findViewById(R.id.graphic_overlay);
if (graphicOverlay == null) {
    Log.e(OBJECT_DETECTION_CUSTOM, "graphicOverlay is null");
}

Next, we check if we already have the permissions our app needs and if not, we ask for them at runtime:

    if (allPermissionsGranted()) {
        createCameraSource(selectedModel);
    } else {
        getRuntimePermissions();
    }
}

The CameraSource class is called to get the camera view onto the screen using our model:

    if (allPermissionsGranted()) {
        createCameraSource(selectedModel);
    } else {
        getRuntimePermissions();
    }
}

When the model, stored in the assets folder, runs, we call the LocalModel class to build the model and detect objects. CustomObjectDetectorOptions is typically used for live detection of the most prominent objects in the camera’s field of vision.

    try {
        if (OBJECT_DETECTION_CUSTOM.equals(model)) {
            Log.e(OBJECT_DETECTION_CUSTOM, "Using Custom Object Detector Processor");
            LocalModel localModel =
                    new LocalModel.Builder()
                            .setAssetFilePath("model.tflite")
                            .build();
            CustomObjectDetectorOptions customObjectDetectorOptions =
                    PreferenceUtils.getCustomObjectDetectorOptionsForLivePreview(this, localModel);
            cameraSource.setMachineLearningFrameProcessor(
                    new ObjectDetectorProcessor(this, customObjectDetectorOptions));
        } else {
            Log.e(OBJECT_DETECTION_CUSTOM, "Unknown model: " + model);
        }
    } catch (Exception e) {
        Log.e(OBJECT_DETECTION_CUSTOM, "Can not create image processor: " + model, e);
        Toast.makeText(
             getApplicationContext(),
                "Can not create image processor: " + e.getMessage(),
                Toast.LENGTH_LONG)
                .show();
    }
}

Start the camera as soon as the permissions have been granted and the app shows the preview.

private void startCameraSource() {
    if (cameraSource != null) {
        try {
            if (preview == null) {
                Log.e(TAG, "resume: Preview is null");
            }
            if (graphicOverlay == null) {
                Log.e(TAG, "resume: graphOverlay is null");
            }
            preview.start(cameraSource, graphicOverlay);
        } catch (IOException e) {
            Log.e(TAG, "Unable to start camera source.", e);
            cameraSource.release();
            cameraSource = null;
        }
    }
}

The following code maintains the Android UI lifecycle in case of an interruption:

@Override
public void onResume() {
    super.onResume();
    Log.e(TAG, "onResume");
    createCameraSource(selectedModel);
    startCameraSource();
}

App pauses are handled as follows:

@Override
protected void onPause() {
    super.onPause();
    preview.stop();
}

Killing the app destroys the camera preview:

@Override
public void onDestroy() {
    super.onDestroy();
    if (cameraSource != null) {
        cameraSource.release();
    }
}

After an onCreate call grants access to the required permissions:

    private String[] getRequiredPermissions() {
    	try {
        	PackageInfo info =
                	this.getPackageManager()
                        	.getPackageInfo(this.getPackageName(), PackageManager.GET_PERMISSIONS);
        	String[] ps = info.requestedPermissions;
        	if (ps != null && ps.length > 0) {
            	return ps;
        	} else {
            	return new String[0];
        	}
    	} catch (Exception e) {
        	return new String[0];
    	}
	}

	private boolean allPermissionsGranted() {
    	for (String permission : getRequiredPermissions()) {
        	if (!isPermissionGranted(this, permission)) {
            	return false;
        	}
    	}
    	return true;
	}

	private void getRuntimePermissions() {
    	List<String> allNeededPermissions = new ArrayList<>();
    	for (String permission : getRequiredPermissions()) {
        	if (!isPermissionGranted(this, permission)) {
                allNeededPermissions.add(permission);
        	}
    	}

    	if (!allNeededPermissions.isEmpty()) {
        	ActivityCompat.requestPermissions(
                	this, allNeededPermissions.toArray(new String[0]), PERMISSION_REQUESTS);
    	}
	}

	@Override
	public void onRequestPermissionsResult(
        	int requestCode, @NonNull String[] permissions, @NonNull int[] grantResults) {
    	Log.i(TAG, "Permission granted!");
    	if (allPermissionsGranted()) {
        	createCameraSource(selectedModel);
    	}
    	super.onRequestPermissionsResult(requestCode, permissions, grantResults);
	}

	private static boolean isPermissionGranted(Context context, String permission) {
    	if (ContextCompat.checkSelfPermission(context, permission)
            	== PackageManager.PERMISSION_GRANTED) {
        	Log.i(TAG, "Permission granted: " + permission);
        	return true;
    	}
    	Log.i(TAG, "Permission NOT granted: " + permission);
    	return false;
	}

}

You’ll see quite a few instances of Log.e… in the code. This is just to make the Logcat inspection stand out visually. When I do the e- Error log, the traces appear in red - easy to see.

Other Class Files

Next Steps

In the next article, we’ll carry out real-time testing of our app on an Android device.