Table of Contents
-We will hack a RC car.
-So what? This is the oldest hack
-We Will control the car through Mobile.
-LoL, I have seen many Bluetooth hack for RC cars. Nothing new.
-We will use Gesture to Control the car
-I have seen Intel Real Sense RC car hack demo too
- We will have voice control facility
- Micosoft Speech SDK is older than my first wallet
- You can control the car with normal RC remote also.
- Oh, So you are planning to write 5 articles on each of thse already existing hack?
-Nope. Our car will be multi-modality. All in one project. Sounds interesting?
-Yeah, a little. But so what? Making different modules and adding it to car's control isn't innovative
-Our car will also respond to human emotion. And, Our car can run automatically, avoiding obstacles.
- Never heard that!
- But wait..... what is the use? After all what problem it solves, beside being a cool hack?
-Our car will be a video streaming car. It will broadcast live events to Youtube. So you can use it as a videographer car.
Since the first DIY project was ever made on planet earth, the first thing hacker probably tried was to control some kind of vehicle with some kind of control modality. In fact last time there was an IoT contest in our Codeproject, I had written a Grand prize winning multi-modality control robotic framework called UROBI.
So, if I had written another multi-modality project, I would have not been able to complete it as i would have got bored myself. Even if I could some how motivate myself to complete writing, there would have been hardly any takers ( other than the judges, because they would have been forced to read!!!!).
To avoid, wastage of precious digital bytes and valuable time of our coder and hacker friends I wanted to carry out a challenging IoT project that would be fun and would solve a problem.
So, I was not interested to write a DIY project. IoT has eveolved. With new features, machines and APIs IoT can solve real world problem. Till now, mounting a camera in some kind of robotic vehicle was not that difficult, but to actually use that camera to live stream event to Youtube was never attempted ( to best of my knowledge). Imagine, you have organized a home party and your YouCar keeps broadcasting that party live to Youtube, moving around the room automatically, responding to your voice, gestures, mobiles. That's what we are going to build in this article- YouCar.
Sounds interesting? Read On.
Codeproject Bonus: Like most of my IoT local services, this one will have a C# client doing all the processing for you. So, even if you are not a robotic or DIY guy, but a C# programmer, this one will be for you to.
We will use Intel Edison as our IoT device, because firstly I am an Intel Software Innovator and secondly Intel has been kind enough to provide me with an Edison device for free to do my experiements. Wheter you are well versed with Intel Edison or not, following tutorial in general ( and Part A in particular) will help you get started with Edison well.
Intel Edison Getting Started- Refer part A ( Article Coming Soon)
From motivation section it is obvious that we are going to build ( or hack) a RC car that will respond to multiple different devices and commands. So, it is important to understand the architecture before we start making it.
Figure 2.1 : Control Block Diagram of the car
Figure 2.2 Hack Concept
First look at figure 2.2. We can see a RC car and it's corresponding remote. These RC cars are operated at various standard RF frequencies. 27MHz is a very popular frequency choice for toy RC car. The car has two isolated circuits: a transmitter at the remote. Each of the switch in the remote generates a different pulse at the tuned frequency. The car circuit has the receiver circuit which receives these pulses and accordingly controls a H-Bridge which drives the motors responssibe for movement of the car. In most of the commercial toy cars, there are two motors: One for Right-Left movement and other for forward reverse movement. You can control both the motors simultaneously. So Forward-Right, Forward-Left, Reverse-Right, Reverse-Left becomes four possible movements.
Now, if you want to turn such a car into an IoT car, you need to hack either the transmitter or the remote control or the actual car and connect the H-Bridge directly to your IoT embedded device circuit.
I first built a RC car hack almost a year back to allow my son Rupansh to drive the cars from mobile. Now, do you think kids would really love messy circuits coming out of their favourite cars? Never. So the best hack is the hack of the transmitter.
What we can do is somehow hook four switches of the transmitter ( the remote control) with our IoT board and replicate the functioning of the switch programatically. We are going to use Intel Edison with Arduino compatible Expansion Board as our IoT device here. Figure 2.2 gives a very good idea of what our car will be like.
Now, the question is how do we replicate the functionality of the four remote buttons?
Figure 2.1 helps you understand the concept better. We are going to run a MqTT server in out IoT device. Different apps ( mobile or PC) will be publishing commands to this MqTT channel. Depending upon the commands, the board will programatically control the switches.
The best thing about this all is that you can literally build app to support literally any modality and simply hook up to the MqTT channel subscribed by our Edison device and you are done.
Rememeber, we also need to hook up camera with our car and stream to Youtube. But, first we need to hack the car to get it working programatically.
Ready?
Figure 3.1 Opening the remote
First take a small screw driver and open the remote. You will see the circuit inside the remote. A pair of wires( one red and a black) will come out of the circuit and will go to the battery box. This two wires are your power line. You will see forward and reverse handles and left and right push buttons. We no more need those control handles. So just unscrew and open out the circuit.
Figure 3,2 Isolating Circuit from Remote Body
In Figure 3.2 you can see the islolated circuit after it is unscrewd from the remote body. Now you can push the buttons and can see your car moving around. Observe, how the circuit is powered by the battery box.
Our first step is to cut these two wires from the battery box and power it up from our device
Figure 3.3 Connecting Grove's Power Supply to RC Remote
For that, you need to take a grove connector cable. The cables have two ports at either end, one for connecting the cable to Grove shield and the other for connecting to the Grove component modules like LED. You need to cut one of the ports. So, now one port of the cable is connected to the shield and other end has four wires: Red, Black, Yellow and White. If you observe the labels in any of the ports on the top of Grove shield you will easily understand that Red wire is mapped to Vcc, Black to Ground and Yellow to the pin corresponding to that port. For giving supply to remote RC module, you need to separate red and black wires from it that was connected to the battery box and wire them with the red and black wires of the cable which you just cut to remove port at one end. Figure 3.3 will give you a clearer idea about this connection.
Once you have powered up your RC module with your Grove shield, ensure that the power is driving the remote correctly. Switch on your car, now press the buttons manually to see your car moving.
So, you have just connected your RC module with your Edison board. But in real terms, it is still controlled through it's on board push buttons. Right? So, our next task is to give the control of these push buttons to our Edison board.
Before we understand how to hack them, you are advised to read couple of Subsection of My Arduino beginners guide article.
We are going to replace DC motor with the switch in our current hack.
Figure 9.1 of that article ( in Controlling DC motor Section) should be of interest for you. In that figure, you can see three wires coming out of Arduino: Red from VCC, Black from Ground and Green from Pin 9. All we have to do is take out pin 1 of BC 548 and pin 3 of BC 548 and connect them to remote.
So, if we had to hack the remote for Arduino, it would be like 3.4
Figure 3.4 : RC Remote hack for Arduino
As, Intel Edison is Arduino compatible board, the circuit remains similar. BC 548/547 is connected with a 100Ohm resistor with Yellow wire of Grove cable, red with remote's red( or vcc), cable's black with remote's black. That's it. And you need four transistors and four resistors to bypass all the swicthes.
You can use bread board, but i am more comfortable with soldering in general purpose PCB. After successfully hacking the remote the final circuit looks similar to figure 3.5
Figure 3.5 : Final Circuit of complete RC remote hack
Once you complete the circuit, just connect the port connected parts of the cable on the Grove shield ( D5-Forward, D6-Reverse, D4- Right, D3-Left). The circuit is as shown below.
Figure 3.6 Final Remote hack Connected to Intel Edison
We are still in the process of getting our car to work with Intel Edison and perhaps as a first step with our mobile. So, this coding section will include writing an app for Edison ( the IoT app) and corresponding mobile App ( called the companion App in IoT ecosystem).
In our MqTT section of Part A of Biometric Locker, we had seen how to use a global iot.eclipse.org message broker to effectively work as a bridge between your client app and your IoT App. However, RC car must be more responsive. There is a latency associated with MqTT messages however small that might be. This latency is extremely irritating in terms of user experience. You want the car to be responsive and responding to commands in no time. So, instead of relying on global broker, we need to run a local broker in Intel Edison. Fortunately that is also extremely easy.
Mosquitto is an IoT MqTT message broker that runs on Linux. So let's install it first
Installing Mosquitto MqTT Broker in Edison
- In Edison Shell, type and enter following command to download Mosquitto bundle in your /home/root
wget http:
tar xzf mosquitto-1.3.5.tar.gz
cd mosquitto-1.3.5
make WITH_SRV=no
That's it. You Intel Edison will run a Mosquitton MqTT broker at port 1883 on start up.
Open two PuTTY terminals and connect to your board through SSH. In one shell, subscribe to message. let's say we use a channel by name rupam/data here. CHANNEl_NAME here will be rupam/data
mosquitto_sub -h EDISON_IP -p 1883 -t CHANNEL_NAME
In the other terminal publish the message.
mosquitto_pub -h EDISON_IP -p 1883-t CHANNEL_NAME -m "SOME_MESSAGE"
That's it, you will see the published command or message appearing in first shell where you had subscribed.
It is always a good idea to have MyMQTT App if you are an Android user and aspiring to be an IoT developer. In MyMqtt, subscribe to CHANNEL_NAME for broker EDISON_IP.
if you are not sure about EDISON_IP, ifconfig command will produce lots of IP addresses. Look out for wlan0 section to get your device IP address.
Node.js Code
The code is all about subscribing to rupam/data ( or whatever channel you fancy to use) in mqtt module. Then controlling pin D3, D4, D5 and D6 based on the received command.
An important part to know is that, RIGHT and LEFT commands bears meaning only when used in conjunction with FORWARD and REVERSE. Also in either of FORWARD or REVERSE mode, the car may have RIGHT, LEFT or Staright movement 9 no RIGHT or LEFT).
So, accordingly we developed two extra commands: NO and STOP. No- when LEFT=0 and RIGHT=0, STOP when all the switches are 0.
var mqtt = require('mqtt');
var client = mqtt.connect('mqtt://192.168.1.101');
var mraa = require('mraa')
var SPEED=1
var fPin = new mraa.Gpio(6)
var bPin = new mraa.Gpio(5)
var rPin = new mraa.Gpio(3)
var lPin = new mraa.Gpio(4)
fPin.dir(mraa.DIR_OUT)
bPin.dir(mraa.DIR_OUT)
rPin.dir(mraa.DIR_OUT)
lPin.dir(mraa.DIR_OUT)
fPin.write(0)
bPin.write(0)
rPin.write(0)
lPin.write(0)
client.subscribe('rupam/data/#')
client.handleMessage=function(packet,cb) {
var payload = packet.payload.toString()
if (payload === 'FORWARD')
{
bPin.write(0);
fPin.write(SPEED)
}
if (payload === 'REVERSE')
{
fPin.write(0);
bPin.write(SPEED);
}
if (payload === 'RIGHT')
{
lPin.write(0);
rPin.write(1)
}
if (payload === 'LEFT')
{
rPin.write(0); lPin.write(1)
}
if (payload === 'STOP')
{
fPin.write(0)
rPin.write(0)
bPin.write(0)
lPin.write(0)
}
if (payload === 'NO')
{ rPin.write(0)
lPin.write(0)
}
console.log(payload) cb()
}
fPin, bPin,rPin and lPin are the pins where FORWARD, BACK, RIGHT and LEFT switches are connected. Note that we have declared a variable called speed but havn't used it. We will use this later to control the speed of forward and reverse motors by actuating the transistor through PWM. That is why the forward and reverse buttons are hooked to PWM supported pins. But for now, we want to get this car working. Though, You can test the movement with MyMqtt app, it is not a good idea as the car would hit some wall by the time you finish typing STOP.
Note that when in FORWARD mode, reverse motor is turned off and vice versa. When in RIGHT mode, LEFT motor is turned off and vice versa. When STOP command is received all motors are turned off. When NO command is received , LEFT and RIGHT motors are turned off.
Now we are going to build an Android Studio native Android App to Control this Car.
The idea of the app is pretty simple. provide a means to connect to our Mqtt broker and publish messages to our channel. We are going to use Eclipse Paho Mqtt library for communication.
Let us first create our GUI in activity_main.xml layout.
See the layout design in figure 4.1.
Figure 4.1: Design of our Simple UI for controlling the car
We use relative layout to place the components. Here is the xml file,
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:tools="http://schemas.android.com/tools" android:layout_width="match_parent"
android:layout_height="match_parent" android:paddingLeft="@dimen/activity_horizontal_margin"
android:paddingRight="@dimen/activity_horizontal_margin"
android:paddingTop="@dimen/activity_vertical_margin"
android:paddingBottom="@dimen/activity_vertical_margin" tools:context=".MainActivity">
<TextView android:text="Broker" android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:id="@+id/textView" />
<Button
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:text="FORWARD"
android:id="@+id/button"
android:layout_below="@+id/edChannel"
android:layout_centerHorizontal="true" />
<TextView
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:textAppearance="?android:attr/textAppearanceSmall"
android:text="Small Text"
android:id="@+id/textView2"
android:layout_alignParentBottom="true"
android:layout_alignRight="@+id/button"
android:layout_alignEnd="@+id/button" />
<Button
style="?android:attr/buttonStyleSmall"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:text="REVERSE"
android:id="@+id/button2"
android:layout_above="@+id/textView2"
android:layout_centerHorizontal="true"
android:layout_marginBottom="119dp" />
<Button
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:text="LEFT"
android:id="@+id/button3"
android:layout_below="@+id/button"
android:layout_alignParentLeft="true"
android:layout_alignParentStart="true"
android:layout_marginTop="55dp"
android:layout_toStartOf="@+id/editText"
android:layout_toLeftOf="@+id/editText" />
<Button
style="?android:attr/buttonStyleSmall"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:text="RIGHT"
android:id="@+id/button4"
android:layout_alignTop="@+id/button3"
android:layout_alignParentRight="true"
android:layout_alignParentEnd="true" />
<EditText
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:id="@+id/editText"
android:hint="192.168.1.102"
android:layout_alignParentTop="true"
android:layout_alignRight="@+id/button2"
android:layout_alignEnd="@+id/button2"
android:text="192.168.1.101" />
<Button
style="?android:attr/buttonStyleSmall"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:text="Connect"
android:id="@+id/button5"
android:layout_alignTop="@+id/editText"
android:layout_toRightOf="@+id/button"
android:layout_toEndOf="@+id/button" />
<TextView
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:textAppearance="?android:attr/textAppearanceSmall"
android:text="Channel"
android:id="@+id/textView3"
android:layout_below="@+id/editText"
android:layout_alignParentLeft="true"
android:layout_alignParentStart="true" />
<EditText
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:id="@+id/edChannel"
android:layout_below="@+id/editText"
android:layout_alignLeft="@+id/editText"
android:layout_alignStart="@+id/editText"
android:layout_alignRight="@+id/editText"
android:layout_alignEnd="@+id/editText"
android:text="rupam/data" />
</RelativeLayout>
Download the mqtt jar file for android, go to physical directory of the project and paste the jar file in App/libs folder. Create the folder, if not created.
You should now be able to see the jar file under project tab in Android studio ( see figure 4.2)
Figure 4.2 Adding Mqtt jar file to project
Recall we have six states in our Mqtt Node.js app. FORWARD,REVERSE,LEFT,RIGHT,STOP, and NO. So, to control the app, you actually need six buttons. But the remote doesn't have six buttons. Right? When you press the remote button, car will move, when you realse car will stop. So instead of attaching MqTT publish to button click event, we can attach to Touch event and send NO when RIGHT and LEFT buttons are realeased and send STOP when FORWARD/REVERSE buttons are released. Topics are published to channel specified in edChannel.
In MainActivity.java class we create a method Connect() for connecting to the broker. As Android doesn't permit to access network calls from main thread, we create a new class called ConnectionClass that extends AsyncTask and connect to the broker in doInBackground
public class ConnectionClass extends AsyncTask<String, String, String>
{
Exception exc = null;
@Override
protected void onPreExecute() {
super.onPreExecute();
}
@Override protected String doInBackground(String... params) {
try
{
if (Looper.myLooper()==null)
Looper.prepare();
sampleClient = new MqttClient(broker, clientId, persistence);
connOpts = new MqttConnectOptions();
connOpts.setCleanSession(true);
IMqttToken imt=sampleClient.connectWithResult(connOpts);
Log.d("MQTT MODULE.....","....DONE..."+sampleClient.getServerURI()+"--"+imt.getResponse().getPayload());
if(sampleClient.isConnected()) {
return "CONNECTED";
}
else
{
return "Connection Failed.....";
}
}
catch(Exception ex )
{
Log.d("MQTT MODULE", "CONNECTion FAILED " + ex.getMessage() + " broker: " + broker + " clientId " + clientId);
return "FAILED "+ex.getMessage();
}
}
@Override protected void onPostExecute(String result) {
super.onPostExecute(result);
if(result!= null)
{
if(result.Equals("CONNECTED"))
{
isConnected=true;
}
else
{
isConnected=false;
}
tv2.setText(result);
Toast.makeText(MainActivity.this, result, Toast.LENGTH_LONG).show();
}
}
}
In the postExecute method with display the result of connection request with a Toast.
When btnConnect is clicked, execute() method is called with the object of ConnectionClass that triggers the connection in background.
btnConnect.setOnClickListener(new OnClickListener() {
@Override
public void onClick(View v) {
broker="tcp://"+edServer.getText().toString().trim()+":1883";
topic=edChannel.getText().toString().trim();
ConnectionClass con=new ConnectionClass();
con.execute();
}
});
I could actually create six buttons for six states: FORWARD,REVERSE,LEFT,RIGHT and NO and generate respective MqTT commands from onClickListener of the buttons. But that wouldn't be too intuitive. Recall how the physical remote works. When you keep a button pressed the car is in that state and when you release the button, the car is stopped. I wanted to emulate this principle. So instead of creating six buttons, just like our physical remote, we create four buttons and instead of attaching the event with onClickListener, we attach setOnTouchListener to the buttons. When touch is pressed, we generate MqTT message corresponding to the text of the button, when touch is released we generate NO if the released button is LEFT or RIGHT and STOP if released button is FORWARD or REVERSE.
Here is the initialization of all the UI components in mainActivity.java
b = (Button)findViewById(R.id.button);
b2=(Button)findViewById(R.id.button2);
b3=(Button)findViewById(R.id.button3);
b4=(Button)findViewById(R.id.button4);
btnConnect=(Button)findViewById(R.id.button5);
edServer=(EditText)findViewById(R.id.editText);;
edChannel=(EditText)findViewById(R.id.edChannel);
topic=edChannel.getText().toString().trim();
btnConnect.setOnClickListener(new OnClickListener() {
@Override
public void onClick(View v) {
broker="tcp://"+edServer.getText().toString().trim()+":1883";
topic=edChannel.getText().toString().trim();
ConnectionClass con=new ConnectionClass();
con.execute();
}
});
b.setOnTouchListener(this);
b2.setOnTouchListener(this);
b3.setOnTouchListener(this);
b4.setOnTouchListener(this);
tv2=(TextView)findViewById(R.id.textView2);
Now the next part is to handle the touch event. Make the MainActivity.java class to implement OnTouchListener interface.
In onTouch method we handle MotionEvent.ACTION_DOWN and MotionEvent.ACTION_CANCEL to trigger MqTT Messages.
@Override
public boolean onTouch(View v, MotionEvent event) {
Button b=(Button)v;
switch (event.getAction() & MotionEvent.ACTION_MASK) {
case MotionEvent.ACTION_DOWN:
v.setPressed(true);
tv2.setText("ENTERED:"+b.getText()+"-" + (new Date()).toString());
Send(b.getText().toString());
break;
case MotionEvent.ACTION_UP:
case MotionEvent.ACTION_OUTSIDE:
case MotionEvent.ACTION_CANCEL:
v.setPressed(false);
tv2.setText("LEFT:" + b.getText() + "-" + (new Date()).toString());
if(b.getText().toString().trim().equals("LEFT")||b.getText().toString().trim().equals("RIGHT"))
Send("NO");
else
Send("STOP");
break;
case MotionEvent.ACTION_POINTER_DOWN:
break;
case MotionEvent.ACTION_POINTER_UP:
break;
case MotionEvent.ACTION_MOVE:
break;
}
return true;
}
As you can see that MqTT message is actually generated through Send() method.
In Send(), we check if sampleClient is connected or not, if connected we send the message into the channel specified by edChannel EditText.
void Send(String content)
{
final String data=content;
AsyncTask.execute(new Runnable()
{
@Override
public void run()
{
try
{
if(isConnected)
{
MqttMessage message = new MqttMessage(data.getBytes());
message.setQos(qos);
sampleClient.publish(topic, message);
}
Log.d("MQTT MODULE",data+" SENT");
}
catch(Exception ex)
{
}
}
});
}
As Android requires explicit user permission for accessing Network, do not forget to add
<uses-permission android:name="android.permission.INTERNET"/>
before <application> tag.
Figure 4.3: Running the Android App
That's it. You are now ready to rock with your RC car.
Remember to change the broker address in your Node.js App.
Figure 4.4 : My son Rupansh Controlling car with Mobile( Click to watch the demo video)
Intel RealSense is one of the great technologies for short range accurate gesture recognition. It was introduced with immense amount of fan fare. Earlier, gaming was preceived as one of the best use cases for the technology. With time augmented reality and 3D scanning have overtaken the other use cases. However, Intel is backing the technology to be part of IoT as the best use case. In fact Intel's robotic development kit backs RealSense. When you observe recent trends in robotics, you will realize Intel RealSense is slowly but steadily is becoming part of some recent mass market robots. With Asus Zenbo gesture technology is getting redefined. It is in fact the first mass-market robot.
If you observe the video, you will realize how periodic gestures like "zenbo get me that towel" with hand pointing towards the towel can guide the robot to locate physical objects better.
Because I believe that combining voice and hand gesture is one of the best input modality for physical objects which are traditionally being controlled through some kind of remote and switches. So I thought why not try a RealSense app for our car?
The best thing about our design is that the apps can be added on to the architecture without changing any hardware. For instance we can develop a new Android app that can control the LEFT and right direction of the car with Accelerometer. All we have to do for that app is push the detected gesture into our MqTT channel.
So, RealSense app is an independent unit that can be used along with our smartphone Android app or entirely as an independent app. The biggest challenge however is to create a multi modality app that can take voice, face and hand gesture simultaneously.
Those of you who have already worked with RealSense will agree that multiple gesture response of the technology is quite difficult and is a real pain. Most of the samples provided with the SDK are independent use cases. i.e. a different sample for Hand Tracking, different one for 3D segmentation, different one for Speech Synthesis and yet a different one for controlling desktop UIs with realSense ( mouse mode).
So I created a class called TheUltimateRealSense class, where I have carefully combined face, segmentation, hand tracking and speech recognition ( and speech synthesis as well!).
Let us first look at various gestures and then we shall see how they work in the context of our control mechanism.
Figure 4.5 Hand and Emotion detection in RealSense module
here is the workflow and gesture interpretation
- Use can generate Voice commands for FORWARD, REVERSE, LEFT, RIGHT, STOP and NO
- When user has generated voice command for FORWARD, car's forward movement will remain active.
- User can stop the car wither by voice gesture or by showing hand close gesture or by generating Surprise emotion ( I shall explain you the logic behind selecting this emotion for stopping !!)
- use can bring his hand near the camera and away from camera for forward and reverse command respectively.
- hand can be moved to left or right for left and right movement. If hand is nearer to camera while moving the hand to right side, then it is FORWARD+RIGHT. Similarly REVERSE+RIGHT if hand is moved to right at a position away from camera.
- When car's movement was activated by hand, once hands are dropped ( no hand detected), STOP command will be generated. However, if car was started with voice command, then hand dropping will not stop the car. That means, you can give voice command and sit and relax.
We create a new Thread by name DoRendering in MainForm.cs.
private void Start_Click(object sender, EventArgs e)
{
Start.Enabled = false;
Stop.Enabled = true;
stop = false;
System.Threading.Thread thread = new System.Threading.Thread(DoRendering);
thread.Start();
System.Threading.Thread.Sleep(5);
}
an object of TheUltimateRealSense class is created and MainForm's instance is passed to it's MainForm object form. So, the class can access the elements of MainForm.
The class has a pipeline for accessing the RealSense algorithms called pp.
PXCMSenseManager pp;
pp.QueryHand(),pp.Query3DSeg(),pp.QuerySample(),pp.QueryEmotion(),pp.QueryHandSample() are called to check if the installed SDK components support these features and if the attached camera can be used for these implementations.
StreamColorDepth() is the main method where all the detection takes place.
private void DoRendering()
{
rs= new TheUltimateRealSenseClass(this,session,StreamMode.LIVE,ColorType.SEGMENTED);
rs.timer = new FPSTimer(this);
rs.OneTimeInitializationBeforeLooping();
rs.Speak("Welcome to Smart I o T RC Car Demo", 80, 100, 80);
int choice = 0;
while (!rs.Stop)
{
bool success = rs.StreamColorDepth(true);
if (success)
{
lock (this)
{
if (choice == 0)
{
bitmaps[index] = rs.DisplayJoints(rs.nodes, rs.numOfHands, rs.bitmaps[index], true, true, true);
}
if (choice == 1)
{
if (rs.labeledBitmap != null)
{
bitmaps[index] = rs.labeledBitmap;
}
}
if(rs.emotionData!=null)
bitmaps[index] = rs.DisplayEmotionSentimentFaceLocation(rs.emotionData,bitmaps[index]);
}
MainPanel.Invalidate();
UpdatePanel();
}
}
this.Invoke(new DoRenderingEnd(
delegate
{
rs.Finish();
Start.Enabled = true;
Stop.Enabled = false;
MainMenu.Enabled = true;
if (closing) Close();
}
));
}
In StreamColorDeapth(), we first segment the image ( not that this extremely resource consuming method has any bearing, I included it so that I can use this in future for applications that would need a background segmentation).
if (pp.AcquireFrame(synced) < pxcmStatus.PXCM_STATUS_NO_ERROR)
{
projection.Dispose();
return false;
}
frameCounter++;
PXCM3DSeg seg = pp.Query3DSeg();
if (seg == null)
{
pp.ReleaseFrame();
projection.Dispose();
pp.Close();
pp.Dispose();
UpdateStatus("Error: 3DSeg is not available");
return false;
}
PXCMImage segmented_image = seg.AcquireSegmentedImage();
if (segmented_image == null)
{
pp.ReleaseFrame();
projection.Dispose();
pp.Close();
pp.Dispose();
UpdateStatus("Error: 3DSeg did not return an image");
return false;
}
In StreamColorDeapth(), once we segment the image, we query for HandSample which returns a segmented hand image ( if hand is detected). If segmented hand image is present( which we do not use in our case by the way) then we loop through number of detected hands and QueryhandData() which returns hand statistics in ihandData. This is further queried for cente of mass ( hand location and openness. These statistics are passed to HandGestureToCar() for translating hand position into gestures specific to car control.
#region hand gesture related part
if (handData != null)
{
handData.Update();
}
sample = pp.QueryHandSample();
if (sample != null && sample.depth != null)
{
DisplayPicture(sample.depth, handData);
if (handData != null)
{
frameNumber = liveCamera ? frameCounter : pp.captureManager.QueryFrameIndex();
#region display joints
PXCMHandData handOutput = handData; long timeStamp = 0;
nodes = new PXCMHandData.JointData[][] { new PXCMHandData.JointData[0x20], new PXCMHandData.JointData[0x20] };
numOfHands = handOutput.QueryNumberOfHands();
if (numOfHands == 0)
{
form.HandGestureTocar(posX, posY, posZ, 0);
}
for (int i = 0; i < numOfHands; i++)
{
if (handOutput.QueryHandData(PXCMHandData.AccessOrderType.ACCESS_ORDER_BY_TIME, i, out ihandData) == pxcmStatus.PXCM_STATUS_NO_ERROR)
{
if (ihandData != null)
{
HandOpen[i] = ihandData.QueryOpenness();
var pt = ihandData.QueryMassCenterImage();
posX = pt.x;
posY = pt.y;
posZ = ihandData.QueryMassCenterWorld().y*1000;
form.Invoke((MethodInvoker)delegate
{
form.Text = String.Format("X={0} Y={1} Z={2} OpenNess={3}", posX, posY, posZ,HandOpen[i]);
form.HandGestureTocar(posX,posY,posZ,HandOpen[i]);
});
for (int j = 0; j < 0x20; j++)
{
}
}
}
}
#endregion
public void HandGestureTocar(double x,double y,double z,int openness)
{
if (openness > 30)
{
{
if (x < 250 )
{
gesture = "LEFT";
}
else if (x > 420)
{
gesture = "RIGHT";
}
else
{
if (Math.Abs(z) < 28)
{
if (lastGesture.Equals("LEFT") || lastGesture.Equals("RIGHT"))
{
gesture = "NO";
}
else
{
gesture = "FORWARD";
}
}
if (Math.Abs(z) > 40)
{
if (lastGesture.Equals("LEFT") || lastGesture.Equals("RIGHT"))
{
gesture = "NO";
}
else
{
gesture = "REVERSE";
}
}
}
lastY = y;
}
}
else
{
lastY = 0;
if(!TheUltimateRealSenseClass.isVoiceGesture)
gesture = "STOP";
}
if (!gesture.Equals(lastGesture))
{
label1.Invoke((MethodInvoker)delegate
{
label1.Text = gesture;
});
if (connected )
{
lastGesture = gesture;
lastY = 0;
PublishMQTTData(gesture);
}
}
}
based on z-axis, hand's closeness from the camera is determined which is used for forward and reverse motion. If hand's openness is less that 30, then it is assumed to be hand closed gesture. STOP command is generated. If hand is forward check for x-axis and dtermine LEFT and RIGHT. We store the previously detected gesture in lastGesture. If hand was on RIGHT and is now brought ot the center, the car's RIGHT motor must be stopped with NO command. So, we use lastGesture to know what was the last detected gesture. If it was LEFT or RIGHT and currently LEFT/RIGHT is not detected then we generate NO command.
If FORWARD or REVERSE command was generated by voice module than absence of hand doesn't trigger any command. Otherwise STOP command is generated.
isVoiceGesture becomes true when user generates FORWARD, REVERSE, LEFT or RIGHT commands through voice and is reset to false when STOP command is generated by any module.
SURPRISE facial expression will result in STOP command.
Now, a question may arise in your mind "Why do we need expressions to control devices or for that matter car?" Imagine you are partying and broadcasting and your car is just about the hit that little precisous gift your wife had given to you in anneversery, what would you do? Interestingly we often react to such situations, forgetting what to do. Unfortunately Intel RealSense detects that horryfying moment when your car is just about to break your wife's gift as surprise and not "horror". So, for making it easy for the all you men out there, I have incorporated this feature. This is also a demonstration of how machines react to natural expressions.
So, from DisplayEmotionSentimentFaceLocation() method of TheUltimateRealSenseClass, we analyze the emotion and generate STOP command if detected emotion is SURPRISE.
From this method we first QueryNumFaces(), which returns number of faces present in the scene, then for each face we call QueryAllEmotionData() which is rendered using DrawLocation() method.
public Bitmap DisplayEmotionSentimentFaceLocation(PXCMEmotion ft, Bitmap bitmap)
{
int numFaces = ft.QueryNumFaces();
for (int i = 0; i < numFaces; i++)
{
PXCMEmotion.EmotionData[] arrData = new PXCMEmotion.EmotionData[NUM_EMOTIONS];
if (ft.QueryAllEmotionData(i, out arrData) >= pxcmStatus.PXCM_STATUS_NO_ERROR)
{
bitmap=DrawLocation(arrData,bitmap);
}
}
return bitmap;
}
The detected SURPRISE emotion results in STOP MqTT message.
private Bitmap DrawLocation(PXCMEmotion.EmotionData[] data, Bitmap bitmap)
{
lock (this)
{
if (bitmap == null) return null;
Graphics g = Graphics.FromImage(bitmap);
Pen red = new Pen(Color.Red, 3.0f);
Brush brush = new SolidBrush(Color.Red);
Font font = new Font("Tahoma", 11, FontStyle.Bold);
Brush brushTxt = new SolidBrush(Color.Cyan);
Point[] points4 = new Point[]{
new Point((int)data[0].rectangle.x,(int)data[0].rectangle.y),
new Point((int)data[0].rectangle.x+(int)data[0].rectangle.w,(int)data[0].rectangle.y),
new Point((int)data[0].rectangle.x+(int)data[0].rectangle.w,(int)data[0].rectangle.y+(int)data[0].rectangle.h),
new Point((int)data[0].rectangle.x,(int)data[0].rectangle.y+(int)data[0].rectangle.h),
new Point((int)data[0].rectangle.x,(int)data[0].rectangle.y)};
try
{
g.DrawLines(red, points4);
}
catch
{
brushTxt.Dispose();
}
bool emotionPresent = false;
int epidx = -1; int maxscoreE = -3; float maxscoreI = 0;
for (int i = 0; i < NUM_PRIMARY_EMOTIONS; i++)
{
if (data[i].evidence < maxscoreE) continue;
if (data[i].intensity < maxscoreI) continue;
maxscoreE = data[i].evidence;
maxscoreI = data[i].intensity;
epidx = i;
}
if ((epidx != -1) && (maxscoreI > 0.4))
{
try
{
if (EmotionLabels[epidx].Equals("SURPRISE"))
{
form.PublishMQTTData("STOP");
}
g.DrawString(EmotionLabels[epidx], font, brushTxt, (float)(data[0].rectangle.x + data[0].rectangle.w), data[0].rectangle.y > 0 ? (float)data[0].rectangle.y : (float)data[0].rectangle.h - 2 * font.GetHeight());
}
catch
{
brush.Dispose();
}
emotionPresent = true;
}
int spidx = -1;
if (emotionPresent)
{
maxscoreE = -3; maxscoreI = 0;
for (int i = 0; i < (NUM_EMOTIONS - NUM_PRIMARY_EMOTIONS); i++)
{
if (data[NUM_PRIMARY_EMOTIONS + i].evidence < maxscoreE) continue;
if (data[NUM_PRIMARY_EMOTIONS + i].intensity < maxscoreI) continue;
maxscoreE = data[NUM_PRIMARY_EMOTIONS + i].evidence;
maxscoreI = data[NUM_PRIMARY_EMOTIONS + i].intensity;
spidx = i;
}
if ((spidx != -1))
{
try
{
g.DrawString(SentimentLabels[spidx], font, brushTxt, (float)(data[0].rectangle.x + data[0].rectangle.w), data[0].rectangle.y > 0 ? (float)data[0].rectangle.y + font.GetHeight() : (float)data[0].rectangle.h - font.GetHeight());
}
catch
{
red.Dispose();
}
}
}
brush.Dispose();
brushTxt.Dispose();
try
{
red.Dispose();
}
finally
{
font.Dispose();
}
g.Dispose();
}
return bitmap;
}
You can play around with the code and try out various other possibilities like driving the car with SMILE gesture, moving the car with face movement and so on.
The last modality that we would be covering here is voice. One of the facts that you must know about RealSense speech recognition engine is that it is very poor in detecting single phrase command even in a limited vocabulary or dictionary, but is extrmely efficient in detecting compound phrases even in a large dictionary. So If your speech recognition library included "FORWARD" and "REVERSE" as voice commands, the system would no detect that effectively.
This is also due to the fact that the session is shared between different RealSense algorithms. So if your phrase is small, chances are that when you had started speaking, handle was with hand or face module. A compund and big enough phrase make sure that the voice module captures it.
We declare a string[] called cmd and initialize with the commands.
public string[] cmds = new string[] { "STRAIGHT FORWARD", "STOP STOP", "NO NO", "MOVE RIGHT", "MOVE LEFT", "COME REVERSE" };
Voide module or Speech Recognition is initialized when TheUltimateRealSense class is initialized by calling OneTimeInitVoic() method, where we set the default audio device, set the recording volume, create an instance of PXCMSpeechRecognition and attach it to current session. We build a grammer for recognition using cmd. We set OnRecognition() as event handler which is called when a phrase is recognized.
public void OneTimeInitVoice(PXCMSession session)
{
source = session.CreateAudioSource();
if (source == null)
{
VoiceCleanUp();
PrintStatus("Stopped");
return;
}
source.SetVolume(0.2f);
source.SetDevice(vDevices[selectedVdevice]);
PXCMSession.ImplDesc mdesc = new PXCMSession.ImplDesc();
mdesc.iuid = vModules[selectedVmodule];
pxcmStatus sts = session.CreateImpl<PXCMSpeechRecognition>(out sr);
if (sts >= pxcmStatus.PXCM_STATUS_NO_ERROR)
{
PXCMSpeechRecognition.ProfileInfo pinfo;
sr.QueryProfile(0, out pinfo);
sr.SetProfile(pinfo);
if (cmds != null && cmds.GetLength(0) != 0)
{
sr.BuildGrammarFromStringList(1, cmds, null);
sr.SetGrammar(1);
}
PrintStatus("Init Started");
PXCMSpeechRecognition.Handler handler = new PXCMSpeechRecognition.Handler();
handler.onRecognition = OnRecognition;
handler.onAlert = OnAlert;
sts = sr.StartRec(source, handler);
if (sts >= pxcmStatus.PXCM_STATUS_NO_ERROR)
{
PrintStatus("Init OK");
}
else
{
PrintStatus("Failed to initialize");
}
}
else
{
PrintStatus("Init Failed");
}
}
So when you speak a command from the cmd, OnRecognition is called from where we pblish our MqTT message.
void OnRecognition(PXCMSpeechRecognition.RecognitionData data)
{
string s = data.scores[0].sentence;
try
{
s = s.Split(new char[] { ' ' })[1];
isVoiceGesture = true;
VoiceGesture = s;
form.PublishMQTTData(s);
if(s.Equals("STOP")|| s.Equals("NO"))
{
isVoiceGesture = false;
}
}
catch { }
}
When you are generating vocie gesture isVoiceGesture flag is set to true and when STOP or NO command is generated through their respective phrases, the flag is set to false allowing user to use other modalities.
Now, let us see how the message publishing works. Remember that a gesture may get detected in every frame. Even if the fps is about 10, you are invariably be generating a command in every 100ms. That is not a good design as far as remote calls are concerned. Also recall how the physical remote works: Press will keep the car in a particular state and release will reset the state. So all you need to do is check if the detected gesture9 or generated command is different from the previous one or not, only in the case of a new command, bother the broker.
string lastSent = "";
public void PublishMQTTData(string command)
{
label1.Invoke((MethodInvoker)delegate
{
label1.Text = command;
});
if (connected)
{
if (!lastSent.Equals(command))
{
if (command.Equals("STOP") || command.Equals("NO"))
{
TheUltimateRealSenseClass.isVoiceGesture = false;
}
mqtt.Publish(topic, GetBytes(command), 0, false);
lastSent = command;
}
}
}
And that's it. You can tinker around with the code and control the car in different ways suing your workflow/commands and imagination.
So far we have hacked remote of a RC car, connected it with Intel Edison and played with RC cars. But remember our project here is more than just a hack. We want an automated navigation for the car as well as we want to put the entire Edison board and the hack in the car itself so that we can attach a camera to it. The problem with such a setup is that, Edison with camera needs 12v 1.5A supply. So you need to power up Edison with an Esternal battery. The whole setup becomes bulky and heavy. RC cars are generally powered by 3v-6.5v @ 300mA-600mA max external batteries. With that supply the car wouldn't move for even 5 minutes, forget about running for entire duration of your party!
So, you need to cut another cable, plug it to another Grove slot and connect ground and VCC with your actual car's supply. As the car has to 'host' a lot of hardware, I selected a Jeep of my son's collection( not that he was very pleased with reduction in his inventory, but he setteled on account of mobile and video shooting).
To automate the navigation of the car and to add collision avoidance capability we add an IR sensor ( Amazon Link) This has three pins: Vcc, Gnd and Vout. Vout outputs analog voltage which varies as an obstacle's distance changes with respect to the module. When an obstacle comes closer, the voltage is higher. As this is a sensor, we need to connect it with analog port of Grove. So, cut another cable, connect Red to Vcc of the module, Black to Gnd of the module and Yellow to Vout of the module. Adjust the module in the front part of the car. Adjust your Edison board, transistor array board with RF transmitter ( the hacked circuit) and a 12 v battery on your car.
It looks something like below.
Figure 5.1 RC Car after mounting IR, Battery, Edison and Remote hack on our car
The model doesn't look too polished, but that's because I did not get too much time during this contest to polish the model. Hopefully some one of you can create a better car with good mechatronic parts.
Now, we need to implement a collision avoidance logic into it. For collision avoidance we must also be able to control the speed of the car. As you can see the car is heavy at the rare end, a high speed reverse may result in car being disbalanced. We connected forward and reverse transistors with D5 and D6 so that we can use PWM for the motors.
The sensor return a high value, whenever an obstacle is nearer to it. An ideal use case would have been to stop the car as it detects the obstacle. But remember, we want our party to be broadcasted live on Youtube. There will be many people in the party and we do not want the car to stop. We want it to alter the route when it detects an obstacle. We explain the concept with following diagram
Figure 5.2: Routing of the car upon obstacle detection
The car continuesly monitors the IR sensor value, when the value crosses threshold, it detects obstacle. Once obstacle is detected, it stops initiate a sequence of REVERSE+RIGHT. Then it goes FORWARD. Due to previous RIGHT, it will be now parallel to the object but with side wise deviation. Turn it little RIGHT and then LEFT to straighten up the car. It then goes forward. It would also have preferred a reverse sensor but I was delivered with only once when I was experimenting for this car.
The message exchange architecture is explained in figure 5.3
Figure 5.3 Message Exchange Sequence for Automatic Navigation of the Car
Now we need little modification in our code to integrate this message exchange structure.
Here is our modified RC code
var mqtt = require('mqtt');
var client = mqtt.connect('mqtt://192.168.1.9');
var mraa = require('mraa')
var SPEED=1
var fPin = new mraa.Gpio(6)
var bPin = new mraa.Gpio(5)
var rPin = new mraa.Gpio(3)
var lPin = new mraa.Gpio(4)
var sensor=new mraa.Aio(0);
fPin.dir(mraa.DIR_OUT)
bPin.dir(mraa.DIR_OUT)
rPin.dir(mraa.DIR_OUT)
lPin.dir(mraa.DIR_OUT)
fPin.write(0)
bPin.write(0)
rPin.write(0)
lPin.write(0)
client.subscribe('rupam/data/#')
client.handleMessage=function(packet,cb)
{
var payload = packet.payload.toString()
if (payload === 'FORWARD')
{
State=1;
bPin.write(0);
fPin.write(SPEED)
}
if (payload === 'REVERSE')
{
fPin.write(0);
bPin.write(SPEED);
}
if (payload === 'RIGHT')
{
lPin.write(0);
rPin.write(1)
}
if (payload === 'LEFT')
{
rPin.write(0);
lPin.write(1)
}
if (payload === 'STOP')
{
fPin.write(0)
rPin.write(0)
bPin.write(0)
lPin.write(0)
}
if (payload === 'NO')
{
rPin.write(0)
lPin.write(0)
}
console.log(payload)
cb()
}
Loop();
var numRight=0;
var Obstacle=0;
var State=0;
function Loop()
{
setTimeout(Loop,100);
var a=sensor.read();
if(a>100)
{
if(Obstacle==0)
{
Obstacle=30;
fPin.write(0);
}
}
else
{
if(numRight>0)
{
client.publish('rupam/data/','LEFT');
numRight--;
}
}
if(Obstacle>1)
{
Obstacle--;
}
if(Obstacle==1)
{
Obstacle--;
if(State==1)
{
State==0;
client.publish('rupam/data/','REVERSE');
client.publish('rupam/data/','LEFT');
setTimeout(func, 2000);
function func() {
client.publish('rupam/data/','FORWARD');
client.publish('rupam/Data/','RIGHT');
setTimeout(f,200);
function f()
{
client.publish('rupam/data/','NO');
}
}
}
}
console.log(a);
}
Firstly you need to configure your camera as elaborated in this section of Biometric Locker Article.
For Youtube streaming We will use following Architecture:
Figure 6.1: Procee flow of Video Streaming to Youtube.
Firstly you need to install mjpg-streamer in Intel Edison.
opkg install mjpg-streamer
Now run the streamer in one of the SSH Edison shell
mjpg_streamer -i "input_uvc.so -y -n -f 30 -r 320x240" -o "output_http.so -p 8090 -n -w /www/webcam"
Edison will start streaming video at port 8090. You can first test in the browser to see if you are getting the stream or not
use following for url in a modern broser like chrome.
http:
Replace the specified IP address with your EdisonIP address.
You will see continues stream from your car camera.
Now, go to your youtube account. Click on the top right on your profile icon and then go to creator studio ( note you need to have a verified Youtube account for streaming).
Figure 6.2 Creator Studio
Figure 6.4 Selecting Live Stream in Youtube
On the left, select live stream option. Your stream will be opened. At the bottom you will see the link to rtmp stream and a key. Copy the key. All you need is to stream any video into this link with the given key.
Note that our video frames are coming from remote Edison car. One of the problems with mjpg is that it streams only video and not audio and Youtube doesn't permit audio less stream.
So, it is mandatory to add some audio with the incoming stream and re-encode the video before we can send that to youtube.
As usual, for video mixing, audio capturing and other video-audio related stuff we will use the rockstarr ffmpeg
As we need to capture and mix some audio with our stream, first find out the list of audio recording devices.
ffmpeg.exe -list_devices true -f dshow -i dummy
Figure 6.4 Filst of Audio Devices
Your microphone will be shown something like the green marked rectangle. Copy that.
You can test recording audio using:
<code>ffmpeg.exe -f dshow -i audio="Microphone (Realtek High Definition Audio)" outputAud.wav</code>
Finally a last command in ffmpeg will ensure the streaming of your Edison car's frames to youtube.
ffmpeg.exe -f dshow -i audio="Microphone (Realtek High Definition Audio)" -f mjpeg -r 8 -i http:
The only things you need to change in the above command is your Edison's IP address and YOUR_KEY. You can also configure Edison to 640x480, in which case in the above command profile part you must change 320x240 t0 640x480.
After a gap of about 10 seconds, your stream will start appearing in youtube as shown in figure 6.5
Figure 6.5 Live Stream in Youtube.
I broadcasted one of the Intel Events ( Intel Software Innovator's meet here in India, bangalore). You can watch the recorded stream here: Intel Software Innovator's Meet- Live Streaming from Edison ( now it is recorded version as the event and stream both got over !)
That's it. build the YouCar move it with Gesture/ Samrt Phone/Voice and live telecast events to Youtube.
Here are some not-so professional photos I captured with not so high defination camera. But none the less, gives you an Idea of the hack and the cool car we built with full IoT and gesture integration.
I started this project about an year back just to give some enjoyment to my son by hacking RC cars. In fact I started my migration from "Embedded-IoT" to pure IoT about the same time. Intel has been really kind to have provided me with kits to do some cool stuff. Guys like Syed, Wendy Boswell and Sourav lahoti from Intel and Abhishek Nandy, another Intel blackbelt and a deer friend have helped me to push the limit of capabilities of Edison board and my IoT knowledge. I have built several projects and prototypes over last few months by teaming up with my wife Moumita. With the help of these people, a simple RC car was developed into featured robot with many real time features that is crucial for even a commercial robot. Intel Edison- Youtube broadcaster was indeed a separate project. But I wanted to achieve a complex software and hardware stack integration to test the capabilities of Intel IoT features.
Though this version has performance issues like very quick drainage of battery, jitter while mobile phones are closer, I believe this can provide you with a good start with your next IoT project for your kid or a project that you may want to showcase to your friends.
I hope you enjoy the project as much as I enjoyed making it. Do leave a comment with your suggestion/criticism ( or appriciation :) ).
gasshopper.iics is a group of like minded programmers and learners in codeproject. The basic objective is to keep in touch and be notified while a member contributes an article, to check out with technology and share what we know. We are the "students" of codeproject.
This group is managed by Rupam Das, an active author here. Other Notable members include Ranjan who extends his helping hands to invaluable number of authors in their articles and writes some great articles himself.
Rupam Das is mentor of Grasshopper Network,founder and CEO of Integrated Ideas Consultancy Services, a research consultancy firm in India. He has been part of projects in several technologies including Matlab, C#, Android, OpenCV, Drupal, Omnet++, legacy C, vb, gcc, NS-2, Arduino, Raspberry-PI. Off late he has made peace with the fact that he loves C# more than anything else but is still struck in legacy style of coding.
Rupam loves algorithm and prefers Image processing, Artificial Intelligence and Bio-medical Engineering over other technologies.
He is frustrated with his poor writing and "grammer" skills but happy that coding polishes these frustrations.
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
