Using Wildcard with Grammar Builder – Kinect Speech Recognition

Developing a speech enabled application using Kinect for Windows SDK is really fun. To build an extensive Kinect based speech enabled application along with dynamically loading and unloading grammar, use of wildcard is tremendous important. With the use of wildcard, we can ignore of the speech for whose we really don’t not care about. This work in an exactly same way as we used wildcard in other areas such as searching files.

The GrammarBuilder object can be constructed with either Choices or by mixing both Choices and WildCards. With the wildcard, you can say anything for the current element; where the wildcard has been placed within the grammar builder.

For instance, “draw a red circle” and “draw one Blue Square“; will be recognized if we define the second word as a wildcard.

We can use “AppendWildcard()" method to append a wildcard within the current speech sequence, in text context of current grammarBuilder.

grammarBuilder.AppendWildcard();

So the overall code may looks like as shown below:

// create the grammarBuilder
var grammarBuilder = new GrammarBuilder { Culture = (recognizerInfo as RecognizerInfo).Culture };

// first say Draw
grammarBuilder.Append(new Choices("draw"));

// Wildcard - You can say one, a, some ... whatever !
grammarBuilder.AppendWildcard();

// creating a choices object
var colorObjects = new Choices();
colorObjects.Add("red");
colorObjects.Add("green");
colorObjects.Add("blue");

// New Grammar builder for color
grammarBuilder.Append(colorObjects);

// Another Grammar Builder for object
grammarBuilder.Append(new Choices("circle", "square"));

When the speech are recognized above the threshold values of confidence level, you can parse them using following code snippet using the SpeechRecognizedEventArgs.

System.Collections.ObjectModel.ReadOnlyCollection words = e.Result.Words;
if (words[0].Text == "draw")
{
// skip - Word[1] - As its wildcard
string colorObject = words[2].Text;
switch (colorObject)
{
case "red": objectColor = Colors.Red;
break;
case "green": objectColor = Colors.Green;
break;
case "blue": objectColor = Colors.Blue;
break;
default:
return;
}

var shapeString = words[3].Text;
switch (shapeString)
{
case "circle":
DrawCirecle();
break;
case "square":
DrawSquare();
break;
}

When the wildcard is defined, you can skip that word as you don’t want to recognized it. In the above code block, we have done the same by skipping the “Word[1]” position.  Using a debugger visualizer you can find the following set of  text if you says “Draw a green circle” or “Draw one green circle” or even “draw big green circle”

The pictorial representation below shows how we can use grammar builder, grammar along with Wildcard in a combined way with speech recognizer in Kinect for Windows SDK.

Hope this will help you to build some really cool speech enabled application using Kinect.

Check out my all posts on Kinect for Windows SDK