I got an email today about the new Microsoft Azure Cognitive Services. I checked it out and what I found, really blew my mind.

What Are Cognitive Services?

These new cloud services allow developers to easily add machine recognition to software and websites. So for example, the Steemit developers could check for hostile or violent content and then refuse to allow a user to post such content to the website.

The Facebook Killer

Surely everyone has heard about the Facebook Killer, who uploaded a video of a murder to Facebook last week. Facebook let the video stream off of their website for 2 hours before they finally took it down.

That was quite a mess and rather disgraceful to the the deceased and their family.

Perhaps Facebook should be looking into the Emotion API on Azure that is capable of detecting happiness, neutrality, sadness, contempt, anger, disgust, fear, and surprise.

Image Recognition - "a man swimming in a pool of water"

I have multiple Azure accounts, I play in the cloud a lot, so it seems that I am perpetually logged into Azure in some way. So if you are logged in, the Image Recognition Demonstration will allow you to upload any image you wish, and then the web page will tell you what the image is and consists of.

The following image was the first test I ran this morning...

The Cognitive Image Service returned the following information in JSON format about the image...

{ "Tags": [ "water", "sport", "swimming", "pool", "man", "riding", "blue", "top", "ocean", "young", "wave", "bird", "game", "large", "standing", "body", "frisbee", "board", "playing" ], 

"Captions": [ { "Text": "a man swimming in a pool of water", "Confidence": 0.7850108 } ] }

[ { "Name": "water", "Confidence": 0.9996443 }, 

{ "Name": "sport", "Confidence": 0.950499237 },

{ "Name": "swimming", "Confidence": 0.9062818 }, 

{ "Name": "pool", "Confidence": 0.878758848 }, 

{ "Name": "water sport", "Confidence": 0.631849468 } ]

The tags it assigned to the image are impressive and accurate but the caption it wrote about the image is what really took me by surprise. It is saying that with 70% confidence, this is an image of, "a man swimming in a pool of water".

I am really blown away by how much the computer has recognized about this image. It realizes that it is a man, not a woman. It understands the verb, to swim, and knows what swimming looks like in action. Lastly, it recognizes the swimming pool full of water. Amazing.

Image Recognition - "a bunch of yellow flowers"

The 2nd test image I ran through the service is below.

The Cognitive Image Service returned the following information in JSON format about the image...

Description    { "Tags": [ "animal", "insect", "small", "sitting", "table", "frog", "yellow", "top" ], 

"Captions": [ { "Text": "a bunch of yellow flowers", "Confidence": 0.138596877 } ] }

 [ { "Name": "animal", "Confidence": 0.9838378 }, 

{ "Name": "insect", "Confidence": 0.9138046 }, 

{ "Name": "frog", "Confidence": 0.316563278 } ]

This one was really interesting. Again the tags are impressive but the text caption it wrote was, "a bunch of yellow flowers". I am pretty sure that this Preying Mantis wants all of it's prey to think it is simply, "a bunch of yellow flowers". So it is easy to see how the computer might fall for the same trick! However, the computer has it's doubts as it is only 13% sure that this is, "a bunch of yellow flowers", even thought it sure does look like it.

The last 3 names it gives really say a lot. It is 98% sure that this is an animal. It is 91% sure that this is an insect, and finally it is 31% sure that this is an image of a frog.

Other Cool Cognitive Services

The list of Cognitive Services offered by Azure goes on and it seems to be growing. If you scroll down on the Image Recognition demo, you will see there are services that can turn images of hand writing into editable text. It can even translate between languages and many of these services apply to video, not only still images.

There are tools for moderation in all forms and even a nifty little service for creating thumbnail images.

The Microsoft Strategy

I am finally starting to see the new Microsoft Strategy take place. It has been so long since I have seen a viable strategy come from the company. I have been so confused by products like Windows 8, and the open sourcing of .Net. These things are all fine but I just didn't understand where they were going.

I am starting to see Microsoft's new position and attitude now.