For fun I used the Twilio API and Clarfai to attempt to identify what is in an text message (MMS) and show you what is in the picture.

#Setup
The things you have to do before you can deploy this are the following:

1. Sign up for a twilio account at http://www.twilio.com
2. Buy a Phone number from Twilio
3. Get an API Key

from flask import Flask, request
from twilio.twiml.messaging_response import MessagingResponse
from clarifai.rest import ClarifaiApp
from clarifai.rest import Image as ClImage
import logging

Lets start at the beginning. First we have a set of imports that we need to start using Flask, Clarfai and Twilio. I started by making a flask app and then creating an instance of the API for clarfai.

clarifai_app = ClarifaiApp() # Put API_KEY in parenthesis. 

app = Flask(__name__)

From flask we are only handling requests and we still need to create the flask app to bind routes to so we can create the application. We need from twilml the messaging response because we will be sending and recieving information from twilio.

Also, Clarfai has a rest client in python so we will need the ClarifaiApp to connect to Clarfai and also the Image object from clarfai which we will use to upload the image. I have added the logging API due to the fact that I want to know whats going on with the server. I will show how to set that up later.

Here you need to create the Clarfai_app object with your API key so you can access Clarfai. After you have completed the signup process at https://clarifai.com/developer/account/signup/ then create a new application at https://clarifai.com/developer/account/applications and then replace your API key that you have with PUTAPIKEYHERE .

Routes

Now lets go to the main method hellomonkey. First we want to accept POST requests. (I use the GET here for debugging purposes.

@app.route("/", methods=['GET', 'POST'])
def hello_monkey():

##Processing MMS messages
Now lets get some results from clarfai. First we get the model that we want (I am using the general model but this is configurable).

 # get the general model
    model = clarifai_app.models.get("general-v1.3")
    mediaurl = request.values.get('MediaUrl0', None)
    print(mediaurl) 

Also, we get the mediaurl from the request which is posted to us by Twilio. The next step is to predict with that model.

# predict with the model
    model = clarifai_app.models.get('general-v1.3')
    image = ClImage(url=mediaurl)
    #image = ClImage(url='https://upload.wikimedia.org/wikipedia/commons/thumb/3/39/Phat_Thai_kung_Chang_Khien_street_stall.jpg/240px-Phat_Thai_kung_Chang_Khien_street_stall.jpg')
    responsejson = model.predict([image])

Here the mediaurl is passed to ClImage which will give us a json response with a set of concepts. I also hardcoded an image of Pad Thai to make my development easier when I wasn't testing the twilio part. Additionally I made a helper method to cut through the information to return the concepts which is below.

def get_concept(result_json, index): 
    """ Returns a given concept given an index"""
    return  result_json['outputs'][0]["data"]['concepts'][index]['name']

##Reponse (SMS)
We want to get the first five concepts so I do that and then I return it as a messaging response. This message response will send a text message to twilio.

print(responsejson)
print("3rd Concept" +get_concept(responsejson, 2))
print("4rd Concept" +get_concept(responsejson, 3))
print("5th Concept" +get_concept(responsejson, 4))
resp = MessagingResponse()
resp.message("I think it might be one of the following:  " +
get_concept(responsejson, 0) + " or " +
get_concept(responsejson, 1) + " or " +
get_concept(responsejson, 2) + " or " +
get_concept(responsejson, 3))
print(resp.message)
return str(resp)

UX Design

When we get the results from the REST call to Clarfai we are given a rather large json response. You can see one at https://www.clarifai.com/developer/guide/predict#predict . For my application I chose to return to the user the first five predictions which clarfai calls concepts. Note that the returned set of concepts is sorted by the confidence that clarfai assigns to each concept. I mainly did this because when I showed it to my mom she said that the first one didn't classify it correctly. I could improve this system by preprocessing the image or maybe making the user give tighter bounds.

The last piece is when you have this code above running on a server you have to tell twilio where to access it. Below is where you do that.

Put in the servers ip address or public DNS record. To test send an Picture to the phone number you purchased from twilio.

Code

I have all of the source code at https://github.com/zitterbewegung/mms2text

https://github.com/zitterbewegung/mms2text/blob/master/flask-app.py is the file with all of the things discussed in this document. It also has everything put together nicely.