AWS Comprehend, Translate and Transcribe

At the re:invent2017 AWS presented a lot of new services (read all the announcements here: re:Invent 2017 Product Announcements). In this post we are going to see three new services related to the language processing.

  • Amazon Comprehend
  • Amazon Translate
  • Amazon Transcribe

These new services are listed within the Machine Learning section.

Amazon Comprehend

Amazon Comprehend is a natural language processing (NLP) service that uses machine learning to find insights and relationships in text. Amazon Comprehend identifies the language of the text; extracts key phrases, places, people, brands, or events; understands how positive or negative the text is; and automatically organizes a collection of text files by topic.

You can use the Amazon Comprehend APIs to analyze text and use the results in a wide range of applications including voice of customer analysis, intelligent document search, and content personalization for web applications.

The service constantly learns and improves from a variety of information sources, including product descriptions and consumer reviews – one of the largest natural language data sets in the world – to keep pace with the evolution of language.

You can read more about it here: AWS Comprehend and here Amazon Comprehend – Continuously Trained Natural Language Processing.
Watch the video from the AWS re:Invent Launchpad: Amazon Comprehend.

This service is available, here you can find few examples (using Boto3 Python AWS SDK).

Instantiate a new client.

Detect the dominant language in your text.

Detect the entities in your text.

Detect the key phrases in your text.

Get the sentiment in your text.

Amazon Translate

Amazon Translate is a neural machine translation service that delivers fast, high-quality, and affordable language translation. Neural machine translation is a form of language translation automation that uses machine learning and deep learning models to deliver more accurate and more natural sounding translation than traditional statistical and rule-based translation algorithms. Amazon Translate allows you to easily translate large volumes of text efficiently, and to localize websites and applications for international users.

The service is still in preview, watch the launch video here: AWS re:Invent 2017: Introducing Amazon Translate

You can read more about it here: Introducing Amazon Translate – Real-time Language Translation.

Amazon Transcribe

Amazon Transcribe is an automatic speech recognition (ASR) service that makes it easy for developers to add speech to text capability to their applications. Using the Amazon Transcribe API, you can analyze audio files stored in Amazon S3 and have the service return a text file of the transcribed speech.

Amazon Transcribe can be used for lots of common applications, including the transcription of customer service calls and generating subtitles on audio and video content. The service can transcribe audio files stored in common formats, like WAV and MP3, with time stamps for every word so you can easily locate the audio in the original source by searching for the text. Amazon Transcribe is continually learning and improving to keep pace with the evolution of language.

The service is still in preview, watch the launch video here: AWS re:Invent 2017: Introducing Amazon Transcribe
You can read more about it here: Amazon Transcribe – Accurate Speech To Text At Scale.

This is an example of how to use this service (code written by @jrhunt and taken from here).
Note that the API for Transcribe (while in preview) is subject to change (this code may not be the final version of the API):

Output of the speech recognition:

I am looking forward to use these new services, they are easy to use, they easily integrate with the AWS world and they can add powerful features to your applications.

Machine learning with Amazon Rekognition and Elasticsearch – Skedler Blog

I published a new blog post on the Skedler Blog.
In the post we are going to see how to build a machine learning system to perform the image recognition task and use Elasticsearch as search engine to search for the labels identified within the images.

The components that we used are the following:

  • Elasticsearch
  • Kibana
  • Skedler Reports and Alerts
  • Amazon S3 bucket
  • Amazon Simple Queue Service (eventually you can replace this with AWS Lambda)
  • Amazon Rekognition

System Architecture:


You can read the full post here: Machine learning with Amazon Rekognition and Elasticsearch

Please share the post and let me know your feedbacks.

Elasticsearch Machine Learning: U.S. / U.K. Foreign Exchange Rate

At the beginning of May 2017 Elastic announced the first release of machine learning features for the Elastic Stack, available via X-Pack.

The machine learning features of X-Pack (Platinum/Enterprise subscription) are focused on providing Time Series Anomaly Detection capabilities using unsupervised machine learning.

In this post we are going to see an example of time series anomaly detection using the machine learning features of Elasticsearch.

To use this features you need at least the version 5.4.0 of Elasticsearch, Kibana and X-Pack.
In this post I am not going to show how to install the stack components. I used the following:

  • Elasticsearch 5.4.1
  • Kibana 5.4.1
  • X-Pack 5.4.1 (installed both in ES and Kibana)

Here you can find the installation steps:

The machine learning feature is enabled by default on each node, here you can find more details about further configurations: Machine Learning Settings

We are going to use the following dataset: U.S. / U.K. Foreign Exchange Rate.
It represents the daily foreign exchange rate between U.S. Dollar and U.K. Pound between April 1971 and beginning of June 2017.

This is a sample of the data:

We will index the documents (around 16k) in a time-based index called foreignexchangerate-YYYY (where YYYY represents the year of the document).
The time-based index is necessary to use the machine learning feature. The Configured time field of the index will be used as time-aggregation by the feature.
I did not find a way (AFAIK) to use a not time-based index and select a date field while creating a machine learning job.

This is how each time-based index looks like:

Once we indexed our documents, and once we added the index pattern to Kibana, we can create our first machine learning job.


To create a new Job, select the Machine Learning section from the left menu of Kibana (if you do not see it, maybe you have the wrong Kibana version or you did not install X-Pack into Kibana).

You can now choose between Single Metric or Multi metric job, we will choose Single Metric job (for the foreignexchangerate-* index pattern).

We will use the whole time series and a 3 days rolling exchange_rate average. The idea is to aggregate the series by 3 days, compute the average of the exchange rate and spot anomalies.


One we configure the job, we can create it. The machine learning model will be build using our time series and the aggregation/metric we specified.


We can now inspect the anomalies detected using the Anomaly Explorer or the Single Metric View, both from the ML Jobs dashboard.


I checked some of the anomalies automatically identified and almost all of them make sense (I found drop in the exchange rate due to events like Brexit or EU Crisis).

So far we see all the analysis inside Kibana but the machine learning feature comes also with a set of APIs, so you can integrate the time-series anomaly detection with your application.
Here you can find the details about the APIs: ML APis.

In this post we saw a simple example of how to create and run a machine learning job inside Elasticsearch. There are a lot of other aspects like the multi-metric and advanced-metric that I think are important.

The machine learning features are pretty new and I think (and hope!) that Elastic will invest a lot of resources to improve and extend it.

I am going to run some other tests on the ML features and I would like to run some anomaly detection algorithms (statistical and ML based) on the same dataset to benchmark and compare the Elasticsearch results, if you want to collaborate and help me (or if you have some knowledge/background about time series anomaly detection) drop me a line 🙂 .

Twitter sentiment analysis with Amazon Rekognition

During the AWS re:Invent event, a new service for image analysis has been presented: Amazon Rekognition. Amazon Rekognition is a service that makes it easy to add image analysis to your applications. With Rekognition, you can detect objects, scenes, and faces in images. You can also search and compare faces. Rekognition’s API enables you to quickly add sophisticated deep learning-based visual search and image classification to your applications.
You can read more here: Amazon Rekognition and here: AWS Blog.

In this post we are going to see how to use the AWS Rekognition service to analyze some Tweets (in particular the media attached to the Tweet with the Twitter Photo Upload feature) and extract sentiment information. This task is also called emotion detection and sentiment analysis of images.
The idea is to track the sentiments (emotions, feelings) of people who are posting Tweets (with media) about a given topic (defined by a set of keywords or hashtags).

Given a set of filters (keywords or hashtags) we will use the Twitter Streaming API to get some real-time Tweets and we will analyze the media (images) within the Tweet.
AWS Rekognition gives us information about the faces in the picture, such has the emotions identified, the gender of the subjects, the presence of a smile and so on.

For this example we are going to use Python3.4 and the official AWS Python SDK Boto3.

First, define a new client connection to the AWS Rekognition service.

I assume you know how to work with the Twitter Streaming API, I reported an example, by the way you can find the code of this project in this Github repository: tweet-sentiment-aws-reko.
Example of Tweet dowload using the Streaming API and Python Tweepy:

For each Tweet downloaded from the stream (we will discard the Tweet without media), we analyze the image and extract the sentiment information.
To detect faces we use the detect_faces method. It takes the image blob (or the link to an image stored within S3) and returns the details about the image.

To check if at least one face has been found within the image, check if the key ‘FaceDetails‘ exists in the response dictionary.
We can now loop through the identified details and store the results.

We can now print the results to understand the sentiments within the analyzed images, in particular:

  • How many smiling faces there were in pictures?
  • How many men/women?
  • Which emotions were detected? With which confidence? (in average)

This is an example of output (analysis performed on a set of 20 images).

We can use this simple system to track and analyze the sentiment within the photos posted on Twitter. Given a set of posted pictures we can understand if the people in the pictures are happy, are male/female and which kind of emotions are feeling (this can be very useful to understand the reputation of a brand/product).

Note that AWS Rekognition gives also information about the presence of beard, mustaches and sunglasses and can compute the similarity between two faces (this can be useful to understand if there are more pictures representing the same people).

I would like to improve the system, would be interesting to analyze also the text of each Tweet to see if is there a correlation (and how strong) between the sentiment of the text and the sentiment of the image. If you want to contribute do not hesitate to contact me (would be cool also to build a front-end to see the downloaded medias and the results in a nicer way).

Boto3 AWS Rekognition official documentation: Rekognition
Github repository with the full code: tweet-sentiment-aws-reko.

Real-time prediction – Amazon Machine Learning

Amazon AWS provides a service called Amazon Machine Learning. It is a service that makes it easy for developers of all skill levels to use machine learning technology.
It provides tools to create machine learning (ML) models without having to learn complex ML algorithms and technology and to obtain predictions for your application using simple APIs, without having to implement custom prediction generation code, or manage any infrastructure.
Here you can find all the details about Amazon Machine Learning.

In this post we are going to see how create a ML model and how make real-time predictions using AWS APIs through Python.
The data set we are going to use for the demo is the Zoo data set (UCI Machine Learning repository:
It is a simple data set that represents some animals, it has 17 attributes that represent the features of each animal.
Here the attributes with the relative data type.

The data set has 100 rows, here you can see an example of rows (the last value represents the target value, the predicted one):

The target attribute (the one we are going to predict) is the type attribute. It is a numerical attribute that represents classes of animals.
These are the classes, each numeric value corresponds to a set of animals:

The data set (CSV format) has been uploaded to a S3 bucket and it will be used as input for the ML model.

From the Amazon ML service section we can create a new ML model. We need to specify the url of the input file (from S3) or use a previously added data set. We can set the training and evaluation settings but for this demo we are going to use the default settings (recommended by Amazon).


Amazon ML uses the following learning algorithms:

  • For binary classification, Amazon ML uses logistic regression (logistic loss function + SGD).
  • For multiclass classification, Amazon ML uses multinomial logistic regression (multinomial logistic loss + SGD).
  • For regression, Amazon ML uses linear regression (squared loss function + SGD).

You can find more information about the algorithms here.

Once the ML model has been configured, a set of steps will be automatically performed: the data set will be loaded from S3, the model will be built on the data and an evaluation of the model will be performed (by default the evaluation is performed on the 30% of the data set, the remaining 70% is used for the learning task).


Once these steps are completed the ML model has been correctly built and evaluated (yes, is is pretty easy!).
You can explore the ML model performance (Amazon uses the root mean square error as indicator) in the evaluation summary section.


We can now perform the real-prediction task directly online or through the APIs. To enable the APIs integration, we need to enable a real-time prediction endpoint (you can easily do it from the ML model settings section).


We are going now to write a Python script to predict some data using the previously built ML model. The official Amazon AWS SDK for Python is called Boto3. It allows you to use the AWS service as S3 buckets or Machine Learning (see my previous post about Python and Amazon S3 buckets). Here you can find the official Boto3 documentation.

To create a new client connection to the Amazon ML service using the Boto3 client, we have to specify the service we want to connect to (machinelearning this time), the AWs Keys and region (you can fine the region code here)

We need now to create an object that represents our data instance (the one for which we would like to predict the Type attribute).
We specify the attribute key and its value (according to its data type). These attributes represent an honeybee animal: it has 6 legs, it breathes, it is venomous and makes eggs.

We can now call the predict method. We need to specify the ML Model id and the ML model APIs end point.

The result of the predict method contains the predicted value.

For this example the predicted value is 5.75. The data type of the Type attribute is Number so we can round the predicted value to 6.
So given using these input attributes the model classified our animal (a honeybee) correctly as Type 6 (flea, gnat, honeybee, housefly, ladybird, moth, termite, wasp).

Here you can find the documentation by Amazon about real time prediction.
Here you can find the documentation about prediction using Boto3.