Text analytics and sentiment analysis with Microsoft Cognitive Services

By | April 13, 2017

Text analytics (or sentiment analysis) is the automated processing of texts to determine topics, key phrases and the opinion of the writer (positive, negative, neutral).

Microsoft Cognitive Services are a new set of cloud APIs that can be used from practically any technical platform (Android apps, iOS apps, Windows apps, websites, etc) to leverage capabilities such as natural language processing, speech, vision, knowledge exploration, etc. These capabilities, like the whole field of artificial intelligence, are easy for humans but difficult for computers. Now you can leverage the results of Microsoft Research and powerful cloud computing to power your own applications with these capabilities.

Text Analytics API

The Text Analytics API, offered by Microsoft as part of Cognitive Services, has the following functionality:

  • Determine the language of a text.
  • Determine the sentiment of a text. The output is a score between 0 (very negative), 0.5 (neutral) and 1 (very positive).
  • Detect key phrases in a text.
  • Detect the top topics for a collection of texts.

The Text Analytics API itself is built with Azure Machine Learning. No training data is necessary, since it is already trained by Microsoft.

You can easily test it on the Text Analytics API webpage:

Text Analytics API webpage

Getting a Text Analytics API key

To be able to use the Microsoft Cognitive Services APIs you will need a suitable API key. These keys are requested from the Azure Portal. Click New -> Intelligence & Analytics -> Cognitive Services APIs.

Get Cognitive Services API key

Choose API type: Text Analytics API. Also enter an account name and complete the remaining fields. You can select the free pricing tier F0.

Get Cognitive Services API Key Text Analytics

Your account should be created within a few minutes. Open it in the All resources list and select Keys. Copy any of the keys.

Get Cognitive Services API keys

Examples of using the Text Analytics API

I will now give two samples of how to use the Text Analytics API. The first one is written in Python. The second is written in C#.

Using the Text Analytics API from Python

First I include some libraries and do some configuration. Make sure to replace the apiKey with your own from the Azure Portal.

import urllib.request
import json

# Configure API access
apiKey = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
sentimentUri = 'https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/sentiment'
keyPhrasesUri = 'https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/keyPhrases'
languageUri = 'https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/languages'

Next I ask the user for a text, or use a sample text.

# Ask the user for a text
print('Enter a text (or leave blank for default text)')
sampleText = input()
if (sampleText == ''):
    sampleText = 'I really love Azure. It is the best cloud platform.'

Then I prepare headers that are necessary for calling the API.

# Prepare headers
headers = {}
headers['Ocp-Apim-Subscription-Key'] = apiKey
headers['Content-Type'] = 'application/json'
headers['Accept'] = 'application/json'

Now I can call the API and get the results.

# Detect language
postData1 = json.dumps({"documents":[{"id":"1", "text":sampleText}]}).encode('utf-8')
request1 = urllib.request.Request(languageUri, postData1, headers)
response1 = urllib.request.urlopen(request1)
response1json = json.loads(response1.read().decode('utf-8'))
language = response1json['documents'][0]['detectedLanguages'][0]['iso6391Name'] # Sample json: {'errors': [], 'documents': [{'id': '1', 'detectedLanguages': [{'name': 'English', 'score': 1.0, 'iso6391Name': 'en'}]}]}

# Determine sentiment
postData2 = json.dumps({"documents":[{"id":"1", "language":language, "text":sampleText}]}).encode('utf-8')
request2 = urllib.request.Request(sentimentUri, postData2, headers)
response2 = urllib.request.urlopen(request2)
response2json = json.loads(response2.read().decode('utf-8'))
sentiment = response2json['documents'][0]['score'] # Sample json: {'errors': [], 'documents': [{'id': '1', 'score': 0.946106320818458}]}

# Determine key phrases
postData3 = postData2
request3 = urllib.request.Request(keyPhrasesUri, postData3, headers)
response3 = urllib.request.urlopen(request3)
response3json = json.loads(response3.read().decode('utf-8'))
keyPhrases = response3json['documents'][0]['keyPhrases'] # Sample json: {'documents': [{'keyPhrases': ['Azure'], 'id': '1'}], 'errors': []}

Finally I display the results.

#Display results
print('Text: %s' % sampleText)
print('Language: %s' % language)
print('Sentiment: %f' % sentiment)
print('Key phrases: %s' % keyPhrases)

The output looks like this.

Enter a text (or leave blank for default text)

Text: I really love Azure. It is the best cloud platform.
Language: en
Sentiment: 0.985853
Key phrases: [‘best cloud platform’, ‘Azure’]

Using the Text Analytics API in C#

First I include some libraries and do some configuration. I use regular expressions for parsing the results from the API. Make sure to replace the apiKey with your own from the Azure Portal.

using System;
using System.Net;
using System.Text.RegularExpressions;

namespace TextAnalytics1
{
    class Program
    {
        private const string apiKey = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx";
        private const string sentimentUri = "https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/sentiment";
        private const string keyPhrasesUri = "https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/keyPhrases";
        private const string languageUri = "https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/languages";

Next I write the main function, where I start by asking the user for a text, or use a sample text.

        static void Main(string[] args)
        {
            // Ask the user for a text
            Console.WriteLine("Enter a text (or leave blank for default text)");
            var sampleText = Console.ReadLine();
            if (sampleText.Length == 0)
                sampleText = "I really love Azure. It is the best cloud platform.";

Then I prepare headers that are necessary for calling the API.

            // Prepare headers
            var client = new WebClient();
            client.Headers.Add("Ocp-Apim-Subscription-Key", apiKey);
            client.Headers.Add("Content-Type", "application/json");
            client.Headers.Add("Accept", "application/json");

Now I can call the API and get the results. Note that I use Replace and Regex instead of a real JSON parser. In a real application I would recommend using a library like Newtonsoft.Json.NET instead.

            // Detect language
            var postData1 = @"{""documents"":[{""id"":""1"", ""text"":""@sampleText""}]}".Replace("@sampleText", sampleText);
            var response1 = client.UploadString(languageUri, postData1);
            var language = new Regex(@"""iso6391Name"":""(\w+)""").Match(response1).Groups[1].Value;

            // Determine sentiment
            var postData2 = @"{""documents"":[{""id"":""1"", ""language"":""@language"", ""text"":""@sampleText""}]}".Replace("@sampleText", sampleText).Replace("@language", language);
            var response2 = client.UploadString(sentimentUri, postData2);
            var sentimentStr = new Regex(@"""score"":([\d.]+)").Match(response2).Groups[1].Value;
            var sentiment = Convert.ToDouble(sentimentStr, System.Globalization.CultureInfo.InvariantCulture);

            // Detemine key phrases
            var postData3 = postData2;
            var response3 = client.UploadString(keyPhrasesUri, postData2);
            var keyPhrases = new Regex(@"""keyPhrases"":(\[[^\]]*\])").Match(response3).Groups[1].Value;

Finally I display the results.

            Console.WriteLine("Text: {0}", sampleText);
            Console.WriteLine("Language: {0}", language);
            Console.WriteLine("Sentiment: {0}", sentiment);
            Console.WriteLine("Key phrases: {0}", keyPhrases);

The output looks like this.

Enter a text (or leave blank for default text)

Text: I really love Azure. It is the best cloud platform.
Language: en
Sentiment: 0.985853
Key phrases: [‘best cloud platform’, ‘Azure’]

References