Category: Data Management Platform

Using the Lotame Admin API to build a Dashboard – Part 1

Introduction

We here at Lotame like to think we have a pretty nice UI, and we love when our customers make good use of it. However, we also recognize that we are not able to anticipate every business need display information in the exact way that each organization would prefer. Perhaps your business processes dictate that you need to see the data a certain way, or maybe you have an executive level dashboard that provides your CEO a snapshot of how the business is performing without requiring him or her to log into multiple individual applications to see those details.

Enter the Lotame Admin API. Every feature in our Crowd Control UI leverages this ReST based API. That means everything that you can do and every piece of data that you can access in our UI is also available via the Admin API. There are a number of APIs that customers can use, including the Audience Extraction API and Data Collection APIs. However, from here on out we will be discussing the Admin API exclusively.

Planning a Dashboard

In the introductory section I mentioned a pretty common scenario where a company either has, or wants to have, an executive dashboard that the CEO and others can leverage to get a snapshot of how the organization is performing. Maybe you are using some tools from Pentaho or TIBCO Spotfire, or even built something on your own utilizing tools like JSlate. I am going to show you how to use the Lotame Admin ReST API to add additional information to a corporate dashboard. In this case I will build a simple custom dashboard and add widgets that provide a list of:

  • the most recently created audiences
  • the 10 largest audiences by unique count over the last month
  • audiences that match some search criteria
  • month over month audience statistics, and
  • some financial information related to money earned from data selling compared to a financial target.

We could implement these examples in any language we choose – C#, Java, Ruby, PHP, Python, pretty much anything that can parse XML or JSON. While I am partial to Java, this time I am going to use Python to simplify the build and deploy process. I am also going to use JSON for retrieving data as it is generally much easier to use than XML. While there certainly isn’t any requirement to use object oriented code, I am going to go the “object” route here so that I can show you discreet bits of functionality and put together a good example that you can build upon. I’ll be using Python 2.7.x and try to stick to standard libraries. I will also reference some links from our API documentation.

I’m actually going to split this tutorial into two parts so that I can offer as much detail as possible without being overwhelmingly long. I’ll start off by showing you how to get a token and then make some API requests and then I’ll tie it together by adding some UI code that actually calls these methods. At the end of Part 1 you will know how to call the API and make a widget that displays the data, and potentially pull that widget into any existing dashboard you may already use. Part 2 will focus on building more widgets and creating the dashboard using the same technologies.

Now let’s get started!

Connecting to the API

The first thing that you need to do when using the Admin API is authenticate yourself. Not everyone has access to the ReST API, so make sure to call your Lotame representative and ask for access to the Admin API. A couple of things to keep in mind when you do that:

  • You are probably better off asking for a new account for accessing the API than using an existing one. If you use an existing user account, then every programmer that works on the code or configuration for accessing the API will know user’s password. Personally I would ask for an account like api-access@mycompany.com.
  • Having an account that can leverage the API doesn’t mean that you will have access to everything available in the API or mentioned in the documentation. Our API access works just like our UI access. For instance, if your company has access to create and manage audiences and view profile reports, but your contract doesn’t provide you with the ability to manage custom hierarchies, then that will be reflected in what you can do with the API as well.

Now that we have all of that out of the way, let’s create a connection class that we can reuse across all of our widgets. This class is responsible for reading the API username and password out of a configuration file, passing that to the API, and receiving a token that we will use for every successive call. The class will have methods for connecting, disconnecting, verifying the token is valid, and it will hold the token itself. There is a security section in our API Help that describes the token and connection process. (Please note that you will need an existing Crowd Control account to access the API Help). I’m going to place the Connection class in a python file called api.py. You can see where we open and read our configuration file on lines 14 – 18 to get the username (email), password, and base URL for the API. It is best to put these into a configuration file so that you can easily change any of these settings in the future (passwords never belong in code!). Then lines 22 – 24 set up our parameters and make the call to the API for our token. As you look at the code below you will notice that there are also two import statements at lines 3 and 4 that we haven’t talked about yet. We will come back to those.

api.py – Connection Class

from urllib import urlencode
from ConfigParser import SafeConfigParser
from urllib2 import urlopen, Request
import json

class Connection:

    token = ''
    base_url = ''

    def __init__(self):
        # This is where I will pull in the config file
        # Read out the client_id, email, password, and base url
        parser = SafeConfigParser()
        parser.read('config.cfg')
        email = parser.get('api_samples', 'email')
        password = parser.get('api_samples', 'password')
        self.base_url = parser.get('api_samples', 'base_url')

        # Now we can set up the email and password parameters we
        # need to send the the API to get a token
        params = urlencode({'email': email,
                            'password': password})  # urllib.urlencode
        self.token = urlopen(self.base_url, params).read()

The configuration file opened on line 15 above should have a section titled api_samples and the base_url options described below. Remember to enter your e-mail address / username and password lines 3 and 4.

config.cfg

[api_samples]
base_url = https://api.lotame.com
email = USERNAME_FOR_API_CALLS
password = PASSWORD

That’s it to get a token for authorization! Without the comments, the config reading code, and the class setup, it was just a couple of lines to get the token. An important thing to keep in mind about that token is that it has a 30 minute expiration period. However, the expiration period will be reset each time you use it so that it expires 30 minutes after the last use. That means that if some time passes between connections you might want to re-authenticate get a new token. If your token hasn’t expired yet and you make a re-authentication request, the API will return the same token information and reset the expiration time. Now that we have a connection, we can see how easy it is to make some service requests.

A Simple Service Request

We should now be able to make service requests using our new token. In this first example I simply plan to bring back a report of the top audiences for our client. The first thing that I want to do is check out the documentation for the Top Audiences Report. That page provides us the API endpoint and any parameters that we could pass to the API. I’ve included what it looks like as of the time of this article.

TopAudiencesReport

This snippet tells you that you can access the end point at https://api.lotame.com/audstats/reports/topAudiences and that by default it will return the top 100 audiences sorted by the count of uniques (cookies) from yesterday. You don’t need to pass any parameters, but if you want to change the number of records returned, the date range, or the sort order you can. Passing your client id (client_id) is not required, but you may want to use it if you have access to data for multiple organizations within Crowd Control. Now that we know what call to make, let’s write some code to make it.

I’m going to make a new class called ServiceRequests that I am actually going to place in api.py along with our Connection class as these two really do go hand in hand. Since all of our API requests in this article are going to be GET requests (we won’t be updating any records), the first thing we have in the ServiceRequests class is a method that knows how to make GET requests of the API, passing the token. The method also lets the API know we are interested in having JSON data returned, which is why we had an ‘import json‘ statement in the previous code block. By factoring this out we just won’t have to make these same calls over and over again for each of our service requests.

api.py – ServiceRequests Class

class ServiceRequests:
    def _make_get_request(self, rest_url, token):
        """Private method for making all of the actual ReST GET requests.
        The assumption is that all requests desire a JSON response."""

        # set the authorization with our token
        headers = {'Authorization': token}

        # We want JSON, if we don't set this we will get XML by default
        headers['Accept'] = 'application/json'

        json_to_return = json.load(urlopen(Request(rest_url,
                                    headers=headers)))

        return json_to_return

Easy enough, now we can use the _make_get_request to assist us in getting the top audiences. Staying in the ServiceRequests class, we can make our first real service call. The first time through this example I’m just going to show you how to make the service call using the default service settings. Later on we can pass some parameters. Let’s see what that looks like now.

api.py ServiceRequests Class – get_top_audiences

def get_top_audiences(self,
                          connection):

        # Here we create the URL for the ReST request
        rest_url = connection.base_url + '/audstats/reports/topAudiences'

        # We can print out our URL to make sure we got it right
        print 'rest_url = %s' % rest_url

        # Make the request
        return self._make_get_request(rest_url, connection.token)

Line 5 shows us building the URL that we need to make our request, and line 11 makes the actual request. I stuck a print statement in there just so you could see the URL during run time. We still need to build something to parse the JSON that was returned and make it look nice in the browser, but as far as interacting with the API, that is pretty much it.

Building a Simple UI

The last thing we need to do is find a way to display the data. Most of this has little to nothing to do with the API directly, other than parsing the JSON, but we still need a way to display the data and complete our initial example. The UI I am going to build leverages two extra components – flask and bootstap. Flask is a Python microframework that helps you build web applications. Bootstrap is from Twitter and is essentially a collection of CSS and HTML conventions / templates that allow you to (among other things) create consistent UIs. In our case, I use flask for most of the processing and bootstrap to style the UI.

app.py – top_audiences

from api import Connection, ServiceRequests
from flask import Flask, render_template
import os

# create the app
app = Flask(__name__)

# Create Connection and ServiceRequest objects
connection = Connection()
rest_request = ServiceRequests()

# The method and route for the top audiences
@app.route('/top')
def top_audiences():
    """This endpoint returns the top audiences by unique count for the client
    in question.
    For the ReST request we are using the defaults for sort attribute, sort
    order, the number of audiences to return, and the date range.
    """
    json_request = rest_request.get_top_audiences(connection=connection)

    # Get all of the audiences, their targeting codes, and the unique count
    # from the json
    audiences = []
    for aud in json_request['stat']:
        audiences.append({'AudienceName': aud['audienceName'],
                         'audienceTargetingCode': aud['audienceTargetingCode'],
                         'uniques': '{:,}'.format(int(aud['uniques']))})

    # Then we render the appropriate html page, passing the resulting info.
    return render_template('top_audiences.html', audiences=audiences)

# And then we add a bit of code for this to run on port 33911 (just the port
# I chose).
if __name__ == '__main__':

    port = int(os.environ.get('PORT', 33911))
    app.run(host='0.0.0.0', port=port)

On line 32 of the file above you will notice that we want to return our JSON data into a page called ‘top_audiences.html‘. That page is defined below. It in turn leverages an HTML file (layout.html) that pulls in CSS files from bootstrap. All of the class elements that we are leveraging, such as class=”brand” and class=”navbar” are defined in bootstrap.

top_audiences.html

{% extends "layout.html" %}
{% block body %}</pre>
<div class="span6 well  well-small">
<div class="navbar">
<div class="navbar-inner">
<div class="container"><a class="brand" href="#">Top Audiences</a></div>
</div>
</div>
Audience NameTargeting CodeSize {% for audience in audiences %} {% else %}
<ul>
	<li><em>No audiences exist.</em>
 {% endfor %}</li>
</ul>
<table class="table table-striped table-condensed">
<tbody>
<tr>
<td>{{ audience.AudienceName }}</td>
<td>{{ audience.audienceTargetingCode }}</td>
<td align="right">{{ audience.uniques }}</td>
</tr>
</tbody>
</table>
</div>
<pre>
{% endblock %}

I have included the layout HTML file here too for reference, so that you can see how I pulled in the bootstrap CSS. You will notice that you could also have defined your own CSS (line 3) and not used bootstrap at all.

layout.html

<!doctype html>
Top Audiences
<!-- 		<link rel=stylesheet type=text/css href="{{url_for('static', filename='style.css')}}"> -->
		<link href="{{url_for('static', filename='bootstrap/css/bootstrap-custom.css')}}" rel="stylesheet" type="text/css" />
    {% for message in get_flashed_messages() %}</pre>
<div class="flash">{{ message }}</div>
<pre>    {% endfor %}
    <!-- <body class="container"> -->

        {% block body %}

        {%endblock%}

The Results So Far and What Comes Next

Thus far this article has provided you with some of the basics. We created a way to call the authenticate, call the web service, and show those results in a UI. If we just stopped there, then our UI would look something like the screen shot below.

Simple Top Audiences

You could write something like this and then drop it into an IFrame on a dashboard, or pull it in through some other mechanism your dashboard tools support. This is repeatable for every end point available in the API, including those that allow you to manipulate (create, edit, or delete) data. What we haven’t done yet is look at an example that shows you how to manipulate the API call further, allowing you to change the date range, the number of audiences you return, or the sort order. The next couple of sections will demonstrate how you can tweak the code above to allow more flexibility. Eventually I’ll show you how I implemented a simple dashboard.

Tweaking the Service Call with Parameters

In the examples above we looked at how to leverage the service call, but we left it only making defaults. That meant that the call brought us back the top 100 audiences, which isn’t very likely to fit nicely into a dashboard. What I will show you next is how to use parameters with the service call to bring back only the top 10 audiences. First, refer back to the API snippet we included above. It tells us the top audiences endpoint accepts a parameter called page_count, which is set to a default of 100. All we need to do is set that parameter to 10 in the service to get back what we want. That would make the resulting web service call look like:

https://api.lotame.com/audstats/reports/topAudiences?page_count=10

In our code for the ServiceRequests class, we just need to alter the get_top_audiences method to set that parameter before making the request.  The code sample below shows how I modified get_top_audiences to allow for a more dynamic request from the UI components, allowing UI components to make dynamic requests specifying the sort order, the audience attribute used for sorting, the client id, the number of audiences to return, and the time period to use for determining the size of the audiences.  The method itself sets some defaults, including 10 for the audience count, that are then utilized in the service call.  When we look at the entirety of api.py so for (this includes both the Connection class and ServiceRequests class) we end up with:

apy.py – Dynamic Requests

from urllib import urlencode
from urllib2 import urlopen, Request
import json
from ConfigParser import SafeConfigParser

class Connection:
    token = ''
    base_url = ''
    def __init__(self):
        # This is where I will pull in the config file
        # Read out the client_id, email, password, and base url
        parser = SafeConfigParser()
        parser.read('config.cfg')
        email = parser.get('api_samples', 'email')
        password = parser.get('api_samples', 'password')
        self.base_url = parser.get('api_samples', 'base_url')
        # Now we can set up the email and password parameters we
        # need to send the the API to get a token
        params = urlencode({'email': email,
                            'password': password})  # urllib.urlencode
        self.token = urlopen(self.base_url, params).read()

class ServiceRequests:
    def _make_get_request(self, rest_url, token):
        """Private method for making all of the actual ReST GET requests.
        The assumption is that all requests desire a JSON response."""

        # set the authorization with our token
        headers = {'Authorization': token}

        # We want JSON, if we don't set this we will get XML by default
        headers['Accept'] = 'application/json'

        json_to_return = json.load(urlopen(Request(rest_url,
                                                       headers=headers)))
        return json_to_return

    def get_top_audiences(self,
                          connection,
                          sort_order='DESC',
                          sort_attr='uniques',
                          client_id=0,
                          audience_count=10,
                          date_range='LAST_30_DAYS'):
        """Returns the list of top audiences.

        Keyword arguments:

        connection - the api connection object, needed to gain access to the
        api token

        sort_attr - the audience attribute that you want to use as the sort key
        The default is uniques

        sort_order - the order for the sort (ASC, DESC).  Default is DESC

        client_id - the client id for the client that owns the audience. Client
        id can be optional if you only have access to a single client or if you
        want to return all audiences to which you have access.  The default is
        empty.

        audience_count - the number of audiences you want to return.  The
        default from the service is 100, the default for this code is 10.)

        date_rage - the date interval that you want to use for any statistics.
        The default is LAST_30_DAYS, but the service also supports YESTERDAY,
        LAST_3_DAYS, LAST_7_DAYS, MONTH_TO_DATE, and LAST_MONTH.
        """

        # Build up our parameters list for the call
        params = ['sort_order=%s' % sort_order,
                    'sort_attr=%s' % sort_attr,
                    'page_count=%s' % audience_count,
                    'date_range=%s' % date_range]        

        # Add the client ID, if we have it
        if client_id:
            params.append('client_id=%s' % client_id)

        # Here we create the URL for our ReST request with our params
        rest_url = connection.base_url + '/audstats/reports/topAudiences?' + \
                    '&'.join(params)

        # We can print out our URL to make sure we got it right
        print 'rest_url = %s' % rest_url

        # Make the request
        return self._make_get_request(rest_url, connection.token)

This method might seem like it just got a lot more complicated, but it really didn’t. Most of the extra code is either comments (lines 47 – 70) or is just there to build up the URL for the API request. The biggest thing to note is that the method signature now allows you to pass in all of the parameters or use the defaults to build out the service call (split over lines 83 and 84). Nothing at all needs to change in our app.py code, although we could alter it to change one of the defaults in api.py by making the JSON request look like this:

 

json_request = rest_request.get_top_audiences(connection=connection,
                                               client_id=1324,
                                               sort_order='DESC')

Then we just have one little tiny change in the top_audiences.html to reflect the fact that we are only returning the top ten audiences. Line 11 is all that is new.

 

{% extends "layout.html" %}
{% block body %}</pre>
<div class="span6 well  well-small">
<div class="navbar">
<div class="navbar-inner">
<div class="container"><a class="brand" href="#">Top Audiences</a></div>
 <em>(Top 10 Audiences by Unique Count)</em></div>
</div>
Audience NameTargeting CodeSize {% for audience in audiences %} {% else %}
<ul>
	<li><em>No audiences exist.</em>
 {% endfor %}</li>
</ul>
<table class="table table-striped table-condensed">
<tbody>
<tr>
<td>{{ audience.AudienceName }}</td>
<td>{{ audience.audienceTargetingCode }}</td>
<td align="right">{{ audience.uniques }}</td>
</tr>
</tbody>
</table>
</div>
<pre>
{% endblock %}

Following all of the instructions and code samples above will result in a widget that looks like the live example we have hosted at the Lotame API Samples site in Heroku. The sample one should certainly have different data than yours, as you will be using data from your own client.

Next Steps

In the next tutorial, I will show you how to make several other API calls to return additional information and then we will tie it all together by creating a simple example dashboard. The dashboard will contain several pieces of information, including some example Key Performance Indicators (KPIs) that could be used to better understand how your business is tracking against goals. I look forward writing that tutorial and having you come back and read that one as well.

Advertisement

Data – The Foundation of the Internet Food Pyramid

Learning From the Past

I started working 20 years ago, and while times were very different (Internet access was pretty limited, with only 3 million users worldwide, and nobody knew what software as a service would really mean) some things haven’t changed that much.

My first job was with an engineering company, and one of the things that we did there was to help electric providers access and manage their data.  All of these organizations were responsible in some way for generating piles of data.  Not only the information about how their facilities were licensed and built, but notes, documents, specifications and every bit of information you could imagine related to maintaining, operating, and modifying the facility.

It was great, these companies had tons of data about everything they had ever done – they had the power of data (pun intended).  Well, in theory anyway. The problem for most of these companies was twofold.  First, most of them did not have this direct access to this data.  Even though it was about them, many times it wasn’t considered theirs.  The engineering and maintenance firms either considered it their own or convinced the power providers that they (the engineering companies) were in the best position to hold onto this data, usually in systems to which only the engineering firms had access.

At best the electric provider would have a room full of paper documents that reflected the work that had been done.   This made getting access to that information a time consuming, and sometimes frustrating and costly exercise.  This led to the second problem – this data was primarily at rest.  In other words, since accessing the data was time consuming and costly, no one had any incentive to use it to do anything until they absolutely needed to, so the data just sat there most of the time.

Some of our more forward-thinking customers leveraged our services to address these issues, building systems where they could capture; store, retrieve, and more effectively use all of this data.  That gave them real power.  They were in a better position not only to respond to both internal and external requests, they were also in a much better position to pro-actively improve their facilities and processes in a way that they had not previously had the stomach to attack due to the extra time and costs.

Apply Those Lessons Now

Everything happens in cycles, and looking at businesses on the Internet today we see some of the same problems.  Sure, we have widespread connectivity, new tools, high-powered servers and software capable of creating and churning through terabytes of data, but what is anybody doing with it?  Many companies out there find they are not in that different of a position than those electric providers from 20 years ago, collecting piles of data in any number of systems or logs.  But once it is there, what is anyone doing with it?  What are you doing with it?

Take Charge of Your Data

At some point in the lifespan of any company you need to be asking yourself about your data strategy.  You need to think about all of the opportunities that data can afford you, from providing a better service, to attracting more customers, and ultimately improving the bottom line of your company.  Chances are that if you are not answering these questions one of your competitors probably is giving it some consideration, and while I for one do not put much stock in the “competitor x is doing y, so I need to as well” line of thinking, ignoring your data strategy is doing a disservice to both you and your customers.

Today there are any number of tools and vendors out there that will tell you that they can help you with these kinds of data management and usage problems.  One tool that you can add to your arsenal for executing against a data strategy is a Data Management Platform (DMP).  “Great…” you are probably thinking, another buzzword and another vendor to worry about. “What is a DMP and how can one help me anyway?”

Pick the Right Tools and Partners

First, I would suggest that you do not view whomever you choose to work with as a vendor, but rather that you view them (and they view you) as a partner.  Now that we got that out of the way, I am going to ignore any number of the definitions that you might be able to find out there from vendor or industry sites, and I’m going to tell you what I think it is and what it can do to help you.

I view a DMP as a platform that helps you:

  • Determine how your customers or users interact with you through any number of contact points, including a desktop browser, a mobile device, or even other applications you may use in your enterprise;
  • Combine this data with data you may have from other platforms such as a CRM or POS or third party data;
  • Allow you to view that data in a way that makes sense to you;
  • Analyze and report on that data so that you can visualize the relationships between the various ways that your users interact with you;
  • Use all of that data information to create a better user experience for that user, whether that be showing them more targeted content, serving them a more relevant ad, or even provide them a better purchasing or call center experience;
  • Use the data any way you see fit – it is your data after all.

You may choose to use any number of those types of features in combination or note use any of them at all, but a DMP should ultimately provide those options so that it can support your organization as it grows and evolves.  Elaborating a bit from the points above, a DMP shouldn’t be a closed system that collects your data and just sucks it into a black box where you don’t get many, if any choice, of exactly how to view it or use it.  Instead a DMP should be an open platform that allows you to combine data from multiple sources, and allows you to use that data any way you want through APIs and integrations.  You don’t want to be one of those electric providers mentioned above that always had to ask the engineering firm permission to get their own data.  Instead you want to be that company that is in charge of your own data and strategy.   You should be in charge, and you should have options to control the data that is collected, the classification or categorization and lifespan of that data, and the use of the data.  A system that provides defaults for you is great.  But a system that has defaults and provides you the option of customizing them is even better.  One that offers all of that and is easy to use and offers some level of transparency into how the system works is best.

Remember Privacy

And let’s not forget the ePrivacy Directive in Europe, which makes ownership of all data emanating from a given site the responsibility of that site’s owners.  If you do not control your data, but cede even a small amount of control to other companies who pixel your site, the ePrivacy directive is clear – this will become your liability.  It is vital to understand all of the tracking on your own site.  Work with the partner you choose to set up a system to regularly monitor and audit all the code on your sites.  This is more than just a “cookie audit.”  Much of the tracking covered by the ePrivacy Directive doesn’t use cookies at all.  You need to know the actual scripts that run on your pages. If you haven’t obtained a full tracking audit recently, be sure this is your first step.  You’ll be surprised by the results.  Once complete, you’ll need to categorize each tracker as essential or non-essential, and then rank them on a scale of relative intrusiveness.  Bake this into your data strategy at the onset of your planning.

Put Your Data to Work

Once you have a data strategy and start working with a DMP, then you can put that data to work for you, moving from a state of rest to a state of competitive advantage.  Build a strategy, pick a DMP partner, and wow all of your customers through providing them great content, meaningful targeted ads that they are more likely to find interesting, or even person-to-person customer support.