The Blessing (and Curse) of Patents

Recently Yahoo! was awarded a patent for its prediction algorithm, finally exposing what has been behind the curtain for quite some time now.  While I recognize much has been done to improve it over the last four years (the patent was filed in 2007), it highlights some of the dangers of patents themselves.  Before I rant, if you want to read the patent you can get it here, or read Mike Nolet's post explaining it without the legal jargon here.

Yahoo! uses a prediction engine for impression buys through the Right Media Exchange, better aligning creative/campaigns with their publishers such that there is an increase in overall performance and CPM.  Let's completely disregard whether or not it works, or even how it works, as that is not the point of the post.  Rather, let's focus on is the patent filing itself.  As mentioned above, you can go read about it yourself and learn the logistics behind this once hidden secret.  That to me is quite fascinating.  Wall street and the quant traders never expose their secrets.  The whole concept of optimization only holds true because the algorithm is able to exploit a flaw in the marketplace.  Once that flaw is discovered, it is corrected and the algorithm tends to become ineffective.  I stress again that Yahoo! has made changes and (hopefully?) improvements over the last four years, but it still makes me wonder how much of an advantage they hold when their secrets are exposed to the world in such grave detail.  It seems a lot easier to game a system when you know how it works ... I only hope they changed it enough.

Demystifying Real Time Bidding

Real-time bidding (or as we like to call it, RTB) is major buzzword in the ad industry today. It hints of the term "Behavioral Targeting" back a couple of years ago, where anyone who wanted a play in this industry claimed to do it. However, unlike behavioral targeting, RTB is quite real and very effective. Problem is its poorly understood and complicated to build against. I've read a ton of articles dismissing this solution due to lack of clarity (and just sat in another meeting discussing the how-to while typing this blog). What I hope to accomplish is to shed some light on both the advantages and disadvantages of RTB so you can make up your own mind and see if it works for you.

First and foremost, what is it?

Simply stated, real-time bidding brings the ad buy down to the impression level.

Historically, when you buy through an exchange (ADX, RMX, name it) you create bids that are pseudo-static -- every time you see an impression that fits the criteria you are looking for, you bid a set price. There are obviously subtle complexities to all of this but for simplicity lets assume that if bid 25-cents, you will always bid 25 cents. If you want to bid higher or lower, or on different targets, you change bid price and/or create more line items and targets and hope for the best. Thus, bidding in the standard exchange is like bidding for a crate of goods, hoping the crate contains the items you sort of asked for, and praying everything is in good condition.

Real-time bidding is more like buying each product in the crate, at varying prices, and assembling yourself. In other words, you can inspect a wide variety of attributes about the impression and make a dynamic decision with respect to price. You can still do the bulk buy and hope for the best, but you have all the information you need to make a quantitative decision (something you cannot do on historical exchanges without creating thousands of line items).

How does it all work?

When an end-user goes to a site that wants an ad, the real-time exchange receives this request and sends the information to the various bidding machines that are connected to it. Each bidder answers back with price and a creative they want to show. The exchange simply chooses the highest bidder and sends the ad along its merry way. So how is that different? The exchange didn't do any magic here: no predictive modeling, throttling, pacing, budget look-up, etc. The exchange was simply the conduit and final arbiter. The bidder did the bulk of the work, acting like its own mini-exchange, so to speak.

What is on a Bid Request?

A Bid Request contains a wealth of information about an impression so you can bid intelligently. Obviously you can't see everything, but in the old exchange you can't really see anything. Each real-time exchange offers their own flavors of special sauce, but everyone gives you the standard set of information you can target against in existing exchanges.  Some exchanges will also allocate some cookie space and/or third party pixel data, which is always valuable when used correctly.

Seems easy ... its not, right?

Everything is pretty straightforward and seemingly simple. However, if it were simple everyone would do it. Let's start getting into where this gets tricky.

1. You need a strong developer (or team) to build real-time bidder. This is someone with domain expertise, good coding practices, knowledge of network protocols, blah blah blah. They are expensive and drink too much coffee.

2. You have (give or take) 30 to 120 milliseconds to answer. This is really easy if you always bid $1.00 for the same ad over and over again, but lets be realistic, that sort of spits in the face of RTB. Compound this by the ever-continuous pelting of requests for bid and its quite tricky to pull this off in a timely fashion. Shoot for code that can handle 1 billion requests in a single day, per machine.  I've seen some go significantly higher and others way lower, but that number is a good scaling factor.

3. You SHOULD build an algorithm. This doesn't mean you need to hire NASA, or 30 quant guys from Wall Street, but you need something that can logically determine what the value of the impression is for the creative you want to show.  At a minimum, determine some prices for the targets you want to hit and get that into the machine.

4. Each RT exchange is a little different (the next article I will post is actually about exchange normalization, but lets not go there yet). You need to build the network protocols, requests, and responses in a vacuum.  That being said, doing things intelligently across exchanges becomes a little tricky.  Worst case, pick a big exchange, write code for them, and then see where it goes.

5. Data, more data, and even more data.  On the cool side, you have access to everything you bid on (transparency article coming soon too).  If you have a few terabyte storage devices sitting around, go save that data and analyze.  If not, well bust out some investor money and start planning.  It's a wealth of information that significantly improves your chances of successful ad buying.  Be prepared to hire a good data architect to mine this stuff (see issue #1 as these people tend to hang out with those people).

So where does that leave us?

The more and more I sit in meetings to discuss RTB, the clearer the picture gets. Yes its complicated, but the positives far out-weigh the complexities because you can really target effectively at the EXACT time you need to. It also gives you immediate insight into response rates and efficacy. Furthermore, it adds a tremendous amount of transparency to the spot buy, something everyone has wanted for way too long.  Just think long and hard when deciding if you build it yourself, or utilize one on the DSPs out there to be your execution arm.