Koral, Salesforce, and a Video you should watch

I ran into an old friend and co-worker, Mark Suster, at a Valley event last week.  I haven't spoke to Mark in probably a decade, but he has been very successful and is one of the more thoughtful software/business guys I know.

I Googled him after the event and came across a video he did about his venture history and, more importantly, what he learned about through building and selling software to enterprises.  It is absolutely uncanny how similar our experiences were and the lessons we took away.

Watch his video on Scoble's blog here.

Seriously, watch the video.  He is a very thoughtful guy and much more pragmatic than most software execs you will run across.

What I liked:
1. Simple insights that are devastating to the model of his industry - A content management system without folders is almost heresy to anyone but the Web2.0 world, but Mark got it - folders are just places to lose content in at large corporations.
2. The real dynamics of software development - When you charge a lot of money to large enterprises for your software, it will never be usable.  This might actually be a law of physics.

There is more, so watch it.

Both Mark and I met while working on a very large custom software project for a huge utility in Southern California.  It was years long and impossibly expensive.  In the intervening years I started an ecommerce software integration firm (Fort Point Partners) and he started an Internet firm for builders.  We both raised a great deal of money, suffered through the downturn and created new products. His recent venture, Koral, is now part of Salesforce.com.

'Official' at Last: Faster, faster, faster Marketing

"Many people think the technology revolution in
marketing is about Web sites and interactive
advertising. It’s not. Speed and the customer’s
experience with the brand are the two most important
marketing strategies today."

This is the lead-in to a new study by management consultancy Sapient and the Kellogg School of Management at Northwestern University (blogged here). The report, titled The New Marketing-IT Power Partnership, finds that IT and marketing must work more closely together in order to speed innovation and boost the bottom line.

Speed, and customer experience. Hallelujah!

Whether you call it Agile marketing, or Velocity marketing, or just marketing that moves fast enough to notice, it is the future.  And IT's job is to do everything in its power to increase responsiveness and take itself out of the path of the marketing cycle.  Marketers must be able to change programs and campaigns based on the changing needs of our customers, our campaigns will never be as strong or successful as they could be.

Another great quote:

"How does Netflix stay ahead of the curve? They
constantly experiment with how technology can
enhance consumer and partner touch points. In fact,
Netflix makes significant changes to its Web site
every two weeks in order to improve their customers’
experience.

Lets put it more simply.  Your customer, if they have a noteworthy experience with your brand (positive or negative) can publish their opinion in about 5 minutes on a blog, forum, wiki, or review site.  With the current state of IT and Marketing at many large brands, you can publish new information to your own digital media in about 4 weeks to 3 months.

They are running circles around you.

As we're all learning in the age of the Web 2.0, it's all about the customer experience, and we must learn to maintain a site that is as changeable as our customers, and that is as varied as the needs of our various customers.

So I agree, wholeheartedly: Yes, marketing needs to work with IT. If we can understand each other's needs and fears, we will have come a long way. (Perhaps we can even convince them to do away with the holiday lockdown...)

On the other hand, we as marketers need to learn to stand on our own. If our every site need hinges on the ability for IT to work us into their schedules - no matter how closely we work together and how understanding IT has become of our needs - we're still hamstrung. Marketers need the ability to test and to optimize in real time.

This is why we created Offermatica. After building Jcrew, Nike.com, BestBuy.com and about 40 other ecommerce sites, it became clear that no amount of spending on IT was going to solve the problem.  That is why we set out to create a content delivery platform that could move at the speed of marketing and target customers anywhere they went. The holy grail was to eliminate IT from the path of a marketer who had a great idea and wanted to get it in front of a customer fast and learn.

On a final note, the report suggests that marketers be the ones to reach out. It's time to stop working in silos and work as a team.

MSFT to Acquire AQNT - Seattle Roars back

According to Marketwatch this morning, Microsoft is acquiring Aquantive.

Doubtless this deal is directly linked to the DoubleClick aquisition by Google, and it continues the battles between Mountain View and Redmond. With the announcement of 24/7 heading into the hands of WPP, this leaves only a few standing - Accipiter? Zedo?

There are two ways of looking at this frenzy - First, with search highly optimized and relatively stable around the majors, marketing money is searching for a home and display advertising is certainly the largest spend that is still open for competition. With a predicted continuing influx of brand dollars into interactive, owning a greater share of the spend is a good growth bet.

The other side, however, is even more interesting.

What this is also a likely sign of is the maturation of the online media market from a real estate perspective.  For over ten years, there has always been limitless "frontier" of available marketing real estate, and we have finally colonized the last bit of land on the Pacific Coast.

The value of a good display ad network is not the technology (a team of quality engineers could relatively quickly).  It is the network.  Aquantive and Doubleclick have enormous numbers of impressions per month (DCLK is 6B/month).  The tags that drive this traffic are the value.  They are the real estate.  Just getting the tags on the sites took almost ten years.

As recently as a few years ago, it was not unreasonable to start up a new ad serving technology.  Falk AG, Zedo, and others were successful with pricing and technology strategies that drove demand. But this was because there was still new real estate that had not been exploited.

These days, the combination of ad networks like Blue Lithium and Federated Media (and many others) plus the publisher networks have opened up some new real estate in the long tail of small web sites and blogs, but the mainstream of traffic is already represented, and new, large sites like YouTube are rare.

I am glad to see Redmond in the game.  It is a hard business, and the question of what to do with Avenue A/Razorfish dwarfs the prior speculation about what Google should do with the Performics division of DoubleClick.  But strong product development will likely give a real boost to Aquantive's platform and that should benefit advertisers.

Google's Move to TV - Black Box or no?

Google has entered into television advertising with a new partnership with Echostar. The partnership includes the creation of an automated platform for ads running on EchoStar DISH Networks' 125 networks. Google will gain access to DISH ad inventory from across all channels and dayparts; Google's platform will then allow it to sell that inventory and provide measurement on those buys.

The buying of DISH Network ads will work much like AdWords and use a web-based auction system. The real-time reporting allows advertisers to see how their ads performed on a second-by-second basis and adjust creative or daypart scheduling accordingly. Advertisers can target by age or demographic data. Reporting data will be pulled anonymously from the four million DISH boxes currently in use. The program will, according to Google's Eric Schmidt, allow for the ads delivered to be more relevant and therefore more valuable to the viewer and advertisers, because they can be targeted more closely.

On the surface, this sounds like good news for advertisers. After all, commercial data on a second-by-second basis is something advertisers have long been searching for, yet even Nielsen's new commercial minutes ratings do not provide such detailed measurement.

But relevance and the ability to measure viewership alone are not enough. Advertisers will not be acting in their own best interests when they purchase television time via the AdWords ad management platform. Online, advertisers can look at the quality score assigned by Google, along with keyword bid, ad copy and landing page, to try to determine how the minimum price bid was determined and how they can improve how often their ad is served.

But on television, a "black box" process could hamstring advertisers, because the same tools for improvement do not exist.

There's also the problem of testing. Online, Google can be used to good advantage because advertisers can test different combinations of keyword, ad copy and landing page to see which drives the best ROI.

With television, testing is obviously far more cumbersome and costly. So while Google may be able to offer better measurement than the broadcast and cable networks themselves, what exactly are they offering to measure? Without a control ad against which to run a test, the measurement is far less useful. An advertiser may find that 120,000 people watched an ad. But, did they take an action as a result of the ad? Would a different ad from the advertiser have worked better?

As consumers lean more and more toward creating and/or choosing their own media for consumption when and where they want it (via linear television, time-shifted television with Tivo, via online sources like YouTube or broadcast network sites, via mobile devices, etc.) ads that aren't relevant haven't got a chance.

But I would love to see Google TV move beyond the black-box automation that it uses online.  If they do so, I believe that advertisers will be well-served.

How Experimentation Boosted Restoration Hardware's ROI 267%

We're in the third month of what we have been calling "The Year of Optimization." (We're calling it that because it has become apparent to us at Offermatica that the companies that don't focus on optimizing their sites, in order to give their customers exactly what they are looking for online, will die... In the same way, the companies that do consider optimization an ongoing part of their marketing mix will thrive and speed past their competitors.)

As I wrote recently, the first and easiest phase of optimization is simple experimentation, and almost every company already does it to some extent. Any marketer who has tweaked copy, changed images, or tried something new has experimented with optimization.

But such simple tweaking is merely a baby step. As marketers become skilled at testing elements on a page, watching to see what works, and implementing winning elements, they become able to test and optimize for more intricate needs.

For example, a couple of years ago, marketing executives at Restoration Hardware wanted to improve usability on their website. They started with simple tests, experimenting with where they placed certain details, such as product copy, on the product detail page.

Then they tested the placement of key gift items on a page, during the holiday season.

Finally, they tested different cross-sell products on the checkout page.

Overall, by experimenting with various elements, the company was able to increase ROI by 267% over the duration of the test.

But perhaps more important, they learned how to test quickly and often, and they learned that experimentation can lead to valuable knowledge about their consumers.

For an article on the lessons Restoration Hardware learned from testing, click here.

Targeting Made Simple - Optimization Approach #2

You want to sell more?  Be more relevant.  Of the four types of optimization technology that I have been writing about recently, targeting is the simplest way of exploiting differences among your visitors.

For example, if you find that different groups of people - say, visitors who arrive from Yahoo vs. visitors who arrive from Google - respond differently to different layout or copy, for example, then you should make sure you can show those groups different content to make the most of their preferences.

Like testing and experimentation, targeting is a white box approach that allows you to actually watch what is taking place and how different groups of visitors act. Whereas black box approaches use technology that takes place behind the scenes, a white box approach lets marketers set up tests, with a control, learn what visitors are responding to, and make decisions about future interactions based on those actions.

What is it?

Most marketers are already doing targeting to some extent. The bulk of landing page optimization is a combination of experimentation and targeting. When a marketer makes a decision about what page he wants his search traffic to land on (interior site page vs. home page), that's targeting. Rules-based targeting is simply doing it faster, and under more defined circumstances, to get more yield.

Adwords is a perfect targeting example.  You specify which words you want to buy and other rules like type of match and bid and, "voila", you are defining rules-based targeting for your prospects.  Who knew you were already in so deep:

Targeting and rules-based targeting are probably the most prevalent form of optimization. In its simplest form, targeting can be used for landing page optimization by showing specific content to visitors based on the keyword they typed.

And it doesn't end there.  Where do we go from the keyword ad.  Why, to a landing page, which has been targeted based on the keyword.  Here is one we use for the ad above:

But targeting is by no means trivial. Behavioral targeting can be very sophisticated, from profiles or persona-based targeting to scenario-based approaches that can model a complete customer purchase cycle and target content by stage or maturity.

Targeting's strength is its simplicity and transparency. That is why it is likely the most prevalent form of  optimization or personalization in use today.

At its most basic, targeting is about searching for groups of people that respond similarly. First time buyers may respond in one way, while those who buy several times a year respond differently, and those who have yet to make a purchase respond in a different way still. Successful targeting exploits the differences between those groups by showing different things to the different groups.

Keys to successful targeting

In order to perform rules-based targeting successfully, you must be able to do several things:

1. Identify different groups of visitors that will behave differently

Different groups might be those who come via natural search vs. those who come from paid search, email recipients who tend to respond vs. those who don't respond, regular buyers vs. non-buyers, etc.

Of course, you can't target based on information that you don't have. If you don't have access to buying history, then targeting to first-time buyers vs. long-time buyers is not a good choice.

Note that every company likely has at least four different groups of visitors. These groups may be divided according to your offerings (people who search for home mortgages vs. people who shop for car loans) or according to gender, geographical area, or other attributes that are ripe for exploiting.

2. Identify the elements that "matter"

While you may have identified several different groups that have the potential to behave differently, they may only behave differently about certain things. Weekend visitors may act differently than weekday visitors in that weekend visitors respond better to longer-form content while weekday visitors respond better to bullet points. On the other hand, the two groups may respond identically to what image you show - product photograph vs. lifestyle shot.
< 
This step is a matter of choosing, among the hundreds of elements within a site or page, what will make a difference.
 
3. Have the ability to run tests, quickly and easily

Of course, you may isolate several elements that you think will matter among different groups, but you won't know if they do until you run tests to see what works. If you believe that people coming from Google will behave differently than people coming from MSN, you need the ability to have a control, to show the control to a certain number of both groups, and to test your variations to both groups in order to learn whether your hypothesis was correct.
 
4. Have the ability to show different things to different groups of people on an ongoing basis
 
Once you learn that various groups do indeed behave differently, you need the ability to continue to serve those groups different content based on their needs. You must, in other words, be able to set the "rules", to say, "In x circumstance, when a visitor behaves like y, I want to serve z content."

The key here is that you must be able to do this without having to go through changing routers, reconfiguring the page, or any other action that requires software development any time it needs to happen. You must be able to do this on the fly, quickly, in order to be able to use it all the time.

If you can do this quickly, it's yours - it's a marketing decision. If it takes too long, or if it requires IT involvement, it's an IT decision and is no longer useful to marketers.

Luckily, there are tools like Offermatica that allow marketers to lay out their guidelines, saying, "On this spot, on this page, show this content only under these circumstances…"

The good, the bad and the ugly

Targeting doesn't happen by magic. You have to set it up and define the rules. You have to be able to test, and then serve different content on an ongoing basis.

But because almost every company has at least four significant customer types that have measurably different buying behaviors, targeting is invaluable in learning how to recognize them and exploit their different buying types.

Google's Website Optimizer - Let the Games Begin, again.

Last fall, Google announced that it was beta testing its Google Website Optimizer. Beginning today, the optimizer is in general release and available to everybody.

Tom Leung, the product manager for Google's Website Optimizer, says in a ClickZ article: "The main problem we're trying to solve is to get people out of the dark ages in terms of how they develop pages. All too often, they'll just put a page together and maybe the designer will do a few mock-ups, and they'll point to the one they feel is going to be the best one."

Sound familiar? It's the "Test, don't guess" philosophy we've been championing for three years. Google should be telling telling advertisers that optimizing is a smart thing to do, because even though most advertisers realize that testing and optimizing landing pages can improve results substantially, having Google say Do it amounts to a marching order for many. There is virtually no amount of marketing budget that I could spend that would be as validating.

On the other hand, Leung's implication that Google's solution can replace large-scale platforms like ours is misguided. "One of those guys can theoretically handle 100,000 page variations. We go up to 10,000," he says in the ClickZ article. He adds that a marketer testing that many variations would be in the .001 percentile. But (perhaps intentionally) he misses the point. He's right that marketers for the most part don't need to test 100,000 variations.

What marketers do need, however, is the ability to test things beyond the landing page. A marketer needs to be able to test an offer on a landing page and then draw that offer through the user's next step, and his next, to include it on internal pages within the site. They need the ability to test the offer within the site, and to continue to optimize the site's internal pages based on the test's results.

So am I concerned about Google's optimizer? When a company with $10B in cash enters your market, things are going to change, and we over here at Offermatica may be delusional, but we are not stupid.

But I trust that the market is (mostly) rational, and so I am very optimistic that this will be continue the surge in interest for Offermatica's testing and targeting service.

Why? Because quite simply, more people need to do testing, and a free tool for a small business is a great start.  For the brands we serve,

1. Offermatica integrates well with the leaders in analytics, WebTrends, Omniture, and CoreMetrics, and can even deliver content to segments derived from these systems.

2. Google's freebie tool is for Google Adwords. Offermatica is for Google, Yahoo, and MSN  -- not to mention display ads, email, affiliates and any other source of traffic, and can be used both externally and internally within your site.

3. As we have covered in What is Optimization, discrete testing is important, but ongoing content targeting and personalization must follow quickly behind.

4. And, as Bryan Eisenburg pointed out during the eMetrics show, "The Page is Dead". If you want better advertising, you have do more that just single page content-switching.

We continue to be passionate about not only providing the highest-performing testing and targeting application for companies like Intuit and C*net, but also providing the kind of marketing guidance and service through our own team and our partners like Camelot, Zaaz, Range Online Media, Wunderman, Digital Grit and others that are leaders in interactive and search marketing.

I want to be clear. Hats off to the team. Truly. We are passionate about better online marketing, we appreciate Google's participation, and we toast their success. Marketers must get into the mold of testing more often, and it cannot happen fast enough.

There is no looking back now, marketing is now officially beyond the site.

Altima's Guerilla Campaign Fizzles Online

If you read my post last fall about Blackberry's 10-second ad in the 30-second spot, you may understand why this landing page just kills me.

On the face of it, it's a nice page. Good colors, crisp imagery, nice little National Sales Race logo, useful roll-over technology that allows a visitor to explore key features of the Altima.

But. I happened to read yesterday of an out-of-home marketing campaign that Altima is putting into effect this week. The company has hired a guerilla marketing firm to "lose" 20,000 sets of keys in bars and hotspots across the country.

The key rings include three keys and two tags. The first tag directs finders to AltimaKeys.com, a website where they can enter the tag number to win a gas card or some other freebie. The second tag tells finders not to "return" the keys - they are no longer needed because the Altima has push-button ignition technology. That second tag directs people to a different website, NissanUSA.com/Altima. The landing page on that site, seen above, doesn't mention a single word about the contest. It doesn't even specifically point out the push button technology, except in half a sentence, in small text, halfway through a generic paragraph about the car.

Now, I'm trying to imagine that I have found a set of keys with a tag that says I shouldn't bother to return them. Even in today's age of increasingly weird marketing stunts, I'd probably be interested enough to go online to explore what I have found.

I might go to the sweeps page first, enter my tag number, and perhaps win a prize. But what if I went to the Altima page in the hopes of finding out more about push button ignition? Assuming I go there, what do I find? No mention of the very thing I'm looking for, and in fact the very thing the campaign meant to get me looking for. After all, isn't that the point of the whole campaign?

So why, why, why doesn't the Altima home page at least have a visible link that mentions the push button ignition?

Once again, it is clear that the disconnect between the "site" guys and the marketing guys is broad, wide and deep.

Let me back off a bit. Let me cut Altima some slack, seeing as this is an offline campaign. Because the people who have found the keys will be visiting the site by typing in the URL directly, as given on the tag, there is no way for Altima marketing folks to target any of that page's web content specifically for those visitors, right?

On the other hand, a small amount of real estate on that genearl Altima page, geared for those visitors who have found the keys or have heard about the campaign, shouldn't have been too much for the marketing team to ask.

There's hope, though. Maybe, just maybe, Altima is targeting content specifically to people who find keys (or hear about the campaign), want to learn more about the keyless ignition, but don't type in the URL.

If I use search terms based on the campaign, will Altima direct me to a site that perhaps has a small bit of content targeted to people who have typed in the words (for example), keyless ignition?

In Google, I type in the phrases "Altima push button ignition," "Altima keyless ignition," and "Altima pushbutton ignition." Although there are a few organic listings that lead me to an interior Altima page, there are no paid search results. Again, it is clear that the marketing folks and the web folks just aren't connecting here. Cool campaign, lousy follow through.

Of course, you might be saying, there IS a website devoted to the keyless ignition - AltimaKeys.com, the site that the first tag directs people to. So let's look at that:

Nice, right? It says thanks for returning the keys and mentions that, because the car has push-button ignition, "I haven't really been paying attention to my keys."

Then, there's a link to find out more about the Altima, and...

You guessed it. That link goes directly to the same Altima home page as seen above, once again with only the tiniest mention of the push-button ignition.

If Altima's marketing folks had thought to ask the web folks for a small bit of real estate on that page (or if the web folks had granted it), it would have been incredibly, immensely, crazily easy to target content specifically to visitors who clicked through from AltimaKeys.com.

By designating a spot - perhaps the spot where the National Sales Race logo is - as a "content slot" (at Offermatica we call them Mboxes), the content could have been rotated depending on the originating source. Anyone coming from AltimaKeys.com could have seen a small box that shouted, "Learn more about push-button ignition!" with, perhaps, a link to the internal page that talked about that feature.

Anyone else could still have seen the National Sales Race logo.

Would it have improved the results of the campaign? At the least, it would have kept the momentum of the campaign going for one more click. At the least, it would get visitors to the page that talks about the interior of the car, rather than just the exterior. At the least, it would have kept them engaged for a bit longer. So, would it have improved the campaign? You bet it would.

Marketers, we need to raise the bar. I'm not, actually, surprised by this campaign - though I am disheartened by it. It seems to be the norm, to be accepted by marketers that the campaign and the site are different entities.

But in order to take our marketing to the next level, this has got to stop.

Until our campaigns carry through the user experience to the point where the user - not the marketer - says the experience is over, our users will continue to be disappointed. They will find the two or three marketers who are out there who are targeting content specifically to them, on an individual basis, and they will stick with those marketers. The rest of us will be sucking wind, wondering what happened.

Google - Testing using Offermatica won't harm SEO

Updated: Check out Is It Cloaking on searchengineland.com.

One question we get asked at Offermatica is whether conducting tests (A/B, multivariate, and other) will affect SEO ranking.  Our response has always been:

1. We do not alter the underlying HTML that is spidered, and Google has assured us that this is not forbidden provided that the underlying HTML is a legitimate page viewed by a portion of the visitors.
2. Google used Offermatica to test its own AdWords interface in September of 2005, and supported our mbox implementation on their own properties.

I was very comforted today to see this response from the GWA group:

---------- Forwarded message ----------
From: Website Optimizer Beta Advisor <[email protected]>
Date: Mar 21, 2007 10:43 AM
Subject: [Website Optimizer] Does Website Optimizer affect my organic search rankings?
To: Google Website Optimizer Beta <[email protected] >

We've recently noticed some confusion about whether Website Optimizer
affects organic search rankings, so we wanted to take a moment to
clear things up.

Website Optimizer is designed to keep your original content visible in
the HTML source code of your page at all times. As a result, your
original content is visible to crawlers, which means there should be
no major impact on search engine ranking.  However, if you implement
changes to your content after using Website Optimizer, they'll have
the same effects as any content changes that you would typically make
to your website.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Google Website Optimizer Beta" group.
To unsubscribe from this group, send email to [email protected]
For more options, visit this group at http://groups.google.com/group/websiteoptimizer?hl=en

I trust this will settle the issue.  The GWA mechanism and the Offermatica mbox operate the same way.

note: I am not subscribed to the Google Website Optimizer beta.

Experimentation and Testing Exposed - Optimization Approach #1

Last week's post on Optimization was my most popular post ever.  In it, I outlined four main types of experience optimization: Experimentation, Targeting, Predictive, and Social.  All have very high value. All can improve relevance and results. And they are not out of reach.

In this post, we will explore experimentation in more detail. In case you are curious, experimentation also goes by the names of A/B testing, multivariate testing, experimental design, and a number of known aliases. It is the "wise old man" of optimization, because its methods and math are well-established and are the basis of many newer automated approaches.

Testing is a "white box" approach that offers confidence and transparency.  When done thoughtfully, it gives feedback you can trust about what your consumer actually responds to.

And frankly, knowing what the customer responds to might be one of the most strategic parts of marketing.

(One note to consider when reading: if any of this sounds complicated or slightly intimidating, remember that the benefits of testing far outweigh the slight challenges involved in getting testing up and running. Plus, there are tools that allow marketers to rapidly create and deploy different tests - tools that handle all the elements I outline below - so that the marketer can concentrate on discovering what is most effective in helping them reach a stated goal, rather than on constructing a useful test.)

What is it?

Experimentation has a long and storied tradition in Western scientific thought.  You pose a hypothesis, design an experiment that isolates the effect you are investigating, and measure the results.  If the data supports your hypothesis, then you go with it.  If not, you re-think and try again.  The power of testing is in its simplicity - we intuitively learn through observation and experimentation.

In online testing, the objective is to evaluate the effectiveness of various elements of an online offering based on response.  Whether you are testing ad creative, navigation, or promotion, the basic approach is the same.  We explicitly vary certain elements while controlling for others so we can clearly identify what has influenced a consumer. 

In the example above, we show four different form pages from a multivariate test we ran a while back. We varied layout, imagery, and messaging.  (Can you guess which one had the highest conversion?)

Talking to the Monkey-Brain

Testing/experimentation are valuable for one simple reason: you are not your consumer (even in B2B). As a result, you cannot intuit what they want, even if you ask them. That's because focus groups (and other approaches where the consumer is aware of the experiment) lead to misleading results. Testing gives the ability to learn who your customer really is.

When you are doing simultaneous testing, you are measuring unconscious response to different marketing approaches: "Do you like more copy or more images", "Should we do more interactive merchandising?", or "Is a two page or one-page form easier to execute?". 

I like to call it "Talking to the monkey brain". When the consumer is unaware, and giving token attention, you are likely marketing to the older, reactive part of the brain -- the monkey-brain -- not our high-consideration conscious self. The fact is that the monkey brain decides faster, and we use it intuitively when we consume information. So having a dialog with your consumer's unconscious is likely a great way to get usable answers.

Since you are dealing with the monkey-brain, you don't ask for a considered opinion, you just show different versions to different visitors *at the same time* and measure which version leads to a response that is favorable to the outcome you want.

As one scientist put it "The reptilian brain has managed to FOOL the subconscious into believing that IT is in control... when all along it is the reptilian brain that is doing the basic controlling." I don't know that I totally agree, but start testing and you will see that what we consciously predict is infrequently true.

Succeeding with Testing

The real advantage of testing is that most of us have been trained to understand the basics of experimentation, and almost every company already does it to some extent. Think of any page of your site — the home page, a category page, a page that requests more information from the user. Now think of all the times you have tweaked copy, changed images, tried something new. Every time you have done that, you have been experimenting with optimization.

Unfortunately, too many folks are constrained by resources and do it in a relatively sloppy way. They run one version for a week and another for the next week. They change a page and they measure the results against some other day. Or they continually vary elements of a site, ad, or email and do not isolate the variants.

The keys to testing successfully are fourfold:

1. Be able to Set Up Content Quickly - You cannot test with just a reporting tool. You must have the ability to change the web page/ad/mobile phone image that different people see to create the test data. The heart of any testing service must be a content serving and routing system that you can use.
2. Traffic - Testing is the most rigorous approach, and seeks precision.  As such, it requires traffic to achieve results.  Traffic does not have to be massive, but if the population of consumers who experience a test is too small or too varied, then it may be hard to find signal.  Running tests to larger groups or running to more well-defined segments can help.
3. Things that Matter, Matter - Signal is a word for a measurable effect. If it doesn't matter to your consumer what color the button is, there is no signal. If using a video on a landing page cuts conversion 30%, there is a lot of signal. You are searching for signal when you test, so you need to think about alternatives that are sufficiently different.
4. Confidence - The heart of testing is statistics. Regardless of your approach, you must measure statistical confidence or you cannot trust results. High statistical confidence (95% or above) means you can trust that if you ran the same test again you would have the same lift. Low confidence means that results will vary or are entirely random. If you are not measuring confidence, test results are really useless.
5. Frequency - The amount of experimentation you do is just as important as (if not more important than) the things you test. Finding "signal" can be elusive, and reducing the cost of running a test makes it more likely that you will try newer and riskier ideas - the very ideas that are transformative to a marketing organization.

This screen cap shows some of the basics - what are you measuring, traffic, which version offers the best outcome, and what confidence. Note that only the third recipe  has high confidence when compared to control (four bars). This means the changes in the other recipes really didn't have signal. Luckily one branch really nailed it.

With Offermatica, our approach is to simplify the presentation of content across different channels like ads and the site. Our Mbox allows us to change content elements on a page, and the AdBox allows us to change ads out in the network. This simplifies the creation of simple tests (A/B) or complex multivariate and non-orthogonal approaches.

From our perspective, it is critical that marketers be able to rapidly create and deploy different experiences and that the tool manage segments, confidence, and population. 

The Good, the Bad and the Ugly

Testing can provide significant feedback to a marketer through its transparency, and can be applied almost universally to any type of offer, layout, content or messaging. Simple testing can provide instant feedback on decisions and should be an element in *every* decision a marketer makes online.  There are a broad range of approaches (A/B, A..n, multivariate, and others), but they all share the discipline of controlled experimentation and can provide invaluable insight to guide future marketing.

However, testing is rigid by design and is less suited for environments with high variability or where environmental "signal" cannot be controlled. Running a/b tests to figure out what product to offer a given visitor or which article would be tedious, ineffective, and likely impossible.

Testing also requires participation by a marketer or agency - if version A of a test wins and there is no one there to listen, it may as well not have happened.

Testing can be an integral part of the ongoing marketing experience, not simply a project you do periodically. Testing can provide instant feedback on decisions and that it can (and should) be an element in every decision you make online.

Next Post - Targeting