When I started blogging on GameHex, I was looking for kindred spirits in the narrow field of video game research. There aren’t very many game developers and publishers with enough success to sustain a long-term investment in consumer or market research, which made it difficult to exchange best practices. Or war stories. But in my efforts at outreach, I was inadvertently sucked into the broader market research “MRX” social cloud. I met many great people, had many inspiring conversations, dreamed some crazy ideas, and brought all of that energy back to my client-side role.

For many reasons, some professional but mostly personal, I’ve been in the process of flipping back to the vendor side of the bench. My new agency, Insights Meta, is now up and running and we’re doing some cool work with many of my favorite topics: Google Consumer Surveys, survey gamification, and creative data-driven storytelling. I’m very excited about all of these things, and hope to find ways of sharing some of these activities with the broader research community.

The first stop on that tour is the Insights Innovation Exchange conference in Sao Paulo, Brazil. I’ll be speaking there about [insert top secret topic here], and hope to meet up with old and new friends. (PRO TIP: USE DISCOUNT CODE SPEAKERS TO SAVE 20%)

I’d also like to encourage you to check out the Insight Innovation Competition Idea Board, which has some interesting ideas for new methodologies and services. (Be sure to click over and vote for my submission while you’re there!)

Most of my energy is now focused on getting Insights Meta off to a healthy start, which means unfortunately that my GameHex blogging time will remain diminished. I’ll be continuing to blog and post, but most of the content will be channeled through the Insights Meta blog – so change your links.

If you’re interested, just send me a note via the “I’m interested” form below:

[contact-form]

But there’s still that other 10%. And while I have a reasonable degree of confidence about what the sources of error and bias are in my CRM-based sampling efforts, my trust in panel recruiting erodes more and more every month. Consider, for example, a recent vendor selection experience:

1. I decide to run a study in Country A, Country B, and Country C. I request bids from Company X, Company Y, and Company Z.
2. I receive bids for $25 per complete, $20 per complete, and $6 per complete.
3. I contact the local office for Company X in Country A and get an additional quote of $12 per complete.
4. I tell Company Y that their $25 per complete bid has been beaten by a significant margin and get a “new” bid at $15 per complete.

Names obscured to protect the innocent. But nobody was “innocent” in this exchange, because from a distance it becomes obvious that the “value” of a completed survey is completely arbitrary and driven not by data quality or service quality but by a desire to win the bid.

Worse yet, I question whether that completed survey is even worth $6 to begin with. As an experiment, I joined one of the name-brand panels as a “panelist” under a pseudonym one month ago. I completed as little of the registration process as necessary to become qualified to take surveys. (Don’t worry, I haven’t polluted any of your actual work with fake responses. But I’ll come back to that in a moment.) Between August 3 and September 10, I received 25 survey invitations. That’s roughly 5 surveys per week. The panel’s frequency of contact guidelines explicitly say no more than one invitation every two days and no more than 12 per month.

“Oh, but it can’t be that bad! Most panelists are legitimate.” Let’s assume for a moment that this hypothesis is correct, and that panelists are recruited through completely legitimate efforts. For example, perhaps they were on Google and searched for “surveys” (see right).

Hmm. (And by the way: I’ve never been offered $20 to complete a panel survey. Which panel do I need to join?)

Creating a fake panelist account is fast and painless. Identity verification on the Internet is near impossible. But it’s not just respondents committing fraud in this process, the panels themselves are complicit. The lack of transparency in downstream processes invites opportunism, if not straight-up breaking of rules.

Consider: What’s the difference between a $6 respondent and a $25 respondent? Is the $25 respondent substantially better in terms of recruiting practices, data quality, and policy integrity? Or was the $25 respondent simply a $6 respondent that had been purchased from another source and marked up? And how can you tell the difference?

Answer: You can only tell the difference in quality if you are told which panels are being used, and how those panels manage their database. I haven’t found full-service research agencies to be terribly eager to share that sort of information, because:

1. it allows me (with a little bit of sleuthing) to determine the profit margin between the sample cost and the delivered work, and
2. the agency harbors fears of being disintermediated (there’s that word again!).

So what’s a client to do? I have three rules for myself:

1. Always know the source of the sample.
2. Never communicate “statistical margin of error” to internal clients on panel-based surveys.
3. Stay close to the source. Don’t tolerate unreasonable mark-up on panel data, particularly when it’s known to be a pass-through cost from a downstream supplier.

The implications are more subtle, but rather significant:

• Competitive advantage in traditional research is driven by professional relationships, because there are no fundamentally large barriers to entry for new participants. There are no industry standard boards to conform with, no multi-million-dollar investments necessary to get started. Salesmanship is driven by the experience and quality of the people offering services.
• Competitive advantage in technology comes from establishing an official or de facto standard or platform. Establishing these platforms requires capital, relatively large engineering teams, and intellectual property protection.

Jumping forward 10 years: if you believe that technology companies will become major participants in the research economy, then you must also believe that transparency in how these companies do things will be much less than you are accustomed to.

Nowhere is this more obvious than comparing the conferences and trade shows that dominate each respective sector:

• In traditional research, we have venues like TMRE or ESOMAR or TMRTE or MRMW which are run by independent organizations. They attract a broad spectrum of industry participants, both client and vendor side, where there is no clear “top dog” on the show floor. The content spectrum is broad, with a fair amount of “selling”.
• In technology industries, these venues are predominantly platform or company-specific. IBM hosts developer events for their platforms. Oracle hosts developer events for their platforms. Apple hosts developer events. Microsoft hosts developer events. Every attendee understands quite well who the top dog is; the content is more narrowly defined and is generally intended to provide companies that rely on those technologies to remain informed and competitive.

There is no incentive for a platform provider to disclose how they are creating their platform. And that is exactly what Google, Microsoft, IBM, Apple, and others will be trying to do as they experiment with and expand into the research sector. Transparency will still be demanded by their customers, however, and so instead you will see an increasing amount of transparency in the data…just not in how it’s being collected.

The three hypotheses:

• Information should be free. The marginal cost of collecting information is approaching zero. For a technologist, information is simply data. Data has storage costs and processing costs, but is not thought about in terms of acquisition costs. Nothing annoys an engineer more than suggesting that data should be purchased; we may pay for data for a time, if there are no other immediate solutions, but this becomes an engineering challenge: what do I need to build in order to collect that data on my own and convert this marginal cost into fixed cost?
• Insight is expensive. Conversely, turning data into something meaningful and insightful is valuable. That’s because that is the core of what engineers and technologists do: they find scalable and innovative ways to transform raw data into something brilliant.
• Action is priceless. Better yet is creating insights that lead directly to specific actions. This is of obvious value to the technologist, because it creates new data that can be used to build more advanced, real-time, responsive systems.

Shifting most of the perceived value into the “insight” and “action” categories doesn’t necessarily mean that that’s how the business model is written. For example, Google gives away enormous amounts of data, insights, and actionable services in exchange for equally enormous amounts of ad revenue. In games, “freemium” or “free to play” is another way of saying “give away the basic data, but charge for the good stuff.”

In a market research context:

• “Free data” = “do it yourself data.” Using Zoomerang or SurveyGizmo or Google Consumer Surveys may not be literally $0, but compared to the thousands of dollars previously spent on such projects it’s an insignificant difference.
• “Expensive insights” = “interpreted data.” The biggest issue with DIY research, of course, is that someone still needs to analyze the data and draw conclusions as to what it really means. Dropping survey data into a series of PowerPoint charts for each question isn’t insight, though — that’s just converting the data from one format to a different format. Distilling the data into a cohesive story uncovers insights.
• “Priceless action” = “consultative engagement.” When the vendor can deliver specific recommendations for the business, that is priceless. When the discussion at the end of a project engagement focuses on structuring the business problem and the best possible solutions (instead of “here is the data you asked for”), that is high value.

I think. I’m not 100% certain about that. But if you’re asked to register, that’s what that’s all about.

People are now talking about why clients buy cheap research. Or why there can be such a huge gap between the quality of the sales pitch and the quality of the final report. This is not going to sound particularly inspired, but there are some really simple and obvious reasons for all of the above:

• Quality of work output = quality of materials * quality of craftsmen * quality of processes. If you want high quality output, you need great data, great analysts, and great processes and templates for getting the work done. When the quality of the sales pitch is greater than the quality of the report, it’s usually not that hard to figure out why.
• The value of a study is proportional to the value of its business impact. Clients pay more for work that has an incontrovertible impact on their business. If I’m building an early version of a market sizing model, I’m not going to spend tons of money to get super-accurate data. I only need data that’s good enough to help the business make a go/no-go decision on a particular market or category.
• You need great people to make research look great. If we all stop hiring people, developing the talent funnel, and keeping a strong industry-wide bench of players, we will all suffer from significant gaps…such as the current mid-level researcher gap.

• Mobile technology has changed everything.
• Big data is important.
• The biggest threats to the industry are other more technology-savvy industries.

This didn’t really explain the déjà vu, though. It then occurred to me: I already wrote about this 10 months ago. I made eight specific observations about the challenges facing research vendors, and what I would be doing if I were on the vendor side of the relationship:

• Get out of the survey business.
• Find a better way besides surveys to collect data.
• Get out of the syndicated report business.
• Become a data integration super-ninja.
• Sell impact, not methodology.
• Build or buy technology-based scalability.
• Recruit technologists.
• Embrace multi-modal interaction.

Quite unexpectedly, this became one of the more controversial blog posts picked up and cross-posted on Greenbook. My intention had never been to be controversial or stir up trouble; I was only projecting forward what I thought were some of the most probable implications of technology’s current trajectory.

I won’t be shy about this: I left MRMW feeling rather vindicated. Two days of excellent speakers, one by one, reinforced each of the points above:

• “If you’re in the data collection business, find another business.”
• “The leaders of market in research in 2020 could be companies like Google, Apple, or IBM.”
• “Being the gatekeeper for client research is not a smart idea.”
• It’s not about surveys, it’s about inferences from measured behavior and a different way of doing research.”
• “500 million Google survey impressions in the first two months of operation.”

There are plenty of other one-liners sprinkled throughout the event, but the message was consistent: things aren’t going to change, they have changed, and now because of complacency and skepticism many companies are trying to catch up. Even more concerning to me: there are many more vendors not in attendance at this forward-thinking event, blind to the changes or staring at the headlights. None of my current vendors were in attendance, for example, which is somewhat disconcerting.

The message I would share with my vendors (and I know from my weblogs that you all track what I say here) isn’t one of criticism. We are all in a highly competitive environment and there are many conferences to choose from. We often have project and travel conflicts. I don’t need you to go to a bunch of conferences; what I do need, however, is for other smart people to get proactive about the foundational changes happening in data collection. I don’t mind being a scout, but I’m not going to be the canary in the coal mine.

Do not underestimate how quickly a software startup can go from zero to hero. My current fear, and the thing I am thinking about most obsessively at the moment, is identifying the problem spaces where it isn’t already too late to get into the research technology game. Competing with Google, IBM, Apple, Microsoft, and Facebook is not something for the cautious or weak-willed. I already see very practical uses for Google’s polling tool, and so do other teams in my business. The client-side research team disintermediation process has already started, albeit in small ways, and we all need to work together to redefine our roles.

Interesting times.

This is the first business trip in my life where I am traveling without a laptop and feel completely at ease about being able to accomplish my traditional “information worker” tasks. When I finish at the conference and leave for a brief vacation, it will be the first such trip without a physical magazine or book.

These seem like minor little things, obvious points to the frequent traveler. They are not minor changes in behavior. Technology continues to displace products and industries, and to do so with increasing speed. Companies that ride that wave when it hits their industry can become giants; the only alternatives are to be battered ashore or drowned by it.

Mobile is a HUGE wave. In my industry (games) the smartphone is gobbling up the Gameboy / Nintendo DS / PSP mobile gaming market like PacMan dots. It has utterly destroyed a thriving ecosystem in less than 5 years. I don’t think we are anywhere close to understanding or actualizing the full impact of all these personal computers following around every human like a tattoo. But now is the time to think deeply on the subject and to have a few toes in the mobile research waters.