Research Methodologies

According to When Did You Join Twitter, I started messing around with the service 419 days ago. In these 419 days, I’ve seen a few trends and have been a little taken aback from the recent jump in popularity. Its membership has been growing exponentially since the start of 2009. I lump most of the people I follow into a few categories:

1. Twitter Shitters
   

   These are folks who tweet about everything that they’re doing every moment of the day. I just stopped following three such people because they were driving me up the wall with the inane shit that they were posting. These people say that they’re the “real” users of the service. I think they just have too much time on their hands and an over-inflated sense of their own importance. I liken them to LinkedIn “LIONS”. [READ] They don’t really get what the tool is for.

   

2. Lurkers
   

   There have been a lot of studies coming out this week analyzing Twitter’s users. [READ] One interesting stat is that 40% of Tweeters never tweet again after their first day. Some studies are chalking this up to people who have forgotten their login information but I think they’re actually lurkers – people who like to read the tweets but never respond and never tweet. They’re like eStalkers.

3. Marketing Guys
   

   Someone in the last year put the bug in the ear of marketers that Twitter is a great platform for pushing your wares. Pho Citi Noodles (@phociti) in LA is a prime example. This Vietnamese restaurant tweets many times a day trying to get people to come in and eat at their restaurant. I don’t know how effective this kind of strategy is. It actually seems kind of dumb to me.

4. SEO Junkies
   

   I think I fall into this category. SEO Junkies are typically bloggers that are looking to increase the exposure of their blog to the masses by using whatever means possible. I have a plugin for my Wordpress blog that automatically tweets for me when I publish a new blog post. I see Twitter as a platform to increase my exposure and I follow a number of people who use Twitter in the same way. I find their tweets useful and informative and to be frank – I’m glad these people don’t post about what they had for lunch.

5. Facebook Lite Users
   

   The last category I would put people in are Facebook lite users who see the Twitter platform as a social networking tool. I’ve certainly responded to people who reach out to me via Twitter. I’ve had people in the last week as me for recommendations on moving companies, research methodologies, baby strollers and the like. I don’t mind answering these questions with a short reply or direct message. I think this is what Twitter’s founders were probably envisioning when they created the tool back in 2006.

Don’t get me wrong – I’m still not convinced that Twitter is a mature communications tool – especially when compared to other social networking platforms like LinkedIn or Facebook. It’s an interesting tool though and I’ll continue to play around with it. If I end up roaming into category 1 or 3, someone kick me.

When I go out to do research for clients, sometimes I have to send bulk mailers to a large number of people. Earlier this week, I had to send invitations for comments on a new post-secondary education program to several thousand people. Here are some of the better responses I got and how I responded in return:

1. My time is too precious to respond!!!

“I charge for my time just like you. 4 min @ $270.00/ hr.”

My response:

“So… you didn’t fill in the survey, saving you $18 of time that could generate some billable hours but you wasted $4.50 worth of billable hours to tell me that you’re not going to respond to me? Who’s the loser now? I got paid $4 to respond to your note.”

2. YOU’VE UNLEASHED THE FURY!!!!

Edit on July 3: This douchebag is threatening to sue so I’ve removed the actualy text he wrote. Here’s the summary.

“You sent me an e-mail! WAAAAAAAAAAAAAAAAAAAAAAAAAARRRRRGGGGHHHHHHHHH!!!!!” (plus some wildly inappropriate suggestions about my sexuality, etc…)

My response: “Hahahahaha. How old are you?”

3. Looking for love in all the wrong places

“Take me off your list! No way!”

My response:

“Can’t blame a guy for trying. Have a nice long weekend.”

Her response:

“Hmmm… you`re sweet. Tell me about yourself.”

(10 e-mails later, she starts sending pics and asking about my marital status).

Anyhow, that`s the exciting life I lead as a market research professional. Just thought I`d share.

An interesting article in Wired this month talks about how humanity is entering into the “Petabyte Age”. They describe the Petabyte Age as a time where there are sensors everywhere, data pouring in from myriad sources, unlimited data storage capacity and the processing power necessary to fill in the gaps. This article goes on to proclaim the “scientific method obsolete”. I want to argue for and against this idea.

Modelling, like Economics, assumes predictable, rational behaviour in all aspects of our universe. I’ve always had a beef with this supposition – especially on the economics side of the fence. If economic theory and models really worked, we’d have a utopia based on capitalism and the free market. These kinds of models and the political systems based on thse models fall down when they meet with reality because people do not behave rationally. In my experience, people behave irrationally. By extension, patterns in nature are equally as unpredictable.

On the other side of the argument is the fact that a model is only as good as the observations that form the basis of that model. If the observations are fouled, the model is fouled. Given an unlimited number of observations and validation points, you could build a model that could predict any behaviour. At that point – are we talking about a model anymore or omniscience?

That is what the Wired article is suggesting – that our modern ability to collect, store and analyze data about the world around us has made models irrelevant. We can measure and describe almost complete systems now. When you have a comprehensive description of a system, you don’t need a model because you have all of the data.

Models were originally designed to fill in holes in our ability to measure and analyze. Statistics helped us validate the accuracy of our models. The scientific method works as follows:

1. Define the question
2. Gather information and resources
3. Form a hypothesis
4. Collect the data
5. Analyze the data
6. Draw your conclusions and answer your question based on your analyses

The Petabyte Age allows researchers to skip step 3 – the point where models are developed. You can now just observe the data as it is and analyze the system that is in place. Modelling and hypotheses are important when you have limited resources and ability to collect data. When these limitations are taken out of the picture – are hypotheses really necessary? Wired says no.

My conclusion is that you still need to have some concept of what drives a system or motivations individuals in order to analyze even a comprehensive set of data. Otherwise, there is no patterns to guide our decision-making process. You could just say “well, this is how group A behaves” and when someone asks you to guess how Group B is going to behave, you say “we have to analyze their metrics before I can tell you that”.

Models and hypotheses form the foundation of wisdom – insight that can be applied to like situations without extensive data collection and analysis. Without wisdom, we’re all just fumbling around in the dark, hoping to find inspiration. Innovation in the research community comes from drawing on the wisdom of others – wisdom that is often embodied in models and hypotheses.

*********************
Aaron “Crooky” Cruikshank is the Principal and Founder of Friuch Consulting. He has written professionally about science and technology for ten years.

In the past, I’ve discussed the relative merits and challenges of growing your consulting business beyond a sole proprietorship. Since February, my business has grown in leaps and bounds, as I was confident it would and I’ve had to hire help. In my case, I’ve employed the services of two very different kinds of sub-contractors and I’m using some electronic tools to keep my overhead low while getting lots of work done effectively.

Here’s three ways I leverage subcontractors to make my life easier:

1. Help Young Sub-contractors Gain Experience

One of the subcontractors I’m currently using – Melissa Chungfat – is young, talented and works for a very reasonable rate. I’ve known her for several years and she’s a recent graduate from my Alma Mata Simon Fraser University. She’s currently helping me with a research project where the outcomes are unknown and there’s lots of digging through publicly available information involved. It’s not the most exciting assignment in the world but for someone who hasn’t done it before, it’s not dead easy nor boring.

The tradeoff is that she’s only in her early 20s and while talents and very experienced for her age, she hasn’t been kicked in the teeth enough to have any war wounds. This means that I have to do more mentoring and coaching with her than a sub-contractor that’s been around the block a few times. I’m fine with that. I can throw Melissa on projects that more experienced sub-contractors won’t touch because they’ve done them too many times.

It’s a win-win relationship for both of us. I’m helping Melissa gain some resume-padding skills and experience while providing her with a very reasonable wage. I’m getting a motivated, sincere research assistant who works hard and is willing to try new things. I know that I won’t be able to use her in this capacity for long. Within a year, she will have outgrown her need for my patronage and she’ll move on to bigger and better things. In the meantime, it’s a professional relationship that works for both of us.

2. Use Subcontractors with Very Specialized Skills

I’ll be the first to admit that I’m a generalist. This means I’m reasonably capable at a lot of things but not a genius at any one thing. One of the areas where I fall down is in coordination and organizing. My desk looks like a grenade went off on it and I’ve been known to forget about appointments on occasion. I’ve got a sizeable project I’m completing for Terasen Gas at the moment – one that has me interviewing 50 individuals in six weeks on top of my normal workload.

I realized that I could never do 50 interviews in six weeks without some serious organizational assistance. That’s why I hired Cristina Ambrosi from Ripple Effect Business Solutions. She used to be the Executive Assistant to the President of a major Crown Corporation here in Vancouver. If you’ve never met a high-level EA before, you have no idea what an honour it is to be able to retain her services for my humble business.

Cristina knows how to get shit done. I put her on the task of booking all 50 appointments for me. Some of them are face-to-face meetings. Some of them are telephone interviews. All of them are with busy people. She’s amazing at pursuading people to talk to me. My calendar fills up faster than I can keep pace with sometimes. While she’s the more expensive of the two sub-contractors that I’m using right now, she’s worth every penny because I don’t have to supervise her work. I just let her know what needs to be done, set the budget and off she goes.

The benefit to my customers is that the work is getting done on schedule and on budget. If I had to book all of these appointments myself – I doubt that the project would meet its targets. There are other areas where it makes sense to hire a subject matter expert as a sub-contractor. I know a little HTML so I could have technically made my own website given enough time. It didn’t make sense for me to do that so I hired a web designer to do it for me. It’s that simple.

3. Google Apps

I just started using Google Apps as a shared work environment for myself and the three other people who currently help me with my business. If you haven’t seen Google Apps, it’s like Gmail, Google Calendar and Google Docs on steroids for your business where you can send mail from your own domain name and share documents/schedules with your closest associates (whom you can also assign e-mail addresses from your domain name).

In the case of the Terasen job, I have Cristina adding the appointments to a shared calendar in Google Apps which I have synced to my MS Outlook account (which syncs to my Windows Mobile 6 Phone) so that I know within 10 minutes when I have a new appointment booked. It’s a very slick system. I’ve also assigned Cristina a friuch.com e-mail address so that when she contacts interviewees electronically, it looks like she’s an employee, not a virtual assistant.

I can also share and edit documents that read like Word, Excel and PowerPoint documents with all of my sub-contractors. This lets me work with them primarily over the phone and via e-mail. This cuts down on the number of meetings we have to have and keeps costs lower for my clients.

*********************
Aaron “Crooky” Cruikshank is the Principal and Founder of Friuch Consulting. He has written professionally about science and technology for ten years.

I’ve recently started a new avenue of my consulting practice – research coaching. I can’t take credit for the idea because to be honest, my client coined the idea. At a meeting where I was proposing a normal research project, it became clear that the available funds for the project had shrunk by as much as 70% – the client could no longer afford a normal research and analytics package (normally valued at between $15-30K). I really, really like this client and they weren’t scared of getting their hands dirty so I offered them an alternative off the top of my head. My proposal:

- They do as much of the legwork on the research as possible
- I design the research and advise them on implementation
- They can use my specialized subscription tools
- I will walk them through the analysis process and give them my opinion on the research outcomes

Here’s why this is a good arrangement for them:

1. Cost

Hiring someone like me as a Research Coach helps cut the up-front cash costs by as much as 75%. There’s a caveat here though – when someone hires me to produce a complete research package, 95% of what I bill them for is my time. When you take on a Research Coach and agree to do the bulk of the legwork on a research project, you’re spending your own time – whatever that is worth.

The client that just retained my services as a Research Coach is bootstrapping their business and has already done some pre-research so they’re keen to get their hands dirty to ensure the success of their business. This is a great approach.

2. Business Intelligence

There are many advantages to writing your own business plan, doing your own market research and doing your own lead generation. The biggest advantage is the fact that you get to know your business inside and out. It’s sad when you go to a Venture Capital forum and you see some drop-in CEO giving their business pitch when he clearly only understands their core business at a theoretical level.

You need to be able to answer curveball questions that come up when you’re defending your business ideas. People are going to ask questions about the assumptions that you’ve used to come up with your revenue projections or your total addresable market. If you’ve done most of your own market research, you’re going to have these numbers ready in your head.

3. New Skills

The skills you’ll pick up while learning how to do market research will help you in other aspects of your life and business. Market research skills are valuable in policy work (if you end up in a regulated industry). They’re also handy when you want to explore new market opportunities as your business takes off or there’s a change in the market.

4. Street Cred

Doing your own market reseach will earn you a lot of respect amongst your peers – it’s hard work. A lot of your entrepreneur peers have gone through this themselves and in some cases, without the help of a Research Coach. Doing your own market research = instant entrepreneur street cred. Give it a shot!!

*********************
Aaron “Crooky” Cruikshank is the Principal and Founder of Friuch Consulting. He has written professionally about science and technology.

I am currently in a situation where I have to go back and re-survey a population almost a full year after the last time I contacted them. It’s difficult enough to nail down a decent sized population for an initial study but its even more difficult to go back and look at a similar population a year later. My secret – I always ask respondents on a survey if they’d be willing to let me contact them in future for follow-up information.
This keeps me in compliance with the Privacy Act and it gives me a good intro when I do need to get back in touch with someone.

For example: I have just taken on a new contract with the University of British Columbia to help them develop a new graduate degree. I did the phase II research (after initial concept) in Spring 2007. I reported out on my findings at the time but identified a few gaps and recommended a follow-up focus group.

Just a few weeks ago, I got the greenlight to go ahead and do those focus groups. Rounding up participants was going to be a challenge. Fortuntely, I asked the respondents to the Spring 2007 survey if I could contact them again and had now I have a ready-made list of 100+ respondents complete with phone numbers and e-mail addresses. Best of all – they’ve all previously given me permission to contact them again.

I started calling and e-mailing folks earlier this week and I couldn’t be happier with the results. The majority of the respondents were actually happy to hear from me and my two focus groups are filling up fast. One of them is almost full after only two days and the other is half full.

It just goes to show – an ounce of prevention (in this case – leaving the door open for future contact) is worth a pound of cure (me pulling my hair out trying to identify research subjects on short notice).

*********************
Aaron “Crooky” Cruikshank is the Principal and Founder of Friuch Consulting. He has written professionally about science and technology for ten years.

I have recently started using a new research tool – Jigsaw. Jigsaw is essentially a DIY contact list builders. Before Jigsaw, if you wanted to pull together a list of say… Business Analysts in North America, you would either have to try and dig up those contacts using a clever combination of Google and LinkedIn plus 100s of wasted hours OR you could find a BA magazine and rent their subscriber list for an exhorbitant fee. Jigsaw is not only cost effective but it’s a better option in many regards.

I want to note here that I got to this party a little late. Jigsaw has been around since 2003 and while I was an early adopter of other platforms such as LinkedIn, I did not get on the Jigsaw train until this year.

1. Bad data

When you buy a list from someone, there’s always a percentage of bad data in the database. Let’s face it, most databases are complete garbage after about a year. People change jobs, companies go under, etc… When you rent a list from someone, there are typically no guarantees as to the quality. It’s “buyer beware”. As much as 20% of that list might be shit.

Jigsaw is no different in terms of list quality – there is bad data in some places in Jigsaw. I can’t really think of a way to avoid having shit data in a contacts database that doesn’t involve magic. However, Jigsaw will credit your account for every bad contact. How sweet is that? So that means if I buy a list of 1500 BAs and 300 of them bounce back when I try to mail them, I just go into my Jigsaw account, note which ones were bad and I get credits that I can then use to buy new contacts that hopefully won’t be bad. If they are bad – REFUND!

2. Filtering

The filtering options that you have with a list bought from a publication or other source of that type are very limited. They’ll say “you can filter by state, by NAICS code and by company size” or something to that effect.

With Jigsaw, I get to preview every single contact. For example, if I want to buy a list of VP Marketing folks, I use the List Builder tool (see below) and it lets me filter by all kinds of fields:

Once I have a sample I like, I can go in and manually filter it to exclude companies I don’t want on the list and many other things.

3. Ownership

The nice thing about getting a list from Jigsaw is that once you buy it, it’s yours. You get it as a downloadable file that you can use as many times as you want. This is in comparison to list renting where the list owner has a lot more control over how you use the list.

4. Cost

You can buy a list of contacts from Jigsaw for about $1/contact. That sounds like a lot but when I did the math on what it would take me to generate these names myself, I was coming in at around $5/contact. That’s assuming I could dig up as many contacts as I need for the research.

I can’t give you a quote on what a comparable list from a publishing house would be worth but I can guarantee you – it would be a lot more. I’ve seen a one-time rental for a 3000 contact list go for $9000 and up.

So in a nutshell, I love Jigsaw. Try it! You might like it.

*********************
Aaron “Crooky” Cruikshank is the Principal and Founder of Friuch Consulting. He has written professionally about science and technology for ten years.

I mentioned in a posting last week that I recently wrote a test for a job with the RCMP as a Criminal Intelligence Analyst. Part of the test was an essay question that asked something along the lines of “statistics are all well and good but why is it a good idea to go out and gather qualitative data?” My answer was something along the lines of the following:

During my grad school studies during my Advanced Research Methodologies course, I decided to take a run at some data in the FBI Murder Database. (Fascinating database if you haven’t had the opportunity to muck around in it.) While looking at some of the categories of data that were available for analysis, I noticed that “infanticide” was listed as a subset of murder.

If you’re not familiar with the term – infanticide is when someone kills a child that is less than 12 months old. If you don’t have kids (I do), you need to understand that this is a truly heinous crime because no matter how gifted your infant is, there isn’t a hope in hell that they could defend themselves against malicious intent. Infanticide is the moral equivalent of raping and then killing someone in a vegetative state.

I was looking at the issue of infanticide from the perspective of a policy wonk and there are few more prickly pear of policy problems (P4s) out there. (And no, Gordon Campbell, I’m not trying to say that P3 +1 = Infanticide. Everyone knows that P3 + 1 = Armageddon).

When you look at a behavioural trend like infanticide from the numbers, you automatically try to find a correlation between the incidence of infanticide and other factors. In this case, I saw some correlation between the relationship of the accused to the victim and the socio-economic status of the family of the victim.

These numbers might lead a wonk to believe that the root cause of infanticide is socio-economic status and the accompanying problems that complicate family dynamics. I propose that those hypotheses are wrong.

What I found was that there have been numerous studies around infanticide where the accused murderers were interviewed and it turns out that there is a marked difference between the motivation to kill and the gender of the accused.

The papers pointed in the direction of a biochemical/psychological motivation for women to nurture crying/colicky babies that men lack. When women kill babies, it tends to be done with malice and forethought. When men kill babies, it’s more often a crime of passion and an impulse. The men in the studies could be said to have momentarily lost their grip on reality while trying to manage a squalling infant and accidentally killed the child by shaking or striking it.

Back to the original question – why is qualitative data an important acoutrement to quantitative data? Because if you just looked at the statistics, you might think the best policy to reduce infanticide is to work on socio-economic status.

In fact, I believe that to reduce the (larger) proportion of infanticide incidences committed by men, there is a need for a public education campaign that lets men know how they might react to a squalling child. A little forethought and perspective could go a long way to save a few childrens’ lives.

By way of analogy, that is how I feel about the relationship between quantitative and qualitative research. Quantitative research is the metrics. Qualitative research is the context.

*********************
Aaron “Crooky” Cruikshank is the Principal and Founder of Friuch Consulting. He has written professionally about science and technology for ten years.

I had the pleasure of having dinner with Steve Thomson from Thomson and Associates last night. In addition to being a great guy, Steve has a consulting practice that is very similar to mine and overlaps in some areas in terms of the clients we work with.

Steve and I spent our time swapping war stories over beer and food but it was only the day after that we discovered our mutual passion for giving away tips of the market research and intelligence business for fun and profit.

My half-assed attempt at this is my Career Search Intelligence talk that I give periodically at Canadian Career Moves and organizations of that genus.

Steve, on the other hand, has a very slick presentation that shows high tech companies how to bootstrap their market intelligence efforts.

Steve’s given me permission to share this presentation with you. Click here to download it.

I’m hoping to work with Steve more in the future because we seem to have highly compatible personalities and work ethics so don’t be surprized if you hear more about him on this blog.

*********************
Aaron “Crooky” Cruikshank is the Principal and Founder of Friuch Consulting. He has written professionally about science and technology for ten years.

There are lots of reasons why you might want to do qualitative or quantitative research and just as many reasons why you need to be accountable for your work. This posting covers some of these reasons and how you can cover your ass in case a client wants to know what you’ve been doing.

1. Due dilligence

Whenever you are doing market research where investment dollars might be involved, you need to generate a paper-trail. In this post-Enron era, it’s difficult to take a leak without due dilligence.

What does due dilligence around market research look like? Mostly it’s keeping accurate electronic and print records of who you talked to, when you talked to them and what you talked about.

2. Repeatability

Good research should be repeatable and verifiable. I’ve never had a client re-check my work but if they want to (or if they want to hand the work off to someone else to finish), you need to keep a good paper trail to facilitate that.

3. Auditability

In some cases, you will be doing work for a project that has legal or policy implications. In these cases, the client might be obligated by law to audit your findings. I more frequently find myself in the role of auditor than auditee but it could happen.

Personally, I’ve used two systems – excel spreadsheets and a custom research tracking solution that my friend Phil Carr from Canada Connects in Ottawa built for a nanotechnology project I was working on recently.

The excel system works fine but is vulnerable to your computer’s mortality. If your harddrive dies or someone steals your computer, your data and your auditable paper trail are gone.

One solution you can use to get around this – use Google Docs. The added bonus with google docs is that you can share the spreadsheet with other people working on the project so that lots of people can collaborate.

The software solution that I used was web-based and allowed me to track phone calls and a web survey, attaching responses to phone calls and divinding up the work between several researchers.

This worked well but might be overkill if you’re working with less than 50 research subjects. Once you start to get into really serious research campaigns, you might want to look at a commercial task management solution.

*********************
Aaron “Crooky” Cruikshank is the Principal and Founder of Friuch Consulting. He has written professionally about science and technology