Open-Ended Comments – Analysis Options (Part 2)

This blog is the part 2 of the blog entry on Open-Ended Comments. Part 1 is here.

In the last blog entry we talked about the importance of open-ended
comments and reason for adding them to your surveys if you are not
already doing it. In this blog entry we’ll focus on ways and techniques
you can use to extract the value out of the open-ended comments that
your respondents leave
.

Eyeball browsing:

Believe it or not, the easiest way to analyze open-ended comments is
(you guessed it) reading them! This is the primary reason we have the
“Open-Ended Text Report” within QuestionPro. If you have a ton of
responses, it may make more sense to download the Excel report and read
through the comments. For ease of use, the Excel Report is broken down
by different sheets and each open-ended question has its own sheet.
Reading through the comments while having the analytical data in the
back of your head, gives you a complete picture of how to interpret the
overall survey.

It is generally prudent to support the analytical data with
representative quotes from some comments when preparing an executive
summary. This allows for a personal, impact oriented analysis and
summary of the survey. Which is why journalists typically quote an
established figure – to personalize or put a face on the survey data.



Grouping/Segmentation:

In some cases, you may want to isolate the comments for a particular
data-segment. The most commonly used technique is when respondents are
asked a “closed-ended” question and you want to segment it based on
that. For example, if you are running a customer satisfaction survey
and you ask your customers to choose the region (or department) that
they interacted with – you’ll probably have a question that asks users
which department they interacted with as a closed-ended multiple choice
question. Now, the idea is to filter the results of the open-ended
comments and view the ones that are associated with some key
departments.

This is fairly simple to do within QuestionPro. Just create a
“Data/Segment” for the closed ended question (multiple-choice) and then
click on the “Text Report” for that data-segment. The system will
filter the open ended comments for that particular data-segment.


Keyword Searching:

This technique can be used if users are specific about a particular
topic. The big question here is: what to search for? Since open-ended
comments can be “all over the map” you need to know what kinds of
phrasings your respondents will be using. A classic example here is a
satisfaction survey given to all the attendees of a conference. They
had the basics of conference surveys – Did you like the schedule, the
speakers, hotel accommodation etc. Turns out that one of the big issues
was that the toilet was all dirty and there was apparently a huge line
there as attendees shuttled between presenters.

We were able to search a standard set of keywords (“toilet”, “restroom”
etc.) and determined that over 60% of the comments had to do with that
one issue. This is a good example of having context and the only way
out there is to read the first few comments and decide if the issues
can be framed using a set of keywords. If that is the case, then the
data can be analyzed in the aggregate and deeper insight can be
extracted out.


Text Categorization:

If you’ve been in the Market Research business, you would call this
“Coding” – this is where you pay someone to analyze each text response
and categorize or code them into a set of pre-defined buckets. Quite
frankly, we do not believe this is a reasonable cost/efficient model.
There are numerous flaws associated with this methodology and the model
is very cumbersome. The first issue being that someone (either within
or outside your company) needs to get trained to understand the
categorization model. This is not cheap and really does not scale (it’s
a linear cost model). For example the cost to categorize 1000 comments
would be 10x the cost to categorize 100 comments. The Web, is all about
exponential scaling not linear scaling. You would not be doing online
surveys, if you didn’t buy into the exponential scale concept – would
you? This model fits in with the “Paper survey” or “phone survey”
model, where the cost to conduct a survey is linear – simply because
the costs associated with them are time based – where human beings need
to spend time as data-entry operators.

I am sure there are many valid arguments for using a text
categorization model, in certain situations – when you are paying
through your nose for a niche target demographic (doctors, IT decision
makers, CxO’s etc.). In such cases I think it makes sense to spend the
time and effort to analyze the comments using a text categorization
model.

We at QuestionPro have thought about developing a tool to facilitate
the text categorization model, but so far have pushed it to the
back-burner simply because we don’t think users would use it. If you
think otherwise, please feel free to post a comment or reply to blog
[at] surveyanalytics [dot] com.

Crowdsourcing:

This is HOT new buzzword of the Web 2.0 economy. If you’ve not heard of
crowdsourcing you should climb out of the rock you are in! (just
kidding!) – We just got introduced to the concept about a couple of years ago.
OK – So what is crowdsourcing and how does that apply to open-ended
comments? Crowdsourcing is when you “source” or ask a group of people
to complete a task. Instead of relying on a single “expert” you
basically rely on a “crowd” to come up with the correct answer. To
learn more about crowdsourcing I highly recommend the following:

• The Wisdom of Crowds (Book by James Surowiecki)
Amazon.com‘s Mechanical Turk (Google “mechanical turk”) – http://www.mturk.com/mturk/welcome
• Translation of Facebook.com to Spanish. (instead of hiring translators to translate their site,
they simply asked their users to translate different parts of the site
etc.)

OK, now that we have a quick primer on crowdsourcing, the question is
how can this be applied to open-ended comments? The answer is really a
two part strategy that we think can change the way comments are
solicited and surveys are conducted. Today, the world of surveys is a
uni-directional communication model. It basically means that only the
respondent and you (and in most cases only the researcher) has access
to the comments posted by a respondent. We think this is a wall for
crowdsourcing and serves as an inhibitor to capturing these comments.
We think, a better model for soliciting feedback is a Digg style
open-access paradigm where you let the crowd post comments and then
vote these comments up and down. This basically involves having a
“portal” of some sort where users can browse other comments and cast
their votes on the comments themselves (this is the crowdsouring part)
or post their own open-ended suggestions (this is the feedback part.)

We at QuestionPro are currently in beta mode with a few select
customers on this crowdsourcing paradigm. We’re calling this IdeaScale.
You can find (not a whole lot but a little more) information on this
paradigm here:

http://questionpro.ideascale.com

I hope the options presented here for open-ended feedback is of value
to you as you go about soliciting feedback from your customers,
employees and partners. I am sure there are a few other ways that
you’ve analyzed open-ended feedback and if you feel I’ve missed or
overlooked some other model, please feel free to drop me a note. As
always you can get in touch with us about the blog at :

blog [at] surveyanalytics. [dot] com

Comments are closed.