Personality Characteristics Evident in Product Ratings
I've been curious about the use of "ratings and reviews" that appear on a number of Web sites - in which customers of a site are invited to post their opinions of products as a guide for others who are making a purchasing decision. While I find reviews helpful in a few instances (buying a relatively cheap item and not wanting to invest much time in researching alternatives), I've long had the suspicion that reviews may be skewed.
It's a common fact, but one that few people stop to consider, that evidence based on polls and surveys is presented as a factual representation of a general population - when, in reality, it represents only the opinions and attitudes of the kind of person who is willing to participate in a survey. This is an inherent bias that should give rise to some level of skepticism when an individual is presented "evidence" from one of these studies.
It likewise seems reasonable to assume that the same bias affects the ratings and reviews posted on many popular Web sites (amazon.com, Netflix, and a growing number of e-commerce sites). In effect, that the people who post ratings and reviews fall into a specific psychographic profile, by virtue of their willingness to devote time to typing in their opinions for others to see.
Being the curious type, I looked into this, but couldn't find secondary research. And being the type who has a little too much free time on his hands, who has difficulty falling asleep with an unanswered question rattling around inside his head, and who has just enough training at marketing research to be reasonably dangerous, I decided to look into it for myself.
And since I figured others might be curious about the same, and would sleep better for having some sense of an answer, here's what I found ...
OVERVIEW
This is an exploratory study that seeks to determine the tendencies of individuals who post ratings and reviews on public Web sites in two regards: the Myers-Briggs personality type and the Fog index of readability.
METHODOLOGY
I collected a random sample of 100 comments from the amazon.com site and subjected them to automated analysis using automated tools to determine the Myers-Briggs personality type and Fog readability index of the content of the reviewer's remarks.
I avoided the reviews of books, as I expected that would be inherently skewed (people who read books are a minority and I expect them to be more introverted and literate than the general public) and stuck to ten consumer products: a coffee maker, a television, etc.).
For each product, I sorted the reviews by date and took the ten most recent reviews that contained a reasonable amount of text (about 100 words). This meant skipping the reviews that contained only a few words or sentences, which is a conceded bias of this study (see "caveats"), but fortunately, most reviewers tend to be wordy, so my sense is that the results are fairly good (I didn't have to go through hundreds of reviews to find ten suitable ones).
I recorded the review rating (1 to 5 "stars") and pasted the text into two online tools, one of which assessed personality type and another ran a readability analysis on the passage, and recorded the results.
ANALYSIS
The following analyses were conducted:
Ratings
As shown by the histogram below, ratings are heavily skewed to the positive, with the majority of individuals who rate products giving them the highest possible rating (5 stars), some individuals giving either the lowest possible rating or a "neutral" rating, and very few providing a rating that is only slightly above or below the center.
Since there were 100 samples in the study, the raw count above would coincide with he percentage.
Personality: Holistic
Regarding Myers-Briggs personality types, a bar chart is provided below that shows the count of personality types.
In general, extraverted individuals far outnumber introverted ones, and those who rely upon sensory data outnumber those who reply on intuition.
This will be borne out in further detail in the next four sections, which consider each of the components in isolation.
Since there were 100 samples in the study, the raw count above would coincide with he percentage.
Personality: Extraversion/Introversion
Among reviewers, 75% tended toward extraversion, whereas only 25% tended toward introversion.
Personality: Intuition/Sensation
Among reviewers, 66% tended toward sensory perception, whereas only 25% tended toward intuition.
Personality: Thinking/Feeling
Reviewers were nearly even in terms of their decision-making functions, 52% tended toward thinking, whereas 48% tended toward feeling.
Personality: Judgment/Perception
Reviewers were also nearly even in terms of their lifestyle preference, 54% tended toward judgment, whereas 46% tended toward perception.
Readability Index
The Gunning-Fog metric suggested an average readability index of 4.18, skewed toward the lower end with a mode of 5 and a slight tail toward the upper limit of 13.
INTERPRETATION
Considering the data above, the following conclusions can be drawn:
Ratings
Online product ratings are heavily skewed to the positive, which means that individuals who had a "good" experience with a product are more likely to post a rating than individuals who had a bad experience.
This is largely in line with conventional wisdom (i.e., the "gut feeling" in advance of considering the facts), which suggests that people who have neutral opinions generally do not express them to others, and only those who have had very good or very bad experiences would post their reviews to a Web site.
The only surprise here is that there are far fewer negative than positive reviews, which would seem to suggest a high degree of narcissism among individuals who post reviews to the Web, given that a positive review reflects well on the reviewer ("I made a smart choice in selecting this item") whereas a negative review requires the reviewer to publicly admit having made a bad choice, placing his primary concern with the welfare of others ("I want to warn you against making the same mistake I did").
Personality Types
In terms of the Myers-Briggs classification of personality types,, the categorization of individuals who are extraverted and rely more upon sensation than intuition. Both of these seem to support common logic: an extraverted individual would be more likely to take the initiative to communicate to others, and because the reviews are based on actual experience rather than expectations, the results are skewed to hands-on sensory perception rather than cognitive consideration of what the experience of a product might be.
The remaining characteristics - Thinking versus Feeling and Judging versus Perceiving - were more or less evenly split and, given the small sample size, the level of difference does not seem to be statistically significant, and further analysis based on such a slight difference might be misleading.
Combining the components into personality profiles, the most common reviewer falls into the ESTJ personality type, the "Administrator" archetype, followed closely by the ESFJ ("Teacher") and ESFP ("Politician") personality types. Taken together, the ESF-type personalities constituted 48% of all reviews, which is disproportionate not only to other types, but to the incidence of these personality types among the general population (in which "ES" types are estimated to constitute between 27% and 34% of the population).
It's worth noting that the "ES" group of personality types has marked psychological tendencies toward narcissism and the desire the exert influence over others by means of domination or manipulation.
This would seem to be well in line with the kind of person one might expect to gravitate toward ratings and reviews, as it provides the an outlet for demonstrating their superiority in a situation in which they are likely to have influence over the purchasing behavior of others.
Readability Index
The Gunning-Fog readability index for online reviews showed an average reading level of 4.18, roughly equivalent to the fourth-grade reading level, and the distribution of individual scores supports the conclusion that this is an accurate representation that is largely unaffected by outliers.
There is some controversy over the use of the readability index as a measure of intelligence: some would suggest that the "reading level" of a passage of text reflects upon the intelligence of the writer, whereas the counterpoint is that it is often a reflection of the perceived intelligence of the reader. For example, the fact that newspapers articles are written at the sixth-grade reading level could be interpreted to mean that most journalists are not very bright, or that they are purposefully "dumbing down" the content because they believe people who read the newspaper are not very bright - either of which seems an entirely rational conclusion.
However, online reviews are not written by professionals. It seems therefore less likely that the individual who writes a review would seek to make his writing more readable, but that they would write at their normal level of literacy. If anything, it would seem likely that an individual who posts an online review might attempt to represent themselves as being more intelligent by "smartening up" rather than "dumbing down" their prose in order to make a more positive impression on their readers, especially considering the narcissism and dominance of the prevailing personality types.
As such, the argument that the "readability level" of a passage of text reflects the literacy of the author is, in the case of online reviews, the more credible alternative.
CONCLUSIONS
Taken all in all, the combination of the high level of ratings, prevalence of domineering personality types, and low readability index of rating content, the present analysis paints an unflattering image of the kind of individual who posts ratings and reviews to Web sites: they are self-centered, narcissistic, manipulative, and not very intelligent.
For buyers in the Internet marketplace, this comes as bad news: the ratings on Web sites might seem like a good place to turn for help in making a purchasing decision, but given the intellect and motivation of those who post ratings, chances are that the information posted is unreliable, and is intended merely to feed the egos of those who post ratings and influence others to follow their lead, rather than to provide help to another individual.
For sellers in the same marketplace, there are some positive consequences to posting ratings and reviews: people will say very positive things about products, and those who enjoy posting reviews will be inclined to have a positive experience of the site, given that it feeds their psychological inclinations, even if the ratings have no credibility with those who visit the site to purchase products and are not inclined to post reviews.
CAVEATS
This was an exploratory study done by a layman, and as such, it comes with a handful of caveats:
- The analysis is derived from other analytical tools - Web sites that enabled me to paste in a text sample and get a Myers-Briggs or Gunning-Fog analysis, and any inaccuracies of those tools is carried forward
- I used a convenience sample, a relatively small one (100 reviews), and I expect there is some skew in the nature of products I chose (consumer electronics)
- Because the analysis was based on text, I had to skip reviews containing less than 100 words (and quite a few reviews are a single sentence) because they held insufficient content to be subject to text analysis.
My sense is that a study designed to overcome these limitations would produce more accurate results - but that, nonetheless, the present study remains reasonably valid in spite of these obvious flaws.
SUGGESTIONS FOR FURTHER RESEARCH
Opportunities for further investigation abound.
Primarily, the research should be repeated entirely by someone who has better resources (more accurate text analysis tools) a considerable supply of slave labor (graduate students) on a much more extensive sample, which would not only improve the quality of the primary results.
A more extensive study would also enable cross-tabulation to examine covariance. My sense is it would be interesting to compare the personality type of individuals who give the high ratings versus those who give low ratings, and if possible, to examine the variance in ratings per individual across multiple product categories.