Snapchat: A Design Update Gone Wrong

Abstract

This blog posts shows how releasing a design update that is perceived negatively by the app users can lead to a disaster. Due to the redesign and such tweets, the stock price of Snapchat decreased by 6% or $1.3 billion.

The analyses shown below, come from my research paper The Influence of Design Updates on Users: the Case of Snapchat. Obviously, a huge shoutout to my two co-authors: Lukas Fischer and Roland Holten. If you are interested in the full story, I can definitely recommend reading it. The following sections will focus on the conducted analyses:

1. Description of the Obtained Data Sets

Two data sets were downloaded from the Google Play Store: (1) a data set (2,150,972 observations) that contained the user app store ratings of Snapchat and (2) the text reviews (737,182 observations) of Snapchat that were provided by the app users. In a first step, we excluded all empty text reviews and restricted the language to English. Next, some pre-processing steps were conducted to ensure that we could use proper text mining afterwards:

  • conversion of all characters to lowercase.
  • removal of all numbers and special characters.
  • removal of stop words that did not add any value to the text, such as "and" or "in".

The final descriptives are shown in the table below:

Descriptives of the app store ratings and text reviews of Snapchat
Variable Value
Average review length in characters 49
Standard deviation review length 64
Average rating 3.91
Standard deviation daily rating 0.45

The next thing to do was to split up the data sets into two parts: one containing the ratings and reviews before and one after the design update. As the design update was released on February 6, 2018, this was the date that was used to split the data set. As we can see from the table below, the average app store rating decrease from 4.04 to 3.52. Further the average review length increased:

Descriptive statistics: before and after the update
Before update After update
Average review length in characters 56 69
Standard deviation review length 61 73
Average rating 4.04 3.52
Standard deviation daily rating 0.21 0.69

2. Data Analysis

For the data analysis part, we first aggregate a 7-days moving average of the app store ratings and visualized them using a standard plot:

Average App Store Ratings
Average App Store Ratings

As we can see, at the time of the update - the 6th February of 2018 - the average app store rating decreased tremendously. Next, we use text mining to extract the sentiments and n-grams from the data set. N-grams are sets of words that occur frequently together. So n=1 is, for example, a unigram of one word, while n=2 is a bigram or combination of two words, and n=3 a trigram of or combination of three words. The analyses were conducted with Python library NLTK. A first overview of the n-gram analysis can be found below:

Most common n-grams before the design update
No. Unigram Bigram Trigram
1 app love snapchat social media app
2 love social media favorite social media
3 snapchat worst app multi snap feature
4 update amazing app social media apps
5 friends android users front facing camera
6 filters nice app record multiple videos
7 fun update sucks talk ur friends
8 cool awesome app bad camera quality
9 amazing cool app absolutely love snapchat
10 don fun app social media platform

Most of the words in the table above, are positive and supportive of Snapchat. However, this changes dramatically after the release of the design update of Snapchat:

Most common n-grams after the design update
No. Unigram Bigram Trigram
1 update update sucks update sucks ass
2 app love snapchat social media app
3 snapchat social media update sucks change
4 love friends stories update sucks bring
5 sucks recent update broke don fix
6 hate update ruined favorite social media
7 stories worst update update ruined snapchat
8 don user friendly update ruined app
9 friends update makes update sucks balls
10 version sucks ass snapchat update sucks

The most frequent n-grams change from being positive to negative and name the software update as one of the reasons. In a next step, the Python library VADER was used to generate the average sentiments of the reviews. The results are plotted below:

Average User Reviews Sentiment Analysis
Average User Reviews Sentiment Analysis

The sentiment analysis confirms the results of the app store ratings. Again, a huge drop of sentiments is visible at the release point of the design update.

3. Conclusion

The analyses and plots lead to the conclusion that the design update of Snapchat - that was released on the 6th of February 2018 - had a huge negative impact on the Snapchat's users. We see that both the average app store ratings and the sentiments of the user generated text reviews decreased significantly. Snapchat was forced to reverse some of the design changes introduced and the app store ratings and the sentiments increased afterward to a more normal level.