Module 3 · Product Management Fundamentals
Analysing feedback
Turn 1,500 pieces of feedback from four sources into evidence, count themes properly, spot when one channel's loudness distorts the picture, and weigh feedback against who your users actually are.
About 25 minutes
The problem
Paystream's sales team brings up bulk payroll at every meeting: "every business customer is asking for it". The support lead says verification problems are drowning her team. App reviews complain about fees and crashes. Everyone has a list, and each list is honest. They can't all be the top priority.
Feedback is evidence, but only if you count it carefully and know where it came from.
The concept
Count by theme and source
Tag every item with a theme (the dataset is already tagged), then count by theme and by source. A theme that dominates one source but not the others is a sign of that channel's audience, not of all users.
Weigh against your users
Compare who gives feedback with who your users are. If one segment is 15% of users but gives most of the feedback on a theme, that theme matters to them, not necessarily to everyone.
Severity
App store ratings show how much an issue hurts. One-star reviews about crashes cost downloads.
Requests are not solutions
"Add USSD" is one user's idea of a solution; the problem is "I can't pay when the network is bad". Keep problems and requested solutions apart.
Example
Themes by source:
import pandas as pd
base = "https://academy.cloudtechanalytics.com/datasets/product/"
feedback = pd.read_csv(base + "feedback.csv")
users = pd.read_csv(base + "users.csv")
by_source = pd.crosstab(feedback["theme"], feedback["source"], margins=True).sort_values("All", ascending=False)
by_sourcesource App review Sales team Support ticket Survey All
theme
All 686 150 530 134 1500
Transfer fees 116 10 102 21 249
BVN verification problems 131 9 91 18 249
Bulk payroll payments 46 111 38 10 205
App crashes 98 6 77 19 200
Savings goals 97 1 75 19 192
Split bills 74 9 52 19 154
No USSD or offline access 69 1 53 19 142
Card delivery delays 55 3 42 9 109Most of bulk payroll's feedback comes from one source: the sales team. Who is it coming from, compared with Paystream's users?
payroll = feedback[feedback["theme"] == "Bulk payroll payments"]
print("Bulk payroll feedback by segment:", (payroll["segment"].value_counts(normalize=True) * 100).round(0).to_dict())
print("New users by segment:", (users["segment"].value_counts(normalize=True) * 100).round(0).to_dict())
print("Sales team items that are about payroll:", f"{(feedback[feedback['source'] == 'Sales team']['theme'] == 'Bulk payroll payments').mean():.0%}")Bulk payroll feedback by segment: {'Small business': 77.0, 'Salary earner': 13.0, 'Market trader': 6.0, 'Student': 4.0}
New users by segment: {'Market trader': 35.0, 'Salary earner': 30.0, 'Student': 20.0, 'Small business': 15.0}
Sales team items that are about payroll: 74%Payroll is a real need for small businesses, about one in seven new users, and the sales team hears about it constantly because small businesses are who they talk to. That doesn't make it the biggest problem for Paystream's users overall. Now severity, from app store ratings:
reviews = feedback[feedback["source"] == "App review"]
reviews.groupby("theme")["rating"].agg(reviews="size", average_rating="mean").round(2).sort_values("average_rating")reviews average_rating
theme
App crashes 98 1.72
BVN verification problems 131 1.86
Card delivery delays 55 2.84
Transfer fees 116 2.84
Bulk payroll payments 46 3.22
No USSD or offline access 69 3.25
Savings goals 97 3.25
Split bills 74 3.34Crashes and verification problems get the lowest ratings, and they're also the themes users can't work around: if verification fails, they can't use Paystream at all.
Walkthrough
- Run the cells. Excluding the sales team, what are the top three themes?
- Which segment gives most of the feedback about USSD and offline access?
- Split one theme into the problem and the requested solution.
- Write a reply to the sales team (the task below).
Practice
Practice
What share of the sales team's feedback is about bulk payroll? As a percentage, whole number.
Task
5 minReply to the sales team's "every business customer is asking for payroll" in 50 to 120 words: acknowledge the need, show what the wider feedback says (with numbers), and explain how payroll will be considered.
Your work is checked for
- Acknowledges the need
- Uses numbers
- Mentions other themes (verification, fees, crashes)
- Explains the process (prioritis, RICE, compare, evidence)
- Between 50 and 120 words
Check your understanding
Answer every question to check.