際際滷

際際滷Share a Scribd company logo

[OLD] Take home interview

Tal Raviv
Tal Raviv

This is from a very long time ago. Shared as part of this post: https://medium.com/@talraviv/tearing-down-the-teardown-incl-personal-example-2d8c5ce2dce6

1 of 42
Going from invites to meaningful conversations Tal Raviv Start Talking
How do teams start talking? User model
Modeling this off of teams Ive worked on and how we adopted new tools (admittedly all tech, which is not great) Ideally Id spend a good amount of time learning about this qualitatively (phone chats, time with support, user research) Youve watched The Office Assumptions
Excited about life with slack, first to register Will put in extra effort to help adoption Champion Open to trying new things, next to sign up Positive, but wont make exceptional effort Why Nots Risk-averse, let others dive in first Neutral at best, need to feel motivation Laters
How can we quantify that? KPIs and funnel
At least 10 total messages exchanged by 3+ unique users in 24 hours KPI would be % of teams with at least four invitees to reach that in a cohort Assumption! Ideally Id take todays retained teams, go back in time, ask what typically happens in the first 24h (or, in what frame of time do we see 10, 20, 30 messages written?) What is a meaningful convo?
Champion invites at least four people (given) Champion writes in a public channel Two Why Nots write in a public channel Exchange ten messages within 24h What are the funnel steps?
What might thwart that? Hypotheses
No one to say anything to yet/empty room/blank canvas Expect adoption to just happen because everyone sees what they do Why wouldnt champion write in a public channel?
Not in a public channel by default Why wouldnt the champion write in a public channel? (cont) End of current onboarding leaves me in DM with slackbot
Nothing [interesting] to respond to Not in a public channel by default Why wouldnt two Why Nots write in a public channel?
Conversation not catching momentum, not feeling the real-time Why wouldnt two Why Nots write in a public channel? Subject could be more human Could use a clear call to action Ideally highlights real people talking
Conversation not catching momentum, not feeling the real-time Conversation not interesting or engaging Why wouldnt three users reach 10 messages?
Unsure why slack is necessary Need external credibility/social proof Need to see teammates using it Dont know how to use it Champion fears the Blank Canvas These may well apply to Laters, but assuming theyre not relevant to Why Nots Non Hypotheses
What might improve that? Predictions
May improve if: Let Champion in on what we know about successful adoption Help Champion use their energies to more directly evangelize to Why Nots Hypothesis: Champion expects team to magically adopt slack
May improve if: Direct the end of onboarding to #general Hypothesis: New users start in DM thread instead of public channel
May improve if: Provide Champion examples of easy yet meaningful conversation starters (that make sense even if no one arrived yet) Hypothesis: Initial conversation not interesting enough to catch Hypothesis: Champion has no one to write to
May improve if: Use email to more assertively highlight the early action on slack Put stronger emphasis on enabling desktop notifications Put more emphasis on downloading the desktop/mobile clients Hypothesis: Conversation not catching momentum
What first? Prioritizing
= if this works, itll be impactful = low cost to try (or at least test) = likely to work Would definitely want to pulse the different people on the team for any historical knowledge on all three fronts What makes an experiment worth trying?
Let Champion in on what we know about successful adoption Help Champion use their energies to more directly evangelize to Why Nots Direct the end of onboarding to #general Provide Champion examples of easy yet meaningful conversation starters (that make sense even if no one arrived yet) Use email to more assertively highlight the early action on slack Put stronger emphasis on enabling desktop notifications Put more emphasis on downloading the desktop/ mobile clients Rating each experiment Probably the most assumptions on one slide in this presentation
Winners 1. Provide Champion examples of easy yet meaningful conversation starters (that make sense even if no one arrived yet) 2. Use email to more assertively highlight the early action on slack 3. Put stronger emphasis on enabling desktop notifications
Lets try it Experiments
1. Give conversation starters We want the first conversation a team has on slack to be meaningful and involved. We want to encourage people to talk about topics that: 1) are relevant to all types of teams, 2) dont quickly expire, 3) can discuss before everyone gets on slack, 4) anyone on the team can contribute Tada! Slack itself is a great initial topic that fits all those. A side benefit would be encouraging knowledge sharing of slack. Background
Quick sufficient test 1. Give conversation starters Replace this: With this:
That would only show to users when fewer than three messages have been written in the channel 1. Give conversation starters Once people have started talking, revert back to the standard channel meta text and CTAs:
What to measure I would measure success against our KPI: what % of teams who see this get to 10 msg/3 ppl/24 hours Id also want to measure the intermediate steps in the funnel to see if that suggests why this worked or didnt, and next steps. Id also keep an eye on other established KPIs to make sure it didnt hurt, say, invites or integrations. 1. Give conversation starters
What success looks like 1. Give conversation starters Before () After 4+ invites sent / week 1,000 1,000 1 person writes publicly 50% 50% 2 more people write publicly 50% 50% Exchange 10 msgs in 24h 25% 50% 6.25% (63 teams) 12.5% (125 teams) If successful, we could activate 62 more teams per week Id expect the same number of unique people to exchange messages, but that the # messages exchanged within 24h will increase:
2. Use email to highlight early action Background When a team first adopts slack, not everyone is online at the same time. This likely makes it hard to reach the aha moment of real-time conversation with your team. This test would amplify use of email to create a sense of action happening on slack, and encourage people to visit slack more in the first 24h and participate.
2. Use email to highlight early action Quick sufficient test Focus email copy on whos talking + include a natural call to action Simplest version: Dont show chats, just indicate that people are talking (to amplify significance) (Extra variant to test afterwards: Include all chats and interactions in the email across channels)
Tatiana and Jim are talking on Slack in the #general channel See what theyre saying Your team is collaborating on Slack - a great way for busy teams to make life simpler, more pleasant, and more productive. Too many emails? Eek. Good thing theres plenty of ways to stay posted: Turn o鍖 emails and turn on desktop noti鍖cations Download the desktop app or mobile app Just turn the emails o鍖, thanks
2. Use email to highlight early action Who gets this email, and how often? First 24h: Send when there has been at least one person talking, and at least two hours have gone by since last email After 24h: dial back to every 12 hours Sent to all invited team members, including those not registered If already registered, button directs to that channel If not yet registered, button first takes to register Dont send if theyve downloaded a client or enabled desktop notifications
2. Use email to highlight early action What to measure I would measure success against our KPI: what % of teams who get this email pattern get to 10 msg/3 ppl/24 hours Id also want to measure open and click through rates to see if that suggests why this worked or didnt, and next steps.
What success looks like 2. Use email to highlight early action Before () After 4+ invites sent / week 1,000 1,000 1 person writes publicly 50% 50% 2 more people write publicly 50% 70% Exchange 10 msgs in 24h 25% 50% 6.25% (63 teams) 17.5% (175 teams) If successful, we could activate an extra 112 teams/week Id expect this to increase both the rate of additional teammates messaging on slack, as well as the total messages they exchange
3. Emphasize desktop notifications Quick sufficient test Right now its not an emphasized action. What might happen if we communicated its a priority? Create an inline step that is 100% focused on encouraging user to enable desktop notifications Simple first experiment: put this step after the username is created, but before entering slack. Good copy and personality is key to not spammy feel.
3. Emphasize desktop notifications Slack is 3,924X better* with noti鍖cations * This statement has not been evaluated by the Food and Drug Administration. May contain nuts. OK that might be an overstatement. But you can make sure you never miss a personal message, shoutout, or important Ritual Co鍖ee announcement by letting us to notify you instantly. Feeling unsure? We hear you. You can always control how many you receive, or turn them o鍖 entirely at any time. Give noti鍖cations a shot
3. Emphasize desktop notifications We really recommend enabling noti鍖cations, but no worries, we wont take it personally if not. Remember: Noti鍖cations make Slack super awesome You can always turn these o鍖 later Theres a do not disturb mode anytime you want quiet slack.com would like to send you desktop noti鍖cations Wed be honored if you enabled noti鍖cations
3. Emphasize desktop notifications What to measure I would measure success against our KPI: what % of teams who get this inline step that get to 10 msg/3 ppl/24 hours Id also want to monitor how many people enable notifications compared to before to understand the story, regardless of what happened with the KPI Id want to monitor the rate of conversion from registration to entering the slack app to better understand what we might try as next steps
3. Emphasize desktop notifications What success looks like Id expect this to not significantly affect the first few people, but to definitely increase the number of messages exchanged Before () After 4+ invites sent / week 1,000 1,000 1 person writes publicly 50% 50% 2 more people write publicly 50% 50% Exchange 10 msgs in 24h 25% 70% 6.25% (63 teams) 17.5% (175 teams) If successful, we could activate an extra 112 teams/week
These are assertive tests, and theres definitely many more ideas like these Theres a fine line between good growth and spamminess. And its not a dichotomy - its solved by being human and respectful. Id want to find the balance together with the rest of the product team for how assertive tests can be while everyone feeling comfortable with what ships to users. Thats it. Phew. Off my chest
-Tal Thanks.

More Related Content

[OLD] Take home interview

  • 1. Going from invites to meaningful conversations Tal Raviv Start Talking
  • 2. How do teams start talking? User model
  • 3. Modeling this off of teams Ive worked on and how we adopted new tools (admittedly all tech, which is not great) Ideally Id spend a good amount of time learning about this qualitatively (phone chats, time with support, user research) Youve watched The Office Assumptions
  • 4. Excited about life with slack, first to register Will put in extra effort to help adoption Champion Open to trying new things, next to sign up Positive, but wont make exceptional effort Why Nots Risk-averse, let others dive in first Neutral at best, need to feel motivation Laters
  • 5. How can we quantify that? KPIs and funnel
  • 6. At least 10 total messages exchanged by 3+ unique users in 24 hours KPI would be % of teams with at least four invitees to reach that in a cohort Assumption! Ideally Id take todays retained teams, go back in time, ask what typically happens in the first 24h (or, in what frame of time do we see 10, 20, 30 messages written?) What is a meaningful convo?
  • 7. Champion invites at least four people (given) Champion writes in a public channel Two Why Nots write in a public channel Exchange ten messages within 24h What are the funnel steps?
  • 9. No one to say anything to yet/empty room/blank canvas Expect adoption to just happen because everyone sees what they do Why wouldnt champion write in a public channel?
  • 10. Not in a public channel by default Why wouldnt the champion write in a public channel? (cont) End of current onboarding leaves me in DM with slackbot
  • 11. Nothing [interesting] to respond to Not in a public channel by default Why wouldnt two Why Nots write in a public channel?
  • 12. Conversation not catching momentum, not feeling the real-time Why wouldnt two Why Nots write in a public channel? Subject could be more human Could use a clear call to action Ideally highlights real people talking
  • 13. Conversation not catching momentum, not feeling the real-time Conversation not interesting or engaging Why wouldnt three users reach 10 messages?
  • 14. Unsure why slack is necessary Need external credibility/social proof Need to see teammates using it Dont know how to use it Champion fears the Blank Canvas These may well apply to Laters, but assuming theyre not relevant to Why Nots Non Hypotheses
  • 16. May improve if: Let Champion in on what we know about successful adoption Help Champion use their energies to more directly evangelize to Why Nots Hypothesis: Champion expects team to magically adopt slack
  • 17. May improve if: Direct the end of onboarding to #general Hypothesis: New users start in DM thread instead of public channel
  • 18. May improve if: Provide Champion examples of easy yet meaningful conversation starters (that make sense even if no one arrived yet) Hypothesis: Initial conversation not interesting enough to catch Hypothesis: Champion has no one to write to
  • 19. May improve if: Use email to more assertively highlight the early action on slack Put stronger emphasis on enabling desktop notifications Put more emphasis on downloading the desktop/mobile clients Hypothesis: Conversation not catching momentum
  • 21. = if this works, itll be impactful = low cost to try (or at least test) = likely to work Would definitely want to pulse the different people on the team for any historical knowledge on all three fronts What makes an experiment worth trying?
  • 22. Let Champion in on what we know about successful adoption Help Champion use their energies to more directly evangelize to Why Nots Direct the end of onboarding to #general Provide Champion examples of easy yet meaningful conversation starters (that make sense even if no one arrived yet) Use email to more assertively highlight the early action on slack Put stronger emphasis on enabling desktop notifications Put more emphasis on downloading the desktop/ mobile clients Rating each experiment Probably the most assumptions on one slide in this presentation
  • 23. Winners 1. Provide Champion examples of easy yet meaningful conversation starters (that make sense even if no one arrived yet) 2. Use email to more assertively highlight the early action on slack 3. Put stronger emphasis on enabling desktop notifications
  • 25. 1. Give conversation starters We want the first conversation a team has on slack to be meaningful and involved. We want to encourage people to talk about topics that: 1) are relevant to all types of teams, 2) dont quickly expire, 3) can discuss before everyone gets on slack, 4) anyone on the team can contribute Tada! Slack itself is a great initial topic that fits all those. A side benefit would be encouraging knowledge sharing of slack. Background
  • 26. Quick sufficient test 1. Give conversation starters Replace this: With this:
  • 27. That would only show to users when fewer than three messages have been written in the channel 1. Give conversation starters Once people have started talking, revert back to the standard channel meta text and CTAs:
  • 28. What to measure I would measure success against our KPI: what % of teams who see this get to 10 msg/3 ppl/24 hours Id also want to measure the intermediate steps in the funnel to see if that suggests why this worked or didnt, and next steps. Id also keep an eye on other established KPIs to make sure it didnt hurt, say, invites or integrations. 1. Give conversation starters
  • 29. What success looks like 1. Give conversation starters Before () After 4+ invites sent / week 1,000 1,000 1 person writes publicly 50% 50% 2 more people write publicly 50% 50% Exchange 10 msgs in 24h 25% 50% 6.25% (63 teams) 12.5% (125 teams) If successful, we could activate 62 more teams per week Id expect the same number of unique people to exchange messages, but that the # messages exchanged within 24h will increase:
  • 30. 2. Use email to highlight early action Background When a team first adopts slack, not everyone is online at the same time. This likely makes it hard to reach the aha moment of real-time conversation with your team. This test would amplify use of email to create a sense of action happening on slack, and encourage people to visit slack more in the first 24h and participate.
  • 31. 2. Use email to highlight early action Quick sufficient test Focus email copy on whos talking + include a natural call to action Simplest version: Dont show chats, just indicate that people are talking (to amplify significance) (Extra variant to test afterwards: Include all chats and interactions in the email across channels)
  • 32. Tatiana and Jim are talking on Slack in the #general channel See what theyre saying Your team is collaborating on Slack - a great way for busy teams to make life simpler, more pleasant, and more productive. Too many emails? Eek. Good thing theres plenty of ways to stay posted: Turn o鍖 emails and turn on desktop noti鍖cations Download the desktop app or mobile app Just turn the emails o鍖, thanks
  • 33. 2. Use email to highlight early action Who gets this email, and how often? First 24h: Send when there has been at least one person talking, and at least two hours have gone by since last email After 24h: dial back to every 12 hours Sent to all invited team members, including those not registered If already registered, button directs to that channel If not yet registered, button first takes to register Dont send if theyve downloaded a client or enabled desktop notifications
  • 34. 2. Use email to highlight early action What to measure I would measure success against our KPI: what % of teams who get this email pattern get to 10 msg/3 ppl/24 hours Id also want to measure open and click through rates to see if that suggests why this worked or didnt, and next steps.
  • 35. What success looks like 2. Use email to highlight early action Before () After 4+ invites sent / week 1,000 1,000 1 person writes publicly 50% 50% 2 more people write publicly 50% 70% Exchange 10 msgs in 24h 25% 50% 6.25% (63 teams) 17.5% (175 teams) If successful, we could activate an extra 112 teams/week Id expect this to increase both the rate of additional teammates messaging on slack, as well as the total messages they exchange
  • 36. 3. Emphasize desktop notifications Quick sufficient test Right now its not an emphasized action. What might happen if we communicated its a priority? Create an inline step that is 100% focused on encouraging user to enable desktop notifications Simple first experiment: put this step after the username is created, but before entering slack. Good copy and personality is key to not spammy feel.
  • 37. 3. Emphasize desktop notifications Slack is 3,924X better* with noti鍖cations * This statement has not been evaluated by the Food and Drug Administration. May contain nuts. OK that might be an overstatement. But you can make sure you never miss a personal message, shoutout, or important Ritual Co鍖ee announcement by letting us to notify you instantly. Feeling unsure? We hear you. You can always control how many you receive, or turn them o鍖 entirely at any time. Give noti鍖cations a shot
  • 38. 3. Emphasize desktop notifications We really recommend enabling noti鍖cations, but no worries, we wont take it personally if not. Remember: Noti鍖cations make Slack super awesome You can always turn these o鍖 later Theres a do not disturb mode anytime you want quiet slack.com would like to send you desktop noti鍖cations Wed be honored if you enabled noti鍖cations
  • 39. 3. Emphasize desktop notifications What to measure I would measure success against our KPI: what % of teams who get this inline step that get to 10 msg/3 ppl/24 hours Id also want to monitor how many people enable notifications compared to before to understand the story, regardless of what happened with the KPI Id want to monitor the rate of conversion from registration to entering the slack app to better understand what we might try as next steps
  • 40. 3. Emphasize desktop notifications What success looks like Id expect this to not significantly affect the first few people, but to definitely increase the number of messages exchanged Before () After 4+ invites sent / week 1,000 1,000 1 person writes publicly 50% 50% 2 more people write publicly 50% 50% Exchange 10 msgs in 24h 25% 70% 6.25% (63 teams) 17.5% (175 teams) If successful, we could activate an extra 112 teams/week
  • 41. These are assertive tests, and theres definitely many more ideas like these Theres a fine line between good growth and spamminess. And its not a dichotomy - its solved by being human and respectful. Id want to find the balance together with the rest of the product team for how assertive tests can be while everyone feeling comfortable with what ships to users. Thats it. Phew. Off my chest