狠狠撸

Intro 2 text analytics | Ben Taylor @bentaylordata
Text Analytics Are Awesome!

HIREVUE | TALENT INTERACTION
Agenda
SPAM
Levenshtein distance (word, sentence, cloud)
2
3
4
Text handling, introduction1
Map Reduce / Clustering5
Interview text analytics6
Sentiment

Text handling
Input not expected?

Model ?
M
Input Output

Model ?
M
Input

Model ?
M
Input Output
Stderr:
You’re an idiot &
I don’t like you anymore

Input

HIREVUE | TALENT INTERACTION @BENTAYLORDATA

HIREVUE | TALENT INTERACTION @BENTAYLORDATA
Need to map unstructured text to summary metric

Sentiment
How are you feeling?

Let’s make this easy.
Problem statement:
Expletives + @skullcandy mention?
Good or bad?

Negative Sentiment
? 1048940088:
? "I've got two pairs of Ink'd earbuds by @Skullcandy and they both broke in two weeks. I
$#@&ing hate @Skullcandy! #$#@&You”
? 1054044204:
? “$#@& only one headphone stopped working stupid $#@&ing headphones y is it only one
headphone i blame you @skullcandy”
? 1376767884:
? "@skullcandy never buyin another pair of skull candy headphones this is the fourth pair in the
last 2 months that $#@&ed up”
? 141343855:
? “My headphones blew $#@& you skullcandy -___-”
? 16352011:
? “BAHHHHH My SkullCandys are $#@&ing up AGAIN!”
? 1376767884:
? "@skullcandy $#@& skullcandy"

Positive Sentiment
? 161547390:
? "Getting some skullcandy fix's. #tight #skullcandy #$#@&ingpumped"
? 1306207039:
? "@skullcandy @VegasJarhead @justine_mom $#@& yeah!"
? 1117713458:
? "@skullcandy $#@&in bass is badass",
? 1117713458:
? "@skullcandy ur headphones are bad ass and have awsome $#@&in bass"
? 1086228384:
? "Just bough a pair of Skullcandy supreme sound Hesh's $#@&ING AWSOME!!! the bass is
truly amazing :)"
? 132303540:
? "@K$#@&INGP I thought you were a man not a pussy. Try Skullcandy. Hit me back and I'll
hook you up."

Neutral Sentiment
? 1104061464:
? "@autoerotique @skullcandy #crushers First pair
died after 2 days. Day 2 for new pair. The Alarm is
thrashing my head, un$#@&me these rock”

Conclusion
Sentiment Classification Count
Negative 6
Positive 6
Neutral 1
46% chance tweet is negative, now what?
Welcome to the majority of the sentiment solutions on
the market:
Single-word na?ve Bayesian classification

Positive Sentiment (second pass)
? 161547390:
? "Getting some skullcandy fix's. #tight #skullcandy #$#@&ingpumped"
? 1306207039:
? "@skullcandy @VegasJarhead @justine_mom $#@& yeah!"
? 1117713458:
? "@skullcandy $#@&in bass is badass",
? 1117713458:
? "@skullcandy ur headphones are bad ass and have awsome $#@&in bass"
? 1086228384:
? "Just bough a pair of Skullcandy supreme sound Hesh's $#@&ING AWSOME!!! the bass is truly amazing :)"
? 132303540:
? "@K$#@&INGP I thought you were a man not a pussy. Try Skullcandy. Hit me back and I'll hook you up.”
? 1104061464:
? "@autoerotique @skullcandy #crushers First pair died after 2 days. Day 2 for new pair. The Alarm is
thrashing my head, un$#@&me these rock”

Conclusion
Sentiment Classification Count
Negative 6
Positive ~0
Neutral ~0
~100% chance tweet is negative with tuple assistance. How to find complex
tuples automatically!?
Bayesian bootstrap matrix
Unique words in training cloud
Uniquewordsintrainingcloud

Basic sentiment output
Credit: Ben Peters
Keyword Negative positive
warranty 28.7 1
cant 11.8 1
back 11.8 1
break 11.8 1
after 11.1 1
what 9.1 1
never 9.1 1
Don’t 9.1 1
second 8.4 1
side 8.4 1

SPAM
I can’t handle this

Lost future
customer

SPAM examples:
>80%

SPAM list
Keyword spam good
@nikesb 52.0 1
@lrgskate 52.0 1
live 34.0 1
know 1 28.8
have 1 22.3
pair 1 16.3
earbud 16.1 1
Non-ascii-chars 12.4 1
some 1 11.9
check 1 11.6
Credit: Ben Peters

Training….
Where do you get your training set?
What about @#tags? Misspellings? ? ?

Training….
Where do you get your training set?
What about @#tags? Misspellings? ? ? SPAM?

Manual trainer
http://54.186.199.209/
Credit: Ben Peters

Levenshtein
Now things are getting interesting

The things we take for granted
You type: Awsome
Computer: It’s actually spelled Awesome
① kitten → sitten (substitution of "s" for "k")
② sitten → sittin (substitution of "i" for "e")
③ sittin → sitting (insertion of "g" at the end)

Levenshtein word level
Ref:
I am going skiing tomorrow
Hyp:
I am going skiing on Saturday

Levenshtein word-cloud level
Ref:
alphanumeric_sort(word_cloud_1)
alphanumeric_sort(unique(word_cloud_1))
Hyp:
alphanumeric_sort(word_cloud_2)
alphanumeric_sort(unique(word_cloud_2))
>> wer(str1,str1)
ans = 0
>> wer(strjoin(sort(strsplit(str1,' ')),' '),str1)
ans = 15

MapReduce
Great for
Text processing
i.e. word counts

CLUSTERING
Now things are getting interesting

Group of tweets?
? Once we have categorized tweets we can
build word clouds!!!
Category A
(could be negative sentiment,
low selling areas, etc..)
Category B
(could be positive sentiment,
high selling areas, etc..)
words
words
words
words
words
words
words
words

Levenshtein wordcloud similarity

Cluster 1 example
Camping
VirginGamingBattlefield

Cluster 2 example
Skiing
winterStringray

Cluster 3 example
MMABoxing
Skateboarding

Training a blacklist filter
? Acting…
? Getting…
? Holding…
? Going…
? Brings…
? Turning..
Blacklist dictionary

狠狠撸

Text analytics intro

More Related Content

Text analytics intro