端端舝

2013-10-06 JAECS2013@Tohoku Univ.
緙�賤昴虐噫
卞方月晟惤犯奈正及煦昴
1Sunday, October 6, 13

湮�耀戊奈由旦毛
�薹方仁�I燴六友壬
晟惤�I燴撮胍�腕及斛猁俶

𨈘坰午�I燴
及埻燴卞卩月
毛妏丹砩膽晟惤犯奈正�I燴匹

𨈘坰末白玄及笢匹
睡互お仇勻化中月及井

𨈘坰綎最及
皮仿永弁示永弁旦趙
毛旌仃凶中

升氏卅綎最毛磊化
𨈘坰?�I燴磐彆互
堤月及井毛悝少

Agenda
1. R 匹恅趼蹈�I燴
2. 晟惤犯奈正�I燴及霜木
3. R 卞方月 and 午 but 及𨈘坰
4. 由永弗奈斥瞳蚚卞方月犯奈正�I燴
5. 云歹曰卞

Agenda
5. 云歹曰卞

勾卅仆化杅尹月
ㄗ螿僅桶釬傖ㄘ

pp.87-100 毛淩侔化心月

> sample <-※DONE IS BETTER THAN PERFECT§
> length(sample) #猁匼杅
> nchar(sample) #恅趼杅ㄗ諾啞�漪戈ㄘ
> x <- c(※ABC§,※efg§) #2勾及恅趼蹈
> length(x) #猁匼杅
> nchar(x) #恅趼杅ㄗ跪猁匼卞�仄化ㄘ

> tolower(sample) #湮恅趼毛苤恅趼卞劐𡥼
> toupper(x) #苤恅趼毛湮恅趼卞劐𡥼
> toupper(tolower(sample)) #窣適�I燴1
> tolower(toupper(tolower(sample))) #窣適�I燴2
> #侳嬤說及杅卞蛁砩

> steve <- c(※No one wants to die.§,※Even people who
want to go to heaven don't want to die to get there.§)
> length(steve) # 猁匼杅
> nchar(steve) # 恅趼杅
> strsplit(steve,※※) #恅趼蹈毛諾啞匹ピ曰煦仃
http://news.stanford.edu/news/2005/june15/jobs-061505.html

> steve2 <- unlist(strsplit(steve,※※))
> #伉旦玄囀及猁匼毛磐磁仄ㄠ勾及矛弁玄伙卞
> steve2
> length(steve2)
> table(steve2) #螿僅桶毛釬傖
> as.data.frame(table(steve2)) #桶毛磬卞
http://news.stanford.edu/news/2005/june15/jobs-061505.html

手丹屾仄�耀毛𣷣仆化...

> alice <- readLines(※alice.txt§)
> #readLines() 匹1俴內勾掂心煋心
> head(alice, 50) #50俴醴引匹毛桶尨
> grep(※THE END§, alice) #grep() 匹𨈘坰
> alice[3365:3375] #3370ゴ摽毛桶尨仄化復庲
> alice2 <- alice[41:3370] #掛恅及心毛測⻌
> head(alice2) #癩及凶戶復庲

> alice.words <- unlist (strsplit (alice2, split =
§[[:space:]]+|[[:punct:]]+§))
> #諾啞�?疋伉左玉?市件穴卅升互1勾眕奻適
仁窒煦毛Еピ曰午仄化恅趼蹈毛ピ曰堤仄
> length(alice.words)
> #曆掂萸毛吽中凶�g惤杅毛喲堤

> head(alice.words)
> sum(alice.words ==※§)
> #曆掂萸午旦矢奈旦互窣適仄化�K氏分�磁ㄛ
Еピ勻凶蕣卞公仇反剠ㄗ諾ㄘ午卅月
> alice.words2 <- alice.words[alice.words !=※§]
> # Еピ勻凶磐彆諾午卅月�磁毛壺中化ㄛ陔凶
卅劐杅毛釬勻化婬測⻌
> length(alice.words2)

> sum (alice.words2 ==※the§)
> #※the§及螿僅毛А戶月
> alice.freq <- as.data.frame (table (alice.words2))
> #螿僅桶及釬傖
> alice.sorted <- alice.freq [ order (alice.freq$Freq,
decreasing = TRUE), ]
> # order() 憝杅匹螿僅及詢中�卞�K太杸尹
> nrow(alice.sorted) #俴杅互惤𨘥ㄗtypeㄘ杅

? 勾卅仆月?Еピ月?离𡥼允月互價掛
? �I燴忒�?源楊反ㄠ勾卞癹日卅中
? 引分伊穴趙互斛猁
? Snowball由永弗奈斥匹�𡛟褫夔

Agenda
5. 云歹曰卞

𨈘坰綎最及
皮仿永弁示永弁旦趙
毛旌仃凶中

1) 犯奈正毛掂心煋戈
2) 犯奈正毛煦賤允月
3) 犯奈正毛尹月
4) 杅�毛А戶月
5) 犯奈正毛悵湔允月

1) 犯奈正毛掂心煋戈
> nns <- scan("nns_raw.txt",
what="character")
Read 62959 items

2) 犯奈正毛煦賤允月
> nns_list <- strsplit(nns, " ")
# 旦矢奈旦匹犯奈正毛伉旦玄趙
# 補仄岏袨颷(?)
> nns_unlist <- unlist(nns_list)
# 伉旦玄今木凶犯奈正毛田仿田仿卞煦賤

3) 犯奈正毛尹月
> sort_nns <- sort(nns_unlist)
# 犯奈正及�K太杸尹
> uniq_nns <- unique(sort_nns)
# �K太杸尹凶犯奈正毛引午戶月

4) 杅�毛А戶月
> length(nns_unlist)
[1] 70220 # Token
> nns_all <- table(nns_unlist)
# �g惤珨笊桶及釬傖
> nns_type <- length(uniq_nns)
> nns_type
[1] 7579 # Type

5) 犯奈正毛悵湔允月
> write.table(nns_all,
file="freq.txt", sep="t")
# freq.txt 午中丹靡匹蹈毛正皮Еピ曰卞仄化悵湔

? 升氏卅晟惤?末白玄匹手夎匹仇丹仄凶
�I燴ㄗ杻卞 1)‵3)ㄘ毛仄化中月
? 公及綎最匹ˉ悃ˇ互卅中井蚚陑仄凶中
? 支反曰伊穴趙反仄化中卅中及匹蛁砩

Agenda
5. 云歹曰卞

灍�

R 卞方月 and 午 but 及𨈘坰
ㄗ仇木反剟弘仿白匹允ㄘ

? And/and, But/but 及𨈘坰磐彆毛堵弘仿白匹褫�趙
? nns_raw.txt 毛樓馱仄�g惤卞ピ曰煦仃月
? ピ曰煦仃凶手及井日𨈘坰毛井仃月ㄗ淏�桶政ㄘ
? �I燴卞妏丹憝杅ㄗ犯奈正掂心煋心手云咭木卅仁ㄘ
? strsplit(), unlist(), grep(), length(), barplot()
? 劐杅及笢旯毛呴媆復庲允月
? R 反珨僅場ヽ趙仄化ㄗ悵湔六內ㄘ龰曰瞎心引仄斤丹

湘尹ㄗ丐仁引匹珨瞰匹允ㄘ

> grep("^And,?", nns_unlist, fixed = FALSE, value=TRUE)
> grep("^But,?", nns_unlist, fixed = FALSE, value=TRUE)
# 甲永玄仄凶猁匼毛桶尨
# 淏�桶政及 ^ 毛勾仃化云五ㄛ3恅趼毛漪戈恅趼蹈ㄗunderst§and§ㄘ毛壺俋
# fixed=FALSE 匹��淏�桶政毛瞳蚚ㄛvalue=TRUE 匹猁匼桶尨
> length(grep("^And,?", nns_unlist, fixed = FALSE, value=TRUE))
> length(grep("^But,?", nns_unlist, fixed = FALSE, value=TRUE))
# 甲永玄杅分仃毛桶尨

> length(grep("^And,?", nns_unlist, fixed = FALSE, value=TRUE))
[1] 175
> length(grep("^But,?", nns_unlist, fixed = FALSE, value=TRUE))
[1] 178
> length(grep("^and,?", nns_unlist, fixed = FALSE, value=TRUE))
[1] 1479
> length(grep("^but,?", nns_unlist, fixed = FALSE, value=TRUE))
[1] 260

> freq <- c(175, 1479, 178, 260)
> barplot(freq, names=c("And", "and", "But", "but"), horiz=T, las=1)

? R 匹價掛腔卅𨈘坰?煦昴卅日壬...
1) 犯奈正毛 strsplit() ㄚ unlist() 仄化
2) grep() 午 length() 匹市它件玄
3) barplot() 卅升匹��趙
4) chisq.test() 卅升匹𨈘隅

Agenda
5. 云歹曰卞

Package午反?
? 丐月�I燴?辻夔卞杻趙仄凶皿伕弘仿丞
? baseㄗ價掛由永弗奈斥ㄘ分仃匹手 1,000 眕奻
? 杻忷卅�I燴毛俴丹蕣反ㄛ�e及 package 毛袚樓
? 晟惤�I燴卞杻趙仄凶手及手丐月ㄗex. RMeCabㄘ

灍栳: Snowball package
? 魂蚚倛毛埻倛毛引午戶月辻夔
? stemming 午網壬木月ㄗcf. lemmatization)
? 棗𤩸ロ�午桽磁仄劐𡥼允月方丹卅�I燴
? 灍蕣反方曰詢僅卅�I燴ㄗ匹允互公仇反云￤六ㄘ

> install.packages(※Snowball§) #奶件旦奈伙
> library(Snowball) #掂心煋心
> alice.sn <- SnowballStemmer(tolower(alice.words))
> #苤恅趼卞劐𡥼摽ㄛstemming 毛灍俴ㄗ仄ㄛ磐彆毛測⻌ㄘ
> alice.sn.freq <- as.data.frame(table(alice.sn))
> # 筵�K太及桶毛磬�K太卞
> alice.sn.sorted <- alice.sn.freq[order(alice.sn.freq
$alice.sn),]
> #alice.sn.freq 囀及alice.sn 蹈匹�K太杸尹ㄗ失伙白央矛永玄�ㄘ
>head(alice.sn.sorted)
灍栳: Snowball package

i) tm: Text Mining Package
? http://tm.r-forge.r-project.org/
? http://cran.r-project.org/web/packages/tm/tm.pdf
ii) corpora
? http://www.stefan-evert.de/SIGIL/sigil_R/
? http://cran.r-project.org/web/packages/corpora/corpora.pdf
iii) LanguageR
? http://www.ualberta.ca/~baayen/software.html
? http://cran.r-project.org/web/packages/languageR/languageR.pdf
晟惤�I燴卞杻趙仄凶 packages

゜掛惤犯奈正及煦昴毛
翑仃化仁木月
RMeCab 毛畿賡

RMeCab午反
? 坒泬價𣷣庌互嶱逃仄凶由永弗奈斥
? R 井日 MeCab 毛網太堤仄化゜掛惤
及氾平旦玄毛賤昴今六月
? 賤昴磐彆毛手 R 匹堤薯仄化仁木月
匼ル日仄中皿伕弘仿丞

犯乒珨笊
?RMeCabText() : 白央奶伙賤昴
?RMeCabFreq() : 螿僅摩�
?Ngram() : N-gram 賤昴
?collocate() : 僕お憝�S及煦昴

Agenda
5. 云歹曰卞

氾平旦玄公及手及及復庲仄
ㄗ末白玄及帊�毛眭曰ㄘ
𨈘坰綎最毛隴日井卞允月仇午反
旃噶奻笭猁

公及凶戶卞手
晟惤犯奈正�I燴撮胍毛
輥適仄化悝太引仄斤丹

及旦旦丟

? 允屯化及�I燴?煦昴互 R 及心匹俇磐
? <-> Concordancer + Editor + Excel (+ UNIX) + R
? 皿伕弘仿立件弘及價渙璵傖卞衄�
? --> Python, Perl, ... 升仇匹手妏尹月
? 釬�互藝仄中ㄗㄚMac 卅日白巧件玄手ㄘ
? Excel 及釬�反左乒民乓
及旦旦丟

統蕉恅瓬

統蕉恅瓬ㄗ適ㄘ

Enjoy !
twitter: @sakaue
e-mail: tsakaue<AT>hiroshima-u.ac.jp

端端舝

苀數賤昴遠噫賊卞方月晟逄犯奈正及煦昴

More Related Content

What's hot (20)

More from SAKAUE, Tatsuya (19)

苀數賤昴遠噫賊卞方月晟逄犯奈正及煦昴