�ݺ�ߣ

Search & Analytics 
in Archives using 
Semantic Annotations
Klaus Berberich 
(kberberi@mpi-inf.mpg.de)

/ 27Search & Analytics using Semantic Annotations (Klaus Berberich) 2
Team / About
๏ Research area Text + Time Search & Analytics at MPI-INF 
 
 
 
 
 
 
 
 
+ close collaboration with Jannik Strötgen and others 
๏ Focus: Efﬁcient and effective methods to search and analyze 
text collections with temporal information 
(e.g., temporal expressions, timestamps)
Kai Hui Arunav Mishra Dhruv Guptame

Motivation
๏ Idea of this talk is to give you an overview of our research 
during the past two-and-a-half years without 
diving too deep into technical details 
๏ Semantic annotations as common ground of all our methods
๏ What are they? How do we obtain and process them?
๏ Which models do we use to make sense of them? 
๏ Each of our recent papers presented on at most two slides 
(no technical details, but feel free to ask technical questions)

Outline
๏ Motivation
๏ Archives
๏ Semantic Annotations
๏ Search
๏ Analytics
๏ Outlook

Archives
๏ Newspaper archives
๏ Examples:
๏ The New York Times Annotated Corpus
๏ English Gigaword Corpus
๏ clean language, reliable meta-data (e.g., publication dates)
๏ Web archives
๏ Examples:
๏ ClueWeb09/12
๏ UKGOV
๏ messy language, unreliable meta-data (e.g., no publication dates)

Outline
๏ Motivation
๏ Archives
๏ Search
๏ Analytics
๏ Outlook

Semantic Annotations
๏ Our recent methods leverage semantic annotations to make 
documents more than sequences of text tokens
๏ (disambiguated) mentions of named entities (e.g., persons)
๏ geographic references (e.g., cities or countries)
๏ temporal expressions (e.g., to past dates) 
๏ We rely on existing off-the-shelf tools and some handcrafted
rules to obtain semantic annotations for documents

(Disambiguated) Named Entities
๏ AIDA (https://www.ambiverse.com) as our named entity
recognition and disambiguation (NERD) tool of choice
In Indianola, Mississippi, … , a new museum will begin construction in June to
honor a local boy who has become international star, the blues performer B.B. King.
The B.B. King Museum and Delta Interpretive Center will begin with
restoration of the cotton mill, … , and is expected to be open by 2007, with additions
following. Artifacts from Mr. King's 60-year career will be housed in the museum and
the interpretive center will focus on educational, cultural and character development
programs for Mississippi youth.

(Disambiguated) Named Entities
๏ Named entities are anchored in the YAGO knowledge graph, 
which provides us with additional information about them
๏ semantic types (from WordNet and Wikipedia)
๏ surface forms, keyphrases, and links
๏ general facts
wordnet_guitarist
American_Blues_Guitarist
Riley B. King
Bluesboy
isLocatedIn
hasLatitude 33.43°
hasLongitude -90.63°

Geographic References
๏ Geographic references derived from named entity mentions 
by considering only entities having type yagoGeoEntity
๏ Minimum-bounding rectangle (MBR) indicating geographic
extent of a location determined by making use of isLocatedIn 
relationship and geographic coordinates from YAGO 
isLocatedIn
hasLatitude 33.43°

๏ Outlier removal found to be helpful 
(e.g., Hawaii or Alaska) 
๏ Open question whether richer 
representation (e.g., convex hull) 
would be beneﬁcial
isLocatedIn
hasLatitude 33.43°
isLocatedIn
hasLatitude 34.96°
isLocatedIn
hasLatitude 31.00°

๏ Probabilistic generative model for latitude-longitude pairs 
based on a set of MBRs (e.g., from document or query)
๏ draw a MBR at uniform random
๏ draw a latitude-longitude pair contained in MBR at uniform random

Temporal Expressions
๏ Temporal expressions annotated using
๏ SUTime (http://nlp.stanford.edu/software/sutime.shtml)
๏ HeidelTime (https://github.com/HeidelTime/heideltime) 
๏ Reliable publication dates are crucial for correct resolution 
of relative temporal expressions (e.g., “last year”, “next month”)
In Indianola, Mississippi, … , a new museum will begin construction in June to 
The B.B. King Museum and Delta Interpretive Center will begin with 

๏ Meaning of temporal expressions is often vague (e.g., “in June”)
๏ capture earliest/latest begin/end time point of any precise 
time interval that the temporal expression may refer to
In Indianola, Mississippi, … , a new museum will begin construction in June to 
The B.B. King Museum and Delta Interpretive Center will begin with 
[2005/06/01, 2005/06/30, 2005/06/01, 2005/06/30]
[2007/01/01, 2007/12/31, 2007/01/01, 2007/12/31]

๏ Probabilistic generative model for time intervals 
based on a set of temporal expressions 
(e.g., from document of query)
๏ draw a temporal expression (quadruple) at uniform random
๏ draw a time interval that the temporal expression may refer to
begin time point
endtimepoint
‘05 ‘06 ‘07
‘05‘06‘07

Outline
๏ Motivation
๏ Archives
๏ Search
๏ Analytics
๏ Outlook

Identifying Interesting Time Intervals
๏ Idea: Given a keyword query (e.g., world war), determine
interesting time intervals (e.g., [1914, 1918] or [1939, 1945]) 
that help the user reﬁne the query to explore results
D. Gupta and K. Berberich: Identifying Time Intervals of Interest to Queries, 
CIKM 2014
P [ [tb, te] | q ] =
ÿ
d œ top(q,k)
P [ [tb, te] | d ] · P [ d | q ]

CIKM 2014
P [ [tb, te] | q ] =
ÿ
d œ top(q,k)
P [ [tb, te] | d ] · P [ d | q ]
Generative model 
based on 
temporal expressions

CIKM 2014
P [ [tb, te] | q ] =
ÿ
d œ top(q,k)
P [ [tb, te] | d ] · P [ d | q ]
Generative model 
based on 
temporal expressions
Document likelihood 
~

query likelihood

Temporal Diversiﬁcation of Search Results
๏ Idea: Re-rank documents to cover all interesting time intervals 
(e.g., [1914, 1918] or [1933, 1945]) previously identiﬁed 
for a keyword query (e.g., world war)
D. Gupta and K. Berberich: Diversifying Search Results Using Time, 
ECIR 2016
arg max
R : |R|=k
ÿ
[tb,te]
A
P [ [tb, te] | q ]
A
1 ≠
Ÿ
d œ R
(1 ≠ P [ [tb, te] | d ] · P [ d | q ])
BB

ECIR 2016
Probability that 
time interval [tb,te]  
is interesting for query q
arg max
R : |R|=k
ÿ
[tb,te]
A
P [ [tb, te] | q ]
A
1 ≠
Ÿ
d œ R
(1 ≠ P [ [tb, te] | d ] · P [ d | q ])
BB

ECIR 2016
Probability that user sees 
at least one document in R 
that covers [tb,te]
Probability that 
time interval [tb,te]  
is interesting for query q
arg max
R : |R|=k
ÿ
[tb,te]
A
P [ [tb, te] | q ]
A
1 ≠
Ÿ
d œ R
(1 ≠ P [ [tb, te] | d ] · P [ d | q ])
BB

Linking Wikipedia and News Archives
๏ Idea: Given an excerpt from Wikipedia, automatically retrieve
articles from a news archive that provide in-depth information 
 
 
 
 
๏ Divergence-based re-ranking 
of top-K documents
A. Mishra and K. Berberich: Leveraging Semantic Annotations to Link Wikipedia 
and News Archives, ECIR 2016
The same month, a groundbreaking was held for a new museum,
dedicated to King,[44] in Indianola, Mississippi.[45] The B.B. King Museum
and Delta Interpretive Center opened on September 13, 2008.[46]
KL(QÎD) = KL(QtxtÎDtxt) + KL(QtimeÎDtime)
+KL(QgeoÎDgeo) + KL(QentityÎDentity)

Outline
๏ Motivation
๏ Archives
๏ Search
๏ Analytics
๏ Outlook

Generating Event Digests
๏ Idea: Given an event query (e.g., from a Wikipedia year page), 
automatically generate a summary of the event 
having a user-speciﬁed length 
 
๏ ILP selects sentences from top-k 
pseudo-relevant documents
๏ minimize divergence(s) between 
sentences and event query
๏ cover all temporal expressions,  
geographic references, and 
named entities from event query
๏ adhere to length budget
A. Mishra and K. Berberich: Event Digest: A Holistic View on Past Events, 
SIGIR 2016
1999/08/30 East Timor votes for independence 
from Indonesia in referendum
Figure 1: Example of an event digest.
Event Query
Description: East Timor votes for independence from Indonesia in a referendum
Time: August 30, 1999
Geolocations: 1643084; 1623843; 7289708;
Entities: East_Timor; Indonesia; 1999_East_Timorese_crisis;
Event Digest (with chronological ordering on publication dates)
‚ Publication Date: July 20, 1999 Source Link: http://goo.gl/rJYDiZ
(1) Indonesia is preparing to relinquish control of East Timor after 23 years of
occupation and it believes that independence advocates are highly likely to win a
referendum next month says an authentic internal government report that has been
made available to reporters by advocates of independence. (2) Late next month
estimated 400 000 East Timorese are to choose between broad autonomy within
Indonesia option 1 or independence option 2.
‚ Publication Date: August 29, 1999 Source Link: http://goo.gl/Cz6Jkk
(3) Former president Jimmy Carter whose human rights and diplomacy organi-
zation the Carter Center is monitoring the referendum here said this this month
some top representatives of the government of Indonesia have failed to fulﬁll their
main obligations with regard to public order and security.
‚ Publication Date: November 21, 1999 Source Link: http://goo.gl/hdqYm8
(4) The last time it was East Timor which voted for independence from Indonesia
in August only to be plunged into a spasm of violence that required an Australian
led international military force to quell it. (5) Acehs latest push for independence
began with the fall of President Suharto in May 1998 and accelerated after the
East Timor referendum.
‚ Publication Date: September 24, 2000 Source Link: http://goo.gl/AijWVY
(6) East Timor has been under a transitional United Nations administration since
the Aug. 30 independence vote last year. (7) The groups pillaged East Timor
after last year’s independence vote which freed the territory from military control.
‚ Publication Date: August 24, 2001 Source Link: http://goo.gl/EAGBxC
(8) This vote like the referendum in 1999 is being organized by the United Na-
tions which has continued to administer East Timor a former Portuguese colony
annexed by Indonesia as it struggles to its feet economically and politically.
to it. An event digest in such a case becomes an intermediate level
of linking that presents a holistic view. Excerpts from Wikipedia,
an abstract view, are connected to excerpts in the digest which are
in-turn connected to news articles that give a detailed view as il-
lustrated in Figure 2. As other use cases, since event digests are

Mining Events
๏ Idea: Identify and describe important events for a given query  
 
 
 
 
 
๏ Clustering of sentences from top-k pseudo-relevant documents 
based on a Dirichlet process mixture model (DPM) with
resulting clusters as events described by probability distributions
over keywords, time intervals, geographic coordinates, 
and named entities
D. Gupta, J. Strötgen, and K. Berberich: EventMiner: Mining Events from Annotated
Documents, ICTIR 2016
lord of the rings movie
Entities [YAGO:United_States] [YAGO:Al_Gore]
Table 10: An event cluster for query us presidential elections. The cluster points to the US Presidential Elections in 2000 for which A
Gore ran as vice president.
Keywords [soviet][afghanistan][war][military][beginning][party][forces][union][exhibition][mixed]
Time [01-Jan-1938 , 01-Jan-1938][01-Jan-1980 , 01-Jan-1980][29-Apr-1988 , 29-Apr-1988]
[01-Jan-1979 , 01-Jan-1979][01-Apr-1988 , 01-Apr-1988][29-Jul-1987 , 29-Jul-1987]
[01-Jan-1950 , 01-Jan-1950]
Locations [YAGO:Soviet_Union] [YAGO:Afghanistan] [YAGO:Moscow] [YAGO:Kabul] [YAGO:United_States]
Entities [YAGO:Soviet_Union] [YAGO:Afghanistan] [YAGO:Mohammad_Najibullah] [YAGO:Moscow] [YAGO:Bosniaks]
[YAGO:Kabul] [YAGO:United_States]
Table 11: An event cluster for query soviet afghanistan war. It depicts the Soviet-Afghanistan conﬂict that lasted from 1979 to 1989.1
Keywords [lord][rings][top][movie][motion][opinion][pictures][article][elvis][jackson][trilogy][movies]
Time [15-Dec-2002 , 15-Dec-2002][01-Jan-1987 , 01-Jan-1987][25-Jan-2004 , 25-Jan-2004]
[12-Nov-2002 , 12-Nov-2002][01-Jan-2003 , 31-Dec-2003][01-Jan-1982 , 01-Jan-1982]
[11-Jan-2004 , 11-Jan-2004][28-Dec-2002 , 29-Dec-2002][07-Sep-2003 , 07-Sep-2003]
[01-Dec-2003 , 31-Dec-2003]
Locations [YAGO:Weldon,_Northamptonshire] [YAGO:Wellington]
Entities [YAGO:J._R._R._Tolkien] [YAGO:Weldon,_Northamptonshire] [YAGO:Wellington] [YAGO:Carol_Ann_Lee]
[YAGO:Peter_Jackson]
Table 12: An event cluster for query lord of the rings movie. It captures the location where the movie was shot — Wellington and th
author of the book on which the movie is based on — J. R. R. Tolkien.14
Keywords [iraq][states][united][war][opinion][top][international][relations][defense][armament] [presi-
dent][time][fearful][david]
Time [13-Apr-2006 , 13-Apr-2006][15-Jun-2005 , 15-Jun-2005][16-Jul-2003 , 16-Jul-2003]
[16-Oct-2003 , 16-Oct-2003][30-Jun-2005 , 30-Jun-2005]
Locations [YAGO:New_York_City] [YAGO:Port_Washington,_Wisconsin] [YAGO:Radcliff,_Kentucky] [YAGO:Iraq]
[YAGO:United_States]
Entities [YAGO:Iraq] [YAGO:United_States_Army] [YAGO:Donald_Rumsfeld] [YAGO:United_States_Department_of_Defense]
[YAGO:George_W._Bush] [YAGO:Jim_Folsom] [YAGO:New_York_City] [YAGO:Port_Washington,_Wisconsin]
[YAGO:Radcliff,_Kentucky] [YAGO:United_States]
Table 13: An event cluster for query iraq war. The cluster shows the start of Iraq War in 2003.15
Keywords [iraq][iran][war][oil][international][top][faw][port][east][world][delegate][rafsanjani]
Time [01-Mar-1986 , 31-Mar-1986][01-Sep-1980 , 01-Sep-1980][01-Sep-1980 , 30-Sep-1980]
[01-Jan-1970 , 31-Dec-1970][01-Jan-1980 , 01-Jan-1980][23-Sep-2003 , 23-Sep-2003]
[25-Jan-1991 , 25-Jan-1991][01-Aug-1988 , 31-Aug-1988][17-Mar-2006 , 17-Mar-2006]
[01-Jan-1000 , 01-Jan-1000][01-Jan-1988 , 31-Dec-1988][02-Oct-2003 , 02-Oct-2003]
Locations [YAGO:Iran] [YAGO:Iraq] [YAGO:Geneva]
Entities [YAGO:Iran] [YAGO:Iraq] [YAGO:United_Nations] [YAGO:Akbar_Hashemi_Rafsanjani] [YAGO:Iranian_peoples]
[YAGO:Gulf_War] [YAGO:Geneva] [YAGO:United_Nations_Security_Council] [YAGO:Fao_Landing]
[YAGO:Western_world] [YAGO:Persian_people] [YAGO:Iran-Iraq_War] [YAGO:National_Iraqi_News_Agency]
Table 14: An event cluster for query iraq iran war. It describes the conﬂict between Iran and Iraq that lasted from 1980 to 1988.16
…

Estimating Time Models
๏ Idea: Estimate a time model (i.e., probability distribution over 
time intervals) for excerpts (e.g., sentences or paragraphs) 
that do not contain temporal expressions
A. Mishra and K. Berberich: Estimating Time Models for News Article Excerpts, 
CIKM 2016
B.B. King collaborated with Eric Clapton
on the album Riding with the King in 2000
In 2007 B.B. King played at Eric
Clapton’s second Crossroads Festival
B.B. King played with his long-time
collaborator Eric Clapton

Estimating Time Models
๏ Distribution propagation approach with edge weights based on  
text similarity, conceptual similarity, and contextual similarity
A. Mishra and K. Berberich: Estimating Time Models for News Article Excerpts, 
CIKM 2016
B.B. King collaborated with Eric Clapton
on the album Riding with the King in 2000
In 2007 B.B. King played at Eric
Clapton’s second Crossroads Festival
B.B. King played with his long-time
collaborator Eric Clapton

Outline
๏ Motivation
๏ Archives
๏ Search
๏ Analytics
๏ Outlook

Outlook
๏ Extension of probabilistic generative model for time intervals 
to support different temporal granularities (day, month, year) 
๏ Narrative generation for event digests to make them more
natural for humans (e.g., chronological order, co-references) 
๏ Index structures for documents with semantic (and linguistic)
annotations from which all our methods can potentially proﬁt 
๏ Word embeddings for semantic annotations to have a uniform
representation of text, temporal expressions, geographic
references, and named entities and simplify our methods

Thank You!
Questions?

�ݺ�ߣ

Search & Analytics in Archives using Semantic Annotations

More Related Content

Search & Analytics in Archives using Semantic Annotations