狠狠撸

Entity Summarization with User Feedback
Qingxia Liu1, Yue Chen1, Gong Cheng1, Evgeny Kharlamov2,3, Junyou Li1, and Yuzhong Qu1
1 National Key Laboratory for Novel Software Technology, Nanjing University, China
2 Department of Informatics, University of Oslo, Norway
3 Bosch Center for Artificial Intelligence, Robert Bosch GmbH, Germany

■ RDF Dataset: T
● triple t∈T: <subj, pred, obj>
■ Entity Description: Desc(e)
● Desc(e) = {t∈T: subj(t)=e or obj(t)=e}
● t∈Desc(e): <e, prop, value>
■ Entity Summarization: S(e, k)
● S?Desc(e) , |S|≤k
2020.06 2
Entity Summarization
Solson
Publications
Rich Buckler
Organization
Gary Brosky
Brooklyn
Company
comics
United States-themed
superheroes “Solson Publications”
“1986”
“1987”
type
type
headquarters
industry
label
subject defunct
founded
founder
keyPerson

■ Goal
● to satisfy users’ information needs
■ Limitations
● have large diff from user-created (F1<0.6)
=> do not meet users’ expectations
● summaries are static
=> cannot adjust to users
2020.06 3
Entity Summarization Methods
A lack of mechanisms for improving an entity summary
when its quality could not satisfy users’ information needs.

■ Our proposal: cooperation between summarizer and user
● involve users in the summarization process
● ask users for feedback
● utilize user’s feedback when computing summaries
2020.06 4
Adding User in the Loop
Summarizer
Summarizer
User

■ Research Challenges
[C1] Cooperative process of 2 agents
– How to represent the process?
[C2] Combining results of actions of 2 agents
– How to represent interdependence between summary and feedback?
2020.06 5
Adding User in the Loop
Summarizer
Summarizer
User

■ Cooperation Process
● Summarizer: presents (current) summary Si
● User: crosses off an irrelevant triple as negative feedback fi
● Summarizer: replaces it with a new triple ri ,
presents an improved summary Si+1
● User: … (repeated)
2020.06 6
Cross-Replace Scenario
f1
r1
replaced by
f0 r0
replaced by
S0 S1 S2
Solson Publications

■ [C1] Representation of the Cross-Replace Scenario
● Markov Decision Process (MDP) based modeling
● Summarizer = reinforcement learning agent
2020.06 7
Our Approach: DRESSED
Solson Publications
- Label: “Solson Publications”
- Key person: Rich Bucker
- Subject: United State-themed superheroes
- Extinction year: “1987”
- Founded by: Gary Brodsky
- Industry: Comics
- Name: “Solson Publications”
- Founding year: “1986”
…
Feedback
Replace
Current Summary
Candidate Facts
Setting:
? agent: summarizer
? environment: Desc(e), user

■ [C2] Representation of Triple Interdependence
● Policy Network
– Encoding triples
– Encoding sets of triples
– Encoding triple interdependence
and scoring candidates
– Selecting replacement triple
2020.06 8

■ Policy Learning
● Learning task: to find a policy that maximize the expected reward
● REINFORCE: a standard policy gradient method in reinforcement learning
– to maximize the expected reward
2020.06 9

■ Entity Summarization
(not utilize user feedback)
● FACES-E: rank triples and select top-k as summary
■ Interactive Document Summarization
(Re-compute doc summaries utilizing user feedback)
● IPS
– positive feedback
■ Interactive Document Retrieval
(Re-rank retrieved documents based on user feedback)
● NRF
– negative relevance feedback
– query-based
● PDGD -L, -N
– positive feedback
– SOTA online learning to rank
– utilize user feedback for model selection
(not as input of the model)
2020.06 10
Baselines Adapted for Evaluation
query
model
generate
update
model
generate
update

■ Setting
● 24 participants
● train: Entity Summarization Benchmark (ESBM)
● test: re-sampled entities from
DBpedia and LinkedMDB
■ Results
● DRESSED: in general best-performing
● with FACES-E and DRESSED
– users stop quickly: I < 4
– obtain reasonably good results: Q_stop > 4
– replacement triples selected by DRESSED
significantly better than the ones of FACES-E
2020.06 11
Experiment 1: User Study
Metrics:
? I = number of iterations
? Qrplc = user rating of suggested
replacements (1-5)
? Qstop = user rating of the final
summary after all iterations (1-5)

Experiment 2: Offline Evaluation
■ Benchmarks
● ESBM (ESBM-D, ESBM-L), FED
■ Metrics
● NDCF: for summary sequence
– Normalized Discounted Cumulative F1
● NDCG: for replacement sequence
– Normalized Discounted Cumulative Gain
■ Results
? DRESSED outperforms
? all baselines
? on all datasets
? DRESSED significantly outperforms
? in most cases
2020.06 12

■ Iterations
● DRESSED is consistently
above all the baselines
● DRESSED better exploits
early feedback to quickly
improve computed
summaries
2020.06 13
Experiment 2: Offline Evaluation

■ Our Approach: DRESSED
● showed better performance than baselines
● has the potential to replace static entity cards
■ Future Work
● cross-replace scenario
– can be extended to support other scenarios:
? multi-feedback, without replacement, positive feedback
● extend the scope of entity summary
– deal with paths, more complex structures, RDF sentence
2020.06 14
Conclusion and Future Work

■ Contributions
● The first research effort to improve entity summarization with user feedback.
● A representation of entity summarization with iterative userfeedback.
– cross-replace scenario, MDP
● A representation of set of triples and their interdependence as a novel DNN.
– DRESSED, solve by RL
● The first empirical study of entity summarization with user feedback.
– based on real users and simulated users
■ DRESSED
● Deep Reinforced Entity Summarization with uSer fEedback
● GitHub Repository: nju-websoft/DRESSED
2020.06 15
Take-home Message

Thank you !
Questions ?
2020.06 16

狠狠撸

Entity Summarization with User Feedback (ESWC 2020)

Recommended

More Related Content

Similar to Entity Summarization with User Feedback (ESWC 2020) (20)

Recently uploaded (20)

Entity Summarization with User Feedback (ESWC 2020)