際際滷

際際滷Share a Scribd company logo

CERTH @ MediaEval 2011 Social Event Detection Task

Symeon Papadopoulos
Symeon Papadopoulos

The presentation of the CERTH presentation in the Social Event Detection (SED) task @ MediaEval (Pisa, 2 September 2011).

1 of 14
Download to read offline
CERTH @ MediaEval 2011 Social Event Detection Task Symeon Papadopoulos, Christos Zigkolis, Yiannis Kompatsiaris, Athena Vakali Pisa, 1-2 September 2011
The problem Identify social events in tagged photos collections: Challenge 1: Soccer matches @ Barcelona, Rome Challenge 2: Events @ Paradiso (Amsterdam) and Parc del Forum (Barcelona) Alternative formulation: For each photo of the collection answer the questions: Q1. Is this photo related to a social event of the given types? Q2. If yes, to which event is it related? Points to classification and clustering as methods to address the problem. 2
Approach Q1 Q2 Q1 / Q2 3
Photo Filtering (1) City classification If geo-tagging available (~20%), use it simple nearest-neighbour classifier If not, match against city-specific tag models: Created from processing independent geo-tagged photo collections TAG MODEL SAMPLES Amsterdam (74) Barcelona (57) London (89) Paris (51) Rome (42) amsterdam barcelona london paris rome netherlands catalunya uk france italy holland catalonia united kingdom francia vaticano nederland espa単a great britain versailles italia . . . . . 4
Photo Filtering (2) Soccer/Venue classification In the case of venue classification, use geo-tagging information if available. Match against soccer/venue tag model: Parameter (cf. evaluation) TAG MODEL SAMPLES (baseline) Soccer (53, m1,b) Paradiso (6, m2,b) Parc del Forum (8, m2,b) soccer paradiso parc del forum football names of Spanish FCs concert primavera sound + goal names of Italian FCs festival concert goalkeeper gig festival live music + domain names of scheduled bands (last.fm) knowledge 5
Event Partitioning Very simple implementation: Find all unique dates of photos that passed the first filtering step. For each date, find all associated photos and split them into groups based on the city they are classified (same classifier as in Step 1). Consider the resulting groups of photos, as the set of events. 6
Event Expansion Expand in three ways: Photos having the same owner as one of the owners in the event & captured at the same date. Photos captured at the same location (<200m) with the event center & at the same date (only for geo-tagged photos) Photos belonging to the same cluster (by use of method [1]) & having the same owner as one of the owners in the event (parameter: cluster type) [1] S. Papadopoulos, C. Zigkolis, Y. Kompatsiaris, A. Vakali. Cluster-based Landmark and Event Detection on Tagged Photo Collections. In IEEE Multimedia Magazine 18(1), pp. 52-63, 2011 7
Evaluation (1) Challenge 1 Notation Parameter 1 (p1): m1,b (baseline tag model), m1,+ (extended soccer tag model) Parameter 2 (p2): tt (use photo title + tags), ttd (use photo description + tt) Parameter 3 (p3): (no clustering), T (tag-based clustering), V (visual clustering) 8
Evaluation (2) Challenge 2 Notation Parameter 1 (p1): m2,b (baseline tag model), m2,+ (extended venue tag model) Parameter 3 (p3): (no clustering), T (tag-based clustering), V (visual clustering), H (hybrid clustering) m2,+ was created by adding to baseline the names of the bands that played in these venues in the same month (collected from last.fm API) 9
Failure examples (1) C1 - Run1 / False positives 3559542192 3618132279 3580841609 Title: AVU SOM 77.331 Title: Sant Pere Title: roma 09. Tags: , Campions, Trophy, Tags: Barcelone, Barcelona, Tags: rome, italy coliseum, campnou, soccer, football, Night Ambiance, Light palatino, chuch, soccer, caosasuna, bar巽a, fiesta, statues, art Many of the photo tags Captured at the same Just one of the tags are related to soccer and date and in the vicinity of (soccer) is related to even to a soccer event the event. soccer. (fiesta, champions). Most of the false positives were due to the expansion step (i.e. same day + close by, or same day + same user) 10
Failure examples (2) C1 - Run2 / False negatives 3559542192 3571654936 3583033760 Title: near Tor di Quinto, Title: Barcelona v. Latium, Italy Title: DSC_0029 Manchester United Tags: N/A Tags: FC Barcelona Fiesta Tri Tags: Sigma 10-20mm, F4-5.6 Campions Description: s.s. lazio wins EX DC HSM, barcelona, spain, the coppa italia moo2 Here the event The information could be Event information is information is only inferred from title if our tag encoded in a single tag, present in the photo model contained FC names but we dont tokenize description. from different countries. tags, so we miss it. Most of the false negatives were due to failure in matching the textual metadata of photos to the soccer tag model. 11
Discussion (1) Most important factor: a good tag model to be used for classification Marginal contribution of clustering: expansion by spatio-temporal metadata already captures most related photos tag-based clusters tend to include many of the photos of the same user at the same date visual clusters did not yield further improvements as one would hope (at least with employed visual similarity measure: 500 feature vector from clustering SIFT features) 12
Discussion (2) Future action: study in detail failure cases and make necessary modifications to approach Ways to improve: better topic/entity classification methods better/richer tag models + text matching methods more sophisticated methods: e.g. SVMs, relational learning + more discriminative features (text, visual, social) more elaborate city classification methods or even precise geo-tagging methods 13
Questions 14

More Related Content

CERTH @ MediaEval 2011 Social Event Detection Task

  • 1. CERTH @ MediaEval 2011 Social Event Detection Task Symeon Papadopoulos, Christos Zigkolis, Yiannis Kompatsiaris, Athena Vakali Pisa, 1-2 September 2011
  • 2. The problem Identify social events in tagged photos collections: Challenge 1: Soccer matches @ Barcelona, Rome Challenge 2: Events @ Paradiso (Amsterdam) and Parc del Forum (Barcelona) Alternative formulation: For each photo of the collection answer the questions: Q1. Is this photo related to a social event of the given types? Q2. If yes, to which event is it related? Points to classification and clustering as methods to address the problem. 2
  • 3. Approach Q1 Q2 Q1 / Q2 3
  • 4. Photo Filtering (1) City classification If geo-tagging available (~20%), use it simple nearest-neighbour classifier If not, match against city-specific tag models: Created from processing independent geo-tagged photo collections TAG MODEL SAMPLES Amsterdam (74) Barcelona (57) London (89) Paris (51) Rome (42) amsterdam barcelona london paris rome netherlands catalunya uk france italy holland catalonia united kingdom francia vaticano nederland espa単a great britain versailles italia . . . . . 4
  • 5. Photo Filtering (2) Soccer/Venue classification In the case of venue classification, use geo-tagging information if available. Match against soccer/venue tag model: Parameter (cf. evaluation) TAG MODEL SAMPLES (baseline) Soccer (53, m1,b) Paradiso (6, m2,b) Parc del Forum (8, m2,b) soccer paradiso parc del forum football names of Spanish FCs concert primavera sound + goal names of Italian FCs festival concert goalkeeper gig festival live music + domain names of scheduled bands (last.fm) knowledge 5
  • 6. Event Partitioning Very simple implementation: Find all unique dates of photos that passed the first filtering step. For each date, find all associated photos and split them into groups based on the city they are classified (same classifier as in Step 1). Consider the resulting groups of photos, as the set of events. 6
  • 7. Event Expansion Expand in three ways: Photos having the same owner as one of the owners in the event & captured at the same date. Photos captured at the same location (<200m) with the event center & at the same date (only for geo-tagged photos) Photos belonging to the same cluster (by use of method [1]) & having the same owner as one of the owners in the event (parameter: cluster type) [1] S. Papadopoulos, C. Zigkolis, Y. Kompatsiaris, A. Vakali. Cluster-based Landmark and Event Detection on Tagged Photo Collections. In IEEE Multimedia Magazine 18(1), pp. 52-63, 2011 7
  • 8. Evaluation (1) Challenge 1 Notation Parameter 1 (p1): m1,b (baseline tag model), m1,+ (extended soccer tag model) Parameter 2 (p2): tt (use photo title + tags), ttd (use photo description + tt) Parameter 3 (p3): (no clustering), T (tag-based clustering), V (visual clustering) 8
  • 9. Evaluation (2) Challenge 2 Notation Parameter 1 (p1): m2,b (baseline tag model), m2,+ (extended venue tag model) Parameter 3 (p3): (no clustering), T (tag-based clustering), V (visual clustering), H (hybrid clustering) m2,+ was created by adding to baseline the names of the bands that played in these venues in the same month (collected from last.fm API) 9
  • 10. Failure examples (1) C1 - Run1 / False positives 3559542192 3618132279 3580841609 Title: AVU SOM 77.331 Title: Sant Pere Title: roma 09. Tags: , Campions, Trophy, Tags: Barcelone, Barcelona, Tags: rome, italy coliseum, campnou, soccer, football, Night Ambiance, Light palatino, chuch, soccer, caosasuna, bar巽a, fiesta, statues, art Many of the photo tags Captured at the same Just one of the tags are related to soccer and date and in the vicinity of (soccer) is related to even to a soccer event the event. soccer. (fiesta, champions). Most of the false positives were due to the expansion step (i.e. same day + close by, or same day + same user) 10
  • 11. Failure examples (2) C1 - Run2 / False negatives 3559542192 3571654936 3583033760 Title: near Tor di Quinto, Title: Barcelona v. Latium, Italy Title: DSC_0029 Manchester United Tags: N/A Tags: FC Barcelona Fiesta Tri Tags: Sigma 10-20mm, F4-5.6 Campions Description: s.s. lazio wins EX DC HSM, barcelona, spain, the coppa italia moo2 Here the event The information could be Event information is information is only inferred from title if our tag encoded in a single tag, present in the photo model contained FC names but we dont tokenize description. from different countries. tags, so we miss it. Most of the false negatives were due to failure in matching the textual metadata of photos to the soccer tag model. 11
  • 12. Discussion (1) Most important factor: a good tag model to be used for classification Marginal contribution of clustering: expansion by spatio-temporal metadata already captures most related photos tag-based clusters tend to include many of the photos of the same user at the same date visual clusters did not yield further improvements as one would hope (at least with employed visual similarity measure: 500 feature vector from clustering SIFT features) 12
  • 13. Discussion (2) Future action: study in detail failure cases and make necessary modifications to approach Ways to improve: better topic/entity classification methods better/richer tag models + text matching methods more sophisticated methods: e.g. SVMs, relational learning + more discriminative features (text, visual, social) more elaborate city classification methods or even precise geo-tagging methods 13
  • 14. Questions 14