Activate 2019: Tweaking the Base Score: Lucene/Solr Similarities Explained
?
0 likes?37 views
ºÝºÝߣs from the talk Rafal Kuc and Radu Gheorghe did at Activate 2019. Video link here: https://www.youtube.com/watch?v=kKocQdYGVJM
Activate 2019: Tweaking the Base Score: Lucene/Solr Similarities Explained
- 2. Tweaking the Base Score: Lucene/Solr Similarities Explained Demo: github.com/sematext/activate/tree/master/2019 More info: sematext.com/blog/search-relevance-solr-elasticsearch-similarity Radu Gheorghe Rafa? Ku? www.sematext.com
- 3. Agenda BM25 - Best Match: the default DFR - Divergence From Randomness framework DFI - Divergence From Independence IB - Information-Based models LM - Language Models Custom similarity Putting it all together
- 4. TF*IDF You know, for historical reasons
- 5. BM25 - the TF part freq / (freq + k1 * (1 - b + b * dl / avgdl)) Best for Most ?
- 6. BM25 tunables freq / (freq + k1 * (1 - b + b * dl / avgdl)) k1 - raise or lower ceiling
- 7. BM25 tunables freq / (freq + k1 * (1 - b + b * dl / avgdl)) doc length normalization
- 8. BM25 demo yes, that¡¯s how we look when we give demos
- 9. BM25 Good default. You can tune the weight of freq and docLength.
- 10. Divergence From Randomness Basic Model G, I(n), I(ne), I(F) After Effect L, B Normalization H1, H2, H3, Z, none
- 11. tf * c * avgFieldLength / docFieldLength Divergence From Randomness - H1
- 12. Divergence From Randomness - H1 No normalization, and H1 with c == 1, 3, 5, 7
- 13. tf * log2 (1 + c * (avgFieldLength / docFieldLength)) Divergence From Randomness - H2
- 14. Divergence From Randomness - H2 No normalization, and H2 with c == 1, 3, 5, 7
- 15. tf * (avgFieldLength / docFieldLength)Z Divergence From Randomness - Z
- 16. Divergence From Randomness - Z No normalization, and Z with z == 0.1, 0.2, 0.3, 0.4
- 17. (tf * mu * ((totalTermFreq + 1) / (#?eldTokens + 1))) (docFieldLength + mu) * mu Divergence From Randomness - H3
- 18. Divergence From Randomness - H3 No normalization, and H3 with mu == 1, 3, 5, 7
- 19. DFR demo Only one, I promise
- 20. DFR Framework. Tunable: choose algorithm and tune parameters for both IDF* and docLength. * generic name for importance of this term
- 21. Divergence From Independence expected frequency
- 22. Divergence From Independence docLength*totalTermFrequency/numberOfFieldTokens expected frequency
- 23. DFI: Standardized (actual - expected)/sqrt(expected)
- 24. DFI demo Oh, but don¡¯t remove stopwords*! 1) arbitrarily chops ?eld length 2) stopwords aren¡¯t always stopwords ;)
- 25. DFI Simple. Parameterless. Flexible: works well with various datasets.
- 26. Information Based how much information we get from this term?
- 27. Information Based Distribution Log-Logistic, Smoothed Power-Law Lambda DF, TTF Normalization H1, H2, H3, Z, none
- 28. Information Based - Log-Logistic log( tfn / (lambda + 1) )
- 29. Information Based - Log-Logistic lambda: 0.1 (red), 0.3 (black), 0.8 (blue)
- 30. Information Based - Retrieval Function the average of the document information brought by each query term
- 31. Information Based - Retrieval Function - DF number of matching documents (docFrequency + 1) / (numberOfDocuments + 1)
- 32. Information Based - Retrieval Function - TTF total number of term occurrences (totalTermFrequency + 1) / (numberOfDocuments + 1)
- 33. IB demo
- 34. IB Framework. like DFR. Even has the same normalization options. But newer and, in the paper, better.
- 35. Language Models probability of a term being our term
- 36. Language Models totalTermFreq/totalFieldTokens probability of a term being our term
- 37. Language Models: Jelinek-Mercer log( (1-¦Ë)* tf ) docLength ¦Ë * probability
- 39. LM Two probabilistic models. Similar approach to DFI, but tunable.
- 40. Custom Similarity compute a similarity score using custom code
- 41. Custom Similarity - Activate Similarity Factory public class ActivateSimilarityFactory extends SimilarityFactory { private volatile Similarity similarity; public void init(SolrParams params) { super.init(params); } public Similarity getSimilarity() { if (similarity == null) { similarity = new ActivateSimilarity(); } return similarity; } }
- 42. Custom Similarity - Activate Similarity Factory public class ActivateSimilarityFactory extends SimilarityFactory { private volatile Similarity similarity; public void init(SolrParams params) { super.init(params); } public Similarity getSimilarity() { if (similarity == null) { similarity = new ActivateSimilarity(); } return similarity; } }
- 43. Custom Similarity - Activate Similarity Factory public class ActivateSimilarityFactory extends SimilarityFactory { private volatile Similarity similarity; public void init(SolrParams params) { super.init(params); } public Similarity getSimilarity() { if (similarity == null) { similarity = new ActivateSimilarity(); } return similarity; } }
- 44. Custom Similarity - Similarity public class ActivateSimilarity extends Similarity { public ActivateSimilarity() {} public long computeNorm(FieldInvertState state) { return 1; } public Similarity.SimScorer scorer(float boost, CollectionStatistics collectionStats, TermStatistics... termStats) { return new ActivateSimScorer(); } }
- 45. Custom Similarity - Similarity public class ActivateSimilarity extends Similarity { public ActivateSimilarity() {} public long computeNorm(FieldInvertState state) { return 1; } public Similarity.SimScorer scorer(float boost, CollectionStatistics collectionStats, TermStatistics... termStats) { return new ActivateSimScorer(); } }
- 46. Custom Similarity - Similarity public class ActivateSimilarity extends Similarity { public ActivateSimilarity() {} public long computeNorm(FieldInvertState state) { return 1; } public Similarity.SimScorer scorer(float boost, CollectionStatistics collectionStats, TermStatistics... termStats) { return new ActivateSimScorer(); } }
- 47. Custom Similarity - SimScorer public class ActivateSimScorer extends Similarity.SimScorer { public float score(float freq, long norm) { return freq; } }
- 48. Custom Similarity - SimScorer public class ActivateSimScorer extends Similarity.SimScorer { public float score(float freq, long norm) { return freq; } }
- 50. Custom When you need something special, like disregarding term frequency.
- 53. THANK YOU