8. Built-in middlewares
? Downloader middlewares
o DefaultHeadersMiddleware
o HttpAuthMiddleware
o HttpCacheMiddleware
o RedirectMiddleware
o RetryMiddleware
? Spider middlewares
o DepthMiddleware
o RefererMiddleware
? Scheduler middlewares
o DuplicatesFilterMiddleware
9. Extensions
? 特性
o Scrapy启动时加载的普通class
o 监听各种signal (engine_started, item_scraped,
item_dropped)
? Built-in extensions
o CoreStats
o WebConsole
o …
10. 从网页中提取数据
? CrawlSpider: Rule/Matcher/callback
? 使用XPath进行提取
? Scrapy shell
? Parsley: a selector language, superset of XPath and css3 (
内存泄露)
li.main>a/@href
12. Sphinx特性
? high indexing speed (upto 10 MB/sec on modern CPUs);
? high search speed (avg query is under 0.1 sec on 2-4 GB text collections);
? high scalability (upto 100 GB of text, upto 100 M documents on a single
CPU);
? provides good relevance ranking through combination of phrase proximity
ranking and statistical (BM25) ranking;
? provides distributed searching capabilities;
? provides document exceprts generation;
? provides searching from within MySQL through pluggable storage engine;
? supports boolean, phrase, and word proximity queries;
? supports multiple full-text fields per document (upto 32 by default);
? supports multiple additional attributes per document (ie. groups, timestamps,
etc);
? supports stopwords;
? supports both single-byte encodings and UTF-8;
? supports English stemming, Russian stemming, and Soundex for morphology;
? supports MySQL natively (MyISAM and InnoDB tables are both supported);
? supports PostgreSQL natively.
14. 索引
? 数据源: 数据库, xml, 等等。
o 表的每一行视为一篇文档,
o 可在配置中指定哪些列需要进行索引
? 属性：表的某些列可被指定为文档的属性，不被索引，但可
用来做过滤和排序
15. 索引(2)
索引配置的片段
sql_query = SELECT id, title, content,
author_id, forum_id, post_date FROM my_forum_posts
sql_attr_uint = author_id
sql_attr_uint = forum_id
sql_attr_timestamp = post_date
过滤和排序应用示例
// only search posts by author whose ID is 123
$cl->SetFilter ( "author_id", array ( 123 ) );
// only search posts in sub-forums 1, 3 and 7
$cl->SetFilter ( "forum_id", array ( 1,3,7 ) );
// sort found posts by posting date in descending order
$cl->SetSortMode ( SPH_SORT_ATTR_DESC, "post_date" );
16. 搜索 – 匹配模式
匹配模式
o SPH_MATCH_ALL
o SPH_MATCH_ANY
o SPH_MATCH_PHRASE
o SPH_MATCH_BOOLEAN
o SPH_MATCH_EXTENDED2
最灵活的SPH_MATCH_EXTENDED2
hello | world
hello | -world
@name hello @intro world
"hello world"
aaa << bbb << ccc
"hello world foo"~10
"the world is a wonderful place"/3
"hello world" @title "example program"~5 @body python -(php|perl) @* code
19. 搜索 – SphinxQL: 使用sql语法进行搜索
? searchd实现了mysql的网络协议
? 可将searchd当做mysql服务器使用，通过mysql client连接
SELECT *, @weight*10+docboost AS skey FROM example ORDER BY ske
SELECT * FROM test1 WHERE MATCH('"test doc"/3')
SELECT * FROM test WHERE MATCH('@title hello @body world') OPTION
ranker=bm25, max_matches=3000
