scrapy 中文的资料挺少的，写文章记录一下，以爬cnbeta新闻为例子，
抓取cnbeta的新闻标题+链接。

1.新建scrapy项目

scrapy startproject cnbeta

目录结构：

1
2
3
4
5
6
7
8
9
cnbeta/
├──cnbeta
│  ├──__init__.py
│  ├──items.py
│  ├──pipelines.py
│  ├──settings.py
│  └──spiders
│  └──__init__.py
└──scrapy.cfg

2.定义数据结构
编辑cnbeta/items.py

1
2
3
4
5
6
7
8
9
10
11
12
# Define here the models for your scraped items
#
# See documentation in:
# http://doc.scrapy.org/en/latest/topics/items.html
fromscrapy.item importItem,Field
classCnbetaItem(Item):
# define the fields for your item here like:
# name = Field()
title=Field()
url=Field()

定义了两个字段，分别存储标题和链接

3.编写spider（爬虫）
编辑cnbeta/spiders/cb.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
from scrapy.contrib.spiders import CrawlSpider,Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import Selector
from cnbeta.items import CnbetaItem
classCBSpider(CrawlSpider):
name='cnbeta'
allowed_domains=['cnbeta.com']
start_urls=['http://www.cnbeta.com']
rules=(
Rule(SgmlLinkExtractor(allow=('/articles/.*\.htm',)),
callback='parse_page',follow=True),
)
def parse_page(self,response):
item=CnbetaItem()
sel=Selector(response)
item['title']=sel.xpath('//title/text()').extract()
item['url']=response.url
returnitem

rules指定了含有/articles/.*\.htm的链接都会被匹配.

4.运行爬虫

1
scrapy crawl cnbeta-oresult.json-tjson

将结果输出到result.json -t json指定文件格式为json

5.结果

[{"url":"http://www.cnbeta.com/articles/268661.htm","title":["\u53ea\u97001\u5143\u4e0d\u62a2\u624d\u75af\uff01\u5c0f\u5ea6Wi-Fi\u5957\u88c5\u9707\u64bc\u4ef716\u65e5\u5f00\u62a2_Baidu \u767e\u5ea6_cnBeta.COM"]},
{"url":"http://www.cnbeta.com/articles/268872.htm","title":["\u8c37\u6b4c\u667a\u80fd\u5bb6\u5c45\u8ba1\u5212\u6216\u72af\u7684\u9519\uff1a\u5c01\u95edNest_Google / \u8c37\u6b4c_cnBeta.COM"]},
{"url":"http://www.cnbeta.com/articles/268865.htm","title":["\u4e2d\u56fd\u624b\u673a\u7f51\u6c11\u89c4\u6a21\u8fbe5\u4ebf \u5e74\u589e\u957f8009\u4e07\u4eba_cnBeta \u89c6\u70b9\u89c2\u5bdf_cnBeta.COM"]},
{"url":"http://www.cnbeta.com/articles/268869.htm","title":["\u524d\u82f9\u679c\u9ad8\u7ea7\u526f\u603b\u88c1\u4e3a\u5927\u5b66\u4ee3\u8a00\uff1f\u539f\u662f\u56fe\u7247\u88ab\u76d7\u7528_cnBeta \u4eba\u7269_cnBeta.COM"]},
{"url":"http://www.cnbeta.com/articles/268866.htm","title":["\u6bd4\u7279\u5e01\u4ea4\u6613\u5e73\u53f0\u906d\u9047\u751f\u5b58\u5371\u673a_cnBeta \u89c6\u70b9\u89c2\u5bdf_cnBeta.COM"]},
{"url":"http://www.cnbeta.com/articles/268870.htm","title":["\u76db\u5927\u6e38\u620f\u4f20\u5947\u88ab\u4fb5\u6743\u6848\u65b0\u8fdb\u5c55\uff1a\u8ffd\u52a0\u56db\u540d\u88ab\u544a_cnBeta \u89c6\u70b9\u89c2\u5bdf_cnBeta.COM"]},
{"url":"http://www.cnbeta.com/articles/268867.htm","title":["\u91d1\u878d\u65f6\u62a5\uff1a\u4e2d\u56fd\u8d2b\u56f0\u5730\u533a\u7f51\u8d2d\u589e\u901f\u8d85\u53d1\u8fbe\u5730\u533a_\u7535\u5b50\u5546\u52a1 - B2C / B2B_cnBeta.COM"]},
{"url":"http://www.cnbeta.com/articles/268868.htm","title":["\u79fb\u52a84G\u7248iPhone 5s/5c\u4e0a\u624b\uff1a\u901f\u5ea6\u6539\u53d8\u4f53\u9a8c_Apple iPhone_cnBeta.COM"]},
{"url":"http://www.cnbeta.com/articles/268871.htm","title":["[\u7ec4\u56fe]\u7247\u573a\u63a2\u79d8\u544a\u8bc9\u4f60\u7d22\u5c3c4K\u5f71\u7247\u662f\u600e\u4e48\u70bc\u6210\u7684_SONY \u7d22\u5c3c_cnBeta.COM"]},
{"url":"http://www.cnbeta.com/articles/268875.htm","title":["USB 3.0\u548c\u534

×××：
cnbeta_spider.tar