Scrapy 项目部署问题及解决方案

原创

华科云商小徐 2024-08-13 09:58:37 ©著作权

文章标签 python 配置文件错误处理 文章分类 Python 后端开发

©著作权归作者所有：来自51CTO博客作者华科云商小徐的原创作品，请联系作者获取转载授权，否则将追究法律责任

部署 Scrapy 项目时可能会遇到一些常见问题。以下是几个常见的部署问题及其解决方案：

1、依赖问题

问题：部署后爬虫运行失败，通常是由于缺少依赖库。

2、配置问题

问题：爬虫在部署环境中无法正常运行，可能是由于配置文件错误或缺失。

Scrapy 项目部署问题及解决方案_配置文件

3、数据库连接问题

问题：爬虫运行时无法连接数据库。

4、存储和日志问题

问题：数据存储或日志记录出现问题。

5、定时任务和调度

问题：爬虫任务未按预期调度或运行。

6、错误处理

问题：爬虫在运行时遇到错误或异常。

7、资源限制

问题：爬虫运行时受限于内存或 CPU 资源。

一、问题背景

项目名称：dirbot
项目路径：/var/www/api/scrapy/dirbot
使用版本：Scrapy 0.16.3
Scrapyd 部署：aptitude install scrapyd-0.16

用户在项目目录中使用 scrapy deploy 命令尝试部署项目时遇到以下错误：

/usr/local/lib/python2.7/dist-packages/Scrapy-0.16.3-py2.7.egg/scrapy/settings/deprecated.py:23: ScrapyDeprecationWarning: You are using the following settings which are deprecated or obsolete (ask scrapy-users@googlegroups.com for alternatives):
    BOT_VERSION: no longer used (user agent defaults to Scrapy now)
  warnings.warn(msg, ScrapyDeprecationWarning)
Traceback (most recent call last):
  File "/usr/local/bin/scrapy", line 5, in <module>
    pkg_resources.run_script('Scrapy==0.16.3', 'scrapy')
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 499, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1235, in run_script
    execfile(script_filename, namespace, namespace)
  File "/usr/local/lib/python2.7/dist-packages/Scrapy-0.16.3-py2.7.egg/EGG-INFO/scripts/scrapy", line 4, in <module>
    execute()
  File "/usr/local/lib/python2.7/dist-packages/Scrapy-0.16.3-py2.7.egg/scrapy/cmdline.py", line 131, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "/usr/local/lib/python2.7/dist-packages/Scrapy-0.16.3-py2.7.egg/scrapy/cmdline.py", line 76, in _run_print_help
    func(*a, **kw)
  File "/usr/local/lib/python2.7/dist-packages/Scrapy-0.16.3-py2.7.egg/scrapy/cmdline.py", line 138, in _run_command
    cmd.run(args, opts)
  File "/usr/local/lib/python2.7/dist-packages/Scrapy-0.16.3-py2.7.egg/scrapy/commands/deploy.py", line 76, in run
    f = urllib2.urlopen(req)
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 406, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 444, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not Found

同时，在使用 scrapy deploy -L scrapyd2 命令列出部署项目时也遇到了同样的错误。

二、解决方案