pancx's blog

参考资料

• 构建IP池 https://github.com/xiaosimao/IP_POOL/blob/master/custom_get_ip/get_ip_from_peauland.py

创建Scrapy工程

scrapy startproject douban

编辑scrapy.cfg

[settings]
default = douban.settings

[deploy]
project = douban

创建main.py

from scrapy.cmdline import execute

import sys
import os

### 获取当前工程的路径
# os.path.abspath(): 获取路径的绝对路径
# os.path.dirname(): 获取路径的目录名
project_dir = os.path.dirname(os.path.abspath(__file__))

# 将工程目录添加到环境变量中
sys.path.append(project_dir)

# 通过execute执行系统命令，执行爬虫文件
execute(["scrapy", "crawl", "movie_subject"])

配置用户代理池middleware

pip install fake-useragent