解决：

将settings.py文件中的

ROBOTSTXT_OBEY = False

修改为

ROBOTSTXT_OBEY = True

详细：

settings.py文件中的

ROBOTSTXT_OBEY = True

在terminal输入

>scrapy shell "https://www.baidu.com/"

[scrapy.core.engine] DEBUG: Crawled (200)

robots.txt

> (referer: None)

[scrapy.downloadermiddlewares.robotstxt] DEBUG:

Forbidden by robots.txt

:

由上面可知

Forbidden by robots.txt

，被禁止了

修改settings.py为

ROBOTSTXT_OBEY = False

时，

[scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (meta refresh) to

4b&vit=fps?from=844b&vit=fps&index=&ssid=0&bd_page_type=1&logid=7889072453155394709&pu=sz%401321_480&t_noscript=jump> from

DEBUG: Crawled (200)

&ssid=0&bd_page_type=1&logid=7889072453155394709&pu=sz%401321_480&t_noscript=jump> (referer: None)

此时成功！！