1、

設(shè)定瀏覽器列表。

首先需要定義各種不同類型的瀏覽器，統(tǒng)一保存于一個列表中。我們將其定義在settings.py中

2、

MY_USER_AGENT = [

"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; AcooBrowser; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",\

"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Acoo Browser; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)",\

“Mozilla/4.0 (compatible; MSIE 7.0; AOL 9.5; AOLBuild 4337.35; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)”,\.......]

3、

將settings.py中設(shè)置固定瀏覽器代碼刪除或注釋掉。

# Crawl responsibly by identifying yourself (and your website) on the user-agent

USER_AGENT = "Mozilla/5.0 (Windows NT 10.0;Win64; x64) " \

"AppleWebKit/537.36 (KHTML, like Gecko) " \

"Chrome/68.0.3440.106 Safari/537.36"