Python爬蟲(chóng)IP代理池的實(shí)現(xiàn)-創(chuàng)新互聯(lián)

Python 爬蟲(chóng)IP代理池的實(shí)現(xiàn)

創(chuàng)新互聯(lián)建站主要從事網(wǎng)站建設(shè)、成都網(wǎng)站制作、網(wǎng)頁(yè)設(shè)計(jì)、企業(yè)做網(wǎng)站、公司建網(wǎng)站等業(yè)務(wù)。立足成都服務(wù)安康,十載網(wǎng)站建設(shè)經(jīng)驗(yàn),價(jià)格優(yōu)惠、服務(wù)專(zhuān)業(yè),歡迎來(lái)電咨詢(xún)建站服務(wù):18980820575

很多時(shí)候，如果要多線程的爬取網(wǎng)頁(yè)，或者是單純的反爬，我們需要通過(guò)代理 IP來(lái)進(jìn)行訪問(wèn)。下面看看一個(gè)基本的實(shí)現(xiàn)方法。

代理 IP Python 爬蟲(chóng)IP代理池的實(shí)現(xiàn) 附件.txt 的提取，網(wǎng)上有很多網(wǎng)站都提供這個(gè)服務(wù)?；旧峡煽啃院豌y子是成正比的。國(guó)內(nèi)提供的免費(fèi)IP基本上都是沒(méi)法用的，如果要可靠的代理只能付費(fèi)；國(guó)外稍微好些，有些免費(fèi)IP還是比較靠譜的。

網(wǎng)上隨便搜索了一下，找了個(gè)網(wǎng)頁(yè)，本來(lái)還想手動(dòng)爬一些對(duì)應(yīng)的 IP，結(jié)果發(fā)現(xiàn)可以直接下載現(xiàn)成的txt文件

下載之后，試試看用不同的代理去爬百度首頁(yè)

#！/usr/bin/env python#! -*- coding:utf-8 -*-# Author: Yuan Liimport re,urllib.requestfp=open("c:\\temp\\thebigproxylist-17-12-20.txt",'r')lines=fp.readlines()for ip in lines: try: print("當(dāng)前代理IP "+ip) proxy=urllib.request.ProxyHandler({"http":ip}) opener=urllib.request.build_opener(proxy,urllib.request.HTTPHandler) urllib.request.install_opener(opener) url="http://www.baidu.com" data=urllib.request.urlopen(url).read().decode('utf-8','ignore') print("通過(guò)") print("-----------------------------") except Exception as err: print(err) print("-----------------------------")fp.close()

結(jié)果如下：

C:\Python36\python.exe C:/Users/yuan.li/Documents/GitHub/Python/Misc/爬蟲(chóng)/proxy.py當(dāng)前代理IP 137.74.168.174:80通過(guò)-----------------------------當(dāng)前代理IP 103.28.161.68:8080通過(guò)-----------------------------當(dāng)前代理IP 91.151.106.127:53281HTTP Error 503: Service Unavailable-----------------------------當(dāng)前代理IP 177.136.252.7:3128<urlopen error [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>-----------------------------當(dāng)前代理IP 47.89.22.200:80通過(guò)-----------------------------當(dāng)前代理IP 118.69.61.57:8888HTTP Error 503: Service Unavailable-----------------------------當(dāng)前代理IP 192.241.190.167:8080通過(guò)-----------------------------當(dāng)前代理IP 185.124.112.130:80通過(guò)-----------------------------當(dāng)前代理IP 83.65.246.181:3128通過(guò)-----------------------------當(dāng)前代理IP 79.137.42.124:3128通過(guò)-----------------------------當(dāng)前代理IP 95.0.217.32:8080<urlopen error [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>-----------------------------當(dāng)前代理IP 104.131.94.221:8080通過(guò)

不過(guò)上面這種方式只適合比較穩(wěn)定的 IP源，如果IP不穩(wěn)定的話，可能很快對(duì)應(yīng)的文本就失效了，最好可以動(dòng)態(tài)地去獲取最新的IP地址。很多網(wǎng)站都提供API可以實(shí)時(shí)地去查詢(xún)
還是用剛才的網(wǎng)站，這次我們用 API去調(diào)用，這里需要瀏覽器偽裝一下才能爬取

#！/usr/bin/env python#! -*- coding:utf-8 -*-# Author: Yuan Liimport re,urllib.requestheaders=("User-Agent","Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.22 Safari/537.36 SE 2.X MetaSr 1.0")opener=urllib.request.build_opener()opener.addheaders=[headers]#安裝為全局urllib.request.install_opener(opener)data=urllib.request.urlopen("http://www.thebigproxylist.com/members/proxy-api.php?output=all&user=list&pass=8a544b2637e7a45d1536e34680e11adf").read().decode('utf8')ippool=data.split('\n')for ip in ippool: ip=ip.split(',')[0] try: print("當(dāng)前代理IP "+ip) proxy=urllib.request.ProxyHandler({"http":ip}) opener=urllib.request.build_opener(proxy,urllib.request.HTTPHandler) urllib.request.install_opener(opener) url="http://www.baidu.com" data=urllib.request.urlopen(url).read().decode('utf-8','ignore') print("通過(guò)") print("-----------------------------") except Exception as err: print(err) print("-----------------------------")fp.close()

結(jié)果如下：

C:\Python36\python.exe C:/Users/yuan.li/Documents/GitHub/Python/Misc/爬蟲(chóng)/proxy.py當(dāng)前代理IP 213.233.57.134:80HTTP Error 403: Forbidden-----------------------------當(dāng)前代理IP 144.76.81.79:3128通過(guò)-----------------------------當(dāng)前代理IP 45.55.132.29:53281HTTP Error 503: Service Unavailable-----------------------------當(dāng)前代理IP 180.254.133.124:8080通過(guò)-----------------------------當(dāng)前代理IP 5.196.215.231:3128HTTP Error 503: Service Unavailable-----------------------------當(dāng)前代理IP 177.99.175.195:53281HTTP Error 503: Service Unavailable

因?yàn)橹苯? for循環(huán)來(lái)按順序讀取文本實(shí)在是太慢了，我試著改成多線程來(lái)讀取，這樣速度就快多了

#！/usr/bin/env python#! -*- coding:utf-8 -*-# Author: Yuan Liimport threadingimport queueimport re,urllib.request#Number of threadsn_thread = 10#Create queuequeue = queue.Queue()class ThreadClass(threading.Thread): def __init__(self, queue): threading.Thread.__init__(self)

super(ThreadClass, self).__init__() #Assign thread working with queue self.queue = queue def run(self): while True: #Get from queue job host = self.queue.get()

print (self.getName() + ":" + host) try: # print("當(dāng)前代理IP " + host) proxy = urllib.request.ProxyHandler({"http": host}) opener = urllib.request.build_opener(proxy, urllib.request.HTTPHandler) urllib.request.install_opener(opener) url = "http://www.baidu.com" data = urllib.request.urlopen(url).read().decode('utf-8', 'ignore') print("通過(guò)") print("-----------------------------")

except Exception as err: print(err)

print("-----------------------------") #signals to queue jo

文章標(biāo)題：Python爬蟲(chóng)IP代理池的實(shí)現(xiàn)-創(chuàng)新互聯(lián)
文章來(lái)源：http://muchs.cn/article18/codpgp.html

成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián)，為您提供網(wǎng)站改版、建站公司、網(wǎng)站制作、網(wǎng)站設(shè)計(jì)公司、品牌網(wǎng)站建設(shè)、企業(yè)建站

聲明：本網(wǎng)站發(fā)布的內(nèi)容（圖片、視頻和文字）以用戶(hù)投稿、用戶(hù)轉(zhuǎn)載內(nèi)容為主，如果涉及侵權(quán)請(qǐng)盡快告知，我們將會(huì)在第一時(shí)間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng)，如需處理請(qǐng)聯(lián)系客服。電話：028-86922220；郵箱：631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載，或轉(zhuǎn)載時(shí)需注明來(lái)源：創(chuàng)新互聯(lián)

猜你還喜歡下面的內(nèi)容