python爬蟲中BaseSpider指的是什么-創(chuàng)新互聯(lián)
這篇文章主要介紹了python爬蟲中Base Spider指的是什么，具有一定借鑒價值，需要的朋友可以參考下。希望大家閱讀完這篇文章后大有收獲。下面讓小編帶著大家一起了解一下。
當(dāng)一個事物的個數(shù)很多的時候，我們會把它們歸為一個群，比如人群之類的稱呼。那對于python中的爬蟲來說，我們想要獲取數(shù)據(jù)，就需要大量的爬蟲去支撐，不過這里我們肯定不能也給它們起一個群的稱呼。同樣我們會用類來表示，這就是我們今天要講Base Spider，感興趣的小伙伴也來看看這個爬蟲中的類究竟是什么吧。
Base Spider
不在使用scrapy原有的Spider類，重寫的RedisSpider繼承了Spider和RedisMixin這兩個類，RedisMixin是用來從redis讀取url的類。
當(dāng)我們生成一個Spider繼承RedisSpider時，調(diào)用setup_redis函數(shù)，這個函數(shù)會去連接redis數(shù)據(jù)庫，然后會設(shè)置signals(信號)：
一個是當(dāng)spider空閑時候的signal，會調(diào)用spider_idle函數(shù)，這個函數(shù)調(diào)用schedule_next_request函數(shù)，保證spider是一直活著的狀態(tài)，并且拋出DontCloseSpider異常。
一個是當(dāng)抓到一個item時的signal，會調(diào)用item_scraped函數(shù)，這個函數(shù)會調(diào)用schedule_next_request函數(shù)，獲取下一個request。
from scrapy import signals
from scrapy.exceptions import DontCloseSpider
from scrapy.spiders import Spider, CrawlSpider
from . import connection, defaults
from .utils import bytes_to_str
class RedisMixin(object):
"""Mixin class to implement reading urls from a redis queue."""
redis_key = None
redis_batch_size = None
redis_encoding = None
# Redis client placeholder.
server = None
def start_requests(self):
"""Returns a batch of start requests from redis."""
return self.next_requests()
def setup_redis(self, crawler=None):
"""Setup redis connection and idle signal.
This should be called after the spider has set its crawler object.
"""
if self.server is not None:
return
if crawler is None:
# We allow optional crawler argument to keep backwards
# compatibility.
# XXX: Raise a deprecation warning.
crawler = getattr(self, 'crawler', None)
if crawler is None:
raise ValueError("crawler is required")
settings = crawler.settings
if self.redis_key is None:
self.redis_key = settings.get(
'REDIS_START_URLS_KEY', defaults.START_URLS_KEY,
)
self.redis_key = self.redis_key % {'name': self.name}
if not self.redis_key.strip():
raise ValueError("redis_key must not be empty")
if self.redis_batch_size is None:
# TODO: Deprecate this setting (REDIS_START_URLS_BATCH_SIZE).
self.redis_batch_size = settings.getint(
'REDIS_START_URLS_BATCH_SIZE',
settings.getint('CONCURRENT_REQUESTS'),
)
try:
self.redis_batch_size = int(self.redis_batch_size)
except (TypeError, ValueError):
raise ValueError("redis_batch_size must be an integer")
if self.redis_encoding is None:
self.redis_encoding = settings.get('REDIS_ENCODING', defaults.REDIS_ENCODING)
self.logger.info("Reading start URLs from redis key '%(redis_key)s' "
"(batch size: %(redis_batch_size)s, encoding: %(redis_encoding)s",
self.__dict__)
self.server = connection.from_settings(crawler.settings)
# The idle signal is called when the spider has no requests left,
# that's when we will schedule new requests from redis queue
crawler.signals.connect(self.spider_idle, signal=signals.spider_idle)
def next_requests(self):
"""Returns a request to be scheduled or none."""
use_set = self.settings.getbool('REDIS_START_URLS_AS_SET', defaults.START_URLS_AS_SET)
fetch_one = self.server.spop if use_set else self.server.lpop
# XXX: Do we need to use a timeout here?
found = 0
# TODO: Use redis pipeline execution.
while found < self.redis_batch_size:
data = fetch_one(self.redis_key)
if not data:
# Queue empty.
break
req = self.make_request_from_data(data)
if req:
yield req
found += 1
else:
self.logger.debug("Request not made from data: %r", data)
if found:
self.logger.debug("Read %s requests from '%s'", found, self.redis_key)
def make_request_from_data(self, data):
"""Returns a Request instance from data coming from Redis.
By default, ``data`` is an encoded URL. You can override this method to
provide your own message decoding.
Parameters
----------
data : bytes
Message from redis.
"""
url = bytes_to_str(data, self.redis_encoding)
return self.make_requests_from_url(url)
def schedule_next_requests(self):
"""Schedules a request if available"""
# TODO: While there is capacity, schedule a batch of redis requests.
for req in self.next_requests():
self.crawler.engine.crawl(req, spider=self)
def spider_idle(self):
"""Schedules a request if available, otherwise waits."""
# XXX: Handle a sentinel to close the spider.
self.schedule_next_requests()
raise DontCloseSpider
class RedisSpider(RedisMixin, Spider):
"""Spider that reads urls from redis queue when idle.
Attributes
----------
redis_key : str (default: REDIS_START_URLS_KEY)
Redis key where to fetch start URLs from..
redis_batch_size : int (default: CONCURRENT_REQUESTS)
Number of messages to fetch from redis on each attempt.
redis_encoding : str (default: REDIS_ENCODING)
Encoding to use when decoding messages from redis queue.
Settings
--------
REDIS_START_URLS_KEY : str (default: "<spider.name>:start_urls")
Default Redis key where to fetch start URLs from..
REDIS_START_URLS_BATCH_SIZE : int (deprecated by CONCURRENT_REQUESTS)
Default number of messages to fetch from redis on each attempt.
REDIS_START_URLS_AS_SET : bool (default: False)
Use SET operations to retrieve messages from the redis queue. If False,
the messages are retrieve using the LPOP command.
REDIS_ENCODING : str (default: "utf-8")
Default encoding to use when decoding messages from redis queue.
"""
@classmethod
def from_crawler(self, crawler, *args, **kwargs):
obj = super(RedisSpider, self).from_crawler(crawler, *args, **kwargs)
obj.setup_redis(crawler)
return obj
class RedisCrawlSpider(RedisMixin, CrawlSpider):
"""Spider that reads urls from redis queue when idle.
Attributes
----------
redis_key : str (default: REDIS_START_URLS_KEY)
Redis key where to fetch start URLs from..
redis_batch_size : int (default: CONCURRENT_REQUESTS)
Number of messages to fetch from redis on each attempt.
redis_encoding : str (default: REDIS_ENCODING)
Encoding to use when decoding messages from redis queue.
Settings
--------
REDIS_START_URLS_KEY : str (default: "<spider.name>:start_urls")
Default Redis key where to fetch start URLs from..
REDIS_START_URLS_BATCH_SIZE : int (deprecated by CONCURRENT_REQUESTS)
Default number of messages to fetch from redis on each attempt.
REDIS_START_URLS_AS_SET : bool (default: True)
Use SET operations to retrieve messages from the redis queue.
REDIS_ENCODING : str (default: "utf-8")
Default encoding to use when decoding messages from redis queue.
"""
@classmethod
def from_crawler(self, crawler, *args, **kwargs):
obj = super(RedisCrawlSpider, self).from_crawler(crawler, *args, **kwargs)
obj.setup_redis(crawler)
return obj感謝你能夠認(rèn)真閱讀完這篇文章，希望小編分享python爬蟲中Base Spider指的是什么內(nèi)容對大家有幫助，同時也希望大家多多支持創(chuàng)新互聯(lián)網(wǎng)站建設(shè)公司，，關(guān)注創(chuàng)新互聯(lián)行業(yè)資訊頻道，遇到問題就找創(chuàng)新互聯(lián)網(wǎng)站建設(shè)公司，，詳細(xì)的解決方法等著你來學(xué)習(xí)!
本文題目：python爬蟲中BaseSpider指的是什么-創(chuàng)新互聯(lián)
本文來源：http://muchs.cn/article4/cochie.html
成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián)，為您提供微信公眾號、網(wǎng)站制作、定制網(wǎng)站、App設(shè)計、網(wǎng)站導(dǎo)航、移動網(wǎng)站建設(shè)
聲明：本網(wǎng)站發(fā)布的內(nèi)容（圖片、視頻和文字）以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主，如果涉及侵權(quán)請盡快告知，我們將會在第一時間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場，如需處理請聯(lián)系客服。電話：028-86922220；郵箱：631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載，或轉(zhuǎn)載時需注明來源： 創(chuàng)新互聯(lián)
猜你還喜歡下面的內(nèi)容
- ASP.NETMVC過濾器(五)-創(chuàng)新互聯(lián)
- 怎么修kingsql中RAC節(jié)點(diǎn)故障-創(chuàng)新互聯(lián)
- Linux系統(tǒng)怎么調(diào)整swap大小-創(chuàng)新互聯(lián)
- 微信公眾號點(diǎn)亮圖片的方法-創(chuàng)新互聯(lián)
- 怎么深入理解Flutter上的數(shù)據(jù)監(jiān)控-創(chuàng)新互聯(lián)
- 如何解決vue組件中使用v-for出現(xiàn)告警問題-創(chuàng)新互聯(lián)
- java中接口的定義和實(shí)現(xiàn)方法-創(chuàng)新互聯(lián)
- 人工智能的發(fā)展會不會全面取代軟件開發(fā)崗位 2016-08-22
- 微信小程序憑什么這么火？ 2016-08-09
- 企業(yè)需要做PC端的網(wǎng)站，微網(wǎng)站、app、手機(jī)網(wǎng)站。 2016-10-11
- 網(wǎng)站維護(hù)是網(wǎng)站設(shè)計建成后必須重視的問題 2013-05-30
- 網(wǎng)站維護(hù)要怎么做？ 2016-09-20
- 成都小程序開發(fā)：如何提高小程序的轉(zhuǎn)化率？ 2013-11-11
- 網(wǎng)站建設(shè)網(wǎng)站維護(hù)建設(shè)贏在細(xì)節(jié)、細(xì)心 2013-07-06
- 如何維護(hù)成都網(wǎng)頁設(shè)計公司的服務(wù)器？ 2016-10-27
- 網(wǎng)站維護(hù)需要做什么？ 2016-09-25
- 公司網(wǎng)站維護(hù)的流程詳解 2016-11-12
- 什么樣的公司可以幫助企業(yè)進(jìn)行網(wǎng)站制作 2014-07-13
- 網(wǎng)站維護(hù)的美工需要具備哪些技能？ 2018-11-14