scrapingbee python
v2.0.1:
CrapingBee是一种网络刮擦API，可处理无头浏览器并为您旋转代理。 Python SDK使与CrapingBee的API互动变得更加容易。
您可以使用PIP安装Crapingbee Python SDK。
pip install scrapingbeeCrapingbee Python SDK是围绕请求库的包装器。抓犬支持获取和发布请求。
注册以获取您的API密钥和一些免费的积分以开始使用。
>> > from scrapingbee import ScrapingBeeClient
>> > client = ScrapingBeeClient ( api_key = 'REPLACE-WITH-YOUR-API-KEY' )
>> > response = client . get (
'https://www.scrapingbee.com/blog/' ,
params = {
# Block ads on the page you want to scrape
'block_ads' : False ,
# Block images and CSS on the page you want to scrape
'block_resources' : True ,
# Premium proxy geolocation
'country_code' : '' ,
# Control the device the request will be sent from
'device' : 'desktop' ,
# Use some data extraction rules
'extract_rules' : { 'title' : 'h1' },
# Wrap response in JSON
'json_response' : False ,
# Interact with the webpage you want to scrape
'js_scenario' : {
"instructions" : [
{ "wait_for" : "#slow_button" },
{ "click" : "#slow_button" },
{ "scroll_x" : 1000 },
{ "wait" : 1000 },
{ "scroll_x" : 1000 },
{ "wait" : 1000 },
]
},
# Use premium proxies to bypass difficult to scrape websites (10-25 credits/request)
'premium_proxy' : False ,
# Execute JavaScript code with a Headless Browser (5 credits/request)
'render_js' : True ,
# Return the original HTML before the JavaScript rendering
'return_page_source' : False ,
# Return page screenshot as a png image
'screenshot' : False ,
# Take a full page screenshot without the window limitation
'screenshot_full_page' : False ,
# Transparently return the same HTTP code of the page requested.
'transparent_status_code' : False ,
# Wait, in miliseconds, before returning the response
'wait' : 0 ,
# Wait for CSS selector before returning the response, ex ".title"
'wait_for' : '' ,
# Set the browser window width in pixel
'window_width' : 1920 ,
# Set the browser window height in pixel
'window_height' : 1080
},
headers = {
# Forward custom headers to the target website
"key" : "value"
},
cookies = {
# Forward custom cookies to the target website
"name" : "value"
}
)
>> > response . text
'<!DOCTYPE html><html lang="en"><head>...'CrapingBee需要各种参数来渲染JavaScript，执行自定义JavaScript脚本，使用特定地理位置的高级代理等。
您可以在CrapingBee文档中找到所有受支持的参数。
您可以像通常使用请求库一样发送自定义的cookie和标头。
在这里，如何在其移动分辨率中从CrapingBee博客中检索和存储屏幕截图有所了解。
>> > from scrapingbee import ScrapingBeeClient
>> > client = ScrapingBeeClient ( api_key = 'REPLACE-WITH-YOUR-API-KEY' )
>> > response = client . get (
'https://www.scrapingbee.com/blog/' ,
params = {
# Take a screenshot
'screenshot' : True ,
# Specify that we need the full height
'screenshot_full_page' : True ,
# Specify a mobile width in pixel
'window_width' : 375
}
)
>> > if response . ok :
with open ( "./scrapingbee_mobile.png" , "wb" ) as f :
f . write ( response . content )砂纸是最受欢迎的Python Web刮擦框架。您可以轻松地将Crapingbee的API与零食中间件集成在一起。
客户包括用于5xx响应的重试机制。
>> > from scrapingbee import ScrapingBeeClient
>> > client = ScrapingBeeClient ( api_key = 'REPLACE-WITH-YOUR-API-KEY' )
>> > response = client . get (
'https://www.scrapingbee.com/blog/' ,
params = {
'render_js' : True ,
},
retries = 5
)
