shopify burst crawler

Shopify-爆发-爬虫

用于从 Shopify Burst 网站下载所有库存照片的元信息的简单爬虫：https://burst.shopify.com。支持2种抓取模式：

• BurstCrawler - 开始使用最新照片 URL 进行爬行
• BurstSitemapCrawler - 使用站点地图 URL 开始抓取

用法

两个爬虫都支持相同的 API：

 BurstCrawler crawler = new BurstCrawler ();

// or you could use the Sitemap crawler
crawler = new BurstSitemapCrawler ();

// this will crawl and return the entire bunch of results
// it is slow and may take a lot of time
List < BurstImage > images = crawler . crawl ();

// a streaming version if also available
GenericConsumer < BurstImage > collector = new GenericConsumer < BurstImage >() {

	@ Override
	public boolean consume ( BurstImage image ) {
		// do something with the image
		// ...
		
		// return a true if you want to continue crawling
		// or return, a false to stop the crawling
		return true ;
	}
};
crawler . crawl ( collector );

下载

当前的开发快照JAR可以使用JitPack.io获取，如下所示：

将以下repository添加到 Maven，

< repository >
	< id >jitpack.io</ id >
	< url >https://jitpack.io</ url >
</ repository >

然后添加依赖项，

< dependency >
    < groupId >com.github.sangupta</ groupId >
    < artifactId >shopify-burst-crawler</ artifactId >
    < version >1.0.0-SNAPSHOT</ version >
</ dependency >

版本控制

为了提高发布周期的透明度和洞察力，并努力保持向后兼容性，该项目将尽可能在语义版本控制指南下进行维护。

版本将按以下格式编号：

<major>.<minor>.<patch>

并按照以下准则构建：

• 破坏向后兼容性会导致主要问题
• 在不破坏向后兼容性的情况下添加新内容会带来轻微的影响
• 错误修复和其他更改会影响补丁

有关 SemVer 的更多信息，请访问 http://semver.org/。

执照

 shopify-burst-crawler
Copyright (c) 2017-2019, Sandeep Gupta

https://sangupta.com/projects/shopify-burst-crawler

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.