shopify burst crawler

Shopify-爆發-爬蟲

用於從 Shopify Burst 網站下載所有庫存照片的元資訊的簡單爬蟲：https://burst.shopify.com。支援2種抓取模式：

• BurstCrawler - 使用最新照片 URL 開始抓取
• BurstSitemapCrawler - 使用網站地圖 URL 開始抓取

用法

兩個爬蟲都支援相同的 API：

 BurstCrawler crawler = new BurstCrawler ();

// or you could use the Sitemap crawler
crawler = new BurstSitemapCrawler ();

// this will crawl and return the entire bunch of results
// it is slow and may take a lot of time
List < BurstImage > images = crawler . crawl ();

// a streaming version if also available
GenericConsumer < BurstImage > collector = new GenericConsumer < BurstImage >() {

	@ Override
	public boolean consume ( BurstImage image ) {
		// do something with the image
		// ...
		
		// return a true if you want to continue crawling
		// or return, a false to stop the crawling
		return true ;
	}
};
crawler . crawl ( collector );

下載

目前的開發快照JAR可以使用JitPack.io獲取，如下所示：

將以下repository新增至 Maven，

< repository >
	< id >jitpack.io</ id >
	< url >https://jitpack.io</ url >
</ repository >

然後添加依賴項，

< dependency >
    < groupId >com.github.sangupta</ groupId >
    < artifactId >shopify-burst-crawler</ artifactId >
    < version >1.0.0-SNAPSHOT</ version >
</ dependency >

版本控制

為了提高發布週期的透明度和洞察力，並努力保持向後相容性，該專案將盡可能在語義版本控制指南下進行維護。

版本將以以下格式編號：

<major>.<minor>.<patch>

並按照以下準則建構：

• 破壞向後相容性會導致主要問題
• 在不破壞向後相容性的情況下添加新內容會帶來輕微的影響
• 錯誤修復和其他變更會影響補丁

有關 SemVer 的更多信息，請訪問 http://semver.org/。

執照

 shopify-burst-crawler
Copyright (c) 2017-2019, Sandeep Gupta

https://sangupta.com/projects/shopify-burst-crawler

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.