Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/sangupta/shopify-burst-crawler
Simple crawler to download meta information for all stock pics from Shopify Burst website
https://github.com/sangupta/shopify-burst-crawler
burst crawler java shopify stock-photos
Last synced: 3 months ago
JSON representation
Simple crawler to download meta information for all stock pics from Shopify Burst website
- Host: GitHub
- URL: https://github.com/sangupta/shopify-burst-crawler
- Owner: sangupta
- License: apache-2.0
- Created: 2017-11-02T06:03:18.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2022-09-01T22:34:46.000Z (over 2 years ago)
- Last Synced: 2024-04-16T11:17:29.253Z (9 months ago)
- Topics: burst, crawler, java, shopify, stock-photos
- Language: Java
- Size: 47.9 KB
- Stars: 2
- Watchers: 3
- Forks: 2
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
README
# shopify-burst-crawler
Simple crawlers to download meta information for all stock photos from Shopify Burst
website: https://burst.shopify.com. There are 2 crawling modes supported:
* BurstCrawler - starts crawling using the latest photos URL
* BurstSitemapCrawler - starts crawling using the sitemap URL
## Usage
Both crawlers support same API:
```java
BurstCrawler crawler = new BurstCrawler();
// or you could use the Sitemap crawler
crawler = new BurstSitemapCrawler();
// this will crawl and return the entire bunch of results
// it is slow and may take a lot of time
List images = crawler.crawl();
// a streaming version if also available
GenericConsumer collector = new GenericConsumer() {
@Override
public boolean consume(BurstImage image) {
// do something with the image
// ...
// return a true if you want to continue crawling
// or return, a false to stop the crawling
return true;
}
};
crawler.crawl(collector);
```
## Downloads
The current development snapshot `JAR` can be obtained using `JitPack.io` as:
Add the following `repository` to Maven,
```xml
jitpack.io
https://jitpack.io
```
Then add the dependency as,
```xml
com.github.sangupta
shopify-burst-crawler
1.0.0-SNAPSHOT
```
## Versioning
For transparency and insight into our release cycle, and for striving to maintain backward compatibility,
this project will be maintained under the Semantic Versioning guidelines as much as possible.
Releases will be numbered with the follow format:
`..`
And constructed with the following guidelines:
* Breaking backward compatibility bumps the major
* New additions without breaking backward compatibility bumps the minor
* Bug fixes and misc changes bump the patch
For more information on SemVer, please visit http://semver.org/.
## License
```
shopify-burst-crawler
Copyright (c) 2017-2019, Sandeep Gupta
https://sangupta.com/projects/shopify-burst-crawler
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
```