https://github.com/tirkarthi/pom-parse-deps
A simple script that parses pom files as JSON and inserts them into MongoDB
https://github.com/tirkarthi/pom-parse-deps
Last synced: 4 months ago
JSON representation
A simple script that parses pom files as JSON and inserts them into MongoDB
- Host: GitHub
- URL: https://github.com/tirkarthi/pom-parse-deps
- Owner: tirkarthi
- License: mit
- Created: 2018-03-18T05:31:21.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2018-03-18T05:33:20.000Z (about 7 years ago)
- Last Synced: 2025-01-05T12:42:36.002Z (5 months ago)
- Language: Clojure
- Size: 2.93 KB
- Stars: 0
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# clojars scraper
A simple script that parses pom files as JSON and inserts them into MongoDB
## Requirements
* A MongoDB installation (By default it listens on localhost and no password is required)
* Download all the POM files with `rsync -av --delete clojars.org::clojars my-wonderful-copy-of-clojars --include="*/" --include="*.pom" --exclude="*"`. The script assumes this directory is present at `/home/ubuntu/` . It downloads around 1GB of data.
* Latest clojure deps tool. Refer https://clojure.org/guides/getting_started
## Installation
* Clone the repo
* Run `clj -m scraper`
* Use can use MongoDB shell to query data. Sample structure as below :
```
➜ ~ mongo
> use lein
switched to db lein
> db.clojars.findOne()
{
"_id" : ObjectId("5aab80f424a4d605180c99fb"),
"modelVersion" : "4.0.0",
"groupId" : "audiogum",
"artifactId" : "clj-lazy-json",
"packaging" : "jar",
"version" : "0.0.3",
"name" : "clj-lazy-json",
"description" : "Jackson-based lazy JSON parsing library for Clojure.",
"scm" : {
"tag" : "url",
"attrs" : null,
"content" : [
"https://github.com/audiogum-forks/clj-lazy-json"
]
},
"dependencies" : [
{
"groupId" : "org.clojure",
"artifactId" : "clojure",
"version" : "1.9.0-RC2"
},
{
"groupId" : "org.codehaus.jackson",
"artifactId" : "jackson-core-asl",
"version" : "1.8.6"
}
]
}
```