https://github.com/khulnasoft/ingestify
https://github.com/khulnasoft/ingestify
Last synced: 6 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/khulnasoft/ingestify
- Owner: khulnasoft
- License: agpl-3.0
- Created: 2024-11-27T02:29:26.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2025-01-20T01:48:39.000Z (9 months ago)
- Last Synced: 2025-02-01T02:16:00.912Z (8 months ago)
- Language: Go
- Size: 57.6 KB
- Stars: 0
- Watchers: 3
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
README
# ingestify
> A ingestify consumer of ecosystem APIs.
Ingestify is an ingestor of package releases from multiple ecosystems (each ecosystem is found in [ingestors/](ingestors/)).
When new package releases are found, they are pushed to a shared redis queue for [Khulnasoft.com](https://khulnasoft.com) to process.
## Ingestor Types
* `ingestors.Ingestor`: these are scheduled to ingest new versions at specific intervals (`ingestor.Schedule()`).
* `ingestors.StreamingIngestor`: these are always running in a goroutine, ingesting new releases via a channel
## Ingestor Cursor Patterns
Ingestify has to know where to pick up once it restarts, so there are several methods for storing such a cursor:
* `ingestors.setBookmarkTime()` + `ingestor.getBookmarkTime()` [RECOMMENDED] : reads/sets a `time.Time` to redis (persistent)
* `ingestors.setBookmark()` + `ingestor.getBookmark()`: reads/sets an arbitrary string to redis (persistent)
* `LatestRun`: reads/sets a `time.Time` on the ingestor instance (non-persistent)
## Running Locally
`go run main.go`
## Running Tests
`go test -v ./...`
## Deploying
1. merge PR into `main` branch
2. `tl setenv libraries`
3. `./bin/deploy.sh`