https://github.com/msolimans/wikimovie
https://github.com/msolimans/wikimovie
Last synced: 10 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/msolimans/wikimovie
- Owner: msolimans
- Created: 2024-04-01T06:00:22.000Z (almost 2 years ago)
- Default Branch: master
- Last Pushed: 2024-04-01T15:03:39.000Z (almost 2 years ago)
- Last Synced: 2025-01-29T07:14:36.155Z (11 months ago)
- Language: Go
- Size: 309 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
Monitoring
RateLimiter https://docs.gofiber.io/api/middleware/limiter/
# WIKIMovie
WIKIMovie acts as a data lake of movies that are injected by providers. It provides a query api that allows searching by name, year, cast member, or
### Diagram
Notes:
- The data store is not specified, noticing heavy reads than writes (which requires indexing and multiple read replicas).
- Reads are just querying and searching for movies based on different criteria/fields so I was thinking of something like an inverted index (e.g. Apache Solr or ElasticSearch). inverted index allows fast and efficient search in text fields like cast members, movie name, or genres. it also allows autocomplete, facets (categories with stats), and paging out of the box.
- While MongoDB can be easier to implement, I don't see any complex queries, transactional or ACID features, Document locking or multi-document transactions but let me know if you would like me to do this with MongoDB specifically.
### Tech stack
* Docker compose
* Local stack
* Kubernetes
* EKSCtl
* Skaffold.dev
* AWS
* S3
* SQS
* Terraform
* Elastic search
### Local development
1) In development environment, run `docker-compose` to be able to run commands on the AWS CLI against localstack.
The `AWS_ENDPOINT` and config `aws.endpoint` is only used for localstack purposes. In live environments, since these variables will be unspecified, the AWS client will automatically use the real AWS URLs.
Note: you may need to add entry in `/etc/hosts` with `localhost.localstack.cloud` pointing to `127.0.0.1`
Run command like this to list all queues from sqs:
```
AWS_REGION=us-east-1 aws --endpoint-url=http://localhost.localstack.cloud:4566 sqs list-queues
```
Docker compose will run the following AWS services locally:
- Elastic search
- S3
- SQS
2) For some reason init script was not working in my end so I had to run `./localstack.sh` manually after I spin up `localstack`
Note: to clean up resources, run `./clean.sh`
#### Seeds
To load database seeds in development, run from the root directory:
```
go run cmd/seed/main.go s3
```
Note: we can add exxtra option of sqs here to feed with sqs fixtures directly.
## API
Run the service by `go run ./cmd/service/main.go` if you have go or run it with docker
1) build image `docker build -t wikimovie`
2) run `docker run -it --rm wikimovie`
If you want to override env vars, check the below env vars and pass vars with `-e` to `docker run`
- The image was pushed to Public `ECR`, feel free to pull this in and try if you don't want to build locally
`docker run -it --rm public.ecr.aws/q6s1l7f2/wikimovie:latest`
Notes:
- Default command for this image is `service` but you can override with `worker` to run an instance of worker.
- Use the Postman file in the `postman/Wikimovie.postman_collection.json` for all supported API calls.
### Environment Variables
Expected environment values that are expected to be set in the respective environment.
| KV |
|:-----------------------------------------------------:|
| AWS_REGION |
| AWS_ENDPOINT |
| ELASTICSEARCH_URLS |
| ENV |
| WORKER_QUEUE |
| BUCKET |
Note: secrets can include passwords like the one for elastic search in prod environment.
### What included
- Search Service (implemented with [Fiber](https://github.com/gofiber/fiber) routes)
- Postman collection to help explore and navigate through API
- SQS Client to send/receive messages
- SQS worker to poll messages from SQS
- S3 Client to upload/download files
- K8s Manifest to deploy search service
- Docker compose stack to run localstack with `sqs,s3,es` services
### Enhancements
- Caching (especially for queries that include all params) can be added to enhance response time
- Github actions
- Swagger support
- Metrics in prometheous
- Better logging
- Auth middleware was added for just demo but the logic in there needs to get replaced and integrated with auth providers like Okta/Firebase
- Request Tracing (maybe Jaeger)
- Chaos Monkey Automation
- SSL Certs
- Better unit testing and test coverage reports (Codegov)