https://github.com/miladhzzzz/data-service

https://github.com/miladhzzzz/data-service

Last synced: 11 days ago
JSON representation

• Host: GitHub
• URL: https://github.com/miladhzzzz/data-service
• Owner: miladhzzzz
• Created: 2023-09-05T06:55:30.000Z (over 1 year ago)
• Default Branch: master
• Last Pushed: 2023-09-05T06:55:33.000Z (over 1 year ago)
• Last Synced: 2024-11-05T22:32:16.172Z (about 2 months ago)
• Language: Go
• Size: 41 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
To address the storage problem described, you can use a database to store the CSV file records and create an endpoint to access the objects based on their ID. Here's a high-level approach to handle the requirements: 

 

1. Database Selection: 

   - Choose a database that can handle large datasets efficiently, such as PostgreSQL or Apache Cassandra. 

   - Consider the read and write performance, scalability, and availability of the chosen database. 

 

2. Data Ingestion: 

   - Create a script or a separate service that reads the CSV file every 30 minutes. 

   - Parse the CSV file and extract the records. 

   - Delete the existing data in the database and write the new records. 

 

3. Endpoint Implementation: 

   - Build an HTTP server using Golang to handle incoming requests. 

   - Define an API endpoint, e.g.,  /promotions/{id} , to retrieve the object based on the provided ID. 

   - Implement the logic to query the database using the ID and return the corresponding object if found. 

   - Return a "not found" response if the ID does not exist. 

 

4. Performance Considerations: 

   - To handle a large CSV file, consider processing the file in chunks rather than loading the entire file into memory at once. 

   - Optimize database queries by indexing the ID column and using appropriate query optimization techniques. 

   - Employ caching mechanisms, such as Redis, to reduce the load on the database during peak periods. 

 

5. Scaling and Deployment: 

   - Deploy the application on a cloud platform or a cluster of servers to handle the expected load. 

   - Utilize containerization technologies like Docker and container orchestration tools like Kubernetes for easy deployment and scaling. 

   - Set up auto-scaling rules based on metrics like CPU utilization or request rate to handle peak periods efficiently. 

 

6. Monitoring: 

   - Instrument your application with appropriate monitoring tools like Prometheus or Datadog to collect metrics on request latency, error rates, and resource utilization. 

   - Set up alerts and dashboards to proactively monitor the health and performance of your application. 

 

Remember to continuously test, monitor, and optimize your application to ensure it performs well under various conditions.