Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/billowdev/gorm-chunck-process-query-pagination
WIP: Example CRUD in golang with GORM
https://github.com/billowdev/gorm-chunck-process-query-pagination
go-routine gorm pagination repository-pattern
Last synced: 3 days ago
JSON representation
WIP: Example CRUD in golang with GORM
- Host: GitHub
- URL: https://github.com/billowdev/gorm-chunck-process-query-pagination
- Owner: billowdev
- Created: 2024-09-17T15:15:17.000Z (4 months ago)
- Default Branch: main
- Last Pushed: 2024-09-17T15:16:16.000Z (4 months ago)
- Last Synced: 2024-11-08T09:44:17.543Z (about 2 months ago)
- Topics: go-routine, gorm, pagination, repository-pattern
- Homepage:
- Size: 3.91 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# gorm-chunck-process-query-pagination
## Overview
This project is designed to handle large datasets efficiently using pagination and chunking techniques. This approach ensures improved performance, memory efficiency, and scalability.
## Features
### Transaction Management
The system supports the following features related to transaction management:
- Efficient retrieval of transaction records
- Pagination and chunking to handle large datasets
- Flexible query building for dynamic filtering
## Code Implementation
### BuildQueryConditions
The BuildQueryConditions function dynamically constructs a SQL query string and its corresponding arguments based on the filters provided. This can be useful when you need to build queries for a database query library like GORM.
```go
// BuildQueryConditions builds a query string and arguments based on the provided filters.
// It returns a query string and a slice of arguments to be used with GORM or other database libraries.
func BuildQueryConditions(filters map[string]interface{}) (string, []interface{}) {
if len(filters) == 0 {
return "1=1", nil // Return a base condition if no filters are provided
}
query := "1=1" // Start with a base query
args := []interface{}{}
for field, value := range filters {
query += fmt.Sprintf(" AND %s = ?", field)
args = append(args, value)
}
return query, args
}
```
### Count Total Records
The `CountRecords` function counts the total number of records that match the filters. This is useful for determining pagination parameters.
```go
// CountRecords returns the total number of records that match the filters
func (o CashflowinRepositoryImpl) CountRecords(ctx context.Context, filters map[string]interface{}) (int64, error) {
tx := o.db.WithContext(ctx)
var count int64
// Build the query dynamically based on the filters
query, args, err := helpers.BuildQueryConditions(filters)
if err != nil {
return 0, err
}
// Count the total number of matching records
if err := tx.Model(&models.Transaction{}).Where(query, args...).Count(&count).Error; err != nil {
return 0, err
}
return count, nil
}
```
### Fetch Records in Chunks
The `GetAllByFieldsWithPagination` function retrieves records with pagination and chunking, which helps in managing large datasets efficiently.
```go
// GetAllByFieldsWithPagination fetches records with pagination and chunking
func (o CashflowinRepositoryImpl) GetAllByFieldsWithPagination(ctx context.Context, filters map[string]interface{}, limit *int, offset int) ([]models.Transaction, error) {
tx := o.db.WithContext(ctx)
var data []models.Transaction
// Build the query dynamically based on the filters
query, args, err := helpers.BuildQueryConditions(filters)
if err != nil {
return nil, err
}
// Build the base query with filters
dbQuery := tx.Where(query, args...)
// Apply limit and offset if provided
if limit != nil {
dbQuery = dbQuery.Limit(*limit).Offset(offset)
}
// Fetch records with pagination
if err := dbQuery.Find(&data).Error; err != nil {
return nil, err
}
return data, nil
}
```
### Example Usage
To use the pagination feature, you can count the total records first and then fetch records in chunks:
```go
filters := map[string]interface{}{
"status": "active",
}
totalRecords, err := CashflowinRepo.CountRecords(ctx, filters)
if err != nil {
// Handle error
}
// Define pagination parameters
pageSize := 500
totalPages := int(totalRecords) / pageSize
if totalRecords%int64(pageSize) != 0 {
totalPages++
}
var allRecords []models.Transaction
for page := 0; page < totalPages; page++ {
offset := page * pageSize
limit := pageSize
records, err := CashflowinRepo.GetAllByFieldsWithPagination(ctx, filters, &limit, offset)
if err != nil {
// Handle error
}
allRecords = append(allRecords, records...)
}
```
### Conclusion
By using pagination and chunking techniques, you can efficiently manage and retrieve large datasets, ensuring that your application performs well even with extensive data.