https://github.com/j2gg0s/kfile
use Kafka as Distribute File System, automatic support Append and Expire
https://github.com/j2gg0s/kfile
append distributefilesystem expire kafka
Last synced: 11 months ago
JSON representation
use Kafka as Distribute File System, automatic support Append and Expire
- Host: GitHub
- URL: https://github.com/j2gg0s/kfile
- Owner: j2gg0s
- License: bsd-3-clause
- Created: 2021-06-03T12:54:32.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2021-06-28T11:04:15.000Z (over 4 years ago)
- Last Synced: 2025-01-25T09:44:41.439Z (about 1 year ago)
- Topics: append, distributefilesystem, expire, kafka
- Language: Go
- Homepage:
- Size: 29.3 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# kfile
Use kafka for distribute file storage, support write append and auto expire.
# User story
In our team, we use PostgreSQL/MySQL as data storage, Kafka as message queue.
It's difficult to share file between multi instances under these components.
For use, sharing file is a rare situation, only like export thoundes of records.
We don't want to import one Distributed File System, so we use kafka to save temporary file.
# Usage
Write:
```go
name := "fileA"
writer, err := kfile.NewWriter(
topic, name,
client,
kfile.WithNumPartitions(int32(numPartitions)),
kfile.WithReplicationFactor(int16(replicationFactor)),
kfile.WithTopicRetention(time.Hour*24*31))
if err != nil {
fmt.Println(fmt.Errorf("new file: %w", err))
return
}
for j := 0; j < 100; j++ {
err := writer.Append(
kfile.Line([]byte(fmt.Sprintf("Line %d of file %s\n", j, name))))
if err != nil {
fmt.Println(fmt.Errorf("write: %w", err))
return
}
}
uri, err := writer.Close(10 * time.Second)
if err != nil {
fmt.Println(fmt.Errorf("close file: %w", err))
return
}
fmt.Printf("write to %s\n", uri)
```
Read:
```go.
reader, err := kfile.NewReader(uri, client)
if err != nil {
fmt.Println(fmt.Errorf("new reader: %w", err))
return
}
buf, err := reader.ReadAll(0)
if err != nil {
fmt.Println(fmt.Errorf("read: %w", err))
return
}
fmt.Printf("read from %s\n", uri)
fmt.Println(buf.String())
if _, err := reader.Close(0); err != nil {
fmt.Println(err)
return
}
```
See more in `example/`