https://github.com/mdouchement/iosupport
It provides some io supports for Golang
https://github.com/mdouchement/iosupport
csv-reader csv-sorter golang
Last synced: 11 months ago
JSON representation
It provides some io supports for Golang
- Host: GitHub
- URL: https://github.com/mdouchement/iosupport
- Owner: mdouchement
- License: mit
- Created: 2015-01-16T19:18:05.000Z (about 11 years ago)
- Default Branch: master
- Last Pushed: 2018-08-20T22:43:22.000Z (over 7 years ago)
- Last Synced: 2025-01-07T06:50:04.507Z (about 1 year ago)
- Topics: csv-reader, csv-sorter, golang
- Language: Go
- Homepage:
- Size: 104 KB
- Stars: 2
- Watchers: 4
- Forks: 0
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# iosupport
[](https://circleci.com/gh/mdouchement/iosupport)
[](https://godoc.org/github.com/mdouchement/iosupport)
[](https://goreportcard.com/report/github.com/mdouchement/iosupport)
[](http://opensource.org/licenses/MIT)
It provides some io supports for GoLang:
- Read large files (line length and large amount of lines)
- Parse CSV files according the RFC4180, but:
- It does not support `\r\n` in quoted field
- It does not support comment
- Sort CSV on one or several columns
## Usage
In order to start, go get this repository:
```bash
$ go get github.com/mdouchement/iosupport
```
### Example
- Scanner & wc
```go
package main
import(
"os"
"github.com/mdouchement/iosupport"
)
func main() {
// With local filesystem
file, _ := os.Open("my_file.txt")
defer file.Close()
// Or with HDFS "github.com/colinmarc/hdfs"
// client, _ := hdfs.New("localhost:9000")
// file, _ := client.Open("/iris.csv")
// See scanner.go for more examples
sc := iosupport.NewScanner(file)
sc.EachString(func(line string, err error) {
check(err)
println(line)
})
// See wc.go for more examples
wc := iosupport.NewWordCount(file)
wc.Perform()
println(wc.Chars)
println(wc.Words)
println(wc.Lines)
}
func check(err error) {
if err != nil {
panic(err)
}
}
```
- TSV sort
```go
package main
import(
"os"
"github.com/mdouchement/iosupport"
)
func main() {
sc := func() *iosupport.Scanner {
file, _ := os.Open("iris.csv")
// Or with HDFS "github.com/colinmarc/hdfs"
// client, _ := hdfs.New("localhost:9000")
// file, _ := client.Open("/iris.csv")
return iosupport.NewScanner(file)
}
// See tsv_indexer.go for more examples
indexer = iosupport.NewTsvIndexer(sc, iosupport.HasHeader(), iosupport.Separator(","), iosupport.Fields("col2", "col1")) // scanner, headerIsPresent, separator, fieldsForSorting
defer indexer.CloseIO()
err := indexer.Analyze() // creates lines index
check(err)
indexer.Sort() // sorts indexed lines
ofile, _ := os.Open("my_sorted.tsv")
defer ofile.Close()
indexer.Transfer(ofile) // transfers the input TSV in sorted output TSV
}
func check(err error) {
if err != nil {
panic(err)
}
}
```
## Tests
- Installation
```sh
$ go get github.com/onsi/ginkgo/ginkgo
$ go get github.com/onsi/gomega
$ go get github.com/golang/mock/gomock
```
_ Run tests
```sh
# One shot
$ ginko
# With watch
$ ginkgo watch
```
- Generate package test file
```sh
$ ginkgo bootstrap # set up a new ginkgo suite
$ ginkgo generate my_file.go # will create a sample test file. edit this file and add your tests then...
```
- Benchmarks
```sh
$ go test -run=NONE -bench=ParseFields
```
- Generate mocks
```sh
# go get github.com/golang/mock/mockgen
$ mockgen -package=iosupport_test -source=storage_service.go -destination=storage_service_mock_test.go
```
## License
**MIT**
## Contributing
1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
5. Push to the branch (git push origin my-new-feature)
6. Create new Pull Request