https://github.com/jdeng/golopro

Tiny multithreaded log processing framework in golang
https://github.com/jdeng/golopro

Last synced: 12 days ago
JSON representation

Tiny multithreaded log processing framework in golang

• Host: GitHub
• URL: https://github.com/jdeng/golopro
• Owner: jdeng
• License: mit
• Created: 2013-10-07T23:45:20.000Z (over 11 years ago)
• Default Branch: master
• Last Pushed: 2013-10-08T00:18:41.000Z (over 11 years ago)
• Last Synced: 2024-12-18T22:02:12.197Z (28 days ago)
• Language: Go
• Size: 195 KB
• Stars: 9
• Watchers: 5
• Forks: 3
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
golopro

=======

Tiny multithreaded log processing framework in golang

The default implementation essentially can do the work of cut/awk | sort | uniq -c  for multiple CSV compatible (e.g., Apache access logs) files usingle golang's multithreading. 

## Usage



jack@jack-VirtualBox:~/work/golopro$ ./lopro -help

Usage of ./lopro:

  -comma=",": separator

  -in=".": input directory

  -keys="0": keys, starts with 0

  -out=".": output directory

  -procs=1: number of processes



### Hints

* Use ln -s to link the log files to the input directory

* Compressed the files to save disk I/O

## Customization

There are two interfaces to be implemented.

* the parser



type Parser interface {

  Clone() Parser

  Reset(r io.Reader)

  NextRecord() (int, interface{}, error)

}



Sample implementation of CSV parser: CSVParser



...

func (lp *CSVParser) NextRecord() (length int, record interface{}, err error) {

  r, err := lp.reader.Read()

  return 0, r, err

}

...



* and the report (counting logic)



type Report interface {

  New() Report

  Merge(report Report)

  Clear()


  Name() string

  Add(rec LogRecord)

  Output(path string)

}



A simple counting report: QuickReport



func (qr *QuickReport) Add(rec LogRecord) {

  r, ok := rec.([]string)

  if !ok {

    return

  }


  var key string

  //TODO: construct the key

  ...

  

  qr.result[key] += 1

}



* and probably some tweaks for the main() function



  ...

  parser := NewCSVParser((*comma)[0])

  reportMgr := NewReportManager()

  reportMgr.RegisterReport(NewQuickReport(ks))

  ...