https://github.com/fibo/5m
is a near real time utility, 5mb or 5min
https://github.com/fibo/5m
Last synced: 11 months ago
JSON representation
is a near real time utility, 5mb or 5min
- Host: GitHub
- URL: https://github.com/fibo/5m
- Owner: fibo
- License: mit
- Created: 2016-11-19T23:29:42.000Z (over 9 years ago)
- Default Branch: master
- Last Pushed: 2018-07-06T17:00:13.000Z (about 8 years ago)
- Last Synced: 2025-01-01T09:25:16.248Z (over 1 year ago)
- Language: JavaScript
- Homepage: http://g14n.info/5m
- Size: 27.3 KB
- Stars: 3
- Watchers: 3
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# 5m
> is a near real time utility, 5mb or 5min
[](http://g14n.info/kiss-literate-programming)
[Installation](#installation) |
[Use case](#use-case) |
[Annotated source](#annotated-source) |
[License](#license)
## Installation
```bash
npm i 5m
```
## Use case
> I use this package with [debug] and [aws-sdk] to upload logs on S3.
Then there is a trigger that feeds every file created on S3 to an AWS Lambda
which sends file content to a server that broadcast it using [socket.io].
Yes I know there are other cool tools like CloudWatch, Kinesis, etc.
(but **5m** is free as in speach and free as in beer too ;)
The cool part is that it is possible to trigger an AWS Lambda every time
a new file is created, and for example load it on a database.
Here is an example code to achieve logging and upload on S3.
```js
// File: log.js
//
// Prints to STDOUT if DEBUG environment variable is set properly.
// Uploads logs on some S3 bucket.
//
// const log = require('./log')
// log('Hello world')
//
const AWS = require('aws-sdk')
const debug = require('debug')
const fiveM = require('5m')
const s3 = new AWS.S3()
// Use package name as namespace for logging, just as an example.
const pkg = require('./package.json')
const namespace = pkg.name
const debug5m = debug(`${namespace}:5m`)
const debugPkg = debug(namespace)
const writeOnS3 = (data) => {
const Bucket = 'my-bucket'
const tstamp = new Date().getTime()
const Key = `my/path/${tstamp}.log`
s3.upload({ Bucket, Key }, error => {
if (error) debug5m(error)
else debug5m()
})
}
function log (message) {
const tstamp = new Date().getTime()
const record = `${tstamp} ${namespace} ${message}`
fiveM(namespace, writeOnS3)(record)
debugPkg(message)
}
module.exports = log
```
## Annotated source
Store data in a namespaced bucket, as well as its function to flush data.
```javascript
var bucket = {}
var flush = {}
```
Considering 1 char should be 1 byte, and dates are expressed in milliseconds,
the following constants express *5 MB* and *5 minutes*.
Aproximation is ok, since we want to achieve a near real time.
The timeout used for flushing data can be set, for testing or other purpouse
via the `FIVEM_TIMEOUT_MILLISECONDS` environment variable.
```javascript
const fiveMb = 1024 * 1024 * 5
const fiveMin = 300 * 1000
const flushTimeout = process.env.FIVEM_TIMEOUT_MILLISECONDS || fiveMin
```
Make sure no data is lost on exit.
```javascript
process.on('exit', () => {
for (var namespace in bucket) flush[namespace]()
})
```
Create the **5m** function.
Since *5m* is allowed as npm package name, but not as JavaScript
identifier, using a roman number like *Vm* can be confusing, so maybe
naming the function as *fiveM* is a good idea.
```javascript
/**
* **@param** `{String}` namespace
* **@param** `{Function}` write
*
* **@returns** `{Function}` logger
*/
function fiveM (namespace, write) {
```
Create the namespaced *flush* function: write data and clean up.
```javascript
flush[namespace] = () => {
if (bucket[namespace]) {
write(bucket[namespace])
delete bucket[namespace]
}
}
```
Create the **logger** function.
```javascript
/**
* @param {*} data
*/
return function logger (data) {
```
If necessary, initialize data bucket and set timeout to flush it later.
```javascript
if (typeof bucket[namespace] === 'undefined') {
bucket[namespace] = ''
setTimeout(flush[namespace], flushTimeout)
}
```
Append data to named bucket.
```javascript
bucket[namespace] += data
```
Check if data is bigger than *5 MB*.
```javascript
const exceededSpace = bucket[namespace] && (bucket[namespace].length > 0) && (bucket[namespace].length > fiveMb)
```
If yes, flush it!
```javascript
if (exceededSpace) flush[namespace]()
}
}
```
Export *5m* function.
```javascript
module.exports = fiveM
```
## License
[MIT](http://g14n.info/mit-license)
[debug]: https://www.npmjs.com/package/debug "debug"
[aws-sdk]: https://www.npmjs.com/package/aws-sdk "aws-sdk"
[socket.io]: https://socket.io/ "socket.io"