Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/UnnoTed/fileb0x

a better customizable tool to embed files in go; also update embedded files remotely without restarting the server
https://github.com/UnnoTed/fileb0x

embed embed-files embedded-files filebox files go go-bindata http http-filesystem update virtual-file-system

Last synced: about 2 months ago
JSON representation

a better customizable tool to embed files in go; also update embedded files remotely without restarting the server

Awesome Lists containing this project

README 

        
fileb0x [![Circle CI](https://circleci.com/gh/UnnoTed/fileb0x.svg?style=svg)](https://circleci.com/gh/UnnoTed/fileb0x) [![GoDoc](https://godoc.org/github.com/UnnoTed/fileb0x?status.svg)](https://godoc.org/github.com/UnnoTed/fileb0x) [![GoReportCard](https://goreportcard.com/badge/unnoted/fileb0x)](https://goreportcard.com/report/unnoted/fileb0x)

-------



### What is fileb0x?

A better customizable tool to embed files in go.



It is an alternative to `go-bindata` that have better features and organized configuration.



###### TL;DR

a better `go-bindata`



-------

### How does it compare to `go-bindata`?

Feature                               | fileb0x                       | go-bindata

---------------------                 | -------------                 | ------------------

gofmt                                 | yes (optional)                | no

golint                                | safe                          | unsafe

gzip compression                      | yes                           | yes

gzip decompression                    | yes (optional: runtime)       | yes (on read)

gzip compression levels               | yes                           | no

separated prefix / base for each file | yes                           | no (all files only)

different build tags for each file    | yes                           | no

exclude / ignore files                | yes (glob)                    | yes (regex)

spread files                          | yes                           | no (single file only)

unexported vars/funcs                 | yes (optional)                | no

virtual memory file system            | yes                           | no

http file system / handler            | yes                           | no

replace text in files                 | yes                           | no

glob support                          | yes                           | no (walk folders only)

regex support                         | no                            | yes (ignore files only)

config file                           | yes (config file only)        | no (cmd args only)

update files remotely                 | yes                           | no



-------

### What are the benefits of using a Virtual Memory File System?

By using a virtual memory file system you can have access to files like when they're stored in a hard drive instead of a `map[string][]byte` you would be able to use IO writer and reader.

This means you can `read`, `write`, `remove`, `stat` and `rename` files also `make`, `remove` and `stat` directories.



###### TL;DR 

Virtual Memory File System has similar functions as a hdd stored files would have.



### Features



- [x] golint safe code output



- [x] optional: gzip compression (with optional run-time decompression)



- [x] optional: formatted code (gofmt)



- [x] optional: spread files



- [x] optional: unexporTed variables, functions and types



- [x] optional: include multiple files and folders



- [x] optional: exclude files or/and folders



- [x] optional: replace text in files



- [x] optional: custom base and prefix path



- [x] Virtual Memory FileSystem - [webdav](https://godoc.org/golang.org/x/net/webdav)



- [x] HTTP FileSystem and Handler



- [x] glob support - [doublestar](https://github.com/bmatcuk/doublestar)



- [x] json / yaml / toml support



- [x] optional: Update files remotely



- [x] optional: Build tags for each file



### License

MIT



### Get Started



###### TL;DR QuickStart™



Here's the get-you-going in 30 seconds or less:



```bash

git clone https://github.com/UnnoTed/fileb0x.git

cd fileb0x

cd _example/simple

go generate

go build

./simple

```



* `mod.go` defines the package as `example.com/foo/simple`

* `b0x.yaml` defines the sub-package `static` from the folder `public`

* `main.go` includes the comment `//go:generate go run github.com/UnnoTed/fileb0x b0x.yaml`

* `main.go` also includes the import `example.com/foo/simple/static`

* `go generate` locally installs `fileb0x` which generates `./static` according to `bax.yaml`

* `go build` creates the binary `simple` from `package main` in the current folder

* `./simple` runs the self-contained standalone webserver with built-in files from `public`



 



How to use it?



##### 1. Download



```bash

go get -u github.com/UnnoTed/fileb0x

```



##### 2. Create a config file

First you need to create a config file, it can be `*.json`, `*.yaml` or `*.toml`. (`*` means any file name)



Now write into the file the configuration you wish, you can use the example files as a start.



json config file example [b0x.json](https://raw.githubusercontent.com/UnnoTed/fileb0x/master/_example/simple/b0x.json)



yaml config file example [b0x.yaml](https://github.com/UnnoTed/fileb0x/blob/master/_example/simple/b0x.yaml)



toml config file example [b0x.toml](https://github.com/UnnoTed/fileb0x/blob/master/_example/simple/b0x.toml)



##### 3. Run

if you prefer to use it from the `cmd or terminal` edit and run the command below.



```bash

fileb0x YOUR_CONFIG_FILE.yaml

```



or if you wish to generate the embedded files through `go generate` just add and edit the line below into your `main.go`.

```go

//go:generate fileb0x YOUR_CONFIG_FILE.yaml

```



 

  What functions and variables fileb0x let me access and what are they for?



#### HTTP

```go

var HTTP http.FileSystem

```



##### Type 

[`http.FileSystem`](https://golang.org/pkg/net/http/#FileSystem)



##### What is it?



A In-Memory HTTP File System.



##### What it does?



Serve files through a HTTP FileServer.



##### How to use it?

```go

// http.ListenAndServe will create a server at the port 8080

// it will take http.FileServer() as a param

//

// http.FileServer() will use HTTP as a file system so all your files

// can be avialable through the port 8080

http.ListenAndServe(":8080", http.FileServer(myEmbeddedFiles.HTTP))

```



 

  How to use it with `echo`?



```go

package main



import (

	"github.com/labstack/echo"

	"github.com/labstack/echo/engine/standard"

	// your embedded files import here ...

	"github.com/UnnoTed/fileb0x/_example/echo/myEmbeddedFiles"

)



func main() {

	e := echo.New()



	// enable any filename to be loaded from in-memory file system

	e.GET("/*", echo.WrapHandler(myEmbeddedFiles.Handler))



	// http://localhost:1337/public/README.md

	e.Start(":1337")

}

```



##### How to serve a single file through `echo`?

```go

package main



import (

	"github.com/labstack/echo"



	// your embedded files import here ...

	"github.com/UnnoTed/fileb0x/_example/echo/myEmbeddedFiles"

)



func main() {

	e := echo.New()



	// read ufo.html from in-memory file system

	htmlb, err := myEmbeddedFiles.ReadFile("ufo.html")

	if err != nil {

		log.Fatal(err)

	}



	// convert to string

	html := string(htmlb)



	// serve ufo.html through "/"

	e.GET("/", func(c echo.Context) error {



		// serve as html

		return c.HTML(http.StatusOK, html)

	})



	e.Start(":1337")

}

```



 

  Examples



[simple example](https://github.com/UnnoTed/fileb0x/tree/master/_example/simple) -

[main.go](https://github.com/UnnoTed/fileb0x/blob/master/_example/simple/main.go)



[echo example](https://github.com/UnnoTed/fileb0x/tree/master/_example/echo) -

[main.go](https://github.com/UnnoTed/fileb0x/blob/master/_example/echo/main.go)



```go

package main



import (

	"log"

	"net/http"



  // your generaTed package

	"github.com/UnnoTed/fileb0x/_example/simple/static"

)



func main() {

	files, err := static.WalkDirs("", false)

	if err != nil {

		log.Fatal(err)

	}



	log.Println("ALL FILES", files)



  // here we'll read the file from the virtual file system

	b, err := static.ReadFile("public/README.md")

	if err != nil {

		log.Fatal(err)

	}



  // byte to str

  s := string(b)

  s += "#hello"



  // write file back into the virtual file system

  err := static.WriteFile("public/README.md", []byte(s), 0644)

  if err != nil {

    log.Fatal(err)

  }



	log.Println(string(b))



	// true = handler

	// false = file system

	as := false



	// try it -> http://localhost:1337/public/secrets.txt

	if as {

		// as Handler

		panic(http.ListenAndServe(":1337", static.Handler))

	} else {

		// as File System

		panic(http.ListenAndServe(":1337", http.FileServer(static.HTTP)))

	}

}

```



 



Update files remotely



Having to upload an entire binary just to update some files in a b0x and restart a server isn't something that i like to do...



##### How it works?

By enabling the updater option, the next time that you generate a b0x, it will include a http server, this http server will use a http basic auth and it contains 1 endpoint `/` that accepts 2 methods: `GET, POST`.



The `GET` method responds with a list of file names and sha256 hash of each file.

The `POST` method is used to upload files, it creates the directory tree of a new file and then creates the file or it updates an existing file from the virtual memory file system... it responds with a `ok` string when the upload is successful.

  

##### How to update files remotely?



1. First enable the updater option in your config file:

```yaml

##################

## yaml example ##

##################



# updater allows you to update a b0x in a running server

# without having to restart it

updater:

  # disabled by default

  enabled: false



  # empty mode creates a empty b0x file with just the 

  # server and the filesystem, then you'll have to upload

  # the files later using the cmd:

  # fileb0x -update=http://server.com:port b0x.yaml

  #

  # it avoids long compile time

  empty: false



  # amount of uploads at the same time

  workers: 3



  # to get a username and password from a env variable

  # leave username and password blank (username: "")

  # then set your username and password in the env vars 

  # (no caps) -> fileb0x_username and fileb0x_password

  #

  # when using env vars, set it before generating a b0x 

  # so it can be applied to the updater server.

  username: "user" # username: ""

  password: "pass" # password: ""

  port: 8041

```

2. Generate a b0x with the updater option enabled, don't forget to set the username and password for authentication.

3. When your files update, just run `fileb0x -update=http://yourServer.com:8041 b0x.toml` to update the files in the running server.



 

  Build Tags



To use build tags for a b0x package just add the tags to the `tags` property in the main object of your config file

```yaml

# default: main

pkg: static



# destination

dest: "./static/"



# build tags for the main b0x.go file

tags: "!linux"

```



You can also have different build tags for a list of files, you must enable the `spread` property in the main object of your config file, then at the `custom` list, choose the set of files which you want a different build tag 

```yaml

# default: main

pkg: static



# destination

dest: "./static/"



# build tags for the main b0x.go file

tags: "windows darwin"



# [spread] means it will make a file to hold all fileb0x data

# and each file into a separaTed .go file

#

# example:

# theres 2 files in the folder assets, they're: hello.json and world.txt

# when spread is activaTed, fileb0x will make a file: 

# b0x.go or [output]'s data, assets_hello.json.go and assets_world.txt.go

#

#

# type: bool

# default: false

spread: true



# type: array of objects

custom:

  # type: array of strings

  - files: 

    - "start_space_ship.exe"



    # build tags for this set of files

    # it will only work if spread mode is enabled

    tags: "windows"



  # type: array of strings

  - files: 

    - "ufo.dmg"



    # build tags for this set of files

    # it will only work if spread mode is enabled

    tags: "darwin"

```



the config above will make:

```yaml

ab0x.go                         # // +build windows darwin



b0xfile_ufo.exe.go              # // +build windows

b0xfile_start_space_ship.bat.go # // +build darwin

```



### Functions and Variables



 

  FS (File System)



```go

var FS webdav.FileSystem

```



##### Type

[`webdav.FileSystem`](https://godoc.org/golang.org/x/net/webdav#FileSystem)



##### What is it?



In-Memory File System.



##### What it does?



Lets you `read, write, remove, stat and rename` files and `make, remove and stat` directories...



##### How to use it?

```go

func main() {



	// you have the following functions available

	// they all control files/dirs from/to the in-memory file system!

	func Mkdir(name string, perm os.FileMode) error

	func OpenFile(name string, flag int, perm os.FileMode) (File, error)

	func RemoveAll(name string) error

	func Rename(oldName, newName string) error

	func Stat(name string) (os.FileInfo, error)

	// you should remove those lines ^



	// 1. creates a directory

	err := myEmbeddedFiles.FS.Mkdir(myEmbeddedFiles.CTX, "assets", 0777)

	if err != nil {

		log.Fatal(err)

	}



	// 2. creates a file into the directory we created before and opens it

	// with fileb0x you can use ReadFile and WriteFile instead of this complicaTed thing

	f, err := myEmbeddedFiles.FS.OpenFile(myEmbeddedFiles.CTX, "assets/memes.txt", os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)

	if err != nil {

		log.Fatal(err)

	}



	data := []byte("I are programmer I make computer beep boop beep beep boop")



	// write the data into the file

	n, err := f.Write(data)

	if err == nil && n < len(data) {

		err = io.ErrShortWrite

	}



	// close the file

	if err1 := f.Close(); err == nil {

		log.Fatal(err1)

	}



	// 3. rename a file

	// can also move files

	err = myEmbeddedFiles.FS.Rename(myEmbeddedFiles.CTX, "assets/memes.txt", "assets/programmer_memes.txt")

	if err != nil {

		log.Fatal(err)

	}



	// 4. checks if the file we renamed exists

	if _, err = myEmbeddedFiles.FS.Stat(myEmbeddedFiles.CTX, "assets/programmer_memes.txt"); os.IsExist(err) {

		// exists!



		// tries to remove the /assets/ directory

		// from the in-memory file system

		err = myEmbeddedFiles.FS.RemoveAll(myEmbeddedFiles.CTX, "assets")

		if err != nil {

			log.Fatal(err)

		}

	}



	// 5. checks if the dir we removed exists

	if _, err = myEmbeddedFiles.FS.Stat(myEmbeddedFiles.CTX, "public/"); os.IsNotExist(err) {

		// doesn't exists!

		log.Println("works!")

	}

}

```



 

  Handler



```go

var Handler *webdav.Handler

```



##### Type

[`webdav.Handler`](https://godoc.org/golang.org/x/net/webdav#Handler)



##### What is it?



A HTTP Handler implementation.



##### What it does?



Serve your embedded files.



##### How to use it?

```go

// ListenAndServer will create a http server at port 8080

// and use Handler as a http handler to serve your embedded files

http.ListenAndServe(":8080", myEmbeddedFiles.Handler)

```



 

  ReadFile



```go

func ReadFile(filename string) ([]byte, error)

```



##### Type

[`ioutil.ReadFile`](https://godoc.org/io/ioutil#ReadFile)



##### What is it?



A Helper function to read your embedded files.



##### What it does?



Reads the specified file from the in-memory file system and return it as a byte slice.



##### How to use it?

```go

// it works the same way that ioutil.ReadFile does.

// but it will read the file from the in-memory file system

// instead of the hard disk!

//

// the file name is passwords.txt

// topSecretFile is a byte slice ([]byte)

topSecretFile, err := myEmbeddedFiles.ReadFile("passwords.txt")

if err != nil {

	log.Fatal(err)

}



log.Println(string(topSecretFile))

```



 

  WriteFile



```go

func WriteFile(filename string, data []byte, perm os.FileMode) error

```



##### Type

[`ioutil.WriteFile`](https://godoc.org/io/ioutil#WriteFile)



##### What is it?



A Helper function to write a file into the in-memory file system.



##### What it does?



Writes the `data` into the specified `filename` in the in-memory file system, meaning you embedded a file!



-- IMPORTANT --

IT WON'T WRITE THE FILE INTO THE .GO GENERATED FILE, IT WILL BE TEMPORARY, WHILE YOUR APP IS RUNNING THE FILE WILL BE AVAILABLE,

AFTER IT SHUTDOWN, IT IS GONE.



##### How to use it?

```go

// it works the same way that ioutil.WriteFile does.

// but it will write the file into the in-memory file system

// instead of the hard disk!

//

// the file name is secret.txt

// data should be a byte slice ([]byte)

// 0644 is a unix file permission



data := []byte("jet fuel can't melt steel beams")

err := myEmbeddedFiles.WriteFile("secret.txt", data, 0644)

if err != nil {

	log.Fatal(err)

}

```



 

  WalkDirs



```go

func WalkDirs(name string, includeDirsInList bool, files ...string) ([]string, error) {

```



##### Type

`[]string`



##### What is it?



A Helper function to walk dirs from the in-memory file system.



##### What it does?



Returns a list of files (with option to include dirs) that are currently in the in-memory file system.



##### How to use it?

```go

includeDirsInTheList := false



// WalkDirs returns a string slice with all file paths

files, err := myEmbeddedFiles.WalkDirs("", includeDirsInTheList)

if err != nil {

	log.Fatal(err)

}



log.Println("List of all my files", files)

```