Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/dacr/code-examples-manager

Software tool to manage your notes, scripts, code examples, configs,... to publish them as gists or snippets
https://github.com/dacr/code-examples-manager

ammonite cem code-examples code-examples-manager gist gists github-gists gitlab gitlab-snippets good-code-examples lmdb scala scala-cli scala3 snippet snippets snippets-manager zio zio-lmdb zio2

Last synced: 11 days ago
JSON representation

Software tool to manage your notes, scripts, code examples, configs,... to publish them as gists or snippets

Awesome Lists containing this project

README 

        
# CEM - Code Examples Manager [![][CodeExamplesManagerImg]][CodeExamplesManagerLnk] ![Scala CI][scalaci-master]



Code example manager (CEM) is a software managing your notes, scripts and code examples.

It provides publish mechanisms to [github.com][githubcom] (as [gists][gists]) or

[gitlab.com][gitlabcom] (as [snippets][snippets]). It also automates execution for

testable examples, this is a quite useful to manage a high number of examples.



All my notes, scripts and code examples (my programming knowledge base) are now managed using this tool,

you can take a look to **[my public gists overview on github][mygists]** to illustrate the 

publishing work achieved by CEM.



Current [Code example manager (CEM)][cem] implementation is just a command line tool

which compare locally available examples with already published ones in order to find

what it should do (add, update, do nothing).



![](images/cloudtags.png)



## Why ?



Code examples are very important, each example is most of the time designed to focus

on a particular feature/characteristic of a programming language, a library or a framework.

They help us to quickly test, experiment and remember how bigger project or at least some

parts of them are working.



**[See the rules for good code examples][rules] for more information.**



Managing hundreds of published code example files as gists (github) and/or snippets (gitlab)

is really not easy and time-consuming, in particular if you want to keep them up to date. This

is the main issue addressed by this software. 



- My open-source examples evolution trend :  

  ![](images/created-examples-trend.png)

- and the execution status trend for executable/testable ones :  

  ![](images/testable-examples-status.png)



As you can see through the previous charts, once you have industrialized your notes and code

examples, analytics on your examples become quite easy, and a lot of advanced features become

possible... So stay connected to this project ;)



## Quick start



No particular prerequisites, just a Java >=8 JVM available, and

it will run on your Linux, Windows or MacOSX



Instructions example with github.com publishing configuration :

- Install the [coursier][csget] from @alxarchambault to automate

  the download/update/start of code-examples-manager directly from

  maven repositories

- Customize your configuration (see below for token configuration)

  ```

  export CEM_SEARCH_ROOTS="/home/myuser/myexamples"

  export CEM_GITHUB_TOKEN="xxxxx"

  ```

- Create an example file in `/home/myuser/myexamples` such as `hello.md`

  ```

  

  # Hello world !

  this is just an example

  ```

- Run the following command from your terminal (`cs` is the [coursier][cs] CLI command):

  ```

  cs launch fr.janalyse:code-examples-manager_3:2.4.0

  ```

  - you can even use `cs launch fr.janalyse:code-examples-manager_3:latest.release` to always use the latest release

  - current release is : [![][CodeExamplesManagerImg]][CodeExamplesManagerLnk]

- Check the command output to get the overview URL



## Code examples



In order to be published your code examples must come with a description header

inserted using single line comments. You must provide a unique identifier (UUID)

to each of your example, as well as a summary and publish keywords which define

remote destinations.



Example for languages using `//` for line comments :

```scala

// summary : Simplest scalatest test framework usage.

// keywords : scala, scalatest, pi, @testable

// publish : gist

// authors : David Crosson

// license : Apache

// id : d24d8cb3-45c0-4d88-b033-7fae2325607b

// created-on : 2020-05-31T19:54:52Z

// run-with : scala-cli $file



// ---------------------

//> using scala  "3.1.1"

//> using lib "org.scalatest::scalatest:3.2.10"

// ---------------------



import org.scalatest._, matchers.should.Matchers._



math.Pi shouldBe 3.14d +- 0.01d

```



Supported keys in description header are the following :

- **`summary`** : example summary in one line.

- **`keywords`** : keywords describing your code features (comma separated). Some reserved keywords :

  - `@testable` : allow automatic execution

  - `@fail` : the example is expected to fail when executed

  - `@exclusive` : all testable examples with this flag will be run sequentially (for scripts which open server sockets for example)

- **`publish`** : publish destination keywords (comma separated)

  - the default configuration file provide those activation keywords :

    - `gist` : for github.com

    - `snippet` : for gitlab.com

- **`authors`** : code example authors list (comma separated).

- **`license`** : the example license.

- **`id`** : UUID for this code example. Generated using such commands :

  - with linux command : `uuidgen`

  - with [scala-cli][scl] : `scala-cli https://gist.github.com/dacr/87c9636a6d25787d7c274b036d2a8aad`

  - with [ammonite][amm] : `amm -c 'println(java.util.UUID.randomUUID.toString)'`

- **`attachments`** : List of secondary files (comma separated) which must be published with the current one

  - must be placed in the same directory in your local storage

  - secondary files do not require any headers

- **`created-on`** : The ISO8601 date when this example has been created. Generated using such commands :

  - with linux command : `date -u +"%Y-%m-%dT%H:%M:%S.%3NZ"`

  - with [scala-cli][scl] : `scala-cli https://gist.github.com/dacr/4298fce08e12ba76ab91e9766be52acb`

  - with [ammonite][amm] : `amm -c 'println(java.time.Instant.now.toString)'`

- **`run-with`** : command used to execute this example

  - Only examples with `@testable` keywords are eligible for automated execution

  - on execution the exit code is used to compute execution success or failure

  - Use `$file` (or `$scriptFile`) for example filename substitution

- **`test-with`** : Command to test the example

  - When `@testable` is set as keyword

  - When your example is a "blocking service", you can specify an external command to test it

    - for example : `test-with : curl http://127.0.0.1:8080/docs`



## CEM operations



Code examples manager operations :

- It reads its configuration

- It searches for code examples from the given directories roots

  - select files matching the search pattern and not involved in the ignore mask

  - Selects code examples if and only if they contain a unique identifier (UUID)

- It publishes or updates remote code examples to remote destinations

  - the code example publish scope (`publish` keyword) select target destinations

    - comma separated publish activation keyword (`activation-keyword` parameter in configuration) 

  - It adds or updates a global overview of all published examples for a given destination

    - this summary has its own UUID defined in the configuration file 



## Configuration



The configuration relies on configuration files, a default one named `reference.conf` is provided.

This [default configuration file][referenceconf] defines default values and default behaviors and

allow a simple configuration way based on environment variables which override default values.



### Simplified configuration



| env or property name       | description                                                   | default value

|----------------------------|---------------------------------------------------------------|---------------------------

| CEM_CONFIG_FILE            | Your custom advanced configuration file (optional)            | *undefined*

| CEM_SUMMARY_TITLE          | The generated summary title for all published examples        | Examples knowledge base 

| CEM_SEARCH_ROOTS           | Examples search roots (comma separated)                       | ""

| CEM_SEARCH_PATTERN         | Examples files regular expression pattern                     | ".*"

| CEM_SEARCH_IGNORE_MASK     | Ignore file regular expression                                | "(/[.]bsp)|(/[.]scala.*)|([.]png$)"

| CEM_CHAR_ENCODING          | Chararacter encoding for your examples or notes               | "UTF-8"

| CEM_GITHUB_ENABLED         | To enable or disable standard GITHUB support                  | true

| CEM_GITHUB_ACTIVATION_KEY  | Example publish keyword for github                            | "gist"

| CEM_GITHUB_TOKEN           | Github authentication token for gists API access              | *more information below*

| CEM_GITHUB_API             | Github API http end point                                     | "https://api.github.com"

| CEM_GITLAB_ENABLED         | To enable or disable standard GITLAB support                  | true

| CEM_GITLAB_ACTIVATION_KEY  | Example publish keyword for the gitlab                        | "snippet"

| CEM_GITLAB_TOKEN           | gitlab authentication token for snippets API access           | *more information below*

| CEM_GITLAB_API             | Gitlab API http end point                                     | "https://gitlab.com/api/v4"

| CEM_GITLAB_VISIBILITY      | Gitlab published examples chosen visibility                   | "public"



Configuration examples :

```shell

export CEM_SEARCH_ROOTS="/tmp/someplace,/tmp/someotherplace"

export CEM_SEARCH_PATTERN="[.](sc)|(sh)|(md)|(jsh)$"

export CEM_GITHUB_TOKEN="fada-fada-fada-fada"

```



### Advanced configuration



Take a look to the [configuration file][referenceconf] for more information about advanced configuration.



Once CEM installed you can modify the provided `conf/application.conf` file (whose content is by default

the same as the default [reference.conf][referenceconf] file), remember that any unset parameter in `application.conf`

will default to the value defined in `reference.conf`.



Note : It is also possible to provide a custom configuration file through the `config.file` java property or the

`CEM_CONFIG_FILE` environment variable.



## Authentication tokens



### Gitlab authentication token configuration



Get an access token from gitlab :

- Go to your user **settings**

  - Select **Access tokens**

    - Add a **Personal access token**

      - Enable scopes : `api` and `read_user`

- setup your `CEM_GITLAB_TOKEN` environment variable or `token` parameter in your configuration file

  with the generated token

- **Keep it carefully as it is not possible to retrieve it later.**

- **And of course KEEP IT SECURE**



### Github authentication token configuration



Get an access token from gitlab.com :

- Got to your user **settings**

  - Select **Developer settings**

    - Select **Personal access tokens**

      - Then **generate new token**

        - Enable scopes : `gist` and `read:user`

- setup your `CEM_GITHUB_TOKEN` environment variable or `token` parameter in your configuration file

  with the generated token, the value shown within curl json response

- **Keep it carefully as it is not possible to retrieve it later.**

- **And of course KEEP IT SECURE**



## Project history



- 2019-06 - PoC#1 example proof of concept

- 2019-07 - PoC#2 example proof of concept (in prod)

- 2019-08 - Switch to a real world project

- 2019-09 - In prod for my own usage

- 2020-07 - First public release

- 2021-05 - Full refactoring to use [ZIO][zio] - pure functional

- 2021-06 - Migration to Scala3

- 2021-12 - PoC#3 Search & Execution engines

- 2022-01 - Migrate to ZIO2 and add support for attachments

- 2023-04 - Use ZIO standard configuration

- 2023-05 - Add ZIO LMDB for local storage cache AND data sharing with external applications 



[CodeExamplesManager]:    https://github.com/dacr/code-examples-manager

[CodeExamplesManagerImg]: https://img.shields.io/maven-central/v/fr.janalyse/code-examples-manager_3.svg

[CodeExamplesManagerLnk]: https://mvnrepository.com/artifact/fr.janalyse/code-examples-manager

[scalaci-master]: https://github.com/dacr/code-examples-manager/workflows/Scala%20CI/badge.svg

[mygists]: https://gist.github.com/c071a7b7d3de633281cbe84a34be47f1

[cem]: https://github.com/dacr/code-examples-manager

[amm]: https://ammonite.io/

[scl]: https://scala-cli.virtuslab.org/

[githubcom]: https://github.com/

[gitlabcom]: https://gitlab.com/

[snippets]: https://docs.gitlab.com/ce/user/snippets.html

[gists]: https://docs.github.com/en/github/writing-on-github/creating-gists

[uuid-sc]: https://gist.github.com/dacr/87c9636a6d25787d7c274b036d2a8aad

[scala]: https://www.scala-lang.org/

[lihaoyi]: https://github.com/lihaoyi

[ac2019]: https://www.alpescraft.fr/edition_2019/

[ac2019talk]: https://www.youtube.com/watch?v=61AGIBdG7YE

[referenceconf]: https://github.com/dacr/code-examples-manager/blob/master/src/main/resources/reference.conf

[latest]: https://github.com/dacr/code-examples-manager/releases/latest

[rules]: https://github.com/dacr/the-rules-for-good-code-examples

[cs]: https://get-coursier.io/

[csget]: https://get-coursier.io/docs/cli-installation

[zio]: https://zio.dev/