Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/wallyqs/org-converge

Literate reproducible runs using Org mode and Ruby.
https://github.com/wallyqs/org-converge

Last synced: 2 months ago
JSON representation

Literate reproducible runs using Org mode and Ruby.

Awesome Lists containing this project

README 

        
# -*- mode: org; mode: auto-fill; -*-

#+STARTUP:	showeverything



* Org Converge



  [[https://secure.travis-ci.org/wallyqs/org-converge.png?branch=master]]



** Description



   A framework to create documented reproducible runs using [[http://orgmode.org/worg/org-contrib/babel/Org Babel][Org Mode]],

   borrowing several ideas of what is possible to do with tools

   like =chef-solo=, =rake=, =foreman= and =capistrano=.



** Install



   : gem install org-converge



** Motivation



   [[http://orgmode.org/worg/org-contrib/babel/Org Babel][Org Mode]] has proven to be flexible enough to write [[http://www.jstatsoft.org/v46/i03][reproducible research papers]],

   then I believe that configuring and setting up a server for

   example, is something that could also be done using

   the same approach, given that /converging/ the configuration

   is a kind of run that one ought to be able to reproduce.



   Taking the original Emacs implementation as reference,

   Org Converge uses the [[https://github.com/wallyqs/org-ruby][Ruby implementation of the Org mode parser]]

   to implement and enhance the functionality from Org Babel 

   by adding helpers to define the properties of a run while taking advantage

   of what is already there in the Ruby ecosystem.



** Usage examples



   Org Converge supports the following kind of runs below:



*** Parallel runs



    Each one of the code blocks is run on an independent process.

    This is akin to having a =Procfile= based application, where

    one of the runner command would be to start a web application

    and the other one to start the a worker processes.



    In the following example, we are defining 2 code blocks, one 

    that would be run using Ruby and one more that would be run in Python.

    Notice that the Python block has =:procs= set to 2, meaning that 

    it would spawn 2 processes for this.



#+begin_src sh

  ,#+TITLE: Sample parallel run

  　

  ,#+name: infinite-worker-in-ruby

  ,#+begin_src ruby

  $stdout.sync = true

  loop { puts "working!"; sleep 1; }

  ,#+end_src

  　

  ,#+name: infinite-worker-in-python

  ,#+begin_src python :procs 2

  import sys

  import time

  

  while True:

    print "working too"

    sys.stdout.flush()

    time.sleep(1)

  ,#+end_src

#+end_src



    The above example can be run with the following:

    

    : org-run procfile-example.org



    Sample output of the run:



#+begin_src sh

[2014-06-07T18:05:48 +0900] infinite-worker-in-ruby     -- started with pid 19648

[2014-06-07T18:05:48 +0900] infinite-worker-in-python:1 -- started with pid 19649

[2014-06-07T18:05:48 +0900] infinite-worker-in-python:2 -- started with pid 19650

[2014-06-07T18:05:48 +0900] infinite-worker-in-python:1 -- working too

[2014-06-07T18:05:48 +0900] infinite-worker-in-python:2 -- working too

[2014-06-07T18:05:48 +0900] infinite-worker-in-ruby     -- working!

[2014-06-07T18:05:49 +0900] infinite-worker-in-python:1 -- working too

[2014-06-07T18:05:49 +0900] infinite-worker-in-python:2 -- working too

#+end_src



*** Sequential runs



    In case the code blocks form part of a runlist that should be 

    ran sequentially, it is possible to do this by specifying the

    ~runmode=sequential~ option.



#+begin_src sh

  ,#+TITLE:   Sample sequential run

  ,#+runmode: sequential

  　

  ,#+name: first

  ,#+begin_src sh

  echo "first"

  ,#+end_src

  　

  ,#+name: second

  ,#+begin_src sh

  echo "second"

  ,#+end_src

  　

  ,#+name: third

  ,#+begin_src sh

  echo "third"

  ,#+end_src

  　

  ,#+name: fourth

  ,#+begin_src sh

  echo "fourth"

  ,#+end_src

#+end_src



  In order to specify that this is to be run sequentially, 

  we set the runmode option in the command line:



  : org-run runlist-example.org --runmode=sequential



  Another way of specifying this is via the Org mode file itself:



  #+begin_src org

  ,#+TITLE:   Defining the runmode as an in buffer setting 

  ,#+runmode: sequential

  #+end_src



  Sample output:



#+begin_src sh

  [2014-06-07T18:10:33 +0900] first     -- started with pid 19845

  [2014-06-07T18:10:33 +0900] first     -- first

  [2014-06-07T18:10:33 +0900] first     -- exited with code 0

  [2014-06-07T18:10:33 +0900] second    -- started with pid 19846

  [2014-06-07T18:10:33 +0900] second    -- second

  [2014-06-07T18:10:33 +0900] second    -- exited with code 0

  [2014-06-07T18:10:33 +0900] third     -- started with pid 19847

  [2014-06-07T18:10:33 +0900] third     -- third

  [2014-06-07T18:10:33 +0900] third     -- exited with code 0

  [2014-06-07T18:10:33 +0900] fourth    -- started with pid 19848

  [2014-06-07T18:10:33 +0900] fourth    -- fourth

  [2014-06-07T18:10:33 +0900] fourth    -- exited with code 0

#+end_src



*** Configuration management runs



    For example, using Org Babel tangling functionality we can spread

    config files on a server by writing the following on a ~server.org~ file...



#+begin_src sh

　

Configuration for a component that shoul be run in multitenant mode:

　

,#+begin_src yaml :tangle /etc/component.yml

multitenant: false

status_port: 10004

,#+end_src

#+end_src



    Then run:



   : sudo org-tangle server.org

   

   *Note:* The above tangle command is a reimplementation of the tangling functionality from Org mode using Ruby.

   If you are interested in tangling from the command line without losing functionality and have available a local Emacs Org mode install,

   the following project can be useful: 



*** Idempotent runs



    A run can have idempotency checks (similar to how the execute resource from [[http://docs.opscode.com/resource_execute.html][Chef]] works).



    An example of this, would be when installing packages.  In this example,

    we want to install the =build-essential= package once, and skip it in following runs:



#+begin_src sh

  ,** Installing the dependencies

  　

  Need the following so that ~bundle install~ can compile 

  the native extensions correctly.

  　　

  ,#+name: build-essential-installed

  ,#+begin_src sh

  dpkg -l | grep build-essential

  ,#+end_src

  　 　

  ,#+name: build_essentials

  ,#+begin_src sh :unless build-essential-installed

  apt-get install build-essential -y

  ,#+end_src

　　

  ,#+name: bundle_install

  ,#+begin_src sh

  cd project_path

  bundle install

  ,#+end_src

#+end_src



  Furthermore,since we are using Org mode syntax, it is possible 

  to reuse this setup file by including it into another Org file:



#+begin_src sh

  ,#+TITLE: Another setup

  　

  Include the code blocks from the server into this:

 　　

  ,#+include: "server.org"

  　

  ,#+name: install_org_mode

  ,#+begin_src sh

  apt-get install org-mode -y

  ,#+end_src

#+end_src

#+end_src



Since this a run that involves converging into a state,

it would be run sequentially with idempotency checks applied:



: sudo org-converge setup.org



*** Dependencies based runs



    In this type of runs we use the =:after= and =:before=

    header arguments to specify the prerequisites for a code block to run,

    similar to some of the functioality provided by tools like =rake= 

    (Behind the scenes, these arguments create =Rake= tasks)



    In order for this kind of run to work, it has to be specified

    what is the task that we are converging to by using 

    the =#+final_task:= in buffer setting:



#+begin_src sh

  ,#+TITLE:           Linked tasks example

  ,#+runmode:         tasks

  ,#+final_task:      final

  　

  ,#+name: second

  ,#+begin_src sh :after first

  for i in `seq 5 10`; do 

    echo $i >> out.log

  done

  ,#+end_src

  　

  ,#+name: first

  ,#+begin_src ruby

  5.times { |n| File.open("out.log", "a") {|f| f.puts n } }

  ,#+end_src

  　

  ,#+name: final

  ,#+begin_src python :after second :results output

  print "Wrapping up with Python in the end"

  f = open('out.log', 'a')

  f.write('11')

  f.close()

  ,#+end_src

  　

  ,#+name: prologue

  ,#+begin_src sh :before first :results output

  echo "init" > out.log

  ,#+end_src

#+end_src



    : org-run chained-example.org --runmode=chained



    Instead of using =--runmode= options, it is also possible to just declare in buffer

    that the Org file should be run chained mode.



    #+begin_src org

    ,#+TITLE:   Defining the runmode as an in buffer setting 

    ,#+runmode: chained

    #+end_src

    

    Sample output:



#+begin_src sh

[2014-06-07T18:14:25 +0900] Running final task: final

[2014-06-07T18:14:25 +0900] prologue  -- started with pid 20035

[2014-06-07T18:14:25 +0900] prologue  -- exited with code 0

[2014-06-07T18:14:25 +0900] first     -- started with pid 20036

[2014-06-07T18:14:26 +0900] first     -- exited with code 0

[2014-06-07T18:14:26 +0900] second    -- started with pid 20038

[2014-06-07T18:14:26 +0900] second    -- exited with code 0

[2014-06-07T18:14:26 +0900] final     -- started with pid 20040

[2014-06-07T18:14:26 +0900] final     -- Wrapping up with Python in the end

[2014-06-07T18:14:26 +0900] final     -- exited with code 0

#+end_src



*** Remote runs



    For any of the cases above, it is also possible to specify

    whether the code blocks should be run remotely on another node.

    This is done by using =:dir= in the code block header argument.



#+begin_src sh

  ,#+sshidentityfile:	vagrant/keys/vagrant

  ,#+name: remote-bash-code-block

  　

  ,#+begin_src sh :results output :dir /[email protected]#2222:/tmp

  random_number=$RANDOM

  for i in `seq 1 10`; do echo "[$random_number] Running script is $0 being run from `pwd`"; done

  ,#+end_src

#+end_src



  Note that in order for the above to work, it is also needed to set identity to be used by ssh.



*** Asserted runs



    In case the Org mode file has a results block which represents the expected result, 

    there is an ~org-spec~ command which can be useful to check whether there was a change

    that no longer makes the results from the Org file valid. Example:



#+begin_src sh

  ,#+TITLE:   Expected results example

  　　

  ,#+name: hello

  ,#+begin_src ruby :results output

  10.times do 

    puts "hola"

  end

  ,#+end_src

  　　

  ,#+RESULTS: hello

  ,#+begin_example

  hola

  hola

  hola

  hola

  hola

  hola

  hola

  hola

  hola

  hola

  ,#+end_example

#+end_src



  We can be able to verify whether this is still correct by running:



  : org-spec test.org



#+begin_src sh

Checking results from 'hello' code block:	OK

#+end_src



  As an example, let's say that the behavior of the original code block changed,

  and now says hello 5 times instead. 

  In that case the output would be as follows:



#+begin_src diff

Checking results from 'hello' code block:	DIFF

@@ -1,11 +1,6 @@

-hola

-hola

-hola

-hola

-hola

-hola

-hola

-hola

-hola

-hola

+hello

+hello

+hello

+hello

+hello

#+end_src



** Contributing



The project is still in very early development and a proof of concept at this moment.

But if you feel that it is interesting enough, please create a ticket to start

the discussion.