Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/boomerang-io/flow.loader

Cloud-native workflows
https://github.com/boomerang-io/flow.loader

boomerang java mongo springboot

Last synced: about 6 hours ago
JSON representation

Cloud-native workflows

Awesome Lists containing this project

README 

        
# Flow Loader



> Note: this service only works with Java 11



Every migration starts with creating a ChangeSet (annotated with @ChangeSet). It contains the following attributes:



|Attribute|  Description| Mandatory? |

|--|--|--|

| id | Returns the ChangeSet's id that will be stored in the ChangeSet history table/collection and will the way to identify a ChangeSet. The combination of this field and the author must be unique among the changesets. |Yes |

| order |   Returns the ChangeSet's execution order.|Yes |

| author | Returns the ChangeSet's author. The combination of this and the author must be unique among the changesets. |No |



## Database Support



The Loader has been tested with MongoDB and Azure CosmosDB API for MongoDB.



There is an additional flag for supporting CosmosDB `-Dflow.mongo.cosmosdbttl=true` as [TTL indexes are a known limitation](https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/time-to-live). Additionally, support for CosmosDB means we can't use renameCollection() in our changesets. 



## ChangeLog and Lock Collections



The changelog and lock collections are defined in `/loader/src/main/java/net/boomerangplatform/migration/config/BoomerangFlowConfig.java`



## Creating Collections



https://www.mongodb.com/docs/manual/reference/method/js-database/



To create a collection, use the `db.createCollection()` method. 



```

    db.createCollection(collectionPrefix + "settings");

```



## Adding Documents to Collections

https://www.mongodb.com/docs/manual/reference/method/js-collection/



In `/loader/src/main/resources/flow`, there are several numbered folders. The numbering for the folder matches the changeset in which it is applied. 

 

For example,



```

  @ChangeSet(order = "026", id = "026", author = "Adrienne Hudson")

  public void createFlowSettings(MongoDatabase db) throws IOException {

    db.createCollection(collectionPrefix + "settings");



    final List files = fileloadingService.loadFiles("flow/026/flow_settings/*.json");

    for (final String fileContents : files) {

      final Document doc = Document.parse(fileContents);

      final MongoCollection collection = db.getCollection(collectionPrefix + "settings");

      collection.insertOne(doc);

    }

  }

```

where the documents added with changeset 026 are contained within the `flow/026/flow_settings/` folder in `/loader/src/main/resources/`



## Updating Documents in a Collection

https://www.mongodb.com/docs/manual/reference/method/js-collection/



Modifying existing documents requires identification of the document(s). 



```

  @ChangeSet(order = "064", id = "064", author = "Adrienne Hudson")

  public void updateWorker(MongoDatabase db) throws IOException {

    MongoCollection collection = db.getCollection(collectionPrefix + "settings");

    Document workers = collection.find(eq("name", "Task Configuration")).first();

    List configs = (List) workers.get("config");



    for (Document config : configs) {

      if (config.get("key").equals("worker.image")) {

        config.put("value", "boomerangio/worker-flow:2.8.11");

      }

    }



    workers.put("config", configs);

    collection.replaceOne(eq("name", "Task Configuration"), workers);

  }

```

In this example, we query to find the first document in the `collectionPrefix + "settings` collection with the `"name": "Task Configuration"` with the statement `Document workers = collection.find(eq("name", "Task Configuration")).first();`. Once the document has been identified, the config with `"key":"worker.image"` is updated with `"value":"boomerangio/worker-flow:2.8.11"`

and the update is written back onto the document. 



https://mongodb.github.io/mongo-java-driver/3.4/javadoc/org/bson/Document



```

  @ChangeSet(order = "073", id = "073", author = "Adrienne Hudson")

  public void migrateEnablePersistentStorage(MongoDatabase db) throws IOException {



    final MongoCollection flowWorkflowsCollection =

        db.getCollection(collectionPrefix + "workflows");

    final FindIterable flowWorkflows = flowWorkflowsCollection.find();

    for (final Document flowWorkflow : flowWorkflows) {



      boolean enablePersistentStorage = (boolean) flowWorkflow.get("enablePersistentStorage");



      Document storage =

          (Document) flowWorkflow.get("storage") != null ? (Document) flowWorkflow.get("storage")

              : new Document();



      Document workflowStorage =

          (Document) storage.get("workflow") != null ? (Document) storage.get("workflow")

              : new Document();



      workflowStorage.put("enabled", enablePersistentStorage);

      storage.put("workflow", workflowStorage);



      flowWorkflow.put("storage", storage);

      flowWorkflow.remove("enablePersistentStorage");



      flowWorkflowsCollection.replaceOne(eq("_id", flowWorkflow.getObjectId("_id")), flowWorkflow);



    }

  }

  ```

  In this example, the update applies to every document in the `collectionPrefix + "workflows"` collection. By using the `org.bson.Document.put(String key, Object value)` and `org.bson.Document.remove(Object key)`, object can be created, updated, or removed from a document.  



## Removing Documents in a Collection

https://www.mongodb.com/docs/manual/reference/method/js-database/



To remove a document in a collection, use the `db.collection.findOneAndDelete()` method. 



```

  @ChangeSet(order = "108", id = "108", author = "Adrienne Hudson")

  public void removeTemplates(MongoDatabase db) throws IOException {

    MongoCollection collection = db.getCollection(collectionPrefix + "workflows");

    collection.findOneAndDelete(eq("name", "Looking through planets with HTTP Call "));

    collection.findOneAndDelete(eq("name", "MongoDB email query results"));

  }

```



## How to Run



### Run Local MongoDB w Docker



```

docker run --name local-mongo -d mongo:latest

```



### CLI



```

mvn clean package



java -Dspring.profiles.active={profile} -jar target/loader.jar

```



If you are running a higher version of Java, you will need to locate the Java 11 version by running



```

/usr/libexec/java_home -V

```



Using the output you can then run the particular version, such as



```

/Library/Java/JavaVirtualMachines/ibm-semeru-open-11.jdk/Contents/Home/bin/java -Dspring.profiles.active=flow -jar target/loader.jar

```



### Docker



```

docker run -e JAVA_OPTS="-Dspring.data.mongodb.uri=mongodb://localhost:27017/boomerang -Dflow.mongo.collection.prefix=flow -Dspring.profiles.active=flow" --network host --platform linux/amd64 boomerangio/flow-loader:latest

```