{"id":13736264,"url":"https://github.com/NationalSecurityAgency/emissary","last_synced_at":"2025-05-08T12:32:31.988Z","repository":{"id":37752342,"uuid":"135743458","full_name":"NationalSecurityAgency/emissary","owner":"NationalSecurityAgency","description":"Distributed P2P Data-driven Workflow Framework","archived":false,"fork":false,"pushed_at":"2024-11-15T03:19:48.000Z","size":32375,"stargazers_count":247,"open_issues_count":10,"forks_count":122,"subscribers_count":36,"default_branch":"main","last_synced_at":"2024-11-15T03:28:13.010Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NationalSecurityAgency.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-06-01T17:02:18.000Z","updated_at":"2024-11-15T02:53:32.000Z","dependencies_parsed_at":"2023-10-10T18:48:47.683Z","dependency_job_id":"96a23eaf-e64e-40e2-9c4c-aa2179c3d174","html_url":"https://github.com/NationalSecurityAgency/emissary","commit_stats":null,"previous_names":[],"tags_count":78,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NationalSecurityAgency%2Femissary","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NationalSecurityAgency%2Femissary/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NationalSecurityAgency%2Femissary/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NationalSecurityAgency%2Femissary/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NationalSecurityAgency","download_url":"https://codeload.github.com/NationalSecurityAgency/emissary/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224732129,"owners_count":17360416,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T03:01:18.424Z","updated_at":"2025-05-08T12:32:31.979Z","avatar_url":"https://github.com/NationalSecurityAgency.png","language":"Java","funding_links":[],"categories":["Security Tools"],"sub_categories":[],"readme":"![Emissary Dark Knight - some code just wants to watch the core burn](emissary-knight.png) \n\n[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) [![Maven Central](https://maven-badges.herokuapp.com/maven-central/gov.nsa.emissary/emissary/badge.svg)](https://maven-badges.herokuapp.com/maven-central/gov.nsa.emissary/emissary) [![Java CI with Maven](https://github.com/NationalSecurityAgency/emissary/actions/workflows/maven-ci.yml/badge.svg)](https://github.com/NationalSecurityAgency/emissary/actions/workflows/maven-ci.yml) [![CodeQL](https://github.com/NationalSecurityAgency/emissary/actions/workflows/codeql-analysis.yml/badge.svg)](https://github.com/NationalSecurityAgency/emissary/actions/workflows/codeql-analysis.yml) [![Lint Codebase](https://github.com/NationalSecurityAgency/emissary/actions/workflows/linter.yaml/badge.svg)](https://github.com/NationalSecurityAgency/emissary/actions/workflows/linter.yaml)\n\nTable of Contents\n=================\n\n* [Introduction](#introduction)\n* [Minimum Requirements](#minimum-requirements)\n* [Getting Started](#getting-started)\n* [Contact Us](#contact-us)\n\n\n## Introduction\n\nEmissary is a P2P based data-driven workflow engine that runs in a heterogeneous \npossibly widely dispersed, multi-tiered P2P network of compute resources. Workflow \nitineraries are not pre-planned as in conventional workflow engines, but are discovered as \nmore information is discovered about the data. There is typically no user interaction in an \nEmissary workflow, rather the data is processed in a goal oriented fashion until it reaches \na completion state.\n\nEmissary is highly configurable, but in this base implementation\ndoes almost nothing. Users of this framework are expected to provide\nclasses that extend [emissary.place.ServiceProviderPlace](src/main/java/emissary/place/ServiceProviderPlace.java) to perform\nwork on [emissary.core.IBaseDataObject](src/main/java/emissary/core/IBaseDataObject.java) payloads.\n\nA variety of things can be done and the workflow is managed in\nstages, e.g. [STUDY](DEVELOPING.md#study), [ID](DEVELOPING.md#id), [COORDINATE](DEVELOPING.md#coordinate), \n[TRANSFORM](DEVELOPING.md#transform), [ANALYZE](DEVELOPING.md#analyze), [IO](DEVELOPING.md#io), \n[REVIEW](DEVELOPING.md#review).\n\nThe classes responsible for directing the workflow are the\n[emissary.core.MobileAgent](src/main/java/emissary/core/MobileAgent.java) and classes derived from it, which manage\nthe path of a set of related payload objects through the workflow and\nthe [emissary.directory.DirectoryPlace](src/main/java/emissary/directory/DirectoryPlace.java) which manages the available\nservices, their cost and quality and keep the P2P network connected.\n\n### Maven Site and Javadoc hosted on GitHub Pages: https://code.nsa.gov/emissary/\n\n## Minimum Requirements\n\n- Linux or MacOSX operating system\n- [JDK 11](https://docs.aws.amazon.com/corretto/latest/corretto-11-ug/downloads-list.html)\n- [Apache Maven 3.6.3+](http://maven.apache.org)\n\n## Getting Started\n\nRead through the [DEVELOPING.md](DEVELOPING.md) guide for information on installing required components, pulling the \nsource code, building and running Emissary.\n\n### Building\n\nRun ```mvn clean package``` to compile, test, and package Emissary\n\n```\n[INFO] ------------------------------------------------------------------------\n[INFO] BUILD SUCCESS\n[INFO] ------------------------------------------------------------------------\n[INFO] Total time:  9.132 s\n[INFO] Finished at: 2022-01-10T22:31:05Z\n[INFO] ------------------------------------------------------------------------\n```\n\n### Running\n\nThere is one bash script in Emissary that runs everything.  It is in the top level Emissary directory. The script runs \nthe [emissary.Emissary](src/main/java/emissary/Emissary.java) class which has several [Picocli](https://picocli.info/) \ncommands available to handle different functions.\n\n#### No arguments\n\nIf the *emissary* script is run without any arguments, you will get a listing \nof all the configuration subcommands and a brief description.\n\n```\n./emissary\n```\n\n#### Help\n\nRunning `./emissary help` will give you the same output as running with no arguments.  If you want to see more\ndetailed information on a command, add the command name after help.  For example, see all the \narguments with descriptions for the *server* command, run:\n\n```\n./emissary help server\n```\n\n#### Common parameters\n\nThe rest of commands all have *(-b or --projectBase)* arguments that can be set, but it must match PROJECT_BASE.\n\nThe config directory is defaulted to \u003cprojectBase\u003e/config\nbut can also be passed in with *(-c or --config)*.  When running from the git checkout, you should use\n*target* as the projectBase.  Feel free to modify config files in target/config before you start.\n\nLogging is handled by logback. You can point to a custom file with the *--logbackConfig* argument.\n\nSee the *help -c \u003ccommandName\u003e* for each command to get more info.\n\n#### Server (Standalone)\n\nThis command will start up an Emissary server and initialize all the places, \na pickup place, and drop off filters that are configured.  It will start in \nstandalone mode if *-m or --mode* is not specified.  By default, the number of \nMobileAgents is calculated based on the specs of the machine.  On modern computers, \nthis can be high.  You can control the number of agents with *-a or --agents*.  Here \nis an example run.\n\n```\n./emissary server -a 2\n```\n\nWithout further configuration, it will start on http://localhost:8001.  If you browse to that \nurl, you will need to enter the username and password defined in target/config/jetty-users.properties,\nwhich is emissary and emissary123.\n\nThe default PickUpPlace is configured to read files from _target/data/InputData_.  If you copy\nfiles into that directory, you will see Emissary process them.  Keep in mind, only toUpper and toLower are\nconfigured, so the output will not be too interesting.\n\n#### Agents (Standalone)\n\nThe agents command shows the number of MobileAgents for the configured host and what those\nagents are doing.  By default, the port is 9001, but you can use *-p or --port* to change that.  \nAssuming you are running on 8001 from the server command above, try:\n\n```\n./emissary agents -p 8001\n```\n\n#### Pool (Standalone)\n\nPool is a collapsed view of agents for a node.  It, too, defaults to port 9001.  To run for the \nstandalone server started above run\n\n```\n./emissary pool -p 8001\n```\n\nThis command is more useful for a cluster as it a more digestible view of every node.\n\n#### Env\n\nThe Env Command requires a server to be running.  It will ask the server for some configuration\nvalues, like PROJECT_BASE and BIN_DIR.  With no arguments, it will dump an unformatted json response.\n\n```\n./emissary env\n```\n\nBut you can also dump a response suitable for sourcing in bash.\n\n```\n./emissary env --bashable\n```\n\nStarting  the Emissary server actually calls this endpoint and dumps out $PROJECT_BASE}/env.sh \nwith the configured variables.  This is done so that shell scripts can `source $PROJECT_BASE}/env.sh` \nand then have those variable available without having to worry with configuring them elsewhere.\n\n#### Config\n\nThe config command allows you to see the effective configuration for a specified place/service/class. Since Emissary uses \nflavors, this command will show the resulting configuration of a class after all flavors have been applied. This command can\nbe used to connect to a running Emissary node by specifying the ```-h``` for host (default is localhost) and ```-p``` for\nthe port (default is 8001). To connect to a locally running Emissary on port 8001, any of the following commands will work:\n```\n./emissary config --place emissary.place.sample.ToLowerPlace\n./emissary config --place emissary.place.sample.ToLowerPlace -h localhost -p 8001\n```\n\nOptionally, you can specify offline mode using ```--offline``` to use the configuration files specified in your local \nCONFIG_DIR:\n```\n./emissary config --place emissary.place.sample.ToLowerPlace --offline\n```\n\nIn offline mode, you can provide flavors to see the differences in configurations:\n```\n./emissary config --place emissary.place.sample.ToLowerPlace --offline --flavor STANDALONE,TESTING\n```\n\nThese are useful to see the effective configuration, but we can also run in a verbose mode to \nsee all the configuration files along with the final output. This is controlled with the \n`--detailed` flag:\n```\n./emissary config --place emissary.place.sample.ToLowerPlace --detailed\n```\nor in offline mode:\n```\n./emissary config --place emissary.place.sample.ToLowerPlace --offline --detailed\n```\n\n#### Server (Cluster)\n\nEmissary is fun in standalone, but running cluster is more appropriate for real work.  The way to run clustered\nis similar to the standalone, but you need to *-m cluster* to tell the node to connect to other nodes.  In\nclustered mode Emissary will also start up the PickUpClient instead of the PickUpPlace, so you will need to\nstart a feeder.\n\nLook at the target/config/peers.cfg to see the rendezvous peers.  In this case, there are 3.  Nodes running\non port 8001 and 9001 are just Emissary nodes.  The node running on 7001 is the feeder.  So let's start up\n8001 and 9001 in two different terminals.\n\n```\n./emissary server -a 2 -m cluster\n./emissary server -a 2 -m cluster -p 9001\n```\n\nBecause these nodes all know about ports 8001, 9001 and 7001, you will see errors in the logs as they\ncontinue to try to connect.  \n\nNote, in real world deployments we don't run multiple Emissary processes on the same node.  You can configure the\nhostname with *-h*.\n\n#### Feed (Cluster)\n\nWith nodes started on port 8001 and 9001, we need to start the feeder.  The feed command uses port 7001 by default,\nbut we need to set up a directory that the feeder will read from.  Files dropped into that directory will be available \nfor worker nodes to take and the work should be distributed amongst the cluster.  Start up the feed with\n\n```\nmkdir ~/Desktop/feed1\n./emissary feed -i ~/Desktop/feed1/\n```\n\nYou should be able to hit http://localhost:8001, http://localhost:9001 and http://localhost:7001 in the browser and\nlook at the configured places.  Drop some files in the ~/Desktop/feed1 and see the 2 nodes process them.  It may \ntake a minute for them to start processing\n\n#### Agents (Cluster)\n\nAgents in clustered mode again shows details about the mobileAgents.  It starts at with the node you \nconfigure (localhost:9001 by default), then calls out to all nodes it knows about and gets the same \ninformation.  Run it with:\n\n```\n./emissary agents --cluster\n```\n\n#### Pool (Cluster)\n\nPool in clustered mode also does the same as pool in standalone.  It starts at the node (locahost:9001) by default\nthen goes to all the nodes it knows about and aggregates a collapsed view of the cluster.  Run it with\n\n```\n./emissary pool --cluster\n```\n\n#### Topology (Clustered)\n\nThe topology talks to the configured node (localhost:8001 by default) and talks to every node it knows about.\nThe response is what all those nodes know about, so you can build up a network topology of your cluster.\nRun it with \n\n```\n./emissary topology\n```\n\n#### Running server with SSL\n\nThe keystore and keystore password are in the [emissary.client.EmissaryClient-SSL.cfg](src/main/config/emissary.client.EmissaryClient-SSL.cfg) \nfile.  Included and configured by default is a sample keystore you can use for testing this functionality. We do not \nrecommend using the sample keystore in production environments.  To use your own keystore, change configuration values in the\n[emissary.client.EmissaryClient-SSL.cfg](src/main/config/emissary.client.EmissaryClient-SSL.cfg) file.\n\nStandalone\n\n```\n./emissary server -p 8443 --ssl --disableSniHostCheck\n```\n\nClustered\n```\n./emissary server -p 8443 --ssl --disableSniHostCheck --mode cluster\n./emissary server -p 9443 --ssl --disableSniHostCheck --mode cluster\nmkdir ~/Desktop/feed1\n./emissary feed -p 7443 --ssl --disableSniHostCheck -i ~/Desktop/feed1/\n````\n\n## Contact Us\n\n### General Questions\n\nIf you have any questions or concerns about this project, you can contact us at: EmissarySupport@uwe.nsa.gov\n\n### Security Questions\n\nFor security questions and vulnerability reporting, please refer to [SECURITY.md](SECURITY.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNationalSecurityAgency%2Femissary","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FNationalSecurityAgency%2Femissary","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNationalSecurityAgency%2Femissary/lists"}