{"id":15372097,"url":"https://github.com/xquery/ml-utils","last_synced_at":"2025-04-15T14:06:50.702Z","repository":{"id":141753315,"uuid":"75726639","full_name":"xquery/ml-utils","owner":"xquery","description":"Small collection of command line utilities for working with MarkLogic","archived":false,"fork":false,"pushed_at":"2017-06-06T14:10:32.000Z","size":9971,"stargazers_count":5,"open_issues_count":2,"forks_count":1,"subscribers_count":5,"default_branch":"develop","last_synced_at":"2025-04-15T14:06:33.650Z","etag":null,"topics":["cmake","database","graphing","marklogic"],"latest_commit_sha":null,"homepage":"https://github.com/xquery/ml-utils","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xquery.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-12-06T11:50:37.000Z","updated_at":"2021-11-04T22:50:21.000Z","dependencies_parsed_at":null,"dependency_job_id":"664e2bf3-5761-4f1d-9763-0441e3af679b","html_url":"https://github.com/xquery/ml-utils","commit_stats":{"total_commits":96,"total_committers":1,"mean_commits":96.0,"dds":0.0,"last_synced_commit":"5fef291aa47c51a7c20c8e52359a5f34c835247f"},"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xquery%2Fml-utils","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xquery%2Fml-utils/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xquery%2Fml-utils/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xquery%2Fml-utils/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xquery","download_url":"https://codeload.github.com/xquery/ml-utils/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249085436,"owners_count":21210267,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cmake","database","graphing","marklogic"],"created_at":"2024-10-01T13:49:34.540Z","updated_at":"2025-04-15T14:06:50.695Z","avatar_url":"https://github.com/xquery.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ml-utils\n[![Build Status](https://travis-ci.org/xquery/ml-utils.svg?branch=master)](https://travis-ci.org/xquery/ml-utils)\n\nSmall collection of home grown command line utilities for working with MarkLogic.\n\n\n* [ml-log](#ml-log): retrieve server logs\n* [ml-hist](#ml-hist): retrieve resource metrics (requires gnuplot for graphing)\n* [ml-status](#ml-status): retrieve resource status\n* [ml-xq](#ml-xq): evaluate xquery\n* [ml-js](#ml-js): evaluate javascript\n* [ml-load](#ml-load): load data\n* [ml-config](#ml-config): manage MarkLogic\n* [ml-browse](#ml-browse): browse resources\n\n__WARNING__- Use these utilities at your own risk they are not supported in any way.\n\n## Usage\nDownload [release](https://github.com/xquery/ml-utils/releases) appropriate for your operating system and run the included install script.\n\n### Quick Start\nAll tools look in your home directory for [.ml-utils](etc/.ml-utils) dotfile for MarkLogic connection details.\n\n1) Copy etc/.ml-utils dotfile to your home directory or create ~/.ml-utils with connection details, as follows\n\n```\n# MarkLogic connection details\nhost=localhost\nprotocol=http\nuser=admin\npass=admin\nport=8002\npath=/manage/v2\n```\n\n2) copy bin/* to your bin directory or amend PATH\n\n3) run utils\n\nretrieve history metrics\n```\n\u003e ./ml-hist -r forests -m device-space -f json -p raw\n```\n\nretrieve server logs\n```\n\u003e ./ml-log -n ErrorLog.txt\n```\n\nretrieve resource status\n```\n\u003e ./ml-status -r forests/Documents\n```\n\nevaluate xquery\n```\n\u003e ./ml-js \u003c test.js\n```\n\nevaluate js\n```\n\u003e ./ml-xq \u003c test.xq\n```\n\nload data\n```\n\u003e ./ml-load -u /mytest.json -f /etc/test.json -d Documents\n```\n\nrestart MarkLogic\n```\n\u003e ml-config restart-local-cluster\n```\n\ninstall MarkLogic (require defining ml-config directory path)\n```\n\u003e ml-config install\n```\n\nIf your .ml-utils is not in home directory then you will need to suppy via -c commandline\noption.\n\n```\n\u003e ./ml-xq -c etc/.ml-utils \u003c test.xq\n\n```\n\n## Utilities\n\n### ml-hist\u003ca name=\"ml-hist\"\u003e\u003c/a\u003e\nretrieve history metrics from cluster/server\n\n__NOTE__ - To use graphing capabilities you must install gnuplot (ex. yum install gnuplot).\n\n```\n\u003e./ml-hist -h\n\nml-hist [options]\n    -r    : resources/{resource-name}\n    -c    : config file (ex. /home/jfuller/.ml-utils)\n    -f    : format (xml|json)\n    -p    : period (raw|hour|day)\n    -s    : start date (ex. 2015-03-21T17:38:00)\n    -e    : end date (ex. 2015-03-22T17:58:00)\n    -m    : meter (resource specific)\n    -r    : resource (databases|forests|servers|hosts) / resource-name\n    -o    : graph output destination\n    -g    : custom gnuplot script\n    -v    : verbose (show http call)\n    -q    : quiet (suppress banner)\n```\n\ngenerate jpg graph (note- must use json format)\n```\n\u003e./ml-hist -f json -r forests -m active-fragment-count -o test.jpg\n```\n\ngenerate text graph without any data output (-q option)\n```\n\u003e./ml-hist -f json -r forests -m active-fragment-count -o test.txt -q\n```\n\nuse custom gnuplot with start and end time range\n```\n\u003e  ./ml-hist -f json -s 2015-03-21T17:38:00 -e 2017-03-21T17:38:00 -g ../../etc/basic.gnuplot -r servers/Manage -m request-time -o test.png\n```\n\n### ml-status\u003ca name=\"ml-status\"\u003e\u003c/a\u003e\nretrieve resource statuses\n\n```\n\u003e ./ml-status -h\n\nml-status [options]\n    -r    : resources/{resource-name}\n    -c    : config file (ex. /home/jfuller/.ml-utils)\n    -f    : format (xml|json)\n    -v    : verbose (show http call)\n    -q    : quiet (suppress banner)\n\n```\n\ncluster status (specifying ml-utils config file with -c option)\n```\n\u003e ./ml-status  -c etc/.ml-utils -f json\n```\n\nall forest status\n```\n\u003e ./ml-status -r forests -c etc/.ml-utils -f json\n```\n\nforest status\n```\n\u003e ./ml-status -r forests/Documents -c etc/.ml-utils -f json\n```\n\ndatabase status\n```\n\u003e ./ml-status -r databases/Documents -c etc/.ml-utils -f json\n```\n\nall server status\n```\n\u003e ./ml-status -r servers -c etc/.ml-utils -f json\n```\n\nall host status\n```\n\u003e ./ml-status -r hosts -c etc/.ml-utils -f json\n```\n\n```\n\u003e ./ml-status -r groups/Default -c etc/.ml-utils -f json\n```\n\n### ml-log\u003ca name=\"ml-log\"\u003e\u003c/a\u003e\nretrieve MarkLogic logs\n\n```\n\u003e ./ml-log -h\n\nml-log\n    -c    : config file (ex. /home/jfuller/.ml-utils)\n    -f    : format(xml | json)\n    -t    : host (ex. localhost)\n    -n    : filename (ex. ErrorLog.txt)\n    -s    : start (ex. 2016-10-27T06:00:00)\n    -e    : end (ex. 2016-10-27T06:00:00)\n    -r    : regex (ex. Merged)\n    -v    : verbose (show http call)\n    -q    : quiet (suppress banner)\n```\n\nget log file list\n```\n\u003e ./ml-log\n```\n\nretrieve log file\n```\n\u003e ./ml-log -n ErrorLog.txt\n```\n\nretrieve ErrorLog.txt from specific host node1\n```\n\u003e ./ml-log -n ErrorLog.txt -t node1\n```\n\nretrieve ErrorLog.txt from specific host node1 searching with regex 'Del*'\n```\n\u003e ./ml-log -n ErrorLog.txt -t node1 -r Del*\n```\n\nsearch logs with regex 'Merged' with a start and end time range\n```\n\u003e ./ml-log -n ErrorLog.txt -t node1 -r Merged -s 2016-10-27T06:00:00 -e 2016-10-27T06:00:00\n```\n\nwatch (like tail) a log\n```\n watch \"ml-log -n 8002_ErrorLog.txt -t localhost\"\n```\n\n### ml-load\u003ca name=\"ml-load\"\u003e\u003c/a\u003e\nload data into MarkLogic\n\n```\n\u003e ./ml-load -h\n\nml-load [options] - insert data into database\n    -c    : config file (ex. /home/jfuller/.ml-utils)\n    -u    : uri\n    -f    : filename to upload\n    -d    : database\n    -v    : verbose (show http call)\n    -q    : quiet (suppress banner)\n\n```\n\nload test.json to /mytest.json into Documents database\n```\n\u003e ./ml-load -u /mytest.json -f /etc/test.json -d Documents\n```\n\nload test.xml to /test.xml into Documents database\n```\n\u003e./ml-load -u /test.xml -d Documents -f ../../etc/test.xml\n```\n\n### ml-js\u003ca name=\"ml-js\"\u003e\u003c/a\u003e\nevaluate javascript\n\n```\n\u003e./ml-js -h\n\nml-js database [options] - query database\n    -c    : config file (ex. /home/jfuller/.ml-utils)\n    -f    : format (xml|json)\n    -d    : database\n    -r    : raw output\n    -v    : verbose (show http call)\n    -q    : quiet (suppress banner)\n```\n\nsimple example\n```\n\u003e ./ml-js -j 'xdmp.random()'\n```\n\napply against database (specifying where config file resides with -c)\n```\n\u003e ./ml-js -c etc/.ml-utils -j 1+1 -d Documents\n```\n\npipe in\n```\n\u003e echo \"1+1\" | ./ml-js\n```\n\nevaluate file\n```\n\u003e ./ml-js \u003c text.js\n```\n\n### ml-xq\u003ca name=\"ml-xq\"\u003e\u003c/a\u003e\nevaluate xquery\n\n```\n\u003e ./ml-xq -h\n\nml-xq database [options] - query database\n    -c    : config file (ex. /home/jfuller/.ml-utils)\n    -f    : format (xml|json)\n    -d    : database\n    -r    : raw output\n    -v    : verbose (show http call)\n    -q    : quiet (suppress banner)\n\n```\n\nsimple example\n```\n\u003e ./ml-xq -x 'xdmp:random()'\n```\n\napply against database\n```\n\u003e ./ml-xq -x 1+1 -d Documents\n```\n\npipe in\n```\n\u003e echo \"1+1\" | ./ml-xq\n```\n\nevaluate file\n```\n\u003e ./ml-xq \u003c text.xq\n```\n\nmanaging sequence output\n```\n\u003e echo \"(1,2,3,4)\" | ./ml-xq -r\n```\n\n### ml-config\u003ca name=\"ml-config\"\u003e\u003c/a\u003e\nmanage MarkLogic\n\n\n```\n\u003e ./ml-config -h\n\nml-config commands [options]\n commands : restart-local-cluster | get | get-properties | create | update | install\n    -c    : config file (ex. /home/jfuller/.ml-utils)\n    -r    : {resources}/{resource-name} directory\n    -v    : verbose (show http call)\n    -q    : quiet (suppress banner)\n\n```\n\nrestart MarkLogic cluster\n```\n\u003e ./ml-config restart-local-cluster\n```\n\nget list of databases\n```\n\u003e./ml-config get -r databases\n\n```\n\ncreate forest with name 'myForest'\n```\n\u003e./ml-config create -r forests -n myForest\n```\n\nupdate forest\n```\n\u003e ./ml-config update -r forests/myForest \u003c etc/ml-config/forests/forest.json\n```\n\nget forest 'myForest' properties\n```\n\u003e./ml-config get-properties -r forests/myForest\n```\n\ninstall set of resources based on ml-config directory containing properties\n```\n\u003e./ml-config install\n```\n\n### ml-browse\u003ca name=\"ml-browse\"\u003e\u003c/a\u003e\nTBD browse resources\n\n```\n\u003e ./ml-browse -h\n\n\u003eml-browse [options] - browse resources\n    -c    : config file (ex. /home/jfuller/.ml-utils)\n    -u    : uri\n    -f    : filename to upload\n    -r    : resources\n    -v    : verbose (show http call)\n    -q    : quiet (suppress banner)\n\n```\n\n## Build and deploy\n\nTo build you may have to install some additional deps:\n```\nyum install epel-release\nyum install cmake3\nyum install curl-devel\n```\n\nTo build this set of utilities on linux, osx and windows platforms.\n\n```\n\u003ecmake3 -DCMAKE_BUILD_TYPE=Debug -G \"CodeBlocks - Unix Makefiles\"\n```\n\nNote that running cmake will pull down dependencies.\n\nthen run make, make install.\n\nand to create a release package\n```\n\u003ecpack3 --config CPackConfig.cmake\n```\n\n### Dependencies\nThis project uses the following libs:\n\n* [MarkLogic](https://developer.marklogic.com): requires MarkLogic 8 or greater\n* [curl](https://curl.haxx.se/): for http communication\n* [rapidjson](https://github.com/miloyip/rapidjson): for json munging\n* [loguru](https://github.com/emilk/loguru): for logging\n* [gnuplot-cpp](https://github.com/orbitcowboy/gnuplot-cpp): for speaking to gnuplot\n* [googletest](https://github.com/google/googletest): for testing\n\n## Examples\nThe [examples](examples) folder contains a sample configurations and shellscripts.\n\n## License\n\n[Apache License v2.0](LICENSE)\n\n## Background\nThis project was originally an unofficial little prototype to put MarkLogic Management REST API through its paces, to\nsee how easy it would be to leverage. As it has grown in usefulness (to me), I thought I would\nrelease to wider world. All PR's reviewed and appreciated. Note to self - write more unit tests!\n\nAdditionally, I wanted to try out a few modern twists\n\n* cmake: using cmake in a non trivial project. Using cpack\nto manage releases makes life easier and the DownloadProject pulling in dependencies for building is\na great tool. Cmake is sufficient for building cross platform apps in spite of my 'Makefile muscle memory'.\n\n* clion: instead of emacs I used clion ... it was 'ok'\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxquery%2Fml-utils","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxquery%2Fml-utils","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxquery%2Fml-utils/lists"}