Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/ndexbio/ndexutils

Last synced: about 2 months ago
JSON representation

Host: GitHub
URL: https://github.com/ndexbio/ndexutils
Owner: ndexbio
Created: 2015-01-15T21:41:58.000Z (over 9 years ago)
Default Branch: master
Last Pushed: 2022-10-03T17:04:30.000Z (almost 2 years ago)
Last Synced: 2024-07-01T19:25:16.603Z (3 months ago)
Language: Python
Size: 27.4 MB
Stars: 2
Watchers: 9
Forks: 2
Open Issues: 1
Metadata Files:
- Readme: README.rst
- Changelog: HISTORY.rst
- Contributing: docs/contributing.rst

Awesome Lists containing this project

README

ndexutil
==========

.. _NDEx: https://ndexbio.org
.. _NDEx CX: https://www.home.ndexbio.org/data-model/

**Warning: This repository is for development and features may change.
Please use this at your own risk.**

.. image:: https://img.shields.io/pypi/v/ndexutil.svg
:target: https://pypi.python.org/pypi/ndexutil

.. image:: https://img.shields.io/travis/ndexbio/ndexutil.svg
:target: https://travis-ci.org/ndexbio/ndexutil

.. image:: https://coveralls.io/repos/github/ndexbio/ndexutils/badge.svg?branch=master
:target: https://coveralls.io/github/ndexbio/ndexutils?branch=master

.. image:: https://readthedocs.org/projects/ndexutils/badge/?version=latest
:target: https://ndexutil.readthedocs.io/en/latest/?badge=latest
:alt: Documentation Status

Dependencies
------------

* `ndex2 `_
* `networkx `_
* `biothings_client `_
* `requests `_
* `requests-toolbelt `_
* `pandas `_
* `mygene `_
* `jsonschema `_
* `urllib3 `_
* `ijson `_

Optional Dependencies
---------------------

Installed if ``pip install ndexutil[cytoscape]`` is run

* `py4cytoscape `_

Compatibility
-------------

* Python 3.4+

Installation
------------

.. code-block::

git clone https://github.com/ndexbio/ndexutils
cd ndexutils
make dist
pip install dist/ndexutil*whl

OR via `PyPI `_

.. code-block::

pip install ndexutil

ndexmisctools.py
-----------------

**WARNING:** Please consider this tool alpha and probably contains errors/bugs and could cause data corruption or loss. You have been warned

**ndexmisctools.py** lets caller perform some operations on `NDEx`_ via the
command line.
This tool follows format of ``ndexmisctools.py ``

For more information run ``ndexmisctools.py --help`` and ``ndexmisctools.py --help``

**COMMANDS**:

* **addnodeattrib** - adds an attribute to all nodes in a network

Credentials to `NDEx`_ server must be stored in the configuration (default ``~/.ndexutils.conf``)
and be formatted as follows:

.. code-block::

[addprofile]
user=bob
password=6ea8f0ab0b2e
server=public.ndexbio.org

The following command adds **foo** node attribute with value
``my new value`` to all nodes in the network ``9025480b-6fbc-4efe-9cd8-b575ce49dfda``

.. code-block::

ndexmisctools.py --profile addprofile addnodeattrib --uuid 9025480b-6fbc-4efe-9cd8-b575ce49dfda --name foo --value 'my new value'

.. warning::

This command changes the network in place, so it is a destructive call with NO undo

* **copynetwork** - copies `NDEx`_ network between accounts and even servers

For copying the source and destination credentials must be stored in the configuration (default ``~/.ndexutils.conf``)
and be formatted as follows:

.. code-block::

[mycopyprofile]
source_user=bob
source_password=6ea8f0ab0b2e
source_server=public.ndexbio.org
dest_user=smith
dest_password=4efe9cd8
dest_server=public.ndexbio.org

The following command copies the network ``9025480b-6fbc-4efe-9cd8-b575ce49dfda`` from source credentials defined in configuration to dest

.. code-block::

ndexmisctools.py --profile mycopyprofile copynetwork --uuid 9025480b-6fbc-4efe-9cd8-b575ce49dfda

* **cytoscapelayout** - updates layout for network in `NDEx`_ using Cytoscape

**NOTE:** This tool requires installation of `py4cytoscape `_
and Cytoscape to be activately running (tested with Cytsocape 3.8+)

This command requires four positional parameters. The first is the layout algorithm
to use followed by (``username``, ``password``, and ``server``) which are credentials for
`NDEx`_ server to update the layout on the network.

Any of these credential fields set to **'-'** will
force this tool to obtain the information from (default ``~/.ndexutils.conf``) configuration file
under the profile specified by the ``--profile`` field in this format:

.. code-block::

[]
user =
password =
server =

Using credentials from `myattrib` profile, the following command adds grid
layout to the network ``9025480b-6fbc-4efe-9cd8-b575ce49dfda``

.. code-block::

ndexmisctools.py --profile myattrib networkxlayout grid - - - --uuid 9025480b-6fbc-4efe-9cd8-b575ce49dfda

To get a list of available layouts printed to standard out, replace the layout with ``listlayout``. Just put
fake values in for credentials and any text for the --uuid flag:

.. code-block::

ndexmisctools.py cytoscapelayout listlayout x y z --uuid ignored

* **deletenetwork** - deletes a network or all networks under a given networkset from `NDEx`_

Credentials to `NDEx`_ server must be stored in the configuration (default ``~/.ndexutils.conf``)
and be formatted as follows:

.. code-block::

[deleteprofile]
user=bob
password=6ea8f0ab0b2e
server=public.ndexbio.org

The following command deletes the network with UUID ``9025480b-6fbc-4efe-9cd8-b575ce49dfda`` from `NDEx`_

.. code-block::

ndexmisctools.py --profile deleteprofile deletenetwork --uuid 9025480b-6fbc-4efe-9cd8-b575ce49dfda

* **uploadnetwork** - uploads network to `NDEx`_

This command uploads a network to `NDEx`_

Credentials must be stored in the configuration (default ``~/.ndexutils.conf``)
and be formatted as follows:

.. code-block::

[myattrib]
user=bob
password=6ea8f0ab0b2e
server=public.ndexbio.org

The following command uploads ``foo.cx`` network to `NDEx`_

.. code-block::

ndexmisctools.py --profile myattrib uploadnetwork foo.cx

* **featurednetworkreport** - generates report about featurend networks on `NDEx`_

This command requires one positional parameter ``server``.
For production use ``public.ndexbio.org``

If the positional parameter is set to **'-'**, it will
force this tool to obtain the information from (default ``~/.ndexutils.conf``) configuration file
under the profile specified by the ``--profile`` field in this format:

.. code-block::

[]
server =

The command below queries the production server for the featured networks json file
and then queries `NDEx`_ for all networks that are part of the featured networks.

.. code-block::

ndexmisctools.py featurednetworkreport public.ndexbio.org --output foo.csv

**NOTE:** If a networkset contains more then 500 networks, only the first 500 are examined

* **networkattribupdate** - updates network attributes on network in `NDEx`_

**WARNING:** Currently **name, version, and description** CANNOT be updated with this command.

Credentials must be stored in the configuration (default ``~/.ndexutils.conf``)
and be formatted as follows:

.. code-block::

[myattrib]
user=bob
password=6ea8f0ab0b2e
server=public.ndexbio.org

The following command updates **foo** network attribute on the network ``9025480b-6fbc-4efe-9cd8-b575ce49dfda``

.. code-block::

ndexmisctools.py --profile myattrib networkattribupdate --uuid 9025480b-6fbc-4efe-9cd8-b575ce49dfda --name foo --type string --value 'my new value'

* **networkxlayout** - updates layout for network in `NDEx`_ using networkx library

.. code-block::

[]
user =
password =
server =

By default this tool only updates the ``cartesianLayout`` aspect. To update the full network
use add the ``--updatefullnetwork`` flag.

Using credentials from `myattrib` profile, the following command adds spring
layout to the network ``9025480b-6fbc-4efe-9cd8-b575ce49dfda``

.. code-block::

ndexmisctools.py --profile myattrib networkxlayout spring - - - --uuid 9025480b-6fbc-4efe-9cd8-b575ce49dfda

* **removenodeattrib** - removes an attribute from all nodes in a network

Credentials to `NDEx`_ server must be stored in the configuration (default ``~/.ndexutils.conf``)
and be formatted as follows:

.. code-block::

[removeprofile]
user=bob
password=6ea8f0ab0b2e
server=public.ndexbio.org

The following command removes **foo** node attribute from all nodes
in the network ``9025480b-6fbc-4efe-9cd8-b575ce49dfda``

.. code-block::

ndexmisctools.py --profile removeprofile removenodeattrib --uuid 9025480b-6fbc-4efe-9cd8-b575ce49dfda --name foo

.. warning::

This command changes the network in place, so it is a destructive call with NO undo

* **systemproperty** - updates showcase, visibility, and indexing for single network or all networks in networkset in `NDEx`_

**NOTE:** ``--showcase`` has no effect if network visibility is ``private``

Credentials must be stored in the configuration (default ``~/.ndexutils.conf``)
and be formatted as follows:

.. code-block::

[myattrib]
user=bob
password=6ea8f0ab0b2e
server=public.ndexbio.org

The following command enables showcase and sets indexing to `meta` for network with id ``9025480b-6fbc-4efe-9cd8-b575ce49dfda``

.. code-block::

ndexmisctools.py --profile myattrib systemproperty --uuid 9025480b-6fbc-4efe-9cd8-b575ce49dfda --showcase --indexlevel meta

The following command sets visibility to `public` for all networks in networkset with id ``e9580d43-ec14-4be8-9977-9de88e1d410a``

.. code-block::

ndexmisctools.py --profile myattrib systemproperty --networksetid e9580d43-ec14-4be8-9977-9de88e1d410a --visibility public

* **styleupdate** - update style of network in `NDEx`_

Credentials must be stored in the configuration (default ``~/.ndexutils.conf``)
and be formatted as follows:

.. code-block::

[styleprofile]
user=bob
password=6ea8f0ab0b2e
server=public.ndexbio.org

The following command updates style of network with UUID ``9025480b-6fbc-4efe-9cd8-b575ce49dfda`` with style from network specified by this UUID ``e9580d43-ec14-4be8-9977-9de88e1d410a``

.. code-block::

ndexmisctools.py --profile styleprofile styleupdate --uuid 9025480b-6fbc-4efe-9cd8-b575ce49dfda --styleuuid e9580d43-ec14-4be8-9977-9de88e1d410a

* **tsvloader** - Loads TSV files as networks into `NDEx`_

The **tsvloader** command loads an edge list file in tab separated format (hence TSV) and using a load plan, loads that data as a network into `NDEx `_.
This tool attempts to mimic behavior of the older ``tsv_uploader.py`` script located here: https://github.com/ndexbio/load-content
This new version uses the more memory efficient StreamTSVLoader.

This command requires five positional parameters.
The first three (``username``, ``password``, and ``server``) are credentials for
`NDEx`_ server to upload the network.

Any of these first three credential fields set to **'-'** will
force this tool to obtain the information from (default ``~/.ndexutils.conf``) configuration file
under the profile specified by the ``--profile`` field in this format:

.. code-block::

[]
user =
password =
server =

The forth positional parameter ``tsv_file`` (see _`TSV Loader` section below) should be
set to edge list file in tab separated format and the
fifth or last positional parameter ``load_plan`` should be
set to the load plan. The load plan is a JSON formatted text
file that maps the columns to nodes, edges, and attributes
in the network.

By default this tool does not generate much output to
standard out/error. For more verbosity add three to five ``-v`` parameters
to left of command name **tsvloader** as seen in examples below.

**NOTE:** If using ``--layout`` to run a Cytoscape layout, `py4cytoscape `_
must be installed and Cytoscape to be activately running (tested with Cytsocape 3.8+)

In example below the ``- - -`` tells the program to read the credentials
from the configuration file

.. code-block::

ndexmisctools.py -vvvv tsvloader - - - datafile.tsv load.plan

Here is an example where the name and description of the network is set
and the ``-t`` specifies a template network used to get style and in this case
since ``--copyattribs`` is set the network attributes (minus ``@context``) are
also copied to the new network

.. code-block::

ndexmisctools.py -vvv tsvloader bob xx public.ndexbio.org \
datafile.tsv loadplan.json --uppercaseheader \
-t dafe07ca-0676-11ea-93e0-525400c25d22 \
--name mynetwork --description 'some text' --copyattribs

In this example an alternate header is prepended via the ``--header`` flag.

**NOTE:** The ``--header`` flag does **NOT** remove an existing header

.. code-block::

ndexmisctools.py -vv --profile foo tsvloader - - public.ndexbio.org \
datafile.tsv loadplan.json \
--header 'col1 col2 col3' \
-t some_cx_file.cx \
-u 48a26aa0-0677-11ea-93e0-525400c25d22

If successful ``0`` is returned otherwise there was an error.

.. _TSV_Loader: .

TSV Loader
----------

This module contains the Tab Separated Variable Loader (TSV Loader, see `ndexutil/streamtsvloader.py` module) which generates
an `NDEx CX`_ file from a tab separated
text file of edge data and attributes.

To load data a load plan must be created. This plan tells the loader how to map the
columns in the file to nodes, and edges. This load plan needs to validate against
`this load plan JSON schema `_

**Example TSV file**

.. code-block::

SOURCE TARGET WEIGHT
ABCD AAA1 0.555
GGGG BBBB 0.305

**SOURCE** is the source node, **TARGET** is target node

A schema that could be:

.. code-block::

{
"source_plan":
{
"node_name_column": "SOURCE"
},
"target_plan":
{
"node_name_column": "TARGET"
},
"edge_plan":
{
"default_predicate": "unknown",
"property_columns": [
{
"column_name": "WEIGHT",
"attribute_name": "weight",
"data_type": "double"
}
]
}
}

Example below assumes the following:

* **./loadplan.json** is the load plan in JSON format
* **./style.cx** is a `NDEx CX`_ with a style.

.. code-block::

import ndex2
from ndexutil.tsv.streamtsvloader import StreamTSVLoader

# using ndex2 client library read CX file as NiceCXNetwork object
style_network = ndex2.create_nice_cx_from_file('./style.cx')

loader = StreamTSVLoader('./loadplan.json', style_network)
with open('./input.tsv', 'r') as tsvfile:
with open('./output.cx', 'w') as outfile:
loader.write_cx_network(tsvfile, outfile)

Credits
-------