{"id":19745726,"url":"https://github.com/dataoneorg/d1_python","last_synced_at":"2025-04-09T20:13:09.926Z","repository":{"id":37686551,"uuid":"60103877","full_name":"DataONEorg/d1_python","owner":"DataONEorg","description":"Python components for DataONE clients and servers","archived":false,"fork":false,"pushed_at":"2024-11-22T19:31:08.000Z","size":94709,"stargazers_count":17,"open_issues_count":22,"forks_count":10,"subscribers_count":19,"default_branch":"master","last_synced_at":"2025-03-29T14:46:57.817Z","etag":null,"topics":["client","library","python3","repository"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DataONEorg.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-05-31T16:01:00.000Z","updated_at":"2024-11-22T19:31:12.000Z","dependencies_parsed_at":"2024-12-07T17:10:16.612Z","dependency_job_id":"d85494c6-2ff9-4968-aecc-ac8cbad842d9","html_url":"https://github.com/DataONEorg/d1_python","commit_stats":{"total_commits":3261,"total_committers":18,"mean_commits":"181.16666666666666","dds":"0.48850045998160074","last_synced_commit":"d72a9461894d9be7d71178fb7310101b8ef9066a"},"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataONEorg%2Fd1_python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataONEorg%2Fd1_python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataONEorg%2Fd1_python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataONEorg%2Fd1_python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DataONEorg","download_url":"https://codeload.github.com/DataONEorg/d1_python/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247284944,"owners_count":20913704,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["client","library","python3","repository"],"created_at":"2024-11-12T02:11:03.258Z","updated_at":"2025-04-09T20:13:09.897Z","avatar_url":"https://github.com/DataONEorg.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## d1_python\n\nPython components for DataONE clients and servers.\n\nSee the [documentation on ReadTheDocs](http://dataone-python.readthedocs.io/en/latest/).\n\n[![Build Status](https://travis-ci.org/DataONEorg/d1_python.svg?branch=master)](https://travis-ci.org/DataONEorg/d1_python)\n[![Coverage Status](https://coveralls.io/repos/github/DataONEorg/d1_python/badge.svg?branch=master)](https://coveralls.io/github/DataONEorg/d1_python?branch=master)\n[![Documentation Status](https://readthedocs.org/projects/dataone-python/badge/?version=latest)](http://dataone-python.readthedocs.io/en/latest/?badge=latest)\n[![PyPI version](https://badge.fury.io/py/dataone.common.svg)](https://badge.fury.io/py/dataone.common)\n\n### v2 and v1 API\n\n* DataONE Generic Member Node:\n[PyPI](https://pypi.python.org/pypi/dataone.gmn) \u0026ndash;\n[Docs](http://dataone-python.readthedocs.io/en/latest/gmn/index.html)\n* DataONE Client Library for Python:\n[PyPI](https://pypi.python.org/pypi/dataone.libclient) \u0026ndash;\n[Docs](http://dataone-python.readthedocs.io/en/latest/client/index.html)\n* DataONE Common Library for Python: \u0026ndash;\n[PyPI](https://pypi.python.org/pypi/dataone.common) \u0026ndash;\n[Docs](http://dataone-python.readthedocs.io/en/latest/common/index.html)\n* DataONE Test Utilities:\n[PyPI](https://pypi.python.org/pypi/dataone.test_utilities) \u0026ndash;\n[Docs](http://dataone-python.readthedocs.io/en/latest/test/index.html)\n\n### v1 API\n\n* DataONE Command Line Client (CLI):\n[PyPI](https://pypi.python.org/pypi/dataone.cli) \u0026ndash;\n[Docs](http://dataone-python.readthedocs.io/en/latest/cli/index.html)\n* DataONE ONEDrive:\n[PyPI](https://pypi.python.org/pypi/dataone.onedrive) \u0026ndash;\n[Docs](http://dataone-python.readthedocs.io/en/latest/onedrive/index.html)\n* DataONE Certificate Extensions:\n[PyPI](https://pypi.python.org/pypi/dataone.certificate_extensions)\n* DataONE Gazetteer:\n[PyPI](https://pypi.python.org/pypi/dataone.gazetteer)\n* DataONE Ticket Generator:\n[PyPI](https://pypi.python.org/pypi/dataone.ticket_generator)\n* Google Foresite Toolkit:\n[PyPI](https://pypi.python.org/pypi/google.foresite-toolkit)\n\n### Contributing\n\nPull Requests (PRs) are welcome! Before you start coding, feel free to reach out to us and let us know what you plan to implement. We might be able to point you in the right direction.\n\nWe try to follow [PEP8](https://www.python.org/dev/peps/pep-0008/).\n\nTo help keep the style consistent and commit logs, blame/praise and other code annotations accurate, we autoformat all source with Black, isort and docformatter. A script that wraps up the formatting is available at `./dev_tools/src/d1_dev/src-format.py`. Simply call it before commit.\n\n* [Black](https://black.readthedocs.io/en/stable/) - Standardized source formatting\n* [isort](https://github.com/timothycrosley/isort) - Sort and group imports\n* [docformatter](https://github.com/myint/docformatter) - PEP257 format docstrings \n* [Flake8](http://flake8.pycqa.org/en/latest/) - Lint, code and style validation\n\nConfiguration files for isort (`./.isort.cfg`) and Flake8 (`./.style.yapf`) are included, and show the formatting options we have selected.\n\n### Unit tests\n\nTesting is based on the [pytest](https://docs.pytest.org/en/latest/) unit test framework.\n \n#### Sample files\n\nMost of our tests work by serializing objects generated by the code being tested and comparing them with reference samples stored in files. This allows us to check all properties of generated objects without having to write asserts that check individual properties, eliminating a time consuming and repetitive part of the test writing process.\n\nWhen writing comparisons manually, one will often select a few properties to check, and when those are determined to be valid, the remaining values are assumed to be correct as well. By comparing complete serialized versions of the objects, we avoid such assumptions.\n\nBy storing the expected serialized objects in files instead of in the unit tests themselves, we avoid embedding hard coded documents inside the unit test modules and make it simple to automatically update the expected contents of objects as the code evolves.\n\nWhen unit tests are being run as part of CI or as a normal guard against regressions in a local development environment, any mismatches between actual and expected serialized versions of objects simply trigger test failures. However, when a test is initially created or the serialized version of an object is expected to change, tests can automatically write or update the sample files they use. This function is enabled by starting `pytest` with the `--sample-ask` switch. When enabled, missing or mismatched sample files will not trigger test failures, instead starting an interactive process where differences are displayed together with yes/no prompts for writing or updating the samples. By default, differences are displayed in a GUI window using `kdiff3`, which provides a nice color coded view of the differences.\n\nThe normal procedure for writing a sample based unit test is to just write the test as if the sample already exists, then running the test with `--sample-ask` and viewing and approving the resulting sample, which is then automatically written to a file. The sample file name is displayed, making it easy to find the file in order to add it to tracking so that it can be committed along with the test module.\n\nWhen working on large changes that cause many samples to become outdated, reviewing and approving samples can be deferred until the new code approaches stability. This is done by running the tests with `--sample-update`, which automatically writes or updates samples to match the current results. Then, view and approve the tests with `--sample-review` before committing.\n\nTypically, it is not desirable to track generated files in Git. However, although the sample files are generated, they are an integral part of the units tests, and should be tracked just like the unit tests themselves.\n\nAlso implemented is a simple process for cleaning out unused sample files. Sample files are often orphaned when their corresponding tests are removed or refactored. The process is activated with the `--sample-tidy` switch. When active, the test session starts by moving all sample files from their default directory, `test_docs`, to `test_docs_tidy`. As the sample files are accessed by tests, they are automatically moved back to `test_docs`, and any files remaining in `test_docs_tidy` after a complete test run can be untracked and deleted.\n\nWhen staging `test_docs`, stage the directory, so that new files are included, and deleted files get deleted on the server:\n\n    git add test_utilities/src/d1_test/test_docs\n    git commit -m 'Update samples'\n\n#### DataONE Client to Django test adapter\n\nGMN tests are based on an adapter that enables using d1_client with the Django test framework. The adapter mocks Requests to issue requests through the Django test client.\n\nDjango includes a test framework with a test client that provides an interface that's similar to that of an HTTP client, but calls Django internals directly. The client enables testing of most functionality of a Django app without actually starting the app as a network service.\n\nFor testing GMN's D1 REST interfaces, we want to issue the test requests via the D1 MN client. Without going through the D1 MN client, we would have to reimplement much of what the client does, related to formatting and parsing D1 REST requests and responses.\n\nThis module is typically used in tests running under django.test.TestCase and requires an active Django context, such as the one provided by `./manage.py test`.\n\n#### Command line switches\n\nWe have added some custom functionality to pytest which can be enabled by launching pytest with the following switches:\n\n  * `--sample-ask`: Enable a mode that display diffs and, after user confirmation, can automatically update or write new test sample documents on mismatches.\n\n  * `--pycharm`:\n   \n    * Automatically open files where errors occur and move the cursor to the line of the error\n    \n    * Show syntax highlighted diffs for scripts and data files using PyCharm's powerful diff viewer\n    \n    * Also requires the path to the PyCharm binary to be configured in `DEBUG_PYCHARM_BIN_PATH` in `./conftest.py`. \n\n  * See `./conftest.py` for implementation and notes.\n\n  * `parameterize_dict`: Support for parameterizing test functions by adding a dict class member containing parameter sets.\n\nNote: None of these switches can be used when running tests in parallel with xdist (`-n`, `--dist`, `--tx`).\n\n#### Aliases\n\nI have found the following aliases handy for running the tests. If using the `bash` shell, these can be added to `~/.bashrc`.\n\n* Run tests serially. If the test uses one or more samples, and the result of the test has changed since the last time the sample was updated, open a diff between the current result and the sample in the PyCharm diff viewer and prompt the user for how to handle the mismatch. Options are to update the sample to match the current result and continue, ignore the mismatch for now and continue, or fail the test. When a test fails, stop the test run and open the location of the test failure in PyCharm, with the cursor set to the last line that was executed before the error and is part of d1_python.\n\n      alias p='pytest --exitfirst --random-order-bucket=none --sample-ask --pycharm'\n\n* Same as `p`, but more verbose and disabled capturing of console output. Capturing causes output from tests that passed to be hidden, making it easier to find the output from tests that failed. However, it makes debugging harder, as no output is displayed while stepping through tests that have not yet failed.\n\n      alias pc='pytest --exitfirst --random-order-bucket=none --sample-ask --pycharm -vv --capture=no'\n\n* Run tests in parallel using all available CPU cores, and create a coverage report. This greatly speeds up test execution but cannot be used with the PyCharm integration and interactive sample updates. \n\n      alias pn='pytest -n auto --cov-report=term --cov-report=xml'\n\npytest searches for and runs all tests below the current directory, so starting pytest directly or via one of the aliases from the d1_python directory will run all tests.\n\nTo run a specific test module, add a path to the test module. E.g.:\n\n    p d1_python/lib_common/src/d1_common/tests/test_checksum.py\n \nTo run only a single test, add a filter on the test number. E.g.:\n\n    p d1_python/lib_common/src/d1_common/tests/test_checksum.py -k 1050\n\n#### Debugging tests with PyCharm\n\n* By default, the PyCharm `Run context configuration (Ctrl+Shift+F10)` will generate test configurations and run the tests under the native unittest framework in Python's standard library. This will cause the tests to fail, as they require pytest. To generate pytest configurations by default, set `Settings \u003e Tools \u003e Python Integrated Tools \u003e Default test runner` to pytest. See the [documentation](https://www.jetbrains.com/help/pycharm/2017.1/testing-frameworks.html) for details.\n\n* Generate and run a configuration for a specific test by placing the cursor on a test function name and running `Run context configuration (Ctrl+Shift+F10)`.\n\n* After generating the configuration, debug with `Debug (Shift-F9)`.\n\n* If running the tests outside of PyCharm, launching `pytest` with the `--pycharm` switch will cause `pytest` to attempt to move the cursor in PyCharm to the location of any tests failures as they occur. This should be used with the `--exitfirst` (`-x`) switch.\n\n* Stopping a test that has hit a breakpoint in PyCharm can cause the test database to be left around. On the next run, Django will then prompt the user to type \"yes\" to remove the database. The prompt appears in the PyCharm debug console output. To disable the prompt, go to `Run / Debug Configurations \u003e Edit Configurations \u003e Defaults \u003e Django tests \u003e Options` and add `--noinput`. See the [question on SO](https://stackoverflow.com/questions/34244171) for details.\n\n* `pytest` by default captures `stdout` and `stderr` output for the tests and only shows the output for the tests that failed after all tests have been completed. Since a test that hits a breakpoint has not yet failed, this hides any output from tests being debugged and also hides output from the debug console prompt (where Python script can be evaluated in the current context). To see the output while debugging, go to `Run / Debug Configurations \u003e Edit Configurations \u003e Defaults \u003e pytest \u003e Additional Arguments` and add `--capture=no`. Also add an environment variable `JB_DISABLE_BUFFERING` and set it to `--capture=no --exitfirst --verbose`. Verbosity can also be increased by adding one or more `-v`.\n\n* Each unit test is implicitly wrapped in a database transaction and I have not found a way around this. The effect is that it's cumbersome to check the current state of the database while at a breakpoint or stepping through tests. PyCharm's database tools will only see the database as it was before the test was started. The only workaround I've found is to manually issue queries from within the current context, using the PyCharm console. While stepping through the test, bring up the console,`View \u003e Tool Windows \u003e Python Console`, and click `Show Python Prompt`. Then submit queries with, e.g., `\u003e self.run_django_sql('select count(*) from app_scienceobject')`. Write them in the database console to get the code completion and other features, then copy it into a call in the Python console. If an invalid query is submitted, the current database transaction will be lost. If there is no output when running commands in the console, it's due to the output being captured by pytest. See above.\n\n* The settings in `settings_test.py` are optimized for testing and debugging, while the settings in `settings_template.py` are optimized for production. To use `settings_test.py` when debugging tests in PyCharm, go to `Run / Debug Configurations \u003e Edit Configurations \u003e Defaults \u003e pytest \u003e Environment variables`, add `DJANGO_SETTINGS_MODULE` and set it to `d1_gmn.settings_test`.\n\n* If the `requirements.txt` file is out of date, the `Package requirements` inspection in PyCharm will display a list of missing packages at the top of the screen. Do not follow PyCharm's suggestion to install the missing packages as it's probably the `requirements.txt` file that should be updated instead.  See the section describing how to update dependencies for more information.\n \n\n### Django\n\n* Testing of the GMN Django web app is based on pytest and [pytest-django](https://pytest-django.readthedocs.io/en/latest/).\n\n* The tests use `settings_test.py` for GMN and Django configuration.\n\n* pytest-django forces `settings.DEBUG` to `False` in `pytest_django/plugin.py`. To set `settings.DEBUG`, override it close to where it will be read, e.g., wit `@django.test.override_settings(DEBUG=True)`.\n\n\n#### Django database test fixture\n\nThe GMN tests run in the context of a database that has been prepopulated with randomized data. The fixture file for the database is a JSON file stored in\n\n    ./test_utilities/src/d1_test/test_docs/json/db_fixture.json.bz2\n\nAfter changing any of the ORM classes in models.py, the database test fixture must be regenerated. This will often cause sample files to have to be updated as well, by running the tests with `--sample-update`.\n\nGenerate the fixture file with:\n\n    ./gmn/src/d1_gmn/tests/mk_db_fixture.py\n\nFixtures can be loaded directly into the test database from the JSON files but it's much faster to keep an extra copy of the db as a template and create the test db as needed with Postgres' \"create database from template\" function. So we only load the fixtures into a template database and reuse the template. This is implemented in `./conftest.py`.\n\nCreate template database from fixture with:\n\n    ./gmn/src/d1_gmn/tests/mk_db_template.py\n\nThe template is reused between test runs.\n\nScience object bytes are stored on disk, so they are not captured in the db fixture. If a test needs `get()`, `getChecksum()` and `replica()` to work, it must first create the correct file in GMN's object store or mock object store reads. The bytes are predetermined for a given test PID. See `d1_test.d1_test_case.generate_reproducible_sciobj_str()` and `d1_gmn.app.util.sciobj_file_path()`.\n\n\n### Setting up the development environment\n\nThese instructions are tested on Linux Ubuntu 22.04 and should also work on close derivatives.\n\n#### Install packaged dependencies\n\n```shell\nsudo bash -c '\n  apt update\n  apt -fy dist-upgrade\n  apt install -y \\\n    build-essential \\\n    curl \\\n    gir1.2-gtk-4.0 \\\n    git \\\n    libbz2-dev \\\n    libcairo2-dev \\\n    libffi-dev \\\n    libgirepository1.0-dev \\\n    liblzma-dev \\\n    libncursesw5-dev \\\n    libreadline-dev \\\n    librsync-dev \\\n    libsmbclient-dev \\\n    libsqlite3-dev \\\n    libssl-dev \\\n    libxml2-dev \\\n    libxmlsec1-dev \\\n    libxslt1-dev \\\n    llvm \\\n    make \\\n    openssl \\\n    postgresql \\\n    postgresql-server-dev-all \\\n    python-setuptools \\\n    python3-dev \\\n    tk-dev \\\n    wget \\\n    xz-utils \\\n    zlib1g-dev\n'\n```\n\n#### Install pyenv\n\nIn general, the system version of Python should not be touched. E.g., avoid installing packages with `sudo pip`.\n\npyenv provides a handy way to download and build local versions of Python mostly without sudo and without modifying the system Python environment. The Python environments created and managed by pyenv are stored under `~/.pyenv` by default. pyenv automatically switches between Python environments based on the current directory.\n\nInstall:\n\n    curl https://pyenv.run | bash\n    \n* Follow the instructions on how to activate pyenv automatically.\n* Open a new shell.\n\n#### Set up a virtual environment for d1_python\n\nThis is the environment all of d1_python's packaged Python dependencies will be installed to. It provides the runtime environment for d1_python tests and utilities.\n\nThe `CONFIGURE_OPTS=--enable-shared` setting in the snippet is required for `mod_wsgi` to be able to run from the environment. \n\n    bash -c '\n        pyver=3.11.3\n        CONFIGURE_OPTS=--enable-shared pyenv install ${pyver}\n        pyenv virtualenv ${pyver} d1_python\n        pyenv activate d1_python\n        pip install --upgrade pip wheel\n    '\n\nSelect a location for the d1_python git repository. Change this as needed.\n\n    export d1path=~/dev/d1_python\n\nDownload the source from GitHub and install:\n\n    bash -c '\n        git clone https://github.com/DataONEorg/d1_python.git ${d1path}\n        cd ${d1path}\n        pyenv activate d1_python\n        ./dev_tools/src/d1_dev/setup-all.py --root . develop\n    '\n\n#### Postgres\n\n    sudo apt install --yes postgresql\n\nSet the password of the postgres superuser account:\n\n    sudo passwd -d postgres\n    sudo su postgres -c passwd\n\nWhen prompted for the password, enter a new superuser password (and remember it :-).\n\n    sudo -u postgres createdb -E UTF8 gmn2\n    sudo -u postgres createuser --superuser `whoami`\n\nPyCharm (and other IntelliJ based platforms), are not able to connect to database with local (UNIX) sockets. Postgres' convenient \"peer\" authentication type only works over local sockets. A convenient workaround for this is to set Postgres up to trust local connections made over TCP/IP.\n\n    sudo editor /etc/postgresql/14/main/pg_hba.conf\n\nAdd line:\n\n    host all all 127.0.0.1/32 trust\n\nA similar line for `scram-sha-256` may already be present and, if so, must be commented out:\n\n    # host all all 127.0.0.1/32 scram-sha-256\n\n#### Certificates\n\nCopy the OpenSSL config file from the repository.\n\nMake sure to update the `d1path` if necessary. \n\n    sudo bash -c '\n        export d1path=~/dev/d1_python\n        mv /etc/ssl/openssl.cnf /etc/ssl/openssl.cnf.bak\n        cp ${d1path}/gmn/src/d1_gmn/deployment/openssl.cnf /etc/ssl/openssl.cnf\n    '\n\n#### Tests\n\nRun the tests and verify that they all pass:\n\n    pyenv activate d1_python\n    pip install pytest\n    pytest\n\n#### PyPI\n\nSet up credentials for working with the DataONE account on PyPI:\n\nEdit `~/.pypirc`:\n\n    [server-login]\n    username: dataone\n    password: \u003csecret\u003e\n\n#### Running GMN under Apache\n\nThe setup above is sufficient for testing against GMN using HTTP and the Django test client, which is normally all that is required. However, if testing over HTTPS or in an environment that is closer to production is required, Apache can be set up to host GMN directly from its location in d1_python, using the d1_python virtual environment.\n\nIn such a setup, the GMN source can be open in an IDE and changes made active with an `service apache2 reload`.\n\nNote that `mod_wsgi` can only run from a Python environment compiled with `--enable-shared`, as done in the venv setup above.\n\nThe APT package version of `mod_wsgi` has been compiled to work with the APT package version of Python. The two must be compatible at the ABI level, and Apache can only load a single instance of `mod_wsgi`. So this uninstalls any installed `mod_wsgi` APT package before compiling a new version against the Python environment in which it will be used.\n\n\n    sudo bash -c '\n        service apache2 stop\n        apt remove libapache2-mod-wsgi*\n        apt-get update\n        apt-get install python libexpat1 apache2 apache2-utils ssl-cert apache2-dev\n        setfacl -m u:${SUDO_USER}:w /etc/apache2/mods-available/wsgi.load\n    '\n\n    bash -c '\n        pyenv activate d1_python\n        pip install mod_wsgi\n        mod_wsgi-express module-config \u003e\u003e /etc/apache2/mods-available/wsgi.load\n    '\n    \n    sudo bash -c '\n        a2enmod wsgi\n        service apache2 restart\n    '\n\n### Creating a new release\n\n#### Updating dependencies\n\nUpdate all packages managed by pip:\n\n    ./dev_tools/src/d1_dev/pip-update-all.py\n\nThe DataONE Python stack specifies the versions that were tested in CI builds before release as the lowest required versions, and allows any later versions to be installed as part of regular maintenance.\n\nCheck that there are no package version conflicts:\n\n    pip check \n\nAs updating the versions in the `setup.py` files manually is time consuming and error prone, a script is included that automates the task. The script updates the version information for the dependencies in the `setup.py` files to match the versions of the currently installed dependencies. Update the `setup.py` files with:\n\n    ./dev_tools/src/d1_dev/src-sync-dependencies.py . \u003cversion\u003e\n\nThe `\u003cversion\u003e` argument specifies what the version will be for the release. E.g., `\"2.3.1\"`. We keep the version numbers in sync between all of the packages in the d1_python git repository, so only one version string needs to be specified.\n\nThe current version can be found in any of the `setup.py` files and in the `VERSION` string in `./lib_common/src/d1_common/const.py`\n\nRun the tests, ensure that they pass and update sample files as necessary.\n\nThe `requirements.txt` file contains a list of packages and pinned versions that will be used in CI builds. It designates the exact Python environment in which the unit tests will run in CI builds.\n\nUpdate the `requirements.txt` file:\n\n    ./dev_tools/src/d1_dev/update-requirements-txt.py\n\n\nCommit and push the changes, and check the build on Travis.\n\n#### Building the release packages\n\nAfter successful build, clone a fresh copy, which will be used for building the release packages:\n\nBuilding the release packages from a fresh clone is a simple way of ensuring that only tracked files are released. It is a workaround for the way setuptools works, which is basically that it vacuums up everything that looks like a Python script in anything that looks like a package, which makes it easy to publish local files by accident.\n\nCreate a Python venv to use for build and deploy:\n\n* The package `setup.py` scripts will run in this venv.\n* The venv can be reused indefinitely.\n    \n    pyenv virtualenv \"x.y.z\" venv_build\n\n* Where \"x.y.z\" is one of the versions listed in `pyenv versions`.\n* Pick a version that is close or the same as the version of Python used for testing on Travis.\n\nBuild and publish the packages:\n\n* Download current master from GitHub, create binary wheel packages and push the packages to PyPI.\n\n    pyenv activate venv_build\n    \n    bash -c '\n      bdir=~/d1_python_build\n      rm -rf ${bdir}\n      pip install --upgrade pip\n      pip install --upgrade wheel    \n      git clone git@github.com:DataONEorg/d1_python.git ${bdir}\n      ${bdir}/dev_tools/src/d1_dev/setup-all.py --root ${bdir} bdist_wheel upload\n    '\n\n### Building the documentation\n\nWhen `d1_python` is pushed to GitHub, a signal is sent by GitHub to [ReadTheDocs.org](https://readthedocs.org/), which automatically retrieves the new version of the project from GitHub, builds the documentation and makes it available at\n\nhttp://dataone-python.readthedocs.io/en/latest/\n\nSo it is not absolutely necessary to have a local build environment set up for the documentation, but building locally provides faster feedback when making changes that need to be checked before publishing.\n\n### Troubleshooting\n\nClear out the installed libraries and reinstall:\n\n    sudo rm -rf /usr/local/lib/python2.7/dist-packages/d1_*\n    sudo nano /usr/local/lib/python2.7/dist-packages/easy-install.pth\n    Remove all lines that are: dataone.*.egg and that are paths to your d1_python.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdataoneorg%2Fd1_python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdataoneorg%2Fd1_python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdataoneorg%2Fd1_python/lists"}