{"id":19578590,"url":"https://github.com/materialsproject/mpenv","last_synced_at":"2025-04-27T06:33:57.403Z","repository":{"id":10170880,"uuid":"12254703","full_name":"materialsproject/MPenv","owner":"materialsproject","description":"create a virtual environment for running FireWorks within Materials Project","archived":false,"fork":false,"pushed_at":"2016-11-07T15:36:44.000Z","size":203,"stargazers_count":2,"open_issues_count":1,"forks_count":13,"subscribers_count":42,"default_branch":"master","last_synced_at":"2025-04-04T22:41:45.716Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/materialsproject.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-08-20T21:29:31.000Z","updated_at":"2022-05-30T08:29:00.000Z","dependencies_parsed_at":"2022-08-31T03:11:01.920Z","dependency_job_id":null,"html_url":"https://github.com/materialsproject/MPenv","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/materialsproject%2FMPenv","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/materialsproject%2FMPenv/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/materialsproject%2FMPenv/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/materialsproject%2FMPenv/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/materialsproject","download_url":"https://codeload.github.com/materialsproject/MPenv/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251099351,"owners_count":21536146,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T07:12:05.675Z","updated_at":"2025-04-27T06:33:57.054Z","avatar_url":"https://github.com/materialsproject.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"=====\nMPenv\n=====\n\nThis codebase helps users create a virtual environment for running FireWorks\nwithin Materials Project in a semi-automated way.\n\n\nUser instructions\n=================\n\nWarnings\n--------\n\n1. If you are planning on running VASP or any other commercial code, make sure\n   you have communicated your license to NERSC and have access to running it.\n   The MPEnv will not give you access to any codes for which you are not a\n   licensed user.\n\n2. This only works on NERSC\n\n3. This only works if your NERSC shell is BASH, not CSH. Note that by default\n   NERSC often sets CSH. **Seriously, BASH needs to be your DEFAULT shell. You\n   cannot just start bash within CSH or virtualenv fails at NERSC.** To change\n   your default shell, login to nim.nersc.gov, go to 'Logins by host', then\n   'change login shell', then change your shells. Type `echo $SHELL` when\n   logged in to confirm that your shell is BASH (e.g. `/bin/bash`).\n\n4. After creating your environment, you can't move or rename it. If you need to\n   delete it see the instructions below.\n   \n5. Unfortunately, differences between python versions on cori, edison, and matgen\n   require you to install a separate environment for each machine.  However, you \n   can use the same database configuration files for multiple machines.  For example, you \n   might have wc_surfaces installed on matgen, and can then copy wc_surface_files to create\n   a new environment, say wc_surface_cori_files to install the environment wc_surface_cori.\n\nPart 1 - Install the MPenv code at NERSC and request an environment\n-------------------------------------------------------------------\n\n1. Log into Edison/Cori/Matgen. Note that the matgen nodes (\"Mendel\") are accessible \n   by logging into into ``matgen.nersc.gov`` to submit jobs to Mendel. If you cannot\n   log into ``matgen.nersc.gov``, run ``id`` on another NERSC system (Cori, Edison)\n   and check that you're in the ``matcomp`` group. If not, request to be added\n   to the group by sending an e-mail to an administrator (Patrick Huck). Hopper\n   has been retired as of January 2016.\n\n\n   Also, add the following line to your ``~/.bashrc.ext`` to export the\n   PIP_CERT environment variable to the path of the PEM file (see\n   http://stackoverflow.com/a/28724886)::\n\n    export PIP_CERT=/project/projectdirs/matgen/DigiCertHighAssuranceEVRootCA.pem\n\n2. Load necessary modules::\n\n    # Edison\n    module load python/2.7.9\n    module load numpy/1.9.2\n    module load virtualenv\n    module load virtualenvwrapper\n\n    # Cori\n    module load python\n    module load numpy\n    module load virtualenv\n\n    # matgen\n    module load vim\n    module unload intel\n    module load python/2.7.3\n    module swap numpy numpy/1.9.2\n    module swap scipy scipy/0.14.0\n    module load virtualenv\n\n  .. note::\n    See bullet #4 in Part 4 for the role of the intel module when updating\n    codes on matgen!\n\n3. Create virtual environment and install MPenv code::\n\n    mkdir admin_env\n    virtualenv admin_env\n    source admin_env/bin/activate\n    cd admin_env\n    eval `ssh-agent -s` \u0026\u0026 ssh-add \u003cpath-to-private-github-key\u003e\n    git clone git@github.com:materialsproject/MPenv.git\n    cd MPenv\n    python setup.py develop\n\n  .. note::\n   * If the virtualenv command fails, make sure you have set your *default*\n     shell to be BASH and not CSH.\n   * If the ``git clone`` command fails, make sure your SSH key for the NERSC\n     machine is registered under your GitHub username. This is done by typing\n     ``ssh-keygen -t dsa`` (hit enter at all prompts) and then copying your\n     ``~/.ssh/id_dsa.pub`` file to your Github account (log into github.com,\n     click account settings at top-right, then the 'SSH keys' section).\n\n3. Type ``which mpenv``. If the installation was successful, the system should\n   find an executable.\n\n4. Request an environment from an administrator (currently Patrick Huck). The current procedure is just to send an email with a\n   requested environment name, e.g. ``aj_vasp``. A good environment name should\n   look like ``A_B`` where ``A`` is your initials and ``B`` is some SHORT\n   description that will help you remember what the environment is for. another\n   example: ``wc_surfaces``.\n\n5. An administrator will create a suite of databases hosted at NERSC for you\n   and send you back an archive (a.k.a tarball), let's call this\n   ``aj_vasp_files.tar.gz``. *Do not rename or change this archive in any way*.\n\n6. Once you receive the tarball, move to the next part.\n\nPart 2 - Install MP codes at NERSC\n----------------------------------\n\n1. Upload the tarball you received from an admin (e.g.,\n   ``aj_vasp_files.tar.gz``) via ``scp`` to your home directory at NERSC, log\n   into Edison or matgen, and unpack it (i.e. ``tar -xvzf\n   aj_vasp_files.tar.gz``). Remember to not change this archive or the\n   resulting directory contents!\n\n2. Load the necessary modules::\n\n    # Edison\n    module load python/2.7.9\n    module load numpy/1.9.2\n    module load virtualenv\n    module load virtualenvwrapper\n\n    # Cori\n    module load python\n    module load numpy\n    module load virtualenv\n\n    # matgen\n    module load vim\n    module unload intel\n    module load python/2.7.3\n    module swap numpy numpy/1.9.2\n    module swap scipy scipy/0.14.0\n    module load virtualenv\n\n3. add GitHub ssh-key and activate the admin environment that allows you to use\n   ``mpenv``::\n\n    eval `ssh-agent -s` \u0026\u0026 ssh-add \u003cpath-to-private-github-key\u003e\n    source admin_env/bin/activate\n\n4. Now, you can install your environment. Staying in your home directory, type::\n\n    mpenv aj_vasp\n\n  .. note::\n   * Replace ``aj_vasp`` with whatever environment name you requested, e.g.\n     ``wc_surfaces``.\n   * There is a ``--pymatpro`` option if you need to install pymatpro (people\n     working with meta db builders might need this).\n   * See note in part 1 if ``git clone`` fails here.\n   * The ``rubicon`` git clone might still fail and claim a not-existing repo\n     if you don't have the correct permissions. Contact an administrator to be\n     granted access.\n\n5. A whole bunch of stuff will happen... just wait for it. Hopefully it will\n   succeed at the end and create a new directory with your environment name.\n\n6. Log out and in to NERSC again (or ``source ~/.bashrc.ext``).\n\n7. Activate your environment by typing ``use_\u003cENV_NAME\u003e``, e.g., ``use_aj_vasp``.\n\n8. If you don't need the data within them, reset your databases \n   by typing ``go_testing --clear -n 'reset'``.\n\nIf all this goes OK, your environment should be installed!\n\nPart 3 - Customize your environment\n-----------------------------------\n\nThere are many things about your environment that you can (and might have to)\ncustomize. Here are a few.\n\n1. Go to ``\u003cENV_NAME\u003e/config/config_\u003cMACHINE\u003e`` where ``\u003cENV_NAME\u003e`` is\n   something like ``aj_vasp`` and ``\u003cMACHINE\u003e`` is either ``Mendel``,\n   ``Hopper``, or ``Edison``. Modify ``my_qadapter.yaml`` so that queue scripts\n   are submitted to the queue you want with the walltime, mppwidth, and account\n   you want. You might want to change the queue to \"debug\" for example in order\n   to test your environment. If the ``account`` field says ``jcesr`` but you\n   are not a member of the ``jcesr`` NERSC repository, either delete the\n   ``account`` field or change to an account that you can charge at NERSC. If\n   you are using Hopper to run VASP, you *must* change the mppwidth to 48.\n   Repeat for all machines that you're using.\n\n2. Since ``Mendel`` is using SLURM, you'll need to add ``mpi_cmd`` to\n   ``my_fworker.yaml`` to run VASP on multiple nodes in parallel. Also\n   explicitly add the vasp command to be used on ``Mendel``::\n\n    env:\n        mpi_cmd: srun\n        vasp_cmd: /usr/common/usg/vasp/5.4.1/bin/vasp\n\n3. In your ``.bashrc.ext``, you'll want to add two lines (if not already done\n   by ``mpenv``)::\n\n    export VASP_PSP_DIR=\u003cPATH_TO_POTCARS\u003e\n    export MAPI_KEY=\u003cMAPI_KEY\u003e\n\n   where \u003cPATH_TO_POTCARS\u003e contains your POTCARs dir and MAPI_KEY is your\n   Materials Project API key. See the pymatgen docs for more details. Some\n   features of the code (e.g. VASP input generation) won't work without these.\n   Note that members of the ``matgen`` group at NERSC should be able to set\n   their \u003cPATH_TO_POTCARS\u003e as ``/project/projectdirs/matgen/POTCARs``.\n\n   Starting with pymatgen 5.0, these two environment variables should be saved\n   in ``~/.pmgrc.yaml``. You can make the simple switch via::\n\n   pmg config -a MAPI_KEY $MAPI_KEY VASP_PSP_DIR $VASP_PSP_DIR\n\n4. If you modify your ``bashrc.ext``, remember the changes are not applied\n   unless you type ``source ~/.bashrc.ext``.\n\nPart 4 - Modifying or updating your codebases\n---------------------------------------------\n\n.. note::\n  Currently this only seems to work on Hopper due to strange NERSC updates\n  messing with SSL certs.\n\n1. The codes installed with your environment are in ``\u003cENV_NAME\u003e/codes``. If\n   you modify these codes (e.g. change a workflow in MPWork's ``snl_to_wf()``\n   method) they will modify the behavior of your environment.\n\n2. Use the ``update_codes`` command to pull the latest changes from **all**\n   codes. **Be careful!** If there is a merge conflict or other problem, the\n   script won't tell you; you need to monitor the output to make sure the pull\n   completed OK.\n\n3. You can also ``git pull`` individually within the repos inside\n   ``\u003cENV_NAMES\u003e/codes``. If the version number changed, then you also need to\n   run ``python setup.py develop``.\n\n4. On matgen, you need to *unload* the intel module when trying to build and/or\n   install python code! However, make sure to load the intel module again\n   before running VASP jobs! Alternatively, you can log out and into NERSC\n   again or ``source ~/.bashrc.ext``.\n\nRunning Jobs\n============\n\nAfter getting your environment installed, you might want to run some test jobs.\nSee the `MPWorks page \u003chttps://github.com/materialsproject/MPWorks\u003e`_ for more\ndetails on how to do so.\n\nUpdating your admin environment\n===============================\n\nFrom time to time MPenv will have new features and you will want to update your\nadmin environment. This is different than updating the codes itself - it is\nupdating the code that *installs* the high-throughput codes. You can update\nMPenv without deleting any data you might have accumulated in your database\n(contact an admin if you want your DBs reset). However you should know that\nthis will delete any configuration updates you made to your environment (e.g.,\n``my_qadapter.yaml``). If you want to retain these changes, copy the files you\nneed to another directory and copy/merge them back after upgrading your admin\nenvironment.\n\nWhen you're ready to begin (logged into NERSC):\n\n1. Edit your ``.bashrc.ext`` file - look for the commented section referring to\n   your environment name and delete that section. This will be rewritten when\n   you reinstall the environment along with any new changes. ``mpenv`` will\n   abort if you forget to do this and if the respective section already exists\n   in ``.bashrc.ext``.\n\n2. Log out and in again to ensure a clean BASH environment.\n\n3. Load necessary modules::\n\n    # Edison\n    module load python/2.7.9\n    module load numpy/1.9.2\n    module load virtualenv\n    module load virtualenvwrapper\n\n    # Cori\n    module load python\n    module load numpy\n    module load virtualenv\n\n    # matgen\n    module load vim\n    module unload intel\n    module load python/2.7.3\n    module swap numpy numpy/1.9.2\n    module swap scipy scipy/0.14.0\n    module load virtualenv\n\n4. Add your GitHub sshkey and activate your admin environment::\n\n    eval `ssh-agent -s` \u0026\u0026 ssh-add \u003cpath-to-private-github-key\u003e\n    source admin_env/bin/activate\n\n5. Pull admin environment changes::\n\n    cd admin_env/MPenv\n    git pull\n\n6. Go back to your home directory and reinstall::\n\n    cd ~\n    mpenv aj_vasp\n\n  .. note:: Replace ``aj_vasp`` with whatever environment name you requested,\n  e.g. ``wc_surfaces``. Also, there is a ``--pymatpro`` option if you need to\n  install pymatpro (people working with meta db builders might need this).\n\n8. Log out and in to NERSC again, or ``source ~/.bashrc.ext``.\n\n9. Finally, remember to go back and make any configuration or code changes you need!\n\nDeleting your environment\n=========================\n\nIf you ever want to remove your environment completely (this is different than\nresetting DBs), you should:\n\n#. Contact an administrator to tear down the DB backends\n\n#. Remove the entire directory containing your environment AND your files (e.g.\n   ``aj_vasp`` and ``aj_vasp_files``)\n\n#. Edit your ``.bashrc.ext`` file - look for the commented section referring to\n   your environment name and delete that section.\n\nAdministrator instructions\n==========================\n\nCreating an admin_env\n---------------------\n\n#. Start by creating the admin_env from the instructions listed for users. You\n   might already have one installed if you've created an MPEnv in the past.\n\n#. You will need a directory called admin_env/MP_env/MP_env/private that\n   contains the DB credentials for making an environment. Obtain this from\n   someone who is currently an admin.\n\n#. Once you have the private dir in the correct spot, you have a working\n   admin_env!\n\nManaging an admin_env\n---------------------\n\n#. Activate your ``admin_env`` environment.\n\n#. ``cd`` in your admin_env/MP_env directory, and then run ``git pull`` and\n   (maybe) ``python setup.py develop``.\n\n#. Start in a directory where you archive all the environments that you've\n   made. For me, it is ``$HOME/envs``.\n\n#. Type ``mpdbmake \u003cENV_NAME\u003e \u003cTYPE\u003e`` where \u003cENV_NAME\u003e is the name the user\n   requested and \u003cTYPE\u003e is either ``FW`` or ``MP`` or ``rubicon``.\n\n#. Usually, I tar.gz the resulting DB files and send them to the user by email.\n   But other methods would also be OK. I keep a copy in my envs directory.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaterialsproject%2Fmpenv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaterialsproject%2Fmpenv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaterialsproject%2Fmpenv/lists"}