{"id":22410985,"url":"https://github.com/e3sm-project/esgf_metrics","last_synced_at":"2025-07-25T04:33:11.038Z","repository":{"id":41344275,"uuid":"468544984","full_name":"E3SM-Project/esgf_metrics","owner":"E3SM-Project","description":"A package that parses ESGF Apache logs for E3SM data request metrics.","archived":false,"fork":false,"pushed_at":"2024-05-28T19:59:55.000Z","size":95,"stargazers_count":2,"open_issues_count":9,"forks_count":0,"subscribers_count":135,"default_branch":"main","last_synced_at":"2025-03-27T02:45:53.480Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/E3SM-Project.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-03-10T23:39:18.000Z","updated_at":"2024-05-28T19:59:59.000Z","dependencies_parsed_at":"2024-01-14T23:05:57.169Z","dependency_job_id":"db320868-e2d8-4254-82c5-b50b6c6260b5","html_url":"https://github.com/E3SM-Project/esgf_metrics","commit_stats":{"total_commits":18,"total_committers":3,"mean_commits":6.0,"dds":"0.11111111111111116","last_synced_commit":"f7dc86d107345a5540361bcee1ec3160ee299c07"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/E3SM-Project/esgf_metrics","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/E3SM-Project%2Fesgf_metrics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/E3SM-Project%2Fesgf_metrics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/E3SM-Project%2Fesgf_metrics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/E3SM-Project%2Fesgf_metrics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/E3SM-Project","download_url":"https://codeload.github.com/E3SM-Project/esgf_metrics/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/E3SM-Project%2Fesgf_metrics/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264075390,"owners_count":23553504,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-05T13:12:38.044Z","updated_at":"2025-07-07T12:04:22.573Z","avatar_url":"https://github.com/E3SM-Project.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ESGF Metrics\n\nA repository that parses ESGF Apache Logs and generates E3SM file request metrics for\nNative and CMIP6 formats.\n\nMetrics include:\n\n- Cumulative number of requests\n- Cumulative GB of data downloaded\n\n## Usage\n\n1. Install Docker with docker-compose\n2. Clone this repository\n\n   ```bash\n   git clone https://github.com/tomvothecoder/esgf_metrics.git\n   ```\n\n3. Copy `.env.template` as `.env` and configure the environment variables\n4. Build the Docker containers using `docker-compose`. Containers include `postgres` and `esgf_metrics`.\n\n   ```bash\n   sudo docker-compose up --build\n   ```\n\n5. The `esgf_metrics` container will now automatically run the `esgf_metrics` package\n   using `crontab` at 8:00AM every Tuesday. It will identify new logs, parse them,\n   and generate updated metrics and plots.\n   - There is a separate cronjob on LLNL climate servers that collect access logs from\n     ESGF nodes every day at 10PM. Logs are stored in `/p/cscratch/esgf-http-logs`.\n   - All `esgf_metrics` parsed logs and metrics are stored in the `postgres` service's\n     Postgres database.\n\n## Helpful Commands\n\n- supervisorctl\n\n  ```bash\n  sudo supervisorctl stop all\n  sudo supervisorctl start all\n  sudo supervisorctl restart all\n  sudo supervisorctl status\n  supervisorctl tail -f esgf_metrics stdout\n  ```\n\n- systemctl\n\n  ```bash\n  sudo systemctl start docker\n  sudo systemctl stop docker\n  sudo systemctl restart docker\n  ```\n\n- Check service logs\n\n  ```bash\n  sudo docker-compose logs esgf_metrics\n  sudo docker-compose logs postgres\n  ```\n\n- Check crontab configuration\n\n  ```bash\n   sudo docker exec -ti esgf_metrics bash -c \"crontab -l\"\n  ```\n\n## Development\n\n1. Install Miniconda\n2. Create and activate the Conda environment\n\n   ```bash\n   cd esgf_metrics\n   conda env create -n conda-env/dev.yml\n   conda activate esgf_metrics_dev\n   ```\n\n3. Create a development branch\n\n   ```bash\n   git checkout -b dev-branch\n   ```\n\n4. Update source code and commit changes\n5. Push development branch and open a PR\n\n## How It Works\n\n```txt\n1) Read in logs, here's an example line:\n      \"128.211.148.13 - - [22/Sep/2019:12:01:01 -0700] \"GET /thredds/fileServer/user_pub_work/E3SM/1_0/historical/1deg_atm_60-30km_ocean/land/native/model-output/mon/ens1/v1/20180215.DECKv1b_H1.ne30_oEC.edison.clm2.h0.1850-01.nc HTTP/1.1\" 200 91564624 \"-\" \"Wget/1.14 (linux-gnu)\"\\n\"\n\n2) Split each log line into a list:\n      ['128.211.148.13',\n      '-',\n      '-',\n      '[22/Sep/2019:12:01:01',\n      '-0700]',\n      '\"GET',\n      '/thredds/fileServer/user_pub_work/E3SM/1_0/historical/1deg_atm_60-30km_ocean/land/native/model-output/mon/ens1/v1/20180215.DECKv1b_H1.ne30_oEC.edison.clm2.h0.1850-01.nc',\n      'HTTP/1.1\"',\n      '200',\n      '91564624',\n      '\"-\"',\n      '\"Wget/1.14',\n      '(linux-gnu)\"']\n\n3) Parse each log line for the directory:\n      \"/thredds/fileServer/user_pub_work/E3SM/1_0/historical/1deg_atm_60-30km_ocean/land/native/model-output/mon/ens1/v1/20180215.DECKv1b_H1.ne30_oEC.edison.clm2.h0.1850-01.nc\"\n\n4) Parse directory for the dataset id:\n\n      Before:\n      \"/E3SM/1_0/historical/1deg_atm_60-30km_ocean/land/native/model-output/mon/ens1/v1/\"\n\n      After:\n      # NOTE: Refer to the templates below for how to translate this\n      \"E3SM.1_0.historical.1deg_atm_60-30km_ocean.land.native.model-output.mon.ens1.v1\"\n\n\n5) Parse directory for file id:\n\n      \"20180215.DECKv1b_H1.ne30_oEC.edison.clm2.h0.1850-01.nc\"\n\n6) Parse for additional info (e.g., timestamp, facets)\n```\n\n## Templates for Parsing Logs\n\nThis list below includes an example log line from an Apache log and the project specific templates which can be used to parse log lines.\n\n### E3SM\n\n1. Example Log Line\n   `123.123.123.123 - - [22/Sep/2019:12:01:01 -0700] \"GET /thredds/fileServer/user_pub_work/E3SM/1_0/historical/1deg_atm_60-30km_ocean/land/native/model-output/mon/ens1/v1/20180215.DECKv1b_H1.ne30_oEC.edison.clm2.h0.1850-01.nc HTTP/1.1\" 200 91564624 \"-\" \"Wget/1.14 (linux-gnu)\"\\n`\n\n2. Directory Format Template\n\n   `%(source)s.%(model_version)s.%(experiment)s.%(grid_resolution)s.%(realm)s.%(regridding)s.%(data_type)s.%(time_frequency)s.%(ensemble_member)s`\n\n3. Dataset Template\n\n   `%(root)s/%(source)s/%(model_version)s/%(experiment)s/%(grid_resolution)s/%(realm)s/%(regridding)s/%(data_type)s/%(time_frequency)s/%`\n\n4. [Search API URL](https://esgf-node.llnl.gov/esg-search/search/?offset=0\u0026limit=0\u0026type=Dataset\u0026replica=false\u0026latest=true\u0026project=e3sm\u0026project=ACME\u0026facets=experiment%2Cscience_driver%2Crealm%2Cmodel_version%2Cregridding%2Ctime_frequency%2Cdata_type%2Censemble_member%2Ctuning%2Ccampaign%2Cperiod%2Catmos_grid_resolution%2Cocean_grid_resolution%2Cland_grid_resolution%2Cseaice_grid_resolution%2Cdata_node\u0026format=application%2Fsolr%2Bjson)\n\n### E3SM CMIP6\n\n1. Example Log Line\n\n   `123.123.123.123 - - [14/Jul/2019:06:58:07 -0700] \"GET /thredds/fileServer/user_pub_work/CMIP6/CMIP/E3SM-Project/E3SM-1-0/piControl/r1i1p1f1/Lmon/tran/gr/v20180608/tran_Lmon_E3SM-1-0_piControl_r1i1p1f1_gr_000101-050012.nc HTTP/1.1\" 206 1573717 \"-\" \"Wget/1.20.1 (linux-gnu)`\n\n2. Directory Format Template\n\n   `%(root)s/%(mip_era)s/%(activity_drs)s/%(institution_id)s/%(source_id)s/%(experiment_id)s/%(member_id)s/%(table_id)s/%(variable_id)s/%(grid_label)s/%(version)s`\n\n3. Dataset ID Template\n\n   `%(mip_era)s.%(activity_drs)s.%(institution_id)s.%(source_id)s.%(experiment_id)s.%(member_id)s.%(table_id)s.%(variable_id)s.%(grid_label)s`\n\n4. [Search API URL](https://esgf-node.llnl.gov/esg-search/search/?offset=0\u0026limit=0\u0026type=Dataset\u0026replica=false\u0026latest=true\u0026institution_id=E3SM-Project\u0026project=CMIP6\u0026facets=mip_era%2Cactivity_id%2Cmodel_cohort%2Cproduct%2Csource_id%2Cinstitution_id%2Csource_type%2Cnominal_resolution%2Cexperiment_id%2Csub_experiment_id%2Cvariant_label%2Cgrid_label%2Ctable_id%2Cfrequency%2Crealm%2Cvariable_id%2Ccf_standard_name%2Cdata_node\u0026format=application%2Fsolr%2Bjson)\n\nE3SM CMIP6 Variables Guideline\n\n- Example Log Line\n\n  `123.123.123.123 - - [18/Jul/2019:00:52:54 -0700] \"GET /thredds/fileServer/user_pub_work/E3SM/1_0/cmip6_variables/piControl/CMIP6/CMIP/E3SM-Project/E3SM-1-0/piControl/r1i1p1f1/Amon/prc/gr/v20190206/prc_Amon_E3SM-1-0_piControl_r1i1p1f1_gr_000101-050012.nc HTTP/1.0\" 404 - \"-\" \"Wget/1.12 (linux-gnu)\"`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fe3sm-project%2Fesgf_metrics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fe3sm-project%2Fesgf_metrics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fe3sm-project%2Fesgf_metrics/lists"}