{"id":17059249,"url":"https://github.com/danijar/elements","last_synced_at":"2025-04-06T15:12:09.415Z","repository":{"id":57426018,"uuid":"360323009","full_name":"danijar/elements","owner":"danijar","description":"Building blocks for productive research","archived":false,"fork":false,"pushed_at":"2024-10-18T03:16:35.000Z","size":152,"stargazers_count":44,"open_issues_count":1,"forks_count":6,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-10-20T05:50:40.127Z","etag":null,"topics":["artificial-intelligence","machine-learning","productivity","python","scientific-computing","tools"],"latest_commit_sha":null,"homepage":"https://danijar.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/danijar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-21T22:29:00.000Z","updated_at":"2024-10-18T03:16:39.000Z","dependencies_parsed_at":"2024-04-26T19:39:35.171Z","dependency_job_id":"a3575b13-27aa-4298-976d-0d77ab381c64","html_url":"https://github.com/danijar/elements","commit_stats":{"total_commits":21,"total_committers":1,"mean_commits":21.0,"dds":0.0,"last_synced_commit":"cd5f6646569e13ad7cbd504867fdbfc9f3fd4892"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danijar%2Felements","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danijar%2Felements/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danijar%2Felements/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danijar%2Felements/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/danijar","download_url":"https://codeload.github.com/danijar/elements/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247500468,"owners_count":20948880,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","machine-learning","productivity","python","scientific-computing","tools"],"created_at":"2024-10-14T10:33:36.779Z","updated_at":"2025-04-06T15:12:09.394Z","avatar_url":"https://github.com/danijar.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![PyPI](https://img.shields.io/pypi/v/elements.svg)](https://pypi.python.org/pypi/elements/#history)\n\n# 🔥 Elements\n\nBuilding blocks for productive research.\n\n## Installation\n\n```sh\npip install elements\n```\n\n## Features\n\nElements aims to provide well thought out solutions to common problems in\nresearch code. It is also hackable. If you need to change some of the code, we\nencourage you to fork the corresponding file into your project directory and\nmake edits.\n\n### `elements.Logger`\n\nA logger for array types that is extensible through backends. Metrics are\nwritten in a background thread to not block program execution, which is\nespecially important on cloud services where bucket access is slow.\n\nProvided backends:\n- `TerminalOutput(pattern)`: Print scalars to the terminal. Can filter to\n  fewer metrics via regex.\n- `JSONLOutput(logdir, filename, pattern)`: Write scalars to JSONL files.\n  For example, can be read directly with pandas.\n- `TensorBoardOutput(logdir)`: Scalars, histograms, images, GIFs.\n  Automatically starts new event files when the current one exceeds a size\n  limit to support cloud storage where appding to files requires a full\n  download and reupload.\n- `WandBOutput(pattern, **init_kwargs)`: Strings, histograms, images, videos.\n- `MLFlowOutput(run_name, resume_id)`: Logs all types of metrics to MLFlow.\n\n```python\nstep = elements.Counter()\nlogger = elements.Logger(step, [\n    elements.logger.TerminalOutput(),\n    elements.logger.JSONLOutput(logdir, 'metrics.jsonl'),\n    elements.logger.TensorBoardOutput(logdir),\n    elements.logger.WandBOutput(name='name', project='project'),\n])\n\nstep.increment()\nlogger.scalar('foo', 42)\nlogger.scalar('foo', 43)\nlogger.scalar('foo', 44)\nlogger.vector('vector', np.zeros(100))\nlogger.image('image', np.zeros((800, 600, 3, np.uint8)))\nlogger.video('video', np.zeros((100, 64, 64, 3, np.uint8)))\n\nlogger.add({'foo': 42, 'vector': np.zeros(100)}, prefix='scope')\n\nlogger.write()\n```\n\n### `elements.Config`\n\nAn immutable nested directory to hold configurations. Keys can be accessed via\nattribute syntax. Values are restricted to primitive types that are supported\nby JSON. Types are checked when replacing values in the config.\n\n```python\nconfig = elements.Config(\n    logdir='path/to/dir',\n    foo=dict(bar=42),\n)\n\nprint(config)                      # Pretty printing\nprint(config.foo.bar)              # Attribute syntax\nprint(config['foo']['bar'])        # Dictionary syntax\nconfig.logdir = 'path/to/new/dir'  # Not allowed\n\n# Access a copy of the flattened dictionary underlying the config.\nconfig.flat == {'logdir': 'path/to/dir', 'foo.bar': 42}\n\n# Configs are immutable, so updating them returns a new object.\nnew_config = config.update({'foo.bar': 43})\n\n# Types are enforced when updating configs, but values of other types are\n# allowed as long as they can be converted without loss of information.\nnew_config = config.update({'foo.bar': float(1e5)})  # Allowed\nnew_config = config.update({'foo.bar': float(1.5)})  # Not allowed\n\n# Configs can be saved and loaded in JSON and YAML formats.\nconfig.save('config.json')\nconfig = elements.Config.load('config.json')\n```\n\n### `elements.Flags`\n\nA parser for command line flags similar to `argparse` but faster to use and\nmore flexible. Enforces types and supports nested dictionaries and overwriting\nmultiple flags at once via regex.\n\nA mapping of valid keys and their default values must be provided to infer\ntypes. Because there are defaults for all values, there are no required\narguments that the user must specify on the command line.\n\n```python\n# Create flags parser from default values.\nflags = elements.Flags(logdir='path/to/dir', bar=42)\n\n# Create flags parser from config.\nflags = elements.Flags(elements.Config({\n    'logdir': 'path/to/dir',\n    'foo.bar': 42,\n    'list': [1, 2, 3],\n}))\n\n# Load a config from YAML and overwrite it from it from the command line.\nconfig = elements.Config.load('defaults.yaml')\nconfig = elements.Flags(config).parse()\n\n# Overwrite some of the keys.\nconfig = flags.parse(['--logdir', 'path/to/new/dir', '--foo.bar', '43'])\n\n# Supports syntax with space or equals.\nconfig = flags.parse(['--logdir=path/to/new/dir'])\n\n# Overwrite lists.\nconfig = flags.parse(['--list', '10', '20', '30'])\nconfig = flags.parse(['--list', '10,20,30'])\nconfig = flags.parse(['--list=10,20,30'])\n\n# Set all nested keys that end in 'bar'.\nconfig = flags.parse(['--.*\\.bar$', '43'])\n\n# Parse only known flags.\nconfig, remaining = flags.parse_known(['--logdir', 'dir', '--other', '123'])\nremaining == ['--other', '123']\n\n# Print a help page and terminate the program.\nflags.parse(['--help'])\n\n# Print a help page without terminating the program.\nflags = elements.Flags(logdir='path/to/dir', bar=42, help_exits=False)\nparsed, remaining = flags.parse_known(['--help', '--other=value'])\nremaining == ['--help', '--other=value']\nsecond_parser.parse(remaining)  # Now we exit.\n```\n\n### `elements.Path`\n\nA filesystem abstraction similar to `pathlib` that is extensible to new\nfilesystems. Comes with support for local filesystems and GCS buckets.\n\n```python\npath = elements.Path('gs://bucket/path/to/file.txt')\n\n# String operations\npath.parent                           # gs://bucket/path/to\npath.name                             # file.txt\npath.stem                             # file\npath.suffix                           # .txt\n\n# File operations\npath.read(mode='r')                   # Content of the file as string\npath.read(mode='rb')                  # Content of the file as bytes\npath.write(content, mode='w')         # Write string to the file\npath.write(content, mode='wb')        # Write bytes to the file\nwith path.open(mode='r') as f:        # Create a file pointer\n  pass\n\n# File system checks\npath.parent.glob('*')                 # Get all sibling paths\npath.exists()                         # True\npath.isdir()                          # False\npath.isfile()                         # True\n\n# File system changes\n(path.parent / 'foo').mkdir()         # Creates directory including parents\npath.remove()                         # Deletes a file or empty directory\npath.parent.rmtree()                  # Deletes directory and its content\npath.copy(path.parent / 'copy.txt')   # Makes a copy\npath.move(path.parent / 'moved.txt')  # Moves the file\n```\n\n### `elements.Checkpoint`\n\nHolds a collection of objects that can be saved to and loaded from disk.\n\nEach object attached to the checkpoint needs to implement `save() -\u003e data` and\n`load(data)` methods, where `data` is pickleable.\n\nCheckpoints are written in a background thread to not block program execution.\nNew checkpoints are writing to a temporary path first and renamed to the actual\npath once they are fully written, so that the path always points to a valid\nname even if the program gets terminated while writing.\n\n```python\nstep = elements.Counter()\n\ncp = elements.Checkpoint(directory)\n# Attach objects to the checkpoint.\ncp.step = step\ncp.model = model\n# After attaching the objects we load the checkpoint from disk if it exists\n# and otherwise save an initial checkpoint.\ncp.load_or_save()\n\n# Later on, we can change the objects and then save the checkpoint again.\nshould_save = elements.when.Every(10)\nfor _ in range(100):\n  step.increment()\n  if should_save(step):\n    cp.save()\n\n# We can also load checkpoints or parts of a checkpoint from a different directory.\ncp.load(pretraining_directory, keys=['model'])\nprint(cp.model)\n```\n\n### `elements.Timer`\n\nCollect timing statistics about the run time of different parts of a program.\nMeasures code inside sections and can wrap methods into sections. Returns\nexecution count, execution time, fraction of overall program time, and more.\nThe resulting statisticse can be added to the logger.\n\n```python\ntimer = Timer()\n\ntimer.section('foo'):\n  time.sleep(10)\n\ntimer.wrap('name', obj, ['method1', 'method2'])\nobj.method1()\nobj.method1()\nobj.method1()\nobj.method2()\n\nstats = timer.stats(reset=True, log=True)\nstats == {\n    'foo_count': 1,\n    'foo_total': 10.0,\n    'foo_avg': 10.0,\n    'foo_min': 10.0,\n    'foo_max': 10.0,\n    'foo_frac': 0.92,\n    'name.method1_count': 3,\n    'name.method1_frac': 0.07,\n    # ...\n    'name.method2_frac': 0.01,\n    # ...\n}\n```\n\n### `elements.when`\n\nHelpers for running code at defined times, such as every fixed number of steps\nor seconds or a certain fraction of the time. The counting is robust, so when\nyou skip a step it will run the code the next time to catch up.\n\n```python\nshould = elements.when.Every(100)\nfor step in range(1000):\n  if should(step):\n    print(step)  # 0, 100, 200, ...\n\nshould = elements.when.Ratio(0.3333)\nfor step in range(100):\n  if should(step):\n    print(step)  # 0, 4, 7, 10, 13, 16, ...\n\nshould = elements.when.Once()\nfor step in range(100):\n  if should(step):\n    print(step)  # 0\n\nshould = elements.when.Until(5)\nfor step in range(10):\n  if should(step):\n    print(step)  # 0, 1, 2, 3, 4\n\nshould = elements.when.Clock(1)\nfor step in range(100):\n  if should(step):\n    print(step)  # 0, 10, 20, 30, ...\n  time.sleep(0.1)\n```\n\n### `elements.plotting`\n\nTools for storing, loading, and plotting data with sensible defaults.\n\nData is stored in the `run` format in gzipped JSON files. Each file contains a\nlist of one or more run. A run is a dictionary with the keys `task`, `method`,\n`seed`, `xs`, `ys`. The task, method, and seed are string fields, whereas xs\nand ys are lists of equal length containing numbers for the data to plot.\n\nTake a look at `plotting.py` in the repository to see the list of all available\nfunctions, beyond what is used in this snippet.\n\n```python\nfrom elements import plotting\n\nruns = plotting.load('filename.json.gz')\nplotting.dump(runs, 'filename.json.gz')\n\nbins = np.linspace(0, 1e6, 100)\ntensor, tasks, methods, seeds = plotting.tensor(runs, bins)\ntensor.shape == (len(tasks), len(methods), len(seeds), len(bins))\n\nfig, axes = plotting.plots(len(tasks))\n\nfor i, task in enumerate(tasks):\n  ax = axes[i]\n  for j, method in enumerate(methods):\n    # Aggregate over seeds.\n    mean = np.nanmean(tensor[i, j, :, :], 2)\n    std = np.nanstd(tensor[i, j, :, :], 2)\n    plotting.curve(ax, bins[1:], mean, std, label=method, order=j)\n\nplotting.legend(fig, adjust=True)\n\n# Saves the figure in both PNG and PDF formats and attempts to crop margins off\n# the PDF.\nplotting.save(fig, 'path/to/name')\n```\n\n## Questions\n\nPlease file an [issue on Github](https://github.com/danijar/elements/issues).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanijar%2Felements","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdanijar%2Felements","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanijar%2Felements/lists"}