{"id":13667272,"url":"https://github.com/geophile/marcel","last_synced_at":"2026-01-21T04:42:53.847Z","repository":{"id":39660109,"uuid":"243075528","full_name":"geophile/marcel","owner":"geophile","description":"A modern shell","archived":false,"fork":false,"pushed_at":"2025-11-14T22:11:08.000Z","size":3779,"stargazers_count":351,"open_issues_count":4,"forks_count":13,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-11-15T00:16:34.374Z","etag":null,"topics":["database","postgres","python","scripting","shell"],"latest_commit_sha":null,"homepage":"https://marceltheshell.org","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/geophile.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-02-25T18:48:51.000Z","updated_at":"2025-11-14T22:11:12.000Z","dependencies_parsed_at":"2023-01-31T05:16:05.774Z","dependency_job_id":"4bcf4ddd-48a0-49be-9b8c-1813ca1cb952","html_url":"https://github.com/geophile/marcel","commit_stats":null,"previous_names":[],"tags_count":107,"template":false,"template_full_name":null,"purl":"pkg:github/geophile/marcel","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geophile%2Fmarcel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geophile%2Fmarcel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geophile%2Fmarcel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geophile%2Fmarcel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/geophile","download_url":"https://codeload.github.com/geophile/marcel/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geophile%2Fmarcel/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28626853,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-21T02:47:06.670Z","status":"ssl_error","status_checked_at":"2026-01-21T02:45:44.886Z","response_time":86,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","postgres","python","scripting","shell"],"created_at":"2024-08-02T07:00:34.270Z","updated_at":"2026-01-21T04:42:53.810Z","avatar_url":"https://github.com/geophile.png","language":"Python","readme":"What's New\n----------\n\nThe bad news is that I am the only marcel user, as far as I know.\n\nThe good news is that I am therefore free to make incompatible changes. Which I've just done.\nThe last several versions of marcel\nhave overhauled the namespace implementation to support the spawn\nmodel of multiprocessing (from the `multiprocessing` module). As of this version, it should be\nthe case that marcel's on-disk state, from the marcel namespace, \nand from the workspace's `env.pickle` and `properties.pickle` files, do not rely on\nmarcel-defined types, only builtin Python types. \n\nI had been careful to provide migration of on-disk state across version upgrades. But this became\nunwieldy, because marcel type definitions would change, and sometimes type names, or package structure,\nbut the code had to maintain old names and structure to support migration. Which is just terrible for\ncode cleanliness and maintenance. So I've done an incompatible upgrade. \nOld workspaces need to be discarded.\nHowever, as of version 0.37.0, workspaces should be migrated seamlessly.\n\nMy sincere apologies to me.\n\nMarcel\n======\n\n[Marcel is a shell](https://www.youtube.com/watch?v=VF9-sEbqDvU). \nThe main idea is to rely on piping as the primary\nmeans of composition, as with any Unix or Linux\nshell. However, instead of passing strings from one command to the\nnext, marcel passes Python values: builtin types such as lists,\ntuples, strings, and numbers; but also objects representing files and\nprocesses.\n\nLinux has extremely powerful commands such as `awk` and `find`.  Most\npeople know how to do a few simple operations using these commands.\nBut it is not easy to exploit the full power of these commands\ndue to their reliance on extensive \"sublanguages\" which do:\n\n* __Filtering__: What data is of interest?\n* __Processing__: What should be done with the data?\n* __Formatting__: How should results be presented?\n\nBy contrast, marcel has no sublanguages.  You use marcel operators\ncombined with Python code to filter data, process it, and control\ncommand output.\n\nThe commands and syntax supported by a shell constitute a language\nwhich can be used to create scripts. Of course, in creating a script,\nyou rely on language features that you typically do not use\ninteractively: control structures, data types, and abstraction\nmechanisms (e.g. functions), for example. \nViewed as a programming language, shell scripting languages \nare notoriously bad. I didn't think it was wise to bring another one\ninto the world. So marcel takes a different\napproach, using Python as a scripting language, (see below for more \non scripting).\n\nPipelines\n---------\n\nMarcel provides commands, called _operators_, which do the basic work of a shell. \nAn operator takes a _stream_ of data as input, and generates another stream as output.\nOperators can be combined by pipes, causing one operator's output to be the next operator's input.\nFor example, this command uses the `ls` and `map` operators to list the\nnames and sizes of files in the `/home/jao` directory:\n\n```shell script\nls /home/jao | map (lambda f: (f, f.size))\n``` \n\n* The `ls` operator produces a stream of `File` objects, representing the contents\nof the `/home/jao` directory.\n* `|` is the symbol denoting a pipe, as in any Linux shell.\n* The pipe connects the output stream from `ls` to the input stream of the next\noperator, `map`.\n* The `map` operator applies a given function to each element of the input stream,\nand writes the output from the function to the output stream. The function is enclosed\nin parentheses. It is an ordinary Python function, except that the keyword `lambda` is optional.\nIn this case, an incoming `File` is mapped to a tuple containing the file and the file's size.\n\nA `pipeline` is a sequence of operators connected by pipes. They can be used directly\non the command line, as above. They also have various other uses in marcel. For example,\na pipeline can be assigned to a variable, essentially defining a new operator.\nFor example, here is a pipeline, assigned to the variable `recent`, which selects\n`File`s modified within the past day:\n\n```shell script\nrecent = (| select (file: now() - file.mtime \u003c days(1)) |) \n``` \n\n* The pipeline being defined is bracketed by `(|...|)`. (Without the brackets, marcel would\nattempt to evaluate the pipeline immediately, and then complain because the parameter\n`file` is not bound.)\n* The pipeline contains a single operator, `select`, which uses a function to define\nthe items of interest. In this case, `select` operates on a `File`, bound to the \nparameter `file`. \n* `now()` is a function defined by marcel which gives the current time in seconds since\nthe epoch, (i.e., it is just `time.time()`).\n* `File` objects have an `mtime` property, providing the time since the last content modification.\n* `days()` is another function defined by marcel, which simply maps days to seconds, i.e.,\nit multiplies by 24 * 60 * 60.\n\nThis pipeline can be used in conjunction with any pipeline yielding files. E.g., to locate\nthe recently changed files in `~/git/myproject`:\n\n```shell script\nls ~/git/myproject | recent\n```\n\nFunctions\n---------\n\nAs shown above, a number of operators, like `map` and `select`, take Python functions as \ncommand-line arguments. Functions can also be invoked to obtain the value of an\nenvironment variable.\nFor example, to list the contents of your home directory, you could write:\n\n```shell script\nls /home/(USER)\n```\n\nThis concatenates the string `/home/` with the string resulting from the evaluation of\nthe expression `lambda: USER`. `USER` is a marcel environment variable identifying the\ncurrent user, (so this command is equivalent to `ls ~`).\n\nIf you simply want to evaluate a Python expression, you could use the `map` operator, e.g.\n\n```shell script\nmap (5 + 6)\n```  \n\nwhich prints `11`. Marcel permits the `map` operator to be inferred, \nso this also works:\n\n```shell script\n(5 + 6)\n```\n\nIn general, you can elide `map` from any pipeline.\n\nExecutables\n-----------\n\nIn addition to using built-in operators, you can, of course, call any executable.\nPipelines may contain a mixture of marcel operators and host executables. Piping between\noperators and executables is done via streams of strings.\n\nFor example, this command combines operators and executables. \nIt scans `/etc/passwd` and lists the usernames of \nusers whose shell is `/bin/bash`. \n`cat`, `xargs`, and `echo` are Linux executables. `map` and `select` are marcel operators.\nThe output is condensed into one line through\nthe use of `xargs` and `echo`. \n\n```shell script\ncat /etc/passwd \\\n| map (line: line.split(':')) \\\n| select (*line: line[-1] == '/bin/bash') \\\n| map (user, *_: user) \\\n| xargs echo\n```\n\n* `cat /etc/passwd`: Obtain the contents of the file. Lines are piped to subsequent commands.\n* `map (line: line.split(':'))`: Split the lines at the `:` separators, yielding 7-tuples.\n* `select (*line: line[-1] == '/bin/bash')`: select those lines in which the last field is `/bin/bash`.\n* `map (user, *_: user) |`: Keep the username field of each input tuple.\n* `xargs echo`: Combine the incoming usernames into a single line, which is printed to `stdout`.\n\nShell Features\n--------------\n\nMarcel provides:\n\n* __Command history:__ A `history` operator, rerunning and editing of previous commands,\nreverse search, etc.\n* __Customizable prompts:__ Configured in Python, of course.\n* __Tab completion:__ For operators, flags, and filename arguments.\n* __Help:__ Extensive help facility, providing information on concepts, objects,\nand operators.\n* __Customizable color highlighting:__ The colors used to render output for builtin types such \nas `File` and `Process`, and `help` output can be customized too.\n* __Dynamic reconfiguration:__ Changes to configuration and startup scripts are picked up without restarting.\n\nScripting\n---------\n\nMarcel's syntax for constructing and running pipelines, and defining and using\nvariables and functions, was designed for interactive usage. Instead of extending\nthis syntax to a full-fledged scripting language, marcel provides a Python API,\nallowing Python to be used as the scripting language.\nWhile Python is\nsometimes considered to _already be_ a scripting language, it isn't really. \nExecuting shell commands from Python code is cumbersome. You've got to use\n`os.system`, or `subprocess.Popen`, and write some additional code to\ndo the integration.\n\nMarcel provides a Python module, `marcel.api`,\nwhich brings shell commands into Python in a much cleaner way. For\nexample, to list file names and sizes in `/home/jao`:\n\n```python\nfrom marcel.api import *\n\nfor file, size in ls('/home/jao') | map(lambda f: (f, f.size)):\n    print(f'{file.name}: {size}') \n```\n\nThis code uses the `ls` and\n`map` functions, provided by `marcel.api`. These correspond to the\nmarcel operators `ls` and `map` that you can use on the command\nline. Output from the `ls` is a stream of `File`s, which are piped\nto `map`, which maps files to (file, file size) tuples.  `ls ... |\nmap ...` defines a pipeline (just as on the command line). The\nPython class representing pipelines defines ``iter``, so that\nthe pipeline's output can be iterated over using the standard\nPython `for` loop.\n\n\nInstallation\n------------\n\nTo install marcel locally (i.e., available only to your username):\n\n```shell script\npython3 -m pip install marcel\n```\n\nThis command installs marcel for the current user. To install for the entire system,\nuse `sudo python3 -m pip install --prefix ...` instead. (The value of the `--prefix` flag should\nbe something like `/usr/local`.)\n\nMarcel depends on [dill](https://pypi.org/project/dill/), [psutil](https://pypi.org/project/psutil/),\nand [prompt_toolkit](https://python-prompt-toolkit.readthedocs.io). These packages\nwill be installed automatically if needed, when marcel is installed\nvia pip.\n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeophile%2Fmarcel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgeophile%2Fmarcel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeophile%2Fmarcel/lists"}