{"id":17667472,"url":"https://github.com/lithops-cloud/lithops","last_synced_at":"2026-01-03T09:07:42.250Z","repository":{"id":37033257,"uuid":"130669425","full_name":"lithops-cloud/lithops","owner":"lithops-cloud","description":"A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀","archived":false,"fork":false,"pushed_at":"2025-02-27T11:05:17.000Z","size":13547,"stargazers_count":327,"open_issues_count":11,"forks_count":111,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-04-11T17:50:42.450Z","etag":null,"topics":["big-data","big-data-analytics","cloud-computing","data-processing","distributed","kubernetes","multicloud","multiprocessing","object-storage","parallel","python","serverless","serverless-computing","serverless-functions"],"latest_commit_sha":null,"homepage":"http://lithops.cloud","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lithops-cloud.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2018-04-23T09:02:25.000Z","updated_at":"2025-04-07T21:19:42.000Z","dependencies_parsed_at":"2023-12-21T13:13:50.788Z","dependency_job_id":"fd0e62a9-568e-4584-8356-d858d377b541","html_url":"https://github.com/lithops-cloud/lithops","commit_stats":{"total_commits":2635,"total_committers":44,"mean_commits":59.88636363636363,"dds":0.4922201138519924,"last_synced_commit":"0f22329410203b12b372a0261d1e21e49e8b8266"},"previous_names":["pywren/pywren-ibm-cloud"],"tags_count":77,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lithops-cloud%2Flithops","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lithops-cloud%2Flithops/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lithops-cloud%2Flithops/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lithops-cloud%2Flithops/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lithops-cloud","download_url":"https://codeload.github.com/lithops-cloud/lithops/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248586250,"owners_count":21128997,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big-data","big-data-analytics","cloud-computing","data-processing","distributed","kubernetes","multicloud","multiprocessing","object-storage","parallel","python","serverless","serverless-computing","serverless-functions"],"created_at":"2024-10-23T22:07:22.907Z","updated_at":"2026-01-03T09:07:42.245Z","avatar_url":"https://github.com/lithops-cloud.png","language":"Python","readme":"\u003cp align=\"center\"\u003e\n  \u003ca href=\"http://lithops.cloud\"\u003e\n    \u003ch1 id='lithops' align=\"center\"\u003e\u003cimg src=\"docs/_static/lithops_logo_readme.png\" alt=\"Lithops\" title=\"Lightweight Optimized Processing\"/\u003e\u003c/h1\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca aria-label=\"License\" href=\"https://github.com/lithops-cloud/lithops/blob/master/LICENSE\"\u003e\u003cimg alt=\"\" src=\"https://img.shields.io/github/license/lithops-cloud/lithops?style=for-the-badge\u0026labelColor=000000\"\u003e\u003c/a\u003e\u0026nbsp\u003ca aria-label=\"PyPi\" href=\"https://pypi.org/project/lithops/\"\u003e\u003cimg alt=\"\" src=\"https://img.shields.io/pypi/v/lithops?style=for-the-badge\u0026labelColor=000000\"\u003e\u003c/a\u003e\u0026nbsp\u003ca aria-label=\"Python\" href=\"#lithops\"\u003e\u003cimg alt=\"\" src=\"https://img.shields.io/pypi/pyversions/lithops?style=for-the-badge\u0026labelColor=000000\"\u003e\u003c/a\u003e\u0026nbsp\u003ca href=\"https://pypistats.org/packages/lithops\"\u003e\u003cimg alt=\"PyPI - Downloads\" src=\"https://img.shields.io/pypi/dm/lithops?label=pypi%7Cdownloads\u0026style=for-the-badge\u0026labelColor=000000\"\u003e\u003c/a\u003e\u0026nbsp\u003ca href=\"https://deepwiki.com/lithops-cloud/lithops\" target=\"_blank\" rel=\"noopener\"\u003e\u003cimg alt=\"Ask DeepWiki\" src=\"https://img.shields.io/badge/DeepWiki-Ask%20DeepWiki-blue.svg?style=for-the-badge\u0026labelColor=000000\u0026logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACwAAAAyCAYAAAAnWDnqAAAAAXNSR0IArs4c6QAAA05JREFUaEPtmUtyEzEQhtWTQyQLHNak2AB7ZnyXZMEjXMGeK/AIi+QuHrMnbChYY7MIh8g01fJoopFb0uhhEqqcbWTp06/uv1saEDv4O3n3dV60RfP947Mm9/SQc0ICFQgzfc4CYZoTPAswgSJCCUJUnAAoRHOAUOcATwbmVLWdGoH//PB8mnKqScAhsD0kYP3j/Yt5LPQe2KvcXmGvRHcDnpxfL2zOYJ1mFwrryWTz0advv1Ut4CJgf5uhDuDj5eUcAUoahrdY/56ebRWeraTjMt/00Sh3UDtjgHtQNHwcRGOC98BJEAEymycmYcWwOprTgcB6VZ5JK5TAJ+fXGLBm3FDAmn6oPPjR4rKCAoJCal2eAiQp2x0vxTPB3ALO2CRkwmDy5WohzBDwSEFKRwPbknEggCPB/imwrycgxX2NzoMCHhPkDwqYMr9tRcP5qNrMZHkVnOjRMWwLCcr8ohBVb1OMjxLwGCvjTikrsBOiA6fNyCrm8V1rP93iVPpwaE+gO0SsWmPiXB+jikdf6SizrT5qKasx5j8ABbHpFTx+vFXp9EnYQmLx02h1QTTrl6eDqxLnGjporxl3NL3agEvXdT0WmEost648sQOYAeJS9Q7bfUVoMGnjo4AZdUMQku50McDcMWcBPvr0SzbTAFDfvJqwLzgxwATnCgnp4wDl6Aa+Ax283gghmj+vj7feE2KBBRMW3FzOpLOADl0Isb5587h/U4gGvkt5v60Z1VLG8BhYjbzRwyQZemwAd6cCR5/XFWLYZRIMpX39AR0tjaGGiGzLVyhse5C9RKC6ai42ppWPKiBagOvaYk8lO7DajerabOZP46Lby5wKjw1HCRx7p9sVMOWGzb/vA1hwiWc6jm3MvQDTogQkiqIhJV0nBQBTU+3okKCFDy9WwferkHjtxib7t3xIUQtHxnIwtx4mpg26/HfwVNVDb4oI9RHmx5WGelRVlrtiw43zboCLaxv46AZeB3IlTkwouebTr1y2NjSpHz68WNFjHvupy3q8TFn3Hos2IAk4Ju5dCo8B3wP7VPr/FGaKiG+T+v+TQqIrOqMTL1VdWV1DdmcbO8KXBz6esmYWYKPwDL5b5FA1a0hwapHiom0r/cKaoqr+27/XcrS5UwSMbQAAAABJRU5ErkJggg==\" style=\"vertical-align:middle;\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\nLithops is a Python multi-cloud distributed computing framework that lets you run unmodified Python code at massive scale across cloud, HPC, and on-premise platforms. It supports major cloud providers and Kubernetes platforms, running your code transparently without requiring you to manage deployment or infrastructure.\n\nLithops is ideal for highly parallel workloads—such as Monte Carlo simulations, machine learning, metabolomics, or geospatial analytics—and lets you tailor execution to your priorities: you can optimize for performance using AWS Lambda to launch hundreds of functions in milliseconds, or reduce costs by running the same code on AWS Batch with Spot Instances.\n\n## Installation\n\n1. Install Lithops from the PyPi repository:\n\n    ```bash\n    pip install lithops\n    ```\n\n2. Execute a *Hello World* function:\n  \n   ```bash\n   lithops hello\n   ```\n\n## Configuration\nLithops provides an extensible backend architecture (compute, storage) designed to work with various cloud providers and on-premise platforms. You can write your code in Python and run it unmodified across major cloud providers and Kubernetes environments.\n\n[Follow these instructions to configure your compute and storage backends](config/)\n\n\u003cp align=\"center\"\u003e\n\u003ca href=\"config/README.md#compute-and-storage-backends\"\u003e\n\u003cimg src=\"docs/source/images/multicloud.jpg\" alt=\"Multicloud Lithops\" title=\"Multicloud Lithops\"/\u003e\n\u003c/a\u003e\n\u003c/p\u003e\n\n\n## High-level API\n\nLithops is shipped with 2 different high-level Compute APIs, and 2 high-level Storage APIs\n\n\u003cdiv align=\"center\"\u003e\n\u003ctable\u003e\n\u003ctr\u003e\n  \u003cth\u003e\n    \u003cimg width=\"50%\" height=\"1px\"\u003e\n    \u003cp\u003e\u003csmall\u003e\u003ca href=\"docs/api_futures.md\"\u003eFutures API\u003c/a\u003e\u003c/small\u003e\u003c/p\u003e\n  \u003c/th\u003e\n  \u003cth\u003e\n    \u003cimg width=\"50%\" height=\"1px\"\u003e\n    \u003cp\u003e\u003csmall\u003e\u003ca href=\"docs/source/api_multiprocessing.rst\"\u003eMultiprocessing API\u003c/a\u003e\u003c/small\u003e\u003c/p\u003e\n  \u003c/th\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n\u003ctd\u003e\n\n```python\nfrom lithops import FunctionExecutor\n\ndef double(i):\n    return i * 2\n\nwith FunctionExecutor() as fexec:\n    f = fexec.map(double, [1, 2, 3, 4])\n    print(f.result())\n```\n\u003c/td\u003e\n\u003ctd\u003e\n\n```python\nfrom lithops.multiprocessing import Pool\n\ndef double(i):\n    return i * 2\n\nwith Pool() as pool:\n    result = pool.map(double, [1, 2, 3, 4])\n    print(result)\n```\n\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003c/table\u003e\n\n\u003ctable\u003e\n\u003ctr\u003e\n  \u003cth\u003e\n    \u003cimg width=\"50%\" height=\"1px\"\u003e\n    \u003cp\u003e\u003csmall\u003e\u003ca href=\"docs/api_storage.md\"\u003eStorage API\u003c/a\u003e\u003c/small\u003e\u003c/p\u003e\n  \u003c/th\u003e\n  \u003cth\u003e\n    \u003cimg width=\"50%\" height=\"1px\"\u003e\n    \u003cp\u003e\u003csmall\u003e\u003ca href=\"docs/source/api_storage_os.rst\"\u003eStorage OS API\u003c/a\u003e\u003c/small\u003e\u003c/p\u003e\n  \u003c/th\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n\u003ctd\u003e\n\n```python\nfrom lithops import Storage\n\nif __name__ == \"__main__\":\n    st = Storage()\n    st.put_object(bucket='mybucket',\n                  key='test.txt',\n                  body='Hello World')\n\n    print(st.get_object(bucket='lithops',\n                        key='test.txt'))\n```\n\u003c/td\u003e\n\u003ctd\u003e\n\n```python\nfrom lithops.storage.cloud_proxy import os\n\nif __name__ == \"__main__\":\n    filepath = 'bar/foo.txt'\n    with os.open(filepath, 'w') as f:\n        f.write('Hello world!')\n\n    dirname = os.path.dirname(filepath)\n    print(os.listdir(dirname))\n    os.remove(filepath)\n```\n\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003c/table\u003e\n\u003c/div\u003e\n\nYou can find more usage examples in the [examples](/examples) folder.\n\n## Documentation\n\nFor documentation on using Lithops, see [latest release documentation](https://lithops-cloud.github.io/docs/)\n\nIf you are interested in contributing, see [CONTRIBUTING.md](./CONTRIBUTING.md).\n\n## Additional resources\n\n### Blogs and Talks\n\n* [How to run Lithops over EC2 VMs using the new K8s backend](https://danielalecoll.medium.com/how-to-run-lithops-over-ec2-vms-using-the-new-k8s-backend-4b0a4377c4e9) \n* [Simplify the developer experience with OpenShift for Big Data processing by using Lithops framework](https://medium.com/@gvernik/simplify-the-developer-experience-with-openshift-for-big-data-processing-by-using-lithops-framework-d62a795b5e1c)\n* [Speed-up your Python applications using Lithops and Serverless Cloud resources](https://itnext.io/speed-up-your-python-applications-using-lithops-and-serverless-cloud-resources-a64beb008bb5)\n* [Lithops, a Multi-cloud Serverless Programming Framework](https://itnext.io/lithops-a-multi-cloud-serverless-programming-framework-fd97f0d5e9e4)\n* [CNCF Webinar - Toward Hybrid Cloud Serverless Transparency with Lithops Framework](https://www.youtube.com/watch?v=-uS-wi8CxBo)\n* [Your easy move to serverless computing and radically simplified data processing](https://www.slideshare.net/gvernik/your-easy-move-to-serverless-computing-and-radically-simplified-data-processing-238929020) Strata Data Conference, NY 2019. See video of Lithops usage [here](https://www.youtube.com/watch?v=EYa95KyYEtg\u0026list=PLpR7f3Www9KCjYisaG7AMaR0C2GqLUh2G\u0026index=3\u0026t=0s) and the example of Monte Carlo [here](https://www.youtube.com/watch?v=vF5HI2q5VKw\u0026list=PLpR7f3Www9KCjYisaG7AMaR0C2GqLUh2G\u0026index=2\u0026t=0s)\n\n\u003c!---\n* [Serverless Without Constraints](https://www.ibm.com/cloud/blog/serverless-without-constraints)\n* [Using Serverless to Run Your Python Code on 1000 Cores by Changing Two Lines of Code](https://www.ibm.com/cloud/blog/using-serverless-to-run-your-python-code-on-1000-cores-by-changing-two-lines-of-code)\n* [Decoding dark molecular matter in spatial metabolomics with IBM Cloud Functions](https://www.ibm.com/cloud/blog/decoding-dark-molecular-matter-in-spatial-metabolomics-with-ibm-cloud-functions)\n* [Speed up data pre-processing with Lithops in deep learning](https://developer.ibm.com/patterns/speed-up-data-pre-processing-with-pywren-in-deep-learning/)\n* [Predicting the future with Monte Carlo simulations over IBM Cloud Functions](https://www.ibm.com/cloud/blog/monte-carlo-simulations-with-ibm-cloud-functions)\n* [Process large data sets at massive scale with Lithops over IBM Cloud Functions](https://www.ibm.com/cloud/blog/process-large-data-sets-massive-scale-pywren-ibm-cloud-functions)\n* [Industrial project in Technion on Lithops](http://www.cs.technion.ac.il/~cs234313/projects_sites/W19/04/site/)\n--\u003e\n\n### Papers\n* [Serverful Functions: Leveraging Servers in Complex Serverless Workflows](https://dl.acm.org/doi/10.1145/3700824.3701095)  - ACM Middleware Industrial Track 2024 \n* [Transparent serverless execution of Python multiprocessing applications](https://dl.acm.org/doi/10.1016/j.future.2022.10.038) - Elsevier Future Generation Computer Systems 2023\n* [Outsourcing Data Processing Jobs with Lithops](https://ieeexplore.ieee.org/document/9619947) - IEEE Transactions on Cloud Computing 2022\n* [Towards Multicloud Access Transparency in Serverless Computing](https://www.computer.org/csdl/magazine/so/5555/01/09218932/1nMMkpZ8Ko8) - IEEE Software 2021\n* [Primula: a Practical Shuffle/Sort Operator for Serverless Computing](https://dl.acm.org/doi/10.1145/3429357.3430522) - ACM/IFIP International Middleware Conference 2020. [See presentation here](https://www.youtube.com/watch?v=v698iu5YfWM)\n* [Bringing scaling transparency to Proteomics applications with serverless computing](https://dl.acm.org/doi/abs/10.1145/3429880.3430101) - 6th International Workshop on Serverless Computing (WoSC6) 2020. [See presentation here](https://www.serverlesscomputing.org/wosc6/#p10)\n* [Serverless data analytics in the IBM Cloud](https://dl.acm.org/citation.cfm?id=3284029) - ACM/IFIP International Middleware Conference 2018\n\n\n# Acknowledgements\nThis project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825184 (CloudButton).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flithops-cloud%2Flithops","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flithops-cloud%2Flithops","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flithops-cloud%2Flithops/lists"}