{"id":13366982,"url":"https://github.com/man-group/ArcticDB","last_synced_at":"2025-03-12T18:31:46.752Z","repository":{"id":143112697,"uuid":"564689243","full_name":"man-group/ArcticDB","owner":"man-group","description":"ArcticDB is a high performance, serverless DataFrame database built for the Python Data Science ecosystem.","archived":false,"fork":false,"pushed_at":"2024-10-29T08:21:39.000Z","size":103321,"stargazers_count":1484,"open_issues_count":262,"forks_count":94,"subscribers_count":26,"default_branch":"master","last_synced_at":"2024-10-29T09:19:55.936Z","etag":null,"topics":["big-data","data","data-analysis","data-science","database","dataframe","pandas","quantitative-analysis","quantitative-finance","quantitative-trading"],"latest_commit_sha":null,"homepage":"http://arcticdb.io","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/man-group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-11T09:09:46.000Z","updated_at":"2024-10-28T12:47:23.000Z","dependencies_parsed_at":"2023-09-25T07:30:47.390Z","dependency_job_id":"ef5fdf3d-8160-40e3-8a9e-5c81e0419a4a","html_url":"https://github.com/man-group/ArcticDB","commit_stats":null,"previous_names":[],"tags_count":120,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/man-group%2FArcticDB","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/man-group%2FArcticDB/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/man-group%2FArcticDB/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/man-group%2FArcticDB/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/man-group","download_url":"https://codeload.github.com/man-group/ArcticDB/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242813707,"owners_count":20189281,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big-data","data","data-analysis","data-science","database","dataframe","pandas","quantitative-analysis","quantitative-finance","quantitative-trading"],"created_at":"2024-07-30T00:01:35.561Z","updated_at":"2025-03-12T18:31:46.744Z","avatar_url":"https://github.com/man-group.png","language":"C++","funding_links":[],"categories":["Databases","Python","Data Sources","C++","Python：量化金融第一生态"],"sub_categories":["TimeSeries Analysis","Numerical Libraries \u0026 Data Structures","一、基础数值计算与数据结构"],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"static/ArcticDB Logo Purple Horizontal.svg\"\u003e\n\u003c/p\u003e\n\n---\n\n\u003cdiv align=\"center\"\u003e\n  \u003cp\u003e\u003cb\u003eThree minute ArcticDB demo from PyQuantNews\u003c/b\u003e\u003c/p\u003e\n  \u003ca href=\"https://www.youtube.com/watch?v=5_AjD7aVEEM\"\u003e\n    \u003cimg src=\"static/pqn_video_thumbnail.png\" alt=\"Options data in ArcticDB\" width=\"33%\"\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\n---\n\u003cp align=\"center\"\u003e\n:earth_americas: \u003ca href=\"http://arcticdb.io\"\u003eArcticDB Website\u003c/a\u003e | 📘 \u003ca href=\"https://docs.arcticdb.com\"\u003eArcticDB Docs\u003c/a\u003e | 📰: \u003ca href=\"https://medium.com/arcticdb\"\u003eArcticDB Blog\u003c/a\u003e | :mega: \u003ca href=\"https://www.man.com/man-group-brings-powerful-dataframe-database-product-arcticdb-to-market-with-bloomberg\"\u003ePress Release\u003c/a\u003e | :mega: \u003ca href=\"https://www.bloomberg.com/company/press/man-group-brings-powerful-dataframe-database-product-arcticdb-to-market-with-bloomberg/\"\u003ePress Release\u003c/a\u003e | :busts_in_silhouette: \u003ca href=\"#community\"\u003eCommunity\u003c/a\u003e\n\u003cbr /\u003e\u003cbr /\u003e\n\u003ca href=\"https://github.com/man-group/ArcticDB/actions\"\u003e\u003cimg src=\"https://github.com/man-group/ArcticDB/actions/workflows/build.yml/badge.svg\"/\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n---\n\n**ArcticDB** is a high performance, serverless **DataFrame database** built for the Python Data Science ecosystem.\nLaunched in March 2023, it is the successor to [Arctic](https://github.com/man-group/arctic).\n\nUse of ArcticDB in production (including business or commercial environments) or for a Database Service requires a paid for license from ArcticDB Limited . Please contact info@arcticdb.io for further details.\n\nArcticDB offers an intuitive Python-centric API enabling you to read and write Pandas DataFrames to S3 or LMDB utilising a fast C++ data-processing and compression engine.\n\nArcticDB allows you to:\n\n * **Pandas in, Pandas out**: Read and write Pandas DataFrames, NumPy arrays and native types to S3 and LMDB without leaving Python.\n * **Built for time-series data**: Efficiently index and query time-series data across _billions_ of rows\n * **Time travel**: Travel back in time to see previous versions of your data and create customizable snapshots of the database\n * **Schemaless Database**: Append, update and modify data without being constrained by the existing schema\n * **Optimised for streaming data**: Built in support for efficient sparse data storage\n * **Powerful processing**: Filter, aggregate and create new columns on-the-fly with a Pandas-like syntax\n * **C++ efficiency**: Accelerate analytics though concurrency in the C++ data-processing engine\n\nArcticDB handles data that is big in both row count and column count, so a 20-year history of more than 400,000 unique securities can be stored in a single *symbol*. Each *symbol* is maintained as a separate entity with no shared data which means ArcticDB can scale horizontally across *symbols*, maximising the performance potential of your compute, storage and network.\n\nArcticDB is designed from the outset to be resilient; there is no single point of failure, and persistent data structures in the storage mean that once a version of a *symbol* has been written, it can never be corrupted by subsequent updates. Pulling compressed data directly from  storage to the client means that there is no server to overload, so your data is always available when you need it.\n\n## Quickstart\n\n### Prebuilt binary availability\n\n|                       | PyPI (Python 3.7 - 3.13) | conda-forge (Python 3.9 - 3.13) |\n| --------------------- | - | - |\n| Linux (Intel/AMD)     | ✔️ | ✔️ |\n| Windows (Intel/AMD)   | ✔️ | ➖ |\n| MacOS                 | ➖ | Beta️ |\n\nFor conda-forge see the [release-info](https://github.com/conda-forge/arcticdb-feedstock/?tab=readme-ov-file#current-release-info).\n\n### Storage compatibility\n\n|                       | Linux | Windows | Mac |\n| --------------------- | - | - | - |\n| S3                 | ✔️ | ✔️ | ✔️ |\n| LMDB               | ✔️ | ✔️ | ✔️ |\n| Azure Blob Storage | ✔️ | ✔️ | ➖ |\n\nWe have tested against the following S3 backends:\n- AWS S3\n- Ceph\n- MinIO on Linux\n- Pure Storage S3\n- Scality S3\n- VAST Data S3\n\n### Installation\n\nInstall ArcticDB:\n\n```bash\n$ pip install arcticdb\n```\nor using conda-forge\n```bash\n$ conda install -c conda-forge arcticdb\n```\n\nImport ArcticDB:\n\n```Python\n\u003e\u003e\u003e import arcticdb as adb\n```\n\nCreate an instance on your S3 storage (with or without explicit credentials):\n\n```Python\n# Leave AWS to derive credential information\n\u003e\u003e\u003e ac = adb.Arctic('s3://MY_ENDPOINT:MY_BUCKET?aws_auth=true')\n\n# Manually specify creds\n\u003e\u003e\u003e ac = adb.Arctic('s3://MY_ENDPOINT:MY_BUCKET?region=YOUR_REGION\u0026access=ABCD\u0026secret=DCBA')\n```\n\nOr create an instance on your local disk:\n\n```Python\n\u003e\u003e\u003e ac = adb.Arctic(\"lmdb:///\u003cpath\u003e\")\n```\n\nCreate your first library and list the libraries in the instance:\n\n```Python\n\u003e\u003e\u003e ac.create_library('travel_data')\n\u003e\u003e\u003e ac.list_libraries()\n```\n\nCreate a test dataframe:\n```Python\n\u003e\u003e\u003e import numpy as np\n\u003e\u003e\u003e import pandas as pd\n\u003e\u003e\u003e NUM_COLUMNS=10\n\u003e\u003e\u003e NUM_ROWS=100_000\n\u003e\u003e\u003e df = pd.DataFrame(np.random.randint(0,100,size=(NUM_ROWS, NUM_COLUMNS)), columns=[f\"COL_{i}\" for i in range(NUM_COLUMNS)], index=pd.date_range('2000', periods=NUM_ROWS, freq='h'))\n```\n\nGet the library, write some data to it, and read it back:\n\n```Python\n\u003e\u003e\u003e lib = ac['travel_data']\n\u003e\u003e\u003e lib.write(\"my_data\", df)\n\u003e\u003e\u003e data = lib.read(\"my_data\")\n```\n\nTo find out more about working with data, visit our [docs](https://docs.arcticdb.io)\n\n---\n\n## Documentation\n\nThe source code for the ArcticDB docs are located in the [docs](https://github.com/man-group/ArcticDB/tree/master/docs) folder, and are hosted at [docs.arcticdb.io](https://docs.arcticdb.io).\n\n## License\n\nArcticDB is released under a [Business Source License 1.1 (BSL)](https://github.com/man-group/ArcticDB/blob/master/LICENSE.txt)\n\nBSL features are free to use and the source code is available, but users may not use ArcticDB for production use or for\na Database Service, without agreement with Man Group Operations Limited.\n\nUse of ArcticDB in production or for a Database Service requires a paid for license from ArcticDB Limited\nand is licensed under the ArcticDB Software License Agreement. For more information please contact [info@arcticdb.io](mailto:info@arcticdb.io).\n\nThe BSL is not certified as an open-source license, but most of the [Open Source Initiative (OSI)](https://opensource.org/) criteria are met.\nPlease see version conversion dates in the below table:\n\n| ArcticDB Version | License | Converts to Apache 2.0 |\n| ------------- | ------------- | ------------- |\n| 1.0 | Business Source License 1.1 | Mar 16, 2025 |\n| 1.2 | Business Source License 1.1 | May 22, 2025 |\n| 1.3 | Business Source License 1.1 | Jun  9, 2025 |\n| 1.4 | Business Source License 1.1 | Jun  23, 2025 |\n| 1.5 | Business Source License 1.1 | Jul  11, 2025 |\n| 1.6 | Business Source License 1.1 | Jul  25, 2025 |\n| 2.0 | Business Source License 1.1 | Aug  29, 2025 |\n| 3.0 | Business Source License 1.1 | Sep  13, 2025 |\n| 4.0 | Business Source License 1.1 | Sep  27, 2025 |\n| 4.1 | Business Source License 1.1 | Nov 1, 2025 |\n| 4.2 | Business Source License 1.1 | Nov 12, 2025 |\n| 4.3 | Business Source License 1.1 | Feb 7, 2026 |\n| 4.4 | Business Source License 1.1 | Apr 5, 2026 |\n| 4.5 | Business Source License 1.1 | Aug 14, 2026 |\n| 5.0 | Business Source License 1.1 | Oct 31, 2026 |\n| 5.1 | Business Source License 1.1 | Nov 15, 2026 |\n| 5.2 | Business Source License 1.1 | Jan 27, 2027 |\n\n## Code of Conduct\n\n[Code of Conduct](https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md)\n\nThis project has adopted a Code of Conduct. If you have any concerns about the Code, or behaviour that you have\nexperienced in the project, please contact us at [info@arcticdb.io](mailto:info@arcticdb.io).\n\n## Contributing/Building From Source\n\nWe welcome your contributions to help us improve and extend this project!\n\nPlease refer to the [Contributing](https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md)\npage and feel free to open issues on GitHub.\n\nWe are also always looking for feedback from our dedicated community! If you have used ArcticDB please let us know, we would love to hear about your experience!\n\nOur release process is [documented here](https://github.com/man-group/ArcticDB/wiki/Releasing).\n\n## Community\n\nWe would love to hear how your ArcticDB journey evolves, email us at [info@arcticdb.io](mailto:info@arcticdb.io) or come chat to us on [Twitter](https://www.twitter.com/arcticdb)!\n\nInterested in learning more about ArcticDB? Head over to our [blog](https://medium.com/arcticdb)!\n\nDo you have any questions or issues? Chat to us and other users through our dedicated Slack Workspace - sign up for Slack access on [our website](https://arcticdb.io).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fman-group%2FArcticDB","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fman-group%2FArcticDB","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fman-group%2FArcticDB/lists"}