{"id":28633856,"url":"https://github.com/brimdata/cloud","last_synced_at":"2025-08-01T18:06:38.003Z","repository":{"id":44997513,"uuid":"427066668","full_name":"brimdata/cloud","owner":"brimdata","description":"Brim's Cloud Service","archived":false,"fork":false,"pushed_at":"2022-01-14T18:00:58.000Z","size":19,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-06-12T15:54:04.543Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/brimdata.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-11-11T16:20:53.000Z","updated_at":"2021-12-20T20:52:16.000Z","dependencies_parsed_at":"2022-09-13T13:01:48.942Z","dependency_job_id":null,"html_url":"https://github.com/brimdata/cloud","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/brimdata/cloud","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brimdata%2Fcloud","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brimdata%2Fcloud/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brimdata%2Fcloud/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brimdata%2Fcloud/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/brimdata","download_url":"https://codeload.github.com/brimdata/cloud/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brimdata%2Fcloud/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268272638,"owners_count":24223784,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-01T02:00:08.611Z","response_time":67,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-12T15:39:19.984Z","updated_at":"2025-08-01T18:06:37.966Z","avatar_url":"https://github.com/brimdata.png","language":"Shell","readme":"# Brim Cloud\n\nWelcome to Brim Cloud!\n\nWe've spent the past few years building the open-source\n[Zed project](https://github.com/brimdata/zed)\nand are now building a cloud service to provide an easy-to-use\n[Zed lake](https://github.com/brimdata/zed/tree/main/docs/lake).\n\nLet us know if you would like to be a pre-alpha tester, free of charge.\nYou will be helping us build out our cloud product and your participation\nwill be greatly appreciated.\nTo sign up for the pre-alpha test, please email us at cloud-alpha@brimdata.io\nand we'll get the conversation going.\n\n## Signing Up\n\nWhen you sign up, you will provide us your preferred email\nand we will enable your access via [auth0](https://auth0.com/).\nYou will get a standard account registration email from auth0\nto establish your account.\n\nWe will provide a custom DNS name pointing to a dedicated EC2 instance\nto sign into the service via your auth0 credentials from one of\nour clients.\n\nYour Zed data will be stored in one of our S3 buckets, which we can make\navailable to you for direct migration or backup.  You can copy all of the lake data\nto your own domain (e.g., your own S3 bucket or local filesystem) and run your\nown instance of `zed serve` to precisely recreate the lake from\nthe Brim Cloud.\n\n## Using the Zed Lake\n\nYou can access your Zed lake in the Brim cloud using\n* from your desktop using [the Brim app](#the-brim-app)\n* from the command-line using [`zed` CLI command](#the-cli)\n* from your Python using our [Python client](#the-python-client)\n\nFor the configuration options below, we will assume the host name we provided\nyou at signup is `\u003ccustom\u003e.lake.brimdata.io`.  You should replace\n`\u003ccustom\u003e` with the DNS name we assigned you.\n\nWhen you first connect, there will be a test pool set up for you called \"Welcome\"\nwith some sample Zed data.\n\n### The Brim App\n\nThe Brim App can communicate with and switch between different instances\nof Zed lakes, including the local Zed lake it automatically launches\non `localhost`.  Lake connections are a bit like Slack teams, where Slack\nlets you easily switch between teams.\n\nTo connect to the cloud from the Brim app, click on the lake dropdown\nin the upper left area of the window where you see the down carrot\nand click on _Add Lake..._.  Once the cloud connection has been created,\nyou can use this dropdown to switch between your local lake and the cloud\n(as well as other cloud lakes or lakes that you set up on your own server\nwith `zed serve`).\n\nUnder the _Name_ input, provide a name for the lake connection that you would like to\nappear in listing of available lakes.  Under the Host input, provide the URL\nof the lake service, in this case, `https://\u003ccustom\u003e.lake.brimdata.io`\n\nYou can import some sample queries relevant to the Welcome pools\ninto the app's query library by\ndownloading [queries.json](./queries.json?raw=1).\nYou can simply drag this file onto the **Queries** panel in the Brim app\nand this will add a query folder called \"Welcome\".\nYou may also import this file by clicking on the `+` button\nin the **Queries** panel and selecting _Import from JSON_.\n\n### The CLI\n\nThe `zed` command lets you access all of the functionality exposed to the Brim\nApp on the command line. Type `zed -h` to get online help.\n\nThe quickest way to get the `zed` command on macOS, Linux, or Windows is to [download\na pre-built release](https://github.com/brimdata/zed/releases).\n\nTo use the Brim cloud with `zed`, you can use the `-lake` command-line option\nbut for all the examples here, we will use presume you set the\nfollowing environment variable to point at your cloud instance:\n```\nexport ZED_LAKE=https://\u003ccustom\u003e.lake.brimdata.io\n```\n\u003e Note that the service is available on the default SSL port rather than\n\u003e the `zed serve` default port of 9867.\n\nAuthenticate your lake connection using `zed auth login`:\n```\nzed auth login\n```\nThis will launch your browser with an authentication dialogue pointing\nat the auth0 service for Brim and command will print a confirmation code\non the terminal.  Once you confirm in the auth0 web dialogue, `zed auth login`\nwill report success and you will be logged in.\n\nNote that `zed auth login` will store your access credentials in `$HOME/.zed/credentials.json`.\n\nOnce authenticated, check that the service is working\nby running a `query` command on your \"welcome\" data:\n```\nzed query \"from Welcome\"\n```\n\u003e Note that you can delete the `Welcome` pool at any time.\n\u003e It is just here to help you get started.  You can always\n\u003e reproduce it by creating the pool and copying\n\u003e [the data](welcome.zson) from this repo and optionally dragging the\n\u003e [Welcome queries](queries.json) into your app's query library.\n\n### The Python Client\n\nTo access your lake via the Python client, first authenticate as described\nabove using `zed auth login` as the Python client requires access to your login\ncredentials stored in `$HOME/.zed/credentials.json`.\n\nIf you haven't installed the Python `zed` module, run\n```\npip3 install \"git+https://github.com/brimdata/zed#subdirectory=python/zed\"\n```\nRun a test by creating a pool, loading data from Python, and reading it back:\n```\nimport zed\nimport os\nc = zed.Client(os.environ.get('ZED_LAKE'))\nc.create_pool('test')\nc.load('test', '{a:1}{a:2}{a:3}')\nfor v in c.query('from test'):\n  print(v)\n```\n\n## Performance Caveats\n\nIf you do many small loads, the queries will run slow because the current\nlake implementation does not yet implement the planned design.  See\n[issue 2936](https://github.com/brimdata/zed/issues/2936).\n\nCompaction is coming soon.\n\nIf you pull very large result sets down into the Python client, the network\ntransfer using ZJSON will run much slower than ZNG.  (We plan to add native\nZNG support to the Python client but it's not yet implemented.)  If you want\nto retrieve large data sets from your Zed lake it's best to use `zed query`,\nwhich defaults to the ZNG format (unless you send the output to a terminal,\nin which case the textual ZSON format is default).\n\n## Support\n\nIf you run into problems or have questions, please reach out to us\non our public [Slack channel](https://www.brimdata.io/join-slack).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrimdata%2Fcloud","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrimdata%2Fcloud","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrimdata%2Fcloud/lists"}