{"id":27881776,"url":"https://github.com/src-d/jgscm","last_synced_at":"2025-05-05T05:05:04.535Z","repository":{"id":62572343,"uuid":"65671187","full_name":"src-d/jgscm","owner":"src-d","description":"Jupyter support for Google Cloud Storage","archived":false,"fork":false,"pushed_at":"2019-06-03T12:06:22.000Z","size":63,"stargazers_count":45,"open_issues_count":7,"forks_count":20,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-05-05T05:04:57.121Z","etag":null,"topics":["gcs","google-cloud-storage","jupyter","notebook"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/src-d.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-08-14T14:48:46.000Z","updated_at":"2021-09-01T20:25:11.000Z","dependencies_parsed_at":"2022-11-03T20:38:48.028Z","dependency_job_id":null,"html_url":"https://github.com/src-d/jgscm","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/src-d%2Fjgscm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/src-d%2Fjgscm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/src-d%2Fjgscm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/src-d%2Fjgscm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/src-d","download_url":"https://codeload.github.com/src-d/jgscm/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252442485,"owners_count":21748451,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gcs","google-cloud-storage","jupyter","notebook"],"created_at":"2025-05-05T05:05:03.834Z","updated_at":"2025-05-05T05:05:04.519Z","avatar_url":"https://github.com/src-d.png","language":"Python","readme":"Jupyter FS in Google Cloud Storage [![PyPI](https://img.shields.io/pypi/v/jgscm.svg)](https://pypi.python.org/pypi/jgscm) [![Build Status](https://travis-ci.org/src-d/jgscm.svg?branch=master)](https://travis-ci.org/src-d/jgscm) [![codecov](https://codecov.io/github/src-d/jgscm/coverage.svg)](https://codecov.io/gh/src-d/jgscm)\n========================================\n\nJupyter Google Storage Contents Manager allows working with Jupyter notebooks directly in Google Cloud Storage.\nIt aims to be a complete drop-in replacement for the stock filesystem\n[ContentsManager](http://jupyter-notebook.readthedocs.io/en/latest/extending/contents.html).\nThus JGSCM is only compatible with a relatively modern IPython/Jupyter stack (version 4 and above).\n\nThe root level of the virtual file system is the list of buckets, which\nare presented as directories. In turn, each bucket is presented as an\nordinary folder where users can create files, subdirectories and notebooks.\nBesides, snapshots are completely supported too.\n\nInstallation\n------------\n```\npip install jgscm\n```\nYou must point to jgscm in [Jupyter settings](http://jupyter-notebook.readthedocs.io/en/latest/config.html).\nUsually you'd need to edit `~/.jupyter/jupyter_notebook_config.py` and\ninsert the following:\n```python\nc.NotebookApp.contents_manager_class = 'jgscm.GoogleStorageContentManager'\n# c.GoogleStorageContentManager.project = ''\n# c.GoogleStorageContentManager.keyfile = ''\n```\n`project` and `keyfile` must be set if [gcloud](https://github.com/GoogleCloudPlatform/gcloud-python)\ncannot determine the defaults. Read more about it in one of the next sections.\n\nContributions\n-------------\n...are welcome! See [CONTRIBUTING](CONTRIBUTING.md) and [code of conduct](CODE_OF_CONDUCT.md).\n\nLicense\n-------\nMIT, see [LICENSE](LICENSE.md).\n\nUsage\n-----\nJust launch and use Jupyter as usual.\n\nOn the root level, you may not create files, only directories and the latter\nare considered as new buckets. On deeper levels, there should be no difference.\nYou can change the Google Cloud project in order to work with\ncorresponding buckets in Jupyter configuration (see [Projects and keyfiles](#projects-and-keyfiles)).\n\nGCS treats blobs with names ending with \"/\" as folders and so does this\nproject. Since there is no special handling of directories in GCS,\nwhen you delete some blob which is the only one in it's parent directory,\nand the parent directory was not created explicitly, it will disappear as well.\nThis behavior is similar to some old-style source control systems.\n\nGCS API invocations can take some time. While JGSCM does it's best to reduce\nthe number of calls, they still can introduce substantial delays in\nJupyter UI. Please, be patient.\n\nThere is an ability to specify the starting path instead of the buckets listing:\n```python\nc.GoogleStorageContentManager.default_path = 'path/without/starting/slash'\n```\n(`--notebook-dir` does not seem to work).\n\nCheckpoints\n-----------\nCheckpoints are stored in .ipynb_checkpoints directory as usual. That\nname can be changed via `c.GoogleStorageCheckpoints.checkpoint_dir` in\nJupyter configuration. You can set the dedicated bucket for snapshots via\n`c.GoogleStorageCheckpoints.checkpoint_bucket`.\n\nThe name of each checkpoint is \\\u003cnotebook name\\\u003e-[UUID4](https://en.wikipedia.org/wiki/Universally_unique_identifier).ipynb.\n\nHidden files and directories\n----------------------------\nAs with any UNIX filesystem, files and directories with names starting\nwith dot \"`.`\" are considered hidden by default. You can change this by\nsetting `c.GoogleStorageContentManager.hide_dotted_blobs` to `False`.\n\nProjects and keyfiles\n---------------------\nUsually, if you launch Jupyter in Google Cloud, the default project is picked\nup and no special keys and tokens must be supplied. The same is true\nif you launch Jupyter in the environment which was configured for\nGoogle Cloud (`gcloud init`). If this is not the case, you can explicitly\nset the Google Cloud project and authentication credentials.\n\nOpen Jupyter configuration and set\n```python\nc.GoogleStorageContentManager.project = '...'\nc.GoogleStorageContentManager.keyfile = '...'\n```\n\nRegarding keyfiles, please read the [official documentation](https://cloud.google.com/storage/docs/authentication).\nGo to API Manager / Credentials in cloud console to generate it:\n\n![API Manager / Credentials](docs/keyfile.png)\n\nTesting\n-------\n```\nPYTHONPATH=`pwd` python3 -W ignore::DeprecationWarning jgscm/tests/test.py\n```\nJGSCM writes logs at DEBUG verbosity level (`c.Application.log_level = \"DEBUG\"`).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsrc-d%2Fjgscm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsrc-d%2Fjgscm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsrc-d%2Fjgscm/lists"}