{"id":18351604,"url":"https://github.com/Netflix/nf-data-explorer","last_synced_at":"2025-04-06T11:32:34.565Z","repository":{"id":49760777,"uuid":"378240844","full_name":"Netflix/nf-data-explorer","owner":"Netflix","description":"The Data Explorer gives you fast, safe access to data stored in Cassandra, Dynomite, and Redis.","archived":false,"fork":false,"pushed_at":"2023-04-10T10:57:57.000Z","size":9643,"stargazers_count":434,"open_issues_count":6,"forks_count":41,"subscribers_count":189,"default_branch":"master","last_synced_at":"2025-03-30T02:09:29.183Z","etag":null,"topics":["cassandra","dynomite","nodejs","redis","vuejs"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Netflix.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.MD","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-18T18:55:58.000Z","updated_at":"2025-03-26T02:22:05.000Z","dependencies_parsed_at":"2024-10-12T19:02:40.048Z","dependency_job_id":"733b6734-335d-4a39-aff0-dee661bba743","html_url":"https://github.com/Netflix/nf-data-explorer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Netflix%2Fnf-data-explorer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Netflix%2Fnf-data-explorer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Netflix%2Fnf-data-explorer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Netflix%2Fnf-data-explorer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Netflix","download_url":"https://codeload.github.com/Netflix/nf-data-explorer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247430869,"owners_count":20937874,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cassandra","dynomite","nodejs","redis","vuejs"],"created_at":"2024-11-05T21:32:05.993Z","updated_at":"2025-04-06T11:32:34.544Z","avatar_url":"https://github.com/Netflix.png","language":"TypeScript","readme":"# Netflix Data Explorer\n\nThe Netflix Data Explorer tool allows users to explore data stored in several popular datastores (currently Cassandra, Dynomite, and Redis).\n\n## Quick Start\n\nTo help get you started, we have provided a demo environment that you can run via Docker. You'll need to run a couple of commands to get up and running ([install Yarn](https://yarnpkg.com/getting-started/install) first if you don't already have it).\n\nNote: if using Apple silicon, you will need to run the Terminal using [Rosetta](https://support.apple.com/en-us/HT211861).\n\n```bash\nyarn\nyarn docker:demo\n```\n\nThis will run a production build of the app, a Cassandra instance, and a Redis instance, all locally.\n\n![Docker PS](https://github.com/Netflix/nf-data-explorer/blob/master/.github/docker_ps.png?raw=true 'Docker PS')\n\nNote: the first time you run this command, it will take some time to pull down the images and run a complete build. Once the app starts, your browser will open the app. You may also need to wait a minute while C\\* and Redis startup. Run the command, and then grab some ☕️.\n\nAfter the first invocation, future invocations of `yarn docker:demo` will be much faster.\n\nAlso note: If you see a \"No Hosts Available\" error, it's likely due to C\\* still starting up.\n\n### Docker Configuration\n\nTo run the demo environment, you will need to allocate 2GB of memory to Docker. This is easily done if using Docker for Windows or Mac by adjusting the slider on the [Resources tab of the Preferences page](https://docs.docker.com/docker-for-mac/#resources). Note, 2GB is also the current default for Docker for Window or Mac.\n\nYou can also run `docker system info | grep Memory` to view your allocated memory if you installed Docker via some other means (e.g. VirtualBox/brew/etc). Please see the documentation for your particular Docker installation on how to change these settings.\n\n## Developing/Contributing\n\nIf you are thinking about contributing, please be sure to check out [Contributing Guidelines](https://github.com/Netflix/nf-data-explorer/blob/master/CONTRIBUTING.MD).\n\n## Description\n\nThe Netflix Data Explorer strives to be a turn-key solution for connecting to Cassandra and Dynomite/Redis datastores. It was developed for internal use at Netflix to codify some of our best practices and help our engineers quickly access their data.\n\nWe have provided integration hooks so you can use the Data Explorer in your environment. Since all environments are unique, many configuration overrides can be specified to adapt the app for your particular use case. For instance, you might have clusters with C\\* authentication enabled, or clusters that are discovered by polling a REST service, etc. We've provided seams so you can integrate accordingly.\n\n## Custom Environments\n\nIf you want to experiment connecting to Cassandra or Redis clusters in your environment (not using the provided Docker environment), you will need to generate an overridden config file.\n\n```typescript\n// example custom config file\n\n// only support C* (no Redis)\nexport const SUPPORTED_DATASTORE_TYPES = ['cassandra'];\n\n// discovery settings\nexport const DISCOVERY_PROVIDER = 'FileSystemDiscoveryProvider'; // read a file to get information about available clusters\nexport const DISCOVERY_PROVIDER_FILESYSTEM_SOURCE = 'discovery.json'; // provides cluster discovery information\nexport const ENVIRONMENTS = ['test'];\nexport const REGIONS = ['us-east-1'];\n\n// disable C* authentication\nexport const CASSANDRA_BASE_AUTH_PROVIDER_USERNAME = undefined;\nexport const CASSANDRA_BASE_AUTH_PROVIDER_PASSWORD = undefined;\n\n// hypothetical test environment uses a custom port\nexport const CASSANDRA_PORT = 7199;\n\n// use a custom class to provide C* connection options\nexport const CASSANDRA_CLIENT_OPTIONS_PROVIDER =\n  'CustomCassandraClientOptionsProvider';\n```\n\n### Generating a custom config file via the CLI\n\nWhile a configuration override file can be crafted by hand, we recommend using the provided CLI tool to help you generate it. Please follow all prompts from the tool.\n\n```bash\nyarn # only required if you haven't run yarn up to this point\nyarn setup\n```\n\n![Yarn Setup](https://github.com/Netflix/nf-data-explorer/blob/master/.github/yarn_setup.png?raw=true 'Yarn Setup')\n\nOnce your config file is generated, the CLI will print the startup commands for you.\n\n![Yarn Setup Startup](https://github.com/Netflix/nf-data-explorer/blob/master/.github/yarn_setup_startup.png?raw=true 'Yarn Setup Startup')\n\n### Using the custom config file\n\nOnce you have a custom config file created, you have a few options on how you can use it:\n\n- Demo mode\n- Dev mode\n- Production mode\n\n#### Demo\n\nIf you want to use this custom config file with the Data Explorer docker demo image (e.g., running the Data Explorer in Docker, but using a config file that points to C\\* clusters in your network), you will need to update the `.env` file in the project root.\n\n```bash\n# .env file\nDATA_EXPLORER_CONFIG_NAME=my-custom-config\n```\n\nOnce you've updated the variable, you can re-run the command:\n\n```bash\nyarn docker:demo\n```\n\n#### Dev\n\nTo run in Dev mode directly from source, you can run `yarn dev`.\n\nIf you have a config file created, it will be used automatically. If you have multiple config files, the `yarn dev` command will pause and prompt you to choose one - this is handy if you are switching between environments while developing.\n\n```bash\n# install dependencies\nyarn\n\n# run a local dev server for the UI, start a node server in watch mode, and start a local C* and Redis cluster\nyarn dev\n\n# optional: tail the local C* and Redis cluster logs so you can see what the server is doing\nyarn docker:taillogs\n```\n\nThe UI will be available at [http://localhost:3000](http://localhost:3000). Please note the 3000 port. A WebPack Dev Server serves the UI, and requests are proxied to Node app running on port 80.\n\n#### Production\n\nTo run in Production mode, run the following commands. In Production mode, you will need to export the variable `DATA_EXPLORER_CONFIG_NAME` to use your file.\n\n```bash\n# install dependencies\nyarn\n\n# build for production\nyarn build\n\n# start the app using your config file\nexport DATA_EXPLORER_CONFIG_NAME=my-custom-config \u0026\u0026 yarn start\n```\n\nThe UI will be available at [http://localhost](http://localhost).\n\n## Re-building the Docker image\n\nThe `yarn docker:demo` command will start C\\*, Redis, and a dockerized version of the Data Explorer app. If you make changes to the source and want to rebuild the Data Explorer docker image, simply run:\n\n```bash\nyarn docker:build\n```\n\n## Features\n\n### Cassandra Features\n\nHere are a sampling of some of the features available in the Netflix Data Explorer.\n\n#### Multi-Cluster Access\n\nMulti-cluster access provides easy access to all of the clusters in your environment. The cluster selector in the top nav allows you to switch to any of your discovered clusters quickly.\n\n![Cluster Selector](https://github.com/Netflix/nf-data-explorer/blob/master/.github/cluster_selector.png?raw=true 'Cluster Selector')\n\n#### Explore Your Data\n\nThe Explore view provides a simple way to explore your data quickly. You can query by partition and clustering keys, insert and edit records, and easily export the results or download them as CQL statements.\n\n![Explore View](https://github.com/Netflix/nf-data-explorer/blob/master/.github/explore_view.png?raw=true 'Explore View')\n\nYou can also query and decode binary data.\n\n![Binary Data Support](https://github.com/Netflix/nf-data-explorer/blob/master/.github/blob_support.gif?raw=true 'Binary Data Support')\n\n#### Schema Designer\n\nCreating a new Keyspace and Table by hand can be error-prone\n\nOur schema designer UI streamlines creating a new Table with improved validation and enforcement of best practices.\n\n![Schema Designer](https://github.com/Netflix/nf-data-explorer/blob/master/.github/schema_designer.gif?raw=true 'Schema Designer')\n\n#### Query IDE\n\nThe Query Mode provides a powerful IDE-like experience for writing free-form CQL queries.\n\n![Query IDE](https://github.com/Netflix/nf-data-explorer/blob/master/.github/query_ide.gif?raw=true 'Query IDE')\n\n### Dynomite and Redis Features\n\n#### Key Scanning\n\nBrowsing through keys in an in-memory database like Redis can cause performance problems if you attempt to use the [KEYS](https://redis.io/commands/keys) command. Instead, it's recommended that you perform a cursor-based [SCAN](https://redis.io/commands/scan). The Data Explorer will perform SCANS for you in both Redis and Dynomite environments.\n\n![Dynomite Scan](https://github.com/Netflix/nf-data-explorer/blob/master/.github/dyno_scan.gif?raw=true 'Dynomite Scan')\n\n#### Key Edits\n\nRedis has support for rich data structures like lists, maps, and sorted sets. In addition, the Data Explorer supports viewing, creating, editing, and deleting these entities.\n\n![Editing Keys](https://github.com/Netflix/nf-data-explorer/blob/master/.github/redis_hash_key.png?raw=true 'Editing Keys')\n","funding_links":[],"categories":["Packages"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNetflix%2Fnf-data-explorer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FNetflix%2Fnf-data-explorer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNetflix%2Fnf-data-explorer/lists"}