{"id":13550981,"url":"https://github.com/appbaseio/abc","last_synced_at":"2025-07-03T22:37:48.310Z","repository":{"id":21472109,"uuid":"92479010","full_name":"appbaseio/abc","owner":"appbaseio","description":"Power of appbase.io via CLI, with nifty imports from your favorite data sources","archived":false,"fork":false,"pushed_at":"2023-10-08T11:53:09.000Z","size":34255,"stargazers_count":469,"open_issues_count":29,"forks_count":47,"subscribers_count":10,"default_branch":"dev","last_synced_at":"2025-04-03T01:33:56.896Z","etag":null,"topics":["appbase","cli","elasticsearch","etl"],"latest_commit_sha":null,"homepage":"https://medium.appbase.io/open-sourcing-abc-import-import-mongodb-sql-json-csv-data-into-elasticsearch-8090965c4340","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/appbaseio.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-05-26T06:22:41.000Z","updated_at":"2025-02-11T21:33:21.000Z","dependencies_parsed_at":"2024-01-14T15:23:35.498Z","dependency_job_id":"7b7fd39e-a655-4edd-9ad9-5847e2b7b793","html_url":"https://github.com/appbaseio/abc","commit_stats":null,"previous_names":[],"tags_count":35,"template":false,"template_full_name":null,"purl":"pkg:github/appbaseio/abc","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/appbaseio%2Fabc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/appbaseio%2Fabc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/appbaseio%2Fabc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/appbaseio%2Fabc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/appbaseio","download_url":"https://codeload.github.com/appbaseio/abc/tar.gz/refs/heads/dev","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/appbaseio%2Fabc/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261978909,"owners_count":23239417,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["appbase","cli","elasticsearch","etl"],"created_at":"2024-08-01T12:01:40.644Z","updated_at":"2025-06-26T01:05:35.756Z","avatar_url":"https://github.com/appbaseio.png","language":"Go","readme":"\n[![latest version](https://img.shields.io/badge/version-0.10.1-yellow.svg)](https://github.com/appbaseio/abc/releases/latest)\n[![Go Report Card](https://goreportcard.com/badge/github.com/appbaseio/abc)](https://goreportcard.com/report/github.com/appbaseio/abc)\n[![Travis branch](https://travis-ci.org/appbaseio/abc.svg?branch=dev)]()\n[![license](https://img.shields.io/github/license/appbaseio/abc.svg)]()\n[![ABC Docker Publish](https://github.com/appbaseio/abc/actions/workflows/docker-publish.yaml/badge.svg)](https://github.com/appbaseio/abc/actions/workflows/docker-publish.yaml)\n\n![abc banner image](https://user-images.githubusercontent.com/4047597/29240054-14e0e19a-7f7b-11e7-898b-ba6bad756b1d.png)\n\n# ABC\n\nABC is a command-line tool to interact with appbase.io.\nIt can also serve as a swiss army knife to import data from any popular data source (Postgres, SQL, Mongo) to ElasticSearch.\nThis feature works with minimum configuration and is totally automatic.\nIn certain sources like Postgres and Mongo, you can even keep the database and ElasticSearch cluster in sync such that any change from source gets added in destination as well.\n\n\n1. [Intro](#intro)\n2. [Key Benefits](#key-benefits)\n3. [Getting Started](#getting-started)\n4. [Features](#features)\n\t1. [Appbase features](#appbase-features)\n\t2. [Importer features](#importer-features)\n5. [Development setup](#development)\n\t1. [Local Setup](#local-setup)\n\t2. [Docker Setup](#docker-setup)\n6. [ABC Resources](#abc-resources)\n\t1. [Contributing to ABC](#contributing-to-abc)\n\t2. [Licensing](#licensing)\n\n\u003ca name=\"intro\"\u003e\u003c/a\u003e\n## 1. Intro\n\nABC consists of two parts.\n\n1. Appbase module\n2. Import module (closed source)\n\nTo get the list of all commands supported by ABC, use -\n\n```sh\nabc --help\n```\n\n\n\u003ca name=\"key-benefits\"\u003e\u003c/a\u003e\n## 2. Key Benefits\n\nABC comes with a lots of benefits over any other traditional solution to the same problem. Some of the key points are as follows -\n\n- Whether your data resides in Postgres or a JSON file or MongoDB or in all three places, abc can index the data into Elasticsearch. Besides these, it also supports CSV, MySQL, SQLServer, Kafka and Elasticsearch itself to an Elasticsearch index.\n- It can keep the Elasticsearch index synced in realtime with the data source. (Note: Currently only supported for MongoDB and Postgres)\n- `abc import` is a single line CLI command that allows doing all of the above. It doesn’t require any external dependencies, takes zero lines of code configuration, and runs as an isolated process with a minimal resource footprint.\n- abc also supports configurable user defined transformations for advanced uses to map data types, columns or transform the data itself before it gets indexed into Elasticsearch.\n\n\n\n\u003ca name=\"getting-started\"\u003e\u003c/a\u003e\n## 3. Getting Started\n\nABC can be downloaded as an executable as well as through a Docker image.\n\n#### Using Executable\n\nDownload `abc`'s executable [from releases](https://github.com/appbaseio/abc/releases/latest) for your platform and preferrably put it in a PATH directory.\nThe access it as -\n\n```sh\n\u003e abc\n```\n\nYou should see a list of commands that `abc` supports.\nTry logging in for example.\n\n#### Using Docker\n\nTo use the Docker image, pull it as\n\n```sh\ndocker pull appbaseio/abc\n```\n\nThen create the volume to store config files across containers.\n\n```sh\ndocker volume create --name abc\n```\n\nFinally you should be able to use `abc`\n\n```sh\ndocker run -i --rm -v abc:/root appbaseio/abc\n```\n\nThis command may look too long to you. We can create an alias to make things better.\n\n```sh\n# create alias\nalias abc='docker run -i --rm -v abc:/root appbaseio/abc'\n# run a command\nabc login google\n```\n\n\n\n\u003ca name=\"features\"\u003e\u003c/a\u003e\n## 4. Features\n\nABC's features can be broadly categorized into 2 components.\n\n1. Appbase features\n2. Importer features\n\n\u003ca name=\"appbase-features\"\u003e\u003c/a\u003e\n### 4.1 Appbase features\n\nAppbase features allows you to control your appbase.io account using ABC. You can see them under the *Appbase* heading in the list of commands.\n\n```sh\nCOMMANDS\n  login     login into appbase.io\n  user      get user details\n  apps      display user apps\n  app       display app details\n  create    create app\n  delete    delete app\n  logout    logout session\n  import    import data from various sources into appbase app\n```\n\nYou can look over help for each of these commands using the `--help` switch.\nAlternatively we have detailed docs for them at [docs/appbase folder](docs/appbase).\n\n\n```sh\nabc login --help\n```\n\n#### Example\n\n```sh\n# display all commands\nabc\n# login into system\nabc login google\n# get user details\nabc user\n# get list of apps\nabc apps\n# get details of an app\nabc app MyAppName\n# delete that app\nabc delete MyAppName\n# create it again\nabc create MyAppName\n# view its metrics. It will be 0 as it is a new app\n# here we are using AppID. We can use AppName too.\nabc app -m 2489\n```\n\n\u003ca name=\"importer-features\"\u003e\u003c/a\u003e\n### 4.2 Importer features\n\nABC allows the user to configure a number of data adaptors as sources or sinks. These can be databases, files or other resources. Data is read from the sources, converted into a message format, and then send down to the sink where the message is converted into a writable format for its destination. The user can also create data transformations in JavaScript which can sit between the source and sink and manipulate or filter the message flow.\n\nAdaptors may be able to track changes as they happen in source data. This \"tail\" capability allows a ABC to stay running and keep the sinks in sync.\nFor more details on adaptors, see [Import docs](docs/appbase/import.md).\n\n\n\n\u003ca name=\"development\"\u003e\u003c/a\u003e\n## 5. Development\n\nABC can be built locally via the traditional `go build` or by building a Docker image.\n\n\u003ca name=\"local-setup\"\u003e\u003c/a\u003e\n### 5.1 Local Setup\n\nYou can install ABC by building it locally and then moving the executable to anywhere you like.\n\nTo build it, you will require **Go 1.12** or above installed on your system.\n\n```sh\ngo build ./cmd/abc/...\n./abc --help  # voila, you just built abc from source!\n```\n\n\u003ca name=\"docker-setup\"\u003e\u003c/a\u003e\n### 5.2 Docker Setup\n\n```sh\ngit clone https://github.com/appbaseio/abc\ncd abc\ndocker build -t abc .\ndocker volume create --name abc\n```\n\nVolume is used to store abc config files across containers.\nNow `abc` can be ran through Docker like in the following example which starts google login.  \n\n```sh\ndocker run -i --rm -v abc:/root abc login google\n```\n\nSome more examples\n\n```sh\n# setting alias for easy usage\nalias abc='docker run -i --rm -v abc:/root abc'\n# using alias now :)\nabc user\nabc apps\n```\n\n\u003ca name=\"build-variants\"\u003e\u003c/a\u003e\n### 5.3 Build Variants\n\nThe ABC project you see in this repository is not the complete project. Appbase.io works on a proprietary version of ABC using this project as the base.\n\n\n\n\u003ca name=\"abc-resources\"\u003e\u003c/a\u003e\n## 6. ABC Resources\n\nCheckout the [docs folder](docs/) for details on some ABC commands and topics.\n\n\u003ca name=\"contributing-to-abc\"\u003e\u003c/a\u003e\n\n### 6.1 Contributing to ABC\n\nWant to help out with ABC? Great! There are instructions to get you started [here](CONTRIBUTING.md).\n\n\u003ca name=\"licensing\"\u003e\u003c/a\u003e\n\n### 6.2 Licensing\n\nABC is licensed under the Apache 2.0 License. See [LICENSE](LICENSE) for full license text.\n","funding_links":[],"categories":["Go","cli"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fappbaseio%2Fabc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fappbaseio%2Fabc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fappbaseio%2Fabc/lists"}