{"id":19434134,"url":"https://github.com/mastermindg/farmstead","last_synced_at":"2026-07-03T22:02:50.286Z","repository":{"id":56845591,"uuid":"115543530","full_name":"mastermindg/farmstead","owner":"mastermindg","description":"Farmstead - A data pipeline platform","archived":false,"fork":false,"pushed_at":"2018-02-20T19:53:23.000Z","size":170,"stargazers_count":0,"open_issues_count":4,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-07T23:20:18.297Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mastermindg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-12-27T17:33:33.000Z","updated_at":"2017-12-27T17:34:08.000Z","dependencies_parsed_at":"2022-09-17T22:02:08.104Z","dependency_job_id":null,"html_url":"https://github.com/mastermindg/farmstead","commit_stats":null,"previous_names":[],"tags_count":84,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mastermindg%2Ffarmstead","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mastermindg%2Ffarmstead/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mastermindg%2Ffarmstead/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mastermindg%2Ffarmstead/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mastermindg","download_url":"https://codeload.github.com/mastermindg/farmstead/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240614235,"owners_count":19829307,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T14:44:49.227Z","updated_at":"2025-10-28T16:16:07.913Z","avatar_url":"https://github.com/mastermindg.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# farmstead\n\n[![Gem Version](https://badge.fury.io/rb/farmstead.svg)](https://badge.fury.io/rb/farmstead)\n[![Build Status](https://api.travis-ci.org/mastermindg/farmstead.svg?branch=master)](http://travis-ci.org/mastermindg/farmstead)\n\nFarmstead is a modular data pipeline platform. Farmstead makes creating and deploying a fully-functional data pipeline a snap. Farmstead uses containers to encapsulate the middleware which allows for a super-fast deployment and prototyping process.\n\n\n## Table of Contents\n\n\u003c!-- TOC depthFrom:1 depthTo:6 withLinks:1 orderedList:0 --\u003e\n\n- [Getting started](#getting-started)\n    - [Configuration](#configuration)\n        + [Configuration Options](#config-options)\n    - [Environment Variables](#env)\n    - [Deployment Methods](#deployment)\n- [Architecture](#architecture)\n- [License](#license)\n\n\u003c!-- /TOC --\u003e\n\n\n## Getting Started\n\nTo get started you'll need to install the Farmstead gem:\n\n```ruby\ngem install farmstead\n```\n\nTo create a new Farmstead project:\n\n```\nfarmstead new myproject\n```\n\nOnce the project is created cd to the directory and deploy to get started:\n\n```\ncd myproject\nfarmstead deploy\n```\n\n### Configuration\n\nFarmstead tries to follow Rails conventions. Some of the configuration options are available from the command-line. For instance, to chose a different database techhnology use the -d flag:\n\n```\nfarmstead new myproject -d postgres\n```\n\nThere are default values for database users and passwords. For more advanced usages you can use a configuration file. A configuration file must be written in YAML and is passed to Farmstead via command-line:\n\n```\nfarmstead new myproject -c myproject.yml\n```\n\nAn example configuration file is included.\n\n#### Configuration Options\n\n**Database**\n\nThe default database is MySQL but can be set to either MySQL, Postgres, or SQLLite. Extensions will be available \n\n**Kafka**\n\nThe default is to advertise the IP address assigned to the host. If you're behind a firewall or a load-balancer and want to change it you anything you want. Here's an example:\n\n```yaml\nkafka:\n    - advertise_from_local_ip: false\n    - advertised_ip: 192.168.1.2\n```\n\nYou can use a custom Zookeeper cluster if you have one. Just set the zookeeper_address in the config.\n\nYou can also create custom topics outside of the default Wood, Field, Forest, and Road. \n\n### Environment Variables\n\nFarmstead builds projects on Docker. In order to keep secrets out of the Rails code we use environment varibles. These are read from the .env file at the root of the project. This is IGNORED in Git, so be sure to keep these safe in a deployment/build system (i.e. Jenkins).\n\nKeep in mind that you can always customize the Rails application using further using more environment variables as well.\n\n### Deployment Methods\n\nThe default method is Docker. It is also possible to deploy using Rancher and Kubernetes (on top of Docker). To deploy using another method use the -x flag:\n\n\n```\nfarmstead new myproject -x kubernetes\n```\n\n## Architecture\n\nKafka and Database \n\nETL\n\n* Extract\n* Transform\n* Load\n\nAll of the services are only running a Kafka consumer and producer. There is a Manager service that manages the flow.\n\nProjects are built with their own classes to allow extending the API.\n\n**Classes:**\n\n### Farmstead::Manager\n\nTask scheduling, batch processing, and general flow control. Exposes a very simple web service where you can pull logs and see the data in real-time.\n\n### Farmstead::Extract\n\nExtracts the data from the source.\n\n### Farmstead::Transform\n\nTransforms one or more datasets.\n\n### Farmstead::Load\n\nLoads the data into a database.\n\n**Test**\ncurl -X PUT -H \"Accept: application/json\" -d '{ \"name\": \"test\", \"type\": \"test\", \"module\": \"Test\" }' http://localhost:3000/api/v1/source\n\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmastermindg%2Ffarmstead","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmastermindg%2Ffarmstead","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmastermindg%2Ffarmstead/lists"}