{"id":13558175,"url":"https://github.com/theodi/octopub","last_synced_at":"2026-03-12T09:11:28.129Z","repository":{"id":21523743,"uuid":"24842952","full_name":"theodi/octopub","owner":"theodi","description":"Publish data easily, quickly and correctly","archived":false,"fork":false,"pushed_at":"2023-01-19T10:58:05.000Z","size":4038,"stargazers_count":41,"open_issues_count":78,"forks_count":18,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-12-06T20:46:08.211Z","etag":null,"topics":["data-publication","pusher","rails","ruby","ruby-on-rails"],"latest_commit_sha":null,"homepage":"https://octopub.io/","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/theodi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-10-06T11:18:25.000Z","updated_at":"2022-11-11T02:15:58.000Z","dependencies_parsed_at":"2023-02-10T22:45:55.729Z","dependency_job_id":null,"html_url":"https://github.com/theodi/octopub","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theodi%2Foctopub","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theodi%2Foctopub/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theodi%2Foctopub/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theodi%2Foctopub/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/theodi","download_url":"https://codeload.github.com/theodi/octopub/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247002562,"owners_count":20867477,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-publication","pusher","rails","ruby","ruby-on-rails"],"created_at":"2024-08-01T12:04:47.677Z","updated_at":"2025-12-17T07:35:47.286Z","avatar_url":"https://github.com/theodi.png","language":"Ruby","funding_links":[],"categories":["Ruby","ruby"],"sub_categories":[],"readme":"[![Build Status](http://img.shields.io/travis/theodi/octopub.svg)](https://travis-ci.org/theodi/octopub)\n[![Dependency Status](http://img.shields.io/gemnasium/theodi/octopub.svg)](https://gemnasium.com/theodi/octopub)\n[![Coverage Status](http://img.shields.io/coveralls/theodi/octopub.svg)](https://coveralls.io/r/theodi/octopub)\n[![Code Climate](http://img.shields.io/codeclimate/github/theodi/octopub.svg)](https://codeclimate.com/github/theodi/octopub)\n[![License](http://img.shields.io/:license-mit-blue.svg)](http://theodi.mit-license.org)\n[![Dependency Status](https://dependencyci.com/github/theodi/octopub/badge)](https://dependencyci.com/github/theodi/octopub)\n[![Badges](http://img.shields.io/:badges-7/7-ff6799.svg)](https://github.com/badges/badgerbadgerbadger)\n\n\n# Octopub\n\n[Octopub](http://octopub.io/) is a Ruby on Rails application that provides a simple and frictionless way for users to publish data easily, quickly and correctly on GitHub.\n\n## Summary of features\n\nMore information is in the announcement [blog post](http://theodi.org/blog/removing-barriers-to-publishing-open-data)\n\nThe live instance of Octopub is running at [http://octopub.io/](http://octopub.io/)\n\nFollow the [public feature roadmap](https://trello.com/b/2xc7Q0kd/labs-public-toolbox-roadmap?menu=filter\u0026filter=label:Octopub) for Octopub\n\n## Requirements\n\nThese are the tools and services required to get Octopub fully working for development, testing and production environments. We'll explain how to set these up in the next section.\n\n* Ruby 2.4\n* PostgreSQL\n* Redis/Sidekiq\n* GitHub account\n* AWS account\n* Pusher account\n* [Open Data Certificates](https://certificates.theodi.org/en/) account\n\n## Setup\n\n### Redis/Sidekiq\n\nSidekiq is used for managing the background proccessing of data uploads. To use Sidekiq just install Redis by following the instructions [here](https://redis.io/topics/quickstart), or if you are using Homebrew you can just do ```brew install redis```.\n\n### Environment variables\n\nFor development Octopub uses the [dotenv](https://github.com/bkeepers/dotenv) gem to load environment variables. Create a file called ```.env``` in your project root and paste in the variables below. We'll fill these in as we go along.\n\n```\n# GitHub App Client ID \u0026 secret\nGITHUB_KEY=\nGITHUB_SECRET=\n\n# OAuth access token for GitHub API access\nGITHUB_TOKEN=\n\nS3_BUCKET=\n\nAWS_ACCESS_KEY_ID=\nAWS_SECRET_ACCESS_KEY=\n\nPUSHER_APP_ID=\nPUSHER_KEY=\nPUSHER_SECRET=\nPUSHER_CLUSTER=\n\nBASE_URI=\nODC_API_KEY=\nODC_USERNAME=\n\n# production only\nSMTP_USERNAME=\nSMTP_PASSWORD=\nSMTP_SERVER=\n```\n\n#### GitHub\n\nCreate a GitHub application:\n\n1. Log in to GitHub.\n2. In ```Settings -\u003e Developer settings -\u003e OAuth applications```, create a new OAuth application with a unique name. You can use ```http://octopub.io``` for the homepage and for the callback URL use your local server address, i.e. ```http://localhost:3000```. Click on your OAuth application to see your ```Client ID``` and ```Client Secret```, and update your ```.env``` file:\n\n```\nGITHUB_KEY=\u003cYOUR CLIENT ID\u003e\nGITHUB_SECRET=\u003cYOUR CLIENT SECRET\u003e\n```\n\n3. In ```Settings -\u003e Developer settings -\u003e Personal access tokens```, generate a new token with a sensible description, e.g. octopub_dev_token, and update your ```.env``` file:\n\n```\nGITHUB_TOKEN=\u003cYour token\u003e\n```\n\n#### AWS\n\nCreate an S3 bucket:\n\n1. In AWS go to the S3 service and create a bucket with a sensible name. **Make sure the region is set to EU(Ireland) since Octopub uses this.**\n2. Click on your bucket and go to the ```Permissions``` tab. Click on ```CORS Configuration``` and paste in the configuration below. This will allow your local development version of Octopub to make requests to your S3 bucket.\n```\n\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\u003cCORSConfiguration xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"\u003e\n\u003cCORSRule\u003e\n    \u003cAllowedOrigin\u003ehttp://localhost:3000\u003c/AllowedOrigin\u003e\n    \u003cAllowedMethod\u003eGET\u003c/AllowedMethod\u003e\n    \u003cAllowedMethod\u003ePOST\u003c/AllowedMethod\u003e\n    \u003cAllowedMethod\u003ePUT\u003c/AllowedMethod\u003e\n    \u003cAllowedHeader\u003e*\u003c/AllowedHeader\u003e\n\u003c/CORSRule\u003e\n\u003c/CORSConfiguration\u003e\n```\nGrant permissions to your bucket:\n\n1. In AWS go to the IAM (Identity and Access Management page) service.\n2. Click ```Users```.\n3. Add a new user and give it a name, e.g. octopub-development, and for Access Type select ```Programmatic Access```.\n4. For permissions, select ```Attach existing policies directly``` - this will open a new tab in your browser.\n5. Click ```create your own policy``` and give it a name, e.g. octopub-dev-permissions. Then for the policy document, use the following template, but add your bucket name in place of ```\u003cBUCKETNAME\u003e```.\n ```json\n{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n            \"Sid\": \"AllowAdminAccessToBucketOnly\",\n            \"Action\": [\n                \"s3:*\"\n            ],\n            \"Effect\": \"Allow\",\n            \"Resource\": [\n                \"arn:aws:s3:::\u003cYOUR BUCKET NAME\u003e\",\n                \"arn:aws:s3:::\u003cYOUR BUCKET NAME\u003e/*\"\n            ]\n        }\n    ]\n}\n```\n6. Click ```validate policy``` just to be sure you've not made a typo, then confirm.\n7. Back on the ```Set permissions page```, select the policy you've just created in the table by selecting the checkbox, then click ```Review``` and then click ```Create user```.\n8. Download the CSV file containing your ```Access key ID``` and ```Secret access key``` and update your ```.env``` file:\n\n```\nAWS_ACCESS_KEY_ID=\u003cYOUR ACCESS KEY ID\u003e\nAWS_SECRET_ACCESS_KEY=\u003cYOUR SECRET ACCESS KEY\u003e\nS3_BUCKET=\u003cYOUR BUCKET NAME\u003e\n```\n\n#### Pusher\n\n1. Log in to https://pusher.com or create a free account.\n2. Create a new application and call it something sensible.\n3. Select the ```App Keys``` tab and use the relevant values there to update your ```.env``` file:\n\n```\nPUSHER_APP_ID=\nPUSHER_KEY=\nPUSHER_SECRET=\nPUSHER_CLUSTER=\n```\n\n#### ODC (open data certificate) setup\n\n1. Log in to https://certificates.theodi.org/ or create a free account.\n2. Go to your profile page, copy your API token and update your ```.env``` file:\n\n```\nODC_API_KEY=\u003cAPI TOKEN\u003e\nODC_USERNAME=\u003cYOUR USERNAME (email address you used when signing up)\u003e\n```\n\n## Running the full application locally\n\nAssuming you have completed the setup instructions above...\n\n* Start Redis with ```redis-server```.\n* Start Sidekiq with ```bundle exec sidekiq``` in the application directory.\n* Create the postgresql databases specified in ```config/database.yml``` and run ```rails db:migrate```.\n* Start Octopub with ```rails s``` in the application directory. \n* Navigate to the home page.\n* Sign into octopub with your GitHub account.\n\nCongratulations, you should be signed in! Now try adding some data.\n\n### Checking the Sidekiq queue\n\nStart a rails console session and then...\n\n```\nrequire 'sidekiq/api'\nSidekiq::Queue.new.size\nSidekiq::Queue.new.first\n```\n### Tests\n\nOctopub uses the ```rspec``` test framework and requires the presence of a ```.env```. See earlier section for details as you can (re)use your development variables*\n\nThe test suite can be run with ```bundle exec rspec```.  \n\n\\* Note - the tests use VCR or mocking to allow the tests to be run offline without interfacing with the services.\n\n## Deployment\n\nA commit to master will trigger a TravisCI run; If successful it will automatically deploy to Heroku.\n\n## Caching\n\nThe GitHub organisations are cached for the logged in user. They can be cleared from a console with ```Rails.cache.clear```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftheodi%2Foctopub","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftheodi%2Foctopub","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftheodi%2Foctopub/lists"}