{"id":13626168,"url":"https://github.com/square/shuttle","last_synced_at":"2025-06-13T01:39:04.119Z","repository":{"id":8814279,"uuid":"10511953","full_name":"square/shuttle","owner":"square","description":"String extraction, translation and export tools for the 21st century. \"Moving strings around so you don't have to\"","archived":false,"fork":false,"pushed_at":"2025-04-01T04:55:13.000Z","size":11676,"stargazers_count":660,"open_issues_count":12,"forks_count":102,"subscribers_count":49,"default_branch":"master","last_synced_at":"2025-04-10T15:23:04.791Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/square.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2013-06-05T20:18:58.000Z","updated_at":"2025-04-01T05:00:28.000Z","dependencies_parsed_at":"2024-11-08T15:39:33.258Z","dependency_job_id":"e49d70e4-84a4-4ce1-a04c-ce5c2c3b153a","html_url":"https://github.com/square/shuttle","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/square%2Fshuttle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/square%2Fshuttle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/square%2Fshuttle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/square%2Fshuttle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/square","download_url":"https://codeload.github.com/square/shuttle/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249235062,"owners_count":21235140,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T21:02:11.673Z","updated_at":"2025-04-16T11:31:25.271Z","avatar_url":"https://github.com/square.png","language":"Ruby","funding_links":[],"categories":["Ruby"],"sub_categories":[],"readme":"Shuttle: Magic localization dust [![Build Status](https://travis-ci.org/square/shuttle.svg?branch=master)](https://travis-ci.org/square/shuttle)\n================================\n\n\nShuttle is a website allowing for the automatic extraction and reintegration of\nlocalizable strings in a code base. It also provides an API where articles can be\nsubmitted for translation and retrieved. In addition, it provides a workflow for\ntranslators and reviewers optimized for the efficient processing of many\nstrings. Finally, for project managers, it provides a dashboard allowing them to\nview and manage the progress of a localization effort.\n\nShuttle can be thought of as a Continuous Integration system for translations.\n\nOne typical **Shuttle workflow** is as follows:\n\n1. An engineer makes a commit to a Project, and marks that commit as requiring\n   localization in Shuttle.\n2. Shuttle scans the commit for any localizable strings using its {Importer}s.\n3. New or modified localizable strings are marked as pending translation. They\n   appear on the translators’ dashboard.\n4. Translators translate all these strings. They then appear on the reviewers’\n   dashboard.\n5. Reviewers review and approve translations.\n6. Once all translations applying to a commit are approved for all of a\n   Project's required localizations, the Commit is marked as ready.\n7. When the commit is deployed, Shuttle provides a manifest of translated\n   strings that is downloaded as part of the deploy artifact.\n\nAnother typical **Shuttle workflow** is as follows:\n\n1. An engineer makes an API call using a Project's api_token to submit a\n   new {Article} (this can be an Article, Email, or anything else).\n2. Shuttle parses the {Article}, splits it up into small pieces of strings.\n3. Shuttle determines which strings need translation while optimizing for\n   efficiency and accuracy. These strings appear on the translators’ dashboard.\n4. Translators translate all these strings. They then appear on the reviewers’\n   dashboard.\n5. Reviewers review and approve translations.\n6. Once all translations applying to a {Article} are approved for all of a\n   Project's required localizations, the {Article} is marked as ready.\n7. An engineer makes an API call to retrieve the translated Articles.\n\nThe whole process is extremely parallelizable: while one commit might be pending\ntranslation or review, an engineer can make additional commits with new copy,\nand they will also sit in the queue awaiting translation. Once any commit is\nfully localized, it is marked as ready for release. This is true for\nArticles as well, except, Shuttle doesn't keep versions for Articles and\nany new submission will override the contents of the previous submission.\n\nShuttle will refuse to deliver a manifest for a commit or a Article that\nhas not been fully translated and reviewed. To prevent such deploys, engineers\nshould add a test to their CI script that ensures that the manifest endpoint\ndoes not return 404.\n\n\nGetting Started\n---------------\n\nThe easiest way to use Shuttle is with Docker. You will need to download and\ninstall a Docker environment. If you do not wish to use Docker, you can also\ninstall and configure all the necessary dependencies on your development\nenvironment.\n\n### Using Shuttle with Docker\n\nTo get set up, run `docker-compose build` to generate all the necessary images.\nYou only have to do this once, and each time you make a change to the source\ncode.\n\nTo run the generated images, run `docker-compose up`. Use `docker-compose down`\nto stop the running images. To initialize your database and ElasticSearch\nindexes for the first time, run  `docker-compose run web bin/docker-setup`.\n\nThe Shuttle development database is stored as a Docker volume called\n`shuttle_postgres_data`. ElasticSearch data is stored as a Docker volume called\n`shuttle_es_data`.\n\nVisit [http://localhost:3000](http://localhost:3000) and log in with the\nusername `admin@example.com` and the password `password123`. This user is\ncreated by the seed script.\n\nCreate and import your first project!\n\n### Setting up your development environment without Docker\n\nDeveloping for Shuttle requires Ruby 2.4.6, PostgreSQL 9.4, Redis, Tidy, Sidekiq\nPro ElasticSearch 5.6, and a modern version of libarchive. To run Shuttle for\nthe first time:\n\n1. Clone this project.\n\n2. Install Ruby 2.4.6. If you are using RVM, you can do so using the\n   `.ruby-version` file.\n\n3. Run `brew bundle` to install all dependencies available via Homebrew, which\n   are specified in the `Brewfile`, or install them manually referencing the\n   Brewfile.\n\n4. Buy Sidekiq Pro and place your private source URL in Gemfile as specified by\n   the Sidekiq Pro documentation.\n\n5. Create a PostgreSQL user called `shuttle`, and make it the owner of two\n   PostgreSQL databases, `shuttle_development` and `shuttle_test`:\n\n   ``` sh\n   createuser shuttle\n   createdb -O shuttle shuttle_development\n   createdb -O shuttle shuttle_test\n   ```\n\n6. You will need to tell Bundler where the libarchive install directory is. If\n   you installed libarchive using Homebrew, you can do this by running\n\n   ``` sh\n   bundle config build.libarchive --with-opt-dir=$(brew --prefix libarchive)\n   ```\n\n7. Likewise, for Rugged, you will need to tell Bundler where the libgit2 install\n   directory is. If you installed libgit2 using Homebrew:\n\n   ``` sh\n   bundle config build.rugged --with-opt-dir=$(brew --prefix libgit2)\n   ```\n\n8. Make sure that PostgreSQL, Redis, and ElasticSearch are running. If you\n   installed them via Homebrew, running `brew info postgresql@9.6`,\n   `brew info redis`, and `brew info elasticsearch` will tell you how to run\n   them.\n\n9.  Install the `mailcatcher` gem, which is used to receive emails sent in\n    development. (This gem is not a part of the Gemfile because it's typically\n    installed as part of a global or system-wide gemset to be used with\n    all projects.)\n\n10. Optionally, install the `foreman` gem, which runs all the processes\n    necessary for development.\n\n11. Install all required gems by running `bundle install`.\n\n12. Run `rake db:migrate db:seed` to migrate and seed the database.\n\n13. Run `RAILS_ENV=test rake db:migrate` to setup the test database.\n\n14. Initialize the ElasticSearch index for development by running\n    `rake chewy:reset`.\n\n15. Do the same for the test indexes: `RAILS_ENV=test rake chewy:reset`\n\n16. Verify that all specs pass with `rspec spec`.\n\n#### Starting the server\n\nTo run the development environment, you will need to start the Web server, the\nmail client, and Sidekiq.\n\n* **Running the Web server:** `rails server`\n* **Running the mail client:** `mailcatcher`\n* **Running Sidekiq:** `sidekiq -C config/sidekiq.yml`\n\nThese processes can be launched together by running `foreman start`.\n\nVisit [http://localhost:3000](http://localhost:3000) and log in with the\nusername `admin@example.com` and the password `password123`. This user is\ncreated by the seed script.\n\nCreate and import your first project!\n\n### Adding your first project\n\nYou are now an admin user on Shuttle. You can click the \"Add Project\" button to\nconfigure your first project. You will need at least read-only access to this\nproject's Git repository. Set up the locale and importing settings as necessary.\n\nOnce the project has been added, you can add a commit for it to import strings\nfrom. For starters, try entering \"HEAD\". Once you click \"Add\", you should see\nyour Sidekiq server output start to fill up with importers processing all the\nblobs in your HEAD commit. There may be a delay as the repository is checked out\nto `tmp/repos` for the first time.\n\nRefresh the Shuttle home page. When you click on your project name, it should\nexpand to show your commit. The progress bar should be orange and indeterminate,\nindicating that the commit is being processed. Once processing is finished, the\nSidekiq log will quiet down and the progress bar should change to an empty\n(white) bar, indicating that no translations have been made yet.\n\nClick the progress bar to expand the commit and get detailed status information.\nYou should see three numbers. The badged number on the right is the total number\nof translatable strings found in your project. Inside the progress bar are two\nnumbers separated by a slash. The first number is the number of finished\ntranslations across all required locales (should be zero), and the second number\nis the total number of required translations (should be the number of\ntranslatable strings times the number of required locales, minus any keys that\nare not applicable to certain locales).\n\nThe \"Import and approve a localization\" field allows you to import an existing\nlocalization. For example, if your Rails project already has an \"fr.yml\" file\nthat you want to import into Shuttle (to save your translators the effort of\nretyping all those translations), you can use this field to do it.\n\nThese are the features typically used by users with the \"monitor\" role. As an\nadmin you can also explore and use the tools used by translators: The\ntranslation/review panel, the global search page, and the glossary. Managing\nother users is an admin-specific feature.\n\nOne last important feature that admins have is the ability to visit the\n\"/sidekiq\" URL, which lets them monitor and manage Sidekiq workers.\n\n### Deploying to production\n\nShuttle does not come with a deploy script; you should use whatever deploy\nsystem you are comfortable. Your deploy script should, in addition to copying\nthe code and starting the Rails server,\n\n1. stop and restart the Sidekiq workers (the Sidekiq gem has scripts for this),\n   and\n2. install the cron entries (the Whenever gem has scripts for this).\n\n\nDocumentation\n-------------\n\nComprehensive documentation is written in YARD- and Markdown-formatted comments\nthroughout the source. To view this documentation as an HTML site, run\n`rake yard`.\n\nCoffeeScript libraries are documented using the YARD format as well, but YARD\ndoes not as yet recognize them as documentable files. A `.codoopts` file is\nincluded in case you wish to use [Codo](https://github.com/netzpirat/codo) to\ngenerate the CoffeeScript docs, but as of now Codo does not recognize the ERb\nfiles, and does not use the full set of Markdown syntax features used in the\ndocumentation.\n\n\nArchitecture\n------------\n\n### Models\n\n![Object Model](doc/images/objectmodel.png)\n\nEach {Project} has multiple {Commit Commits} or {Article Articles}.\nWhen a Commit or Article is created, it is scanned by {Importer Importers} for\nlocalizable strings. These strings are represented as {Translation} records.\nA base Translation is created in the project's base locale, and preapproved,\nand pending, untranslated Translations are created for each target locale.\nThese families of Translations are grouped under {Key} records, one for each\nunique key in the Project. The newly created Translations are in the Project's\nbase locale. Future imports reuse the existing Keys if the source copy is unchanged,\notherwise generating new Keys and new Translations for the new source copy.\n{User Users} with the translator role then fill out pending Translations, and\nreviewers approve them.\n\nWhen all of a Commit's/Article's Translations in all of a Project's required\nlocales are marked as approved, the Commit/Article is marked as ready.\nThis Commit's/Article's translated copy can then be exported to a manifest\nfile using an {Exporter}, or localized versions of project files can be\ngenerated and downloaded using a {Localizer}.\n\nArticle keeps an ordered set of {Key Keys} whereas Commit keeps an unordered\nset, so that the exporter can put the Translations back together in the right order.\n\nModels make extensive use of advanced PostgreSQL features for efficiency and\nconvenience. Cached counters are updated using triggers and rules, foreign key\nconstraints and hooks are enforced at the database level, and validations are\nbacked up by corresponding `CHECK` triggers. This helps ensure referential and\ndata integrity even in situations where Rails fails, or outside of the Rails\nstack. See the various migrations to learn more about the triggers, rules, and\nconstraints being used. See the `app/models/concerns` directory for the Active\nRecord mixins that leverage these PostgreSQL features.\n\nObservers are used for more high-level triggers, such as sending emails. See the\nclasses in `app/models/observers` for more.\n\nIssues, Comments and Screenshots make up a light-weight issue-tracking system,\nand can be used by engineers and translators for easy communication and marking\nissues translations (compare to email communication where you don't see the\npreviously raised issues in a translation unless you search for it on an external\nsystem).\n\n### Authentication and Authorization\n\nAuthentication is handled by Devise. Users log in using their email address and\na password.\n\nShuttle uses a role-based authorization system. See the {User} model for details\non the available user roles and their privileges.\n\n### Controllers\n\nFor information about requests and responses, see {ApplicationController}.\n\n### Views\n\nThis is a pretty typical Rails website, save for the views, which are written\nusing Erector. The views forgo the traditional Rails concepts of partials and\ntemplates in favor of analogous OOP concepts more familiar to software\ndevelopers: methods and inheritance. All views inherit from an abstract Erector\nwidget which provides layout; and all views have their content split into\nmultiple private methods.\n\nIn addition to the usual helpers (in `app/helpers`), there are view mixins under\n`app/views/additions` that simplify view coding.\n\nJavaScript files are organized into four possible locations:\n\n* Third-party JavaScript libraries are in `vendor/assets/javascripts` and\n  loaded in the `application.js.coffee` manifest.\n* JavaScript modules or helpers that are not specific to a particular page or\n  site area are in `lib/assets/javascripts` and also loaded in\n  `application.js.coffee`.\n* JavaScript modules or helpers specific to a particular area of the site are in\n  `app/assets/javascripts` and also loaded in `application.js.coffee`.\n* Small JavaScript snippets, glue code, or other code intended to add dynamic\n  behavior to a specific page is in a `.js` file named the same as, and placed\n  alongside, the `.html.rb` view file. For example, if\n  `app/views/projects/new.html.rb` needed a bit of JS glue code, it would be\n  placed in `app/views/projects/new.js`. This code is placed in a `\u003cSCRIPT\u003e` tag\n  at the end of the view by the `application.slim` layout.\n\nCSS files are similarly organized:\n\n* Third-party CSS files are in `vendor/assets/stylesheets` and loaded in the\n   `application.css` manifest.\n* CSS styles or helpers global to the entire website are in\n  `lib/assets/stylesheets` and also loaded in `application.css`.\n* CSS styles specific to a single page or a related group of pages are placed in\n  `app/assets/stylesheets` and also loaded in `application.css`. Each `\u003cBODY\u003e`\n  tag is given a class name equal to the controller name, and an ID equal to\n  the controller and action name separated with a dash. For instance, the\n  `projects/new` action's body would be `\u003cbody class=projects id=projects-new\u003e`.\n\n### Tasks\n\nVarious Rake tasks are available under `lib/tasks`. These include tasks for\nimporting locale data, bulk-importing trados data, server cleanup and other\ndevelopment tasks.\n\n### Others\n\nShuttle also utilizes other design patterns such as:\n\n* `Presenters` (layer between `Controllers` and `Views`),\n* `Mediators` (layer between `Controllers` and `Models`),\n* `Helpers` (for generating views)\n* `Services` (to interact with other services or data stores such as Redis or ElasticSearch)\n\n\nDistributed Processing\n----------------------\n\n![Worker Interactions](doc/images/workers.png)\n\nShuttle is designed to handle large code bases. Fast processing of these code bases is\nachieved through highly parallelizing common tasks which are expected to take a non-trivial\namount of time, using Sidekiq. All Sidekiq workers live in `app/workers`. Best examples\nare CommitImporter and ArticleImporter. These both kick off sub-workers in a batch,\nand run other tasks when all the jobs in the batch are finished.\n\n\nImporting and Exporting\n-----------------------\n\nShuttle provides a number of importer and exporter libraries that are capable of\nextracting strings or generating output in formats such as Rails i18n or iOS\n.strings files. These classes are in `lib/importer` and `lib/exporter`.\n\nSome i18n platforms require that localizations be split across multiple files;\nnormally, exporters only export a single file. To get around this restriction,\nthese exporters will export gzip-compressed tarballs that can be expanded into\nthe project's root directory.\n\n### Localization of files with inline copy\n\nWhile importers can scan an entire project for localizable strings, exporters\nare built on the assumption that the exported file will contain only translated\nstrings, and no other content or metadata. This will not work with, for example,\nxib files, which must be duplicated in their entirety (with the localized copy\nsubstituted for the original copy).\n\nShuttle handles this process with file localization. Importers that work with\nlocalizers (rather than exporters) also record metadata about where in the\nfile the string came from. A localizer (under `lib/localizer`) recreates\nthe original file and substitutes translated copy using the source information.\n\n### Key inclusion and exclusion lists\n\nCertain keys can be excluded by a blacklist or whitelist system. These settings\nare spread about a couple of different places:\n\n**Project-global key exclusions and inclusions**: The `key_exclusions` and\n`key_inclusions` metadata fields on Project are used to control global key\nwhitelisting or blacklisting. Only one or the other should be set.\n\n**Locale-specific key exclusions and inclusions**: The `key_locale_exclusions`\nand `key_locale_inclusions` metadata fields on Project are used to control key\nwhitelisting or blacklisting on a locale-by-locale basis. For each locale, only\na whitelist or a blacklist should be set.\n\n**Commit-specific key exclusions**: If a particular commit has a `.shuttle.yml`\nfile in the project root, it is read and the value `key_exclusions` key is used\nto further filter keys. It should be an array of keys or UNIX-style globs. This\nallows developers to exclude keys on a certain branch until that branch is ready\nfor translation. Note that Keys belong to a Project and have a many-to-many\nrelationship with Commits. Therefore, keys matching a commit-specific exclusion\nwill still be imported, but will simply not be associated with the commit\ncurrently being processed. Because of this, no locale-specific exclusions are\nsupported in this manner.\n\n**Project-global path filtering**: The `only_paths` and `skip_paths` metadata\nfields on Project are used to prevent the importer from descending into certain\npaths.\n\n**Importer-specific path filtering**: The `only_importer_paths` and\n`skip_importer_paths` metadata fields on Project are used to prevent certain\nimporters from descending into specific paths.\n\nSee {Project#skip_key?}, {Project#skip_path?}, and {Commit#skip_key?} for more\ninformation.\n\n\nFencing\n-------\n\nFencing is the act of marking off portions of text as untranslatable (for\nexample, HTML tags). These portions can be moved but not altered or deleted, in\nmost cases. Fencing can also be used to mark off interpolation variables, such\nas `%{count}` in the Ruby i18n format.\n\nA variety of fencers for common formats is provided under `lib/fencer`; they are\nnot subclasses of an abstract superclass, but do all respond to the same\ninformal interface.\n\n\nSpecs\n-----\n\nAll models, controllers, and library files are unit or integration-tested with\nRSpec specs under the `spec` directory. Views and JavaScript files are not\nspecced. Almost all unit tests use factories rather than mocks, putting them\nsomewhat closer to integration tests.\n\nIf you are using Docker, first run `docker-compose -f docker-compose.test.yml build`\nto build the images. This only has to be done once, and each time you make a\nchange to the source code. Run `docker-compose -f docker-compose.test.yml up` to\nstart the environment, and\n`docker-compose -f docker-compose.test.yml run tests bin/docker-tests` to\nactual run tests. Run `docker-compose -f docker-compose.test.yml down` when you\nare done testing to stop the running images.\n\nIf you are not using Docker, run unit tests with the `rspec spec` command.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsquare%2Fshuttle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsquare%2Fshuttle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsquare%2Fshuttle/lists"}