{"id":13426212,"url":"https://github.com/nytimes/library","last_synced_at":"2025-04-07T23:13:29.845Z","repository":{"id":34317921,"uuid":"174578457","full_name":"nytimes/library","owner":"nytimes","description":"A collaborative documentation site, powered by Google Docs.","archived":false,"fork":false,"pushed_at":"2024-08-16T01:04:05.000Z","size":2312,"stargazers_count":1156,"open_issues_count":83,"forks_count":145,"subscribers_count":32,"default_branch":"main","last_synced_at":"2025-03-31T22:26:04.115Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://nyt-library-demo.herokuapp.com/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nytimes.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-03-08T17:15:19.000Z","updated_at":"2025-03-24T07:41:05.000Z","dependencies_parsed_at":"2024-01-18T04:08:48.065Z","dependency_job_id":"66ea3152-6111-4f5b-bc49-ae83d486e194","html_url":"https://github.com/nytimes/library","commit_stats":null,"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nytimes%2Flibrary","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nytimes%2Flibrary/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nytimes%2Flibrary/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nytimes%2Flibrary/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nytimes","download_url":"https://codeload.github.com/nytimes/library/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247744335,"owners_count":20988783,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T00:01:29.078Z","updated_at":"2025-04-07T23:13:29.819Z","avatar_url":"https://github.com/nytimes.png","language":"JavaScript","funding_links":[],"categories":["JavaScript","📚 Learning \u0026 Resources","others"],"sub_categories":[],"readme":"Library ![Supported node versions](https://img.shields.io/badge/dynamic/json?color=informational\u0026label=node\u0026query=%24.engines.node\u0026url=https%3A%2F%2Fraw.githubusercontent.com%2Fnytimes%2Flibrary%2Fmain%2Fpackage.json) [![Tests](https://github.com/nytimes/library/actions/workflows/test.yaml/badge.svg)](https://github.com/nytimes/library/actions/workflows/test.yaml)\n========\n\nA collaborative newsroom documentation site, powered by Google Docs.\n\n\u003c!-- START doctoc generated TOC please keep comment here to allow auto update --\u003e\n\u003c!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --\u003e\n**Table of Contents**\n\n- [Demo Site \u0026 User Guide](#demo-site--user-guide)\n- [Contacting us](#contacting-us)\n- [Community](#community)\n- [Contributing](#contributing)\n- [Questions](#questions)\n- [Development Workflow](#development-workflow)\n  - [Tests](#tests)\n- [Customization](#customization)\n- [Deploying the app](#deploying-the-app)\n  - [Using Heroku](#using-heroku)\n  - [Using Google App Engine](#using-google-app-engine)\n  - [Using Docker](#using-docker)\n  - [Standard Deployment](#standard-deployment)\n- [App structure](#app-structure)\n  - [Server](#server)\n  - [Views](#views)\n  - [Doc parsing](#doc-parsing)\n  - [Listing the drive](#listing-the-drive)\n  - [Auth](#auth)\n  - [User Authentication](#user-authentication)\n\n\u003c!-- END doctoc generated TOC please keep comment here to allow auto update --\u003e\n\n## Demo Site \u0026 User Guide\n\nDocumentation about how to get started with Library is hosted as a working (read only) demo on Heroku. Consult the site for more detailed instructions than this readme about how to get the most out of Library: https://nyt-library-demo.herokuapp.com.\n\n## Contacting us\n\nLove Library? Let us know by [joining our Google Group](https://groups.google.com/forum/#!forum/nyt-library-community) and dropping us a line. You'll also stay up to date with the latest Library features via our release notes, which get sent to this list.\n\n## Community\n\nHere are some of the organizations using Library so far.\n\n[Marketplace](https://www.marketplace.org/)\n\n[The New York Times](http://nytimes.com)\n\n[Northwestern University Knight Lab](https://knightlab.northwestern.edu)\n\n[Star Tribune](http://www.startribune.com)\n\n[WBEZ](https://www.wbez.org)\n\n[The Los Angeles Times Data and Graphics Department](https://twitter.com/palewire/status/1326220493762883585)\n\n## Contributing\n\nSee [CONTRIBUTING.md](https://github.com/nytimes/library/blob/master/CONTRIBUTING.md) for information on how to contribute code and/or documentation on GitHub or on the [demo site](https://nyt-library-demo.herokuapp.com).\n\n## Questions\n\nIf you have questions about how to get your copy of Library up and running, [join our Google Group](https://groups.google.com/forum/#!forum/nyt-library-community), and let us know what you're running into. We also keep an eye on the #proj-library channel in the News Nerdery Slack. We'll do our best to answer your questions.\n\n## Development Workflow\n\n1. Clone and `cd` into the repo:\n\n   `git clone git@github.com:nytimes/library.git \u0026\u0026 cd library`\n\n\n2. From the Google API console, create or select a project, then create a service account with the Cloud Datastore User role. It should have API access to Drive and Cloud Datastore. Store these credentials in `server/.auth.json`.\n\n   - To use oAuth, you will also need to create oAuth credentials.\n   - To use the Cloud Datastore API for reading history, you will need to add in your `GCP_PROJECT_ID`.\n\n3. Install dependencies:\n\n   `npm install --no-optional`\n\n4. Create a `.env` file at the project root. An example `.env` might look like\n\n```bash\n# node environment (development or production)\nNODE_ENV=development\n# Google oAuth credentials\nGOOGLE_CLIENT_ID=123456-abcdefg.apps.googleusercontent.com\nGOOGLE_CLIENT_SECRET=abcxyz12345\nGCP_PROJECT_ID=library-demo-1234\n# comma separated list of approved access domains or email addresses (regex is supported).\nAPPROVED_DOMAINS=\"nytimes.com,dailypennsylvanian.com,(.*)?ar.org,demo.user@demo.site.edu\"\nSESSION_SECRET=supersecretvalue\n\n# Google drive Configuration\n# team or folder (\"folder\" if using a folder instead of a team drive)\nDRIVE_TYPE=team\n# the ID of your team's drive or shared folder. The string of random numbers and letters at the end of your team drive or folder url.\nDRIVE_ID=0123456ABCDEF\n```\nMake sure to not put any comments in the same line as `DRIVE_TYPE` and `DRIVE_ID` vars.\n\nEnsure you share your base drive or folder with the email address associated with the service account created in step 2.\n\n**Be careful!** Setting NODE_ENV to `development` changes the built in behaviors for site authentication to allow accounts other than those in the APPROVED_DOMAINS list. **Never use NODE_ENV=development for your deployed site, only locally.**\n\n5. Start the app:\n\n   `npm run watch`\n\nThe app should now be running at `localhost:3000`. Note that Library requires Node v8 or higher.\n\n### Tests\nYou can run functional and unit tests, which test HTML parsing and routing logic, with `npm test`. A coverage report can be generated by running `npm run test:cover`.\n\nThe HTML parsing tests are based on the [Supported Formats doc](https://docs.google.com/document/d/12bUE9b4_aF_IfhxCLdHApN8KKXwa4gQUCGkHzt1FyRI/edit).  To download a fresh copy of the HTML after making edits, run `node test/utils/updateSupportedFormats.js`.\n\n## Customization\nStyles, text, caching logic, and middleware can be customized to\nmatch the branding of your organization. This is covered in the [customization readme](https://github.com/nytimes/library/blob/master/custom/README.md).\n\nA sample customization repo is provided at [nytimes/library-customization-example](https://github.com/nytimes/library-customization-example).\n\n\n## Deploying the app\nWherever you deploy Library, you'll likely want to [set up a Google service account](https://console.cloud.google.com/iam-admin/serviceaccounts) and [OAuth 2.0 client](https://developers.google.com/identity/protocols/OAuth2) Set up your service account with API access to Drive and Cloud Datastore.\n\nIf you wish to deploy Library with customizations, create a git repo with the files you would like to include. Set the `CUSTOMIZATION_GIT_REPO` environment variable to the cloning URL. Files in the repo and packages specified in the `package.json` will be included in your library installation.\n\nFor more detailed instructions, consult the Getting Started section of the demo site: https://nyt-library-demo.herokuapp.com/get-started\n\n### Using Heroku\n\nThis button can quickly deploy to Heroku:\n[![Deploy](https://www.herokucdn.com/deploy/button.svg)](https://nyti.ms/2CrT2y2)\n\nSet your app's `GOOGLE_APPLICATION_JSON`, `GOOGLE_CLIENT_ID`, and `GOOGLE_CLIENT_SECRET` with values from the service account and Oauth client. Add *\u003cyour-heroku-app-url\\\u003e.com* as an authorized domain in the general OAuth consent screen setup and then add *http://\u003cyour-heroku-app-url\\\u003e.com/auth/redirect* as the callback url in the OAuth credential setup itself.\n\n### Using Google App Engine\nYou can also deploy Library to GAE, using the included `app.yaml`. Note that you will need to enable billing on your GCP project in order to use Google App Engine. More detailed instructions are provided on the [demo site](https://nyt-library-demo.herokuapp.com/get-started/deploying-library-google-app-engine).\n\n### Using Docker [![Dockerhub](https://img.shields.io/docker/v/nytimes/library?logo=docker)](https://hub.docker.com/r/nytimes/library/tags)\nLibrary can be used as a base image for deployment using Docker. This allows you\nto automate building and deploying a custom version of Library during Docker's\nbuild phase. If you create a repo with the contents of your `custom` folder, you\ncould deploy library from that repo with a Dockerfile like the following:\n\n```Dockerfile\nFROM nytimes/library\n\n# copy custom files to library's custom repo\nCOPY . ./custom/\n\n# move to a temporary folder install custom npm packages\nWORKDIR /usr/src/tmp\nCOPY package*.json .npmrc ./\nRUN npm i\n# copy node modules required by custom node modules\nRUN yes | cp -rf ./node_modules/* /usr/src/app/node_modules\n\n# return to app directory and build\nWORKDIR /usr/src/app\nRUN npm run build\n\n# start app\nCMD [ \"npm\", \"start\" ]\n```\n\n### Standard Deployment\nLibrary is a standard node app, so it can be deployed just about anywhere. If you are looking to deploy to a standard VPS, [Digital Ocean's tutorials](https://www.digitalocean.com/community/tutorials/how-to-set-up-a-node-js-application-for-production-on-ubuntu-16-04) are a great resource.\n\n## App structure\n\n### Server\nThe main entry point to the app is `index.js`.\n\nThis file contains the express server which will respond to requests for docs\nin the configured team drive or shared folder. Additionally, it contains logic\nabout issuing 404s and selecting the template to use based on the path.\n\n### Views\nViews (layouts) are located in the `layouts` folder. They use the `.ejs`\nextension, which uses a syntax similar to underscore templates.\n\nBase styles for the views are in the `styles` directory containing Sass files.\nThese files are compiled to CSS and placed in `public/css`.\n\n### Doc parsing\nDoc HTML fetch and parsing is handled by `docs.js`. `fetchDoc` takes the ID of a\nGoogle doc and a callback, then passes the HTML of the document into the\ncallback once it has been downloaded and processed.\n\n### Listing the drive\nTraversing the contents of the NYT Docs folder is handled by `list.js`. There\nare two exported functions:\n\n* `getTree` is an async call that returns a nested hash (tree) of Google Drive\n  Folder IDs mapped to their children. It is used by the server to determine\n  whether a route is valid or not.\n\n* `getMeta` synchronously returns a hash of Google Doc IDs to metadata objects\n  that were saved in the course of populating the tree. This metadata includes\n  edit history, document authors, and parent folders.\n\nThe tree and file metadata are repopulated into memory on an interval (currently 60s). Calling getTree multiple times will not return fresher data.\n\n### Auth\n\nAuthentication with the Google Drive v3 api is handled by the auth.js file, which exposes a single method `getAuth`. `getAuth` will either return an already instantiated authentication client or produce a fresh one. Calling `getAuth` multiple times will not produce a new authentication client if the credentials have expired; we should build this into the auth.js file later to automatically refresh the credentials on some sort of interval to prevent them from expiring.\n\n### User Authentication\n\nLibrary currently supports both Slack and Google Oauth methods. As Library sites are usually intended to be internal to a set of limited users, Oauth with your organization is strongly encouraged. To use Slack Oauth, specify your Oauth strategy in your `.env` file, like so:\n```\n# Slack needs to be capitalized as per the Passport.js slack oauth docs http://www.passportjs.org/packages/passport-slack-oauth2/\nOAUTH_STRATEGY=Slack\n```\nYou will need to provide Slack credentials, which you can do by creating a Library Oauth app in your Slack workspace. After creating the app, save the app's `CLIENT_ID` and `CLIENT_SECRET` in your `.env` file:\n```\nSLACK_CLIENT_ID=1234567890abcdefg\nSLACK_CLIENT_SECRET=09876544321qwerty\n```\nYou will need to add a callback URL to your Slack app to allow Slack to redirect back to your Library instance after the user is authenticated.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnytimes%2Flibrary","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnytimes%2Flibrary","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnytimes%2Flibrary/lists"}