{"id":13819161,"url":"https://github.com/rustyio/super-imap","last_synced_at":"2025-05-16T04:32:43.832Z","repository":{"id":22744505,"uuid":"26089668","full_name":"rustyio/super-imap","owner":"rustyio","description":"SuperIMAP - Monitor inboxes for incoming email, at scale.","archived":false,"fork":false,"pushed_at":"2019-12-28T19:40:02.000Z","size":902,"stargazers_count":240,"open_issues_count":6,"forks_count":41,"subscribers_count":16,"default_branch":"master","last_synced_at":"2024-08-05T08:07:47.812Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rustyio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-11-02T18:45:15.000Z","updated_at":"2024-05-25T21:35:04.000Z","dependencies_parsed_at":"2022-08-05T19:00:57.568Z","dependency_job_id":null,"html_url":"https://github.com/rustyio/super-imap","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rustyio%2Fsuper-imap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rustyio%2Fsuper-imap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rustyio%2Fsuper-imap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rustyio%2Fsuper-imap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rustyio","download_url":"https://codeload.github.com/rustyio/super-imap/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225405545,"owners_count":17469368,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-04T08:00:41.521Z","updated_at":"2024-11-19T18:31:20.543Z","avatar_url":"https://github.com/rustyio.png","language":"Ruby","funding_links":[],"categories":["Happy Exploring 🤘"],"sub_categories":[],"readme":"# SuperIMAP - Version 0.1.2\n\nSuperIMAP helps you build email-driven applications. It takes care of\nconnecting to a customer's IMAP inbox, watching for new email, and\ntriggering a webhook to your application when a new email arrives,\ntypically within seconds.\n\nSuperIMAP is built for\nscale. [FiveStreet.com](https://www.fivestreet.com) built SuperIMAP as\nan alternative to Context.io. It contains a subset of Context.io Lite\nAPI functionality. As of July 2015, the FiveStreet team runs a\nSuperIMAP cluster processing ~400k emails per day for thousands of\nusers.\n\nSuperIMAP is written in Ruby and open sourced under the MIT license. [Why Ruby?](#why-ruby)\n\n[![Deploy](https://www.herokucdn.com/deploy/button.png)](https://heroku.com/deploy?template=https://github.com/rustyio/super-imap)\n\n## Contents\n\n* [Installation](#installation)\n  * [Installing on Heroku](#installing-on-heroku)\n  * [Installing Elsewhere](#installing-elsewhere)\n* [Usage](#usage)\n  * [Set up a Partner](#set-up-a-partner)\n  * [Set up a Partner Connection](#set-up-a-partner-connection)\n  * [Add and Connect a User](#add-and-connect-a-user)\n  * [Add and Connect a User Programatically](#add-and-connect-a-user-programatically)\n  * [Disconnect a User Programatically](#disconnect-a-user-programatically)\n  * [Security](#security)\n* [Webhooks](#webhooks)\n  * [Webhook Security](#webhook-security)\n  * [New Mail Webhook](#new-mail-webhook)\n  * [User Connected Webhook](#user-connected-webhook)\n  * [User Disconnected Webhook](#user-disconnected-webhook)\n* [API](#api)\n  * [Manage Partner Connections](#apiv1connections)\n  * [Manage a Partner Connection](#apiv1connectionsimap_provider_code)\n  * [Manage Users](#apiv1connectionsimap_provider_codeusers)\n  * [Manage a User](#apiv1connectionsimap_provider_codeuserstag)\n* [Operations](#operations)\n  * [Process Types](#process-types)\n  * [Why Ruby?](#why-ruby)\n  * [Environment Variables](#environment-variables)\n  * [Scaling](#scaling)\n  * [Monitoring](#monitoring)\n  * [Performance](#performance)\n  * [Tracer Emails](#tracer-emails)\n* [Appendix](#appendix)\n  * [Understanding OAuth 2.0](#understanding-oauth-20)\n  * [Understanding IMAP](#understanding-imap)\n* [Development Tasks](#development-tasks)\n  * [Running Unit Tests](#running-unit-tests)\n  * [Running Stress Tests](#running-stress-tests)\n  * [Future Work](#future-work)\n  * [Contributions](#contributions)\n* [Changes](#changes)\n* [License](#license)\n\n## Screenshot\n\n![Screenshot](screenshot.png)\n\n## Installation\n\nSuperIMAP was built to run on Heroku, but can run in any environment that supports Rails.\n\n#### Installing on Heroku\n\n1. Provision a new Heroku project.\n2. Add an encrypted database.\n3. Set the `SECRET_KEY_BASE` and `ENCRYPTION_KEY` environment\n   variables to something really long and complicated.\n4. Add a Heroku remote endpoint, and push the code.\n5. Ramp up some workers.\n5. Seed the database with  `heroku run rake db:setup db:seed`\n\nThen, log in as the default user: `admin@example.com` / `password`.\n\n**Remember: Change the username and password immediately!**\n\nThe production Procfile assumes that you are installing on Heroku. As\na result it has multiple definitions for the `imap_client` process\ncorresponding to different sized Heroku dynos. In order for load\nbalancing to work correctly, you should have all `imap_client`\nprocesses use the same dyno size. Do not mix and match boxes.\n\n#### Installing Elsewhere\n\n1. Get the code: `git clone https://github.com/rustyio/super-imap.git`\n2. Update `config/database.yml`. (Use `config/database.yml.example`)\n3. Run `bundle` to install dependencies.\n4. Set the `SECRET_KEY_BASE` and `ENCRYPTION_KEY` environment\n   variables to something really long and complicated.\n5. Seed the database with `RAILS_ENV=production rake db:setup db:seed`\n6. Start the processes: `foreman start -c \"web=1, worker=1, imap_client_1x=1`\n\nThen, log in as the default user: `admin@example.com` / `password`.\n\n**Remember: Change the username and password immediately!**\n\nThe production Procfile assumes that you are installing on Heroku. As\na result it has multiple definitions for the `imap_client` process\ncorresponding to different sized Heroku dynos. In order for load\nbalancing to work correctly, you should ensure that all `imap_client`\nprocesses are the same size and point to the same database.\n\n## Usage\n\n#### Set up a Partner\n\nA single SuperIMAP instance can support multiple applications (and/or multiple environments for a single application.) This is done by creating a new \"Partner\" for every different application (and/or environment).\n\n1. Open the SuperIMAP dashboard.\n2. Click on the \"Partners\" tab. Click \"New Partner\".\n3. Set webhooks to notify your application of the following events:\n   * A new email has arrived.\n   * A user has connected their email account.\n   * A user has disconnected their email account.\n\n#### Set up a Partner Connection\n\n1. Still within the dashboard, click the \"Partners\" tab.\n2. Click on the \"Connections\" link next to your Partner.\n3. Click on \"New Partner Connection\".\n4. Choose an authentication type, and fill out any necessary credentials.\n\nFor Gmail, you will need to get OAuth 2.0 Client credentials here from the [Developer Console](https://console.developers.google.com/project).\n\n#### Add and Connect a User\n\n1. Still within the dashboard, click on \"Partners\".\n2. Click on the \"Connections\" link next to your partner.\n3. Click on the \"Users\" link next to your connection.\n4. Click on the \"New User\" button.\n5. Click the 'Connect' link to connect the user to an IMAP provider.\n6. Send yourself email, and watch the logs!\n\n#### Add and Connect a User Programatically\n\nThe next step is to update your application to handle the process of creating and connecting to SuperIMAP users. Here is some example code:\n\n```ruby\n    require 'rest-client'\n\n    url = \"https://my-app.com/api/v1/connections/GMAIL_OAUTH2/users\"\n    users = RestClient::Resource.new(url, :headers =\u003e {\n      :'x-api-key'  =\u003e \"$API_KEY$\",\n      :content_type =\u003e :json,\n      :accept       =\u003e :json\n    })\n\n    # Create the user.\n    users.post(:tag =\u003e \"MY_USER\")\n\n    # Get the connect url.\n    response = users[\"MY_USER\"].get\n    connect_url = JSON.parse(response)['connect_url']\n\n    # Set up the success and failure callbacks.\n    callbacks = {\n      :success =\u003e \"http://my-app.com/connect_callback?success=1\",\n      :failure =\u003e \"http://my-app.com/connect_callback?failure=1\"\n    }\n\n    # Redirect the user to the connect url.\n    redirect_to connect_url  + \"?\" + callbacks.to_query\n```\n\n#### Disconnect a User Programatically\n\nBelow is sample code to disconnect a user:\n\n```ruby\n    url = \"https://my-host.com/api/v1/connections/GMAIL_OAUTH2/users\"\n    users = RestClient::Resource.new(url, :headers =\u003e {\n      :'x-api-key'  =\u003e \"$API_KEY$\",\n      :content_type =\u003e :json,\n      :accept       =\u003e :json\n    })\n\n    # Later, if you want to disconnect the user.\n    response = users[\"MY_USER\"].get\n    disconnect_url = JSON.parse(response)['disconnect_url']\n\n    # Set up the success and failure callbacks.\n    callbacks = {\n      :success =\u003e \"http://my-app.com/disconnect_callback?success=1\"\n    }\n\n    # Redirect the user to the disconnect url.\n    redirect_to disconnect_url  + \"?\" + callbacks.to_query\n```\n\n#### Security\n\nThis is a good time to mention security. It is a big responsibility to\nhold the keys to someone's email. Treat it with the appropriate amount\nof caution.\n\nIf you use this code:\n\n* *PLEASE* ensure that you use very strong, safeguarded passwords.\n* *PLEASE* use enable 2-factor-authentication for your Heroku account.\n* *PLEASE* make sure your entire database is encrypted at rest.\n\nOther security measures within SuperIMAP:\n\n+ SSL is *required* in production.\n+ Secure fields (e.g. passwords and other credentials) are never exposed via the web interface, and are encrypted in the database.\n+ Passwords are not recoverable by email.\n+ Accounts are locked for an hour after three invalid password attempts.\n\n## Webhooks\n\nSuperIMAP sends new email events (and other events) to your\napplications through webhooks:\n\n+ All webhooks are dispatched through delayed jobs.\n+ Webhooks will be retried up to 6 times, with exponential backoff.\n+ Webhooks will fail if the receiving server takes more than 30 seconds to respond.\n+ Webhooks expect a success response (HTTP code 200 - 206).\n+ A \"Forbidden\" response code of 403 will automatically archive the user.\n+ Any other response codes count as an error, and will trigger a retry.\n\n#### Webhook Security\n\nAll webhooks are signed. You can validate the signature as follows:\n\n```ruby\n    # Parse the incoming JSON body.\n    json_params = JSON.parse(request.raw_post)\n\n    # Calculate expected signature.\n    digest    = OpenSSL::Digest.new('sha256')\n    api_key   = Rails.application.config.super_imap_api_key\n    sha1      = json_params['sha1']\n    timestamp = json_params['timestamp']\n    expected_signature = OpenSSL::HMAC.hexdigest(digest, api_key, \"#{timestamp}#{sha1}\")\n\n    # Get actual signature.\n    actual_signature = json_params['signature']\n\n    # Compare signatures.\n    valid = expected_signature == actual_signature\n```\n\n#### New Mail Webhook\n\nCalled when a new mail arrives in a user's inbox.\n\n+ `timestamp` - Timestamp the webhook was sent. Seconds since Jan 1, 1970.\n+ `sha1` - The SHA1 hash of the rfc822 parameter.\n+ `imap_provider_code` - The IMAP provider code (e.g. \"GMAIL_OAUTH2\")\n+ `user_tag` - The user's tag.\n+ `envelope - The email envelope, including date, subject, from, sender, reply_to, to, cc, bcc, in_reply_to, and message_id.\n+ `rfc822` - The raw body of the email. http://www.w3.org/Protocols/rfc822/\n\n#### User Connected Webhook\n\nCalled when a user has successfully authenticated with an IMAP\nprovider. Only applies to OAuth connections at the moment.\n\n+ `timestamp` - Timestamp the webhook was sent. Seconds since Jan 1, 1970.\n+ `sha1` - The SHA1 hash of the user's tag.\n+ `imap_provider_code` - The IMAP provider code (e.g. \"GMAIL_OAUTH2\")\n+ `user_tag` - The user's tag.\n+ `email` - The email address with which the user authenticated.\n\n#### User Disconnected Webhook\n\nCalled when a user has disconnected from an IMAP provider. Only\napplies to OAuth connections at the moment.\n\n+ `timestamp` - Timestamp the webhook was sent. Seconds since Jan 1, 1970.\n+ `sha1` - The SHA1 hash of the user's tag.\n+ `imap_provider_code` - The IMAP provider code (e.g. \"GMAIL_OAUTH2\")\n+ `user_tag` - The user's tag.\n\n## API\n\nAll API calls are scoped by partner. To authenticate, send the\nPartner's API key using a header or a parameter. (A header is\npreferred because it won't normally appear in HTTP logs.)\n\n```sh\n    # Access the API curl:\n    curl -H \"Accept: json\" \\\n         -H \"x-api-key:APIKEY\" \\\n         https://my-host.com/api/v1/connections\n```\n\n\n```ruby\n    # Access the API using the rest-client gem:\n    url = \"https://my-host.com/api/v1\"\n    resource = RestClient::Resource.new(url, :headers =\u003e {\n      :'x-api-key'  =\u003e \"$API_KEY$\",\n      :content_type =\u003e :json,\n      :accept       =\u003e :json\n    })\n    resource['connections'].get\n```\n___\n\n#### /api/v1/connections\n\n**GET**\n\nGet a list of connections for the specified partner.\n\n**POST**\n\nCreate a new connection.\n\n* `imap_provider_code` is required.\n* Other required parameters depend on the IMAP Provider used.\n\n___\n\n#### /api/v1/connections/:IMAP_PROVIDER_CODE\n\n**GET**\n\nGet information about a given connection.\n\n**PUT**\n\nUpdate settings for a given connection. The required parameters depend on the IMAP provider used.\n\n**DELETE**\n\nDelete a connection and all underlying user data.\n\n___\n\n#### /api/v1/connections/:IMAP_PROVIDER_CODE/users\n\n**GET**\n\nGet a list of users for the specified IMAP Provider.\n\n**POST**\n\nCreate a new user.\n\n* `tag` - Required, a unique tag within the scope of a partner\n  connection, selected by the partner application.\n* Other required parameters depend on the IMAP Provider used.\n\n___\n\n#### /api/v1/connections/:IMAP_PROVIDER_CODE/users/:TAG\n\n**GET**\n\nGet information about the given user, including:\n\n* `email` - The IMAP email address to which the user's account is connected.\n* `connected_at` - The date when the user's account was connected. Present only if connected.\n* `connect_url` - Redirect to this url to connect a user to a provider. For OAuth based IMAP providers, this begins the OAuth dance.\n* `disconnect_url` - Redirect to this url to disconnect a user from a provider.\n\n**PUT**\n\nUpdate a user. The required parameters depend on the IMAP provider used.\n\n**DELETE**\n\nArchive a user. The user can be restored in the web interface, or by\nupdating the user (ie: a PUT request.\n\n## Operations\n\n#### Process Types\n\nSuperIMAP consists of 3 different processes, all written in Ruby / Rails:\n\n+ `web` - Serves the admin interface and the API.\n+ `imap_client` - Handles the task of connecting to IMAP providers and listening for email.\n+ `worker` - Processes background jobs generated by the 'imap_client' process.\n\n#### Environment Variables\n\n+ `ENCRYPTION_KEY` - If provided, used to encrypt passwords and secret keys. Required in production.\n+ `MAX_USER_THREADS` - Change the maximum number of user threads. Default is 500.\n+ `NUM_WORKER_THREADS` - Change the number of worker threads. Default is 5.\n+ `MAX_EMAIL_SIZE` - Change the maximum email size. Default is 1 MiB (1,048,576 bytes).\n+ `TRACER_INTERVAL` - Interval, in seconds, between outgoing tracer emails. Default is 600 seconds (10 minutes).\n+ `NUM_TRACERS` - Number of tracers to send at the end of each tracer interval. Default is 3.\n\n#### Scaling\n\nTo scale SuperIMAP, you will mainly want to increase the number of IMAP\nClient processes. The IMAP Client processes automatically publish a\nheartbeat every 10 seconds. Other instances look for this heartbeat\nand re-calculate which neighboring processes are alive based on any\nprocesses that have published a heartbeat within the last 30 seconds.\n\nThe IMAP Client processes re-balance users every 10 seconds. If no new\ninstances have entered or left the pool, then re-balancing will have\nno effect.\n\nIf a new IMAP Client instance is started, then a small number of users\nwill be taken from each running instance and handed to the new\ninstance. If one of the IMAP Client instances is stopped it is removed\nfrom the pool, then it's users will be evenly distributed to the\nremaining instances (assuming they are still below the\n`MAX_USER_THREADS` threshold.)\n\nThere is no \"master\" process that decides which IMAP Client process\nshould handle a given user. SuperIMAP uses a\n[Rendezvous Hash](http://en.wikipedia.org/wiki/Rendezvous_hashing) to\nallow IMAP Client instances to agree on how to evenly assign users\nwithout any central coordination. The algorithm assumes that all\nSuperIMAP instances have roughly the same number of resources.\n\n#### Monitoring\n\nSuperIMAP publishes some useful monitoring information in the logs.\nThis includes:\n\n+ `imap_client.user_thread.count` - The size of the imap client work queue. Backups may indicate that your servers are overloaded.\n+ `imap_client.total_emails_processed` - The total number of emails processed since the instance was started.\n+ `imap_client.work_queue.length` - The number of user threads. This indicates how many users are connected on a given IMAP Client instance.\n+ `work_queue.latency` - The latency, in seconds, between when an item is added to the work queue and when it is processed.\n\nThese metrics are published in a format that can be consumed by the\nLibrato Add-On in Heroku. See\nhttps://devcenter.heroku.com/articles/librato#custom-log-based-metrics\nfor more information.\n\nApart from keeping an eye on these metrics, SuperIMAP should need no other regular metrics.\n\nYou may also want to keep an eye out for any failing Delayed Job tasks. You can view these from the Admin site.\n\n#### Tracer Emails\n\nSuperIMAP has the ability to give you useful monitoring information\nthrough \"tracer emails\". The system will send a specially formatted\nemail to an account, wait for the incoming email, and log the\nresults. The logs can be accessed through the \"Tracer Logs\" tab.\n\nTo enable Tracer Emails, navigate to a user and check the \"Enable\nTracer\" checkbox. It is recommended that you create a few dummy email\naddresses to use for tracer emails.\n\nBy default, a cluster of three tracers are sent every ten minutes from\neach `imap_client` instance to a random tracer-enabled user managed by\nthat instance.\n\nKeep in mind that this could generate a lot of email. Three emails\nevery ten minutes works out to ~430 emails per day.\n\n#### Performance\n\nSuperIMAP's architecture makes judicious use of system resources:\n\nAll connections to the IMAP server are managed by separate \"user\nthreads\", but these threads sit dormant most of the time. When\nanything interesting happens that requires real work, the operation is\nqueued and handled by a worker in a worker pool. The size of the\nworker pool is controlled by the `NUM_WORKER_THREADS` environment\nvariable. Only worker pools threads, and a small number of other\nsystem threads, require a database connection.\n\nIn terms of tradeoffs, this architecture chooses to slightly degrade\nan individual user's response time in favor of making sure that the\nsystem will not get overloaded when things get rough. When things get\nbusy, the work simply builds up in the queue. The size of the worker\nqueue, and the queue latency, becomes a rough measure of system\nhealth.\n\nTypically, a SuperIMAP box is resource-limited by the number of user\nprocesses that can be started. SuperIMAP requires 2 user processes for\neach user's IMAP connection. On Heroku, the number of user processes\nare limited at 256 for a 1X box, 512 for a 2X box, and 32,767 for a PX\nbox. You can set this at home using `ulimit -u`. Divide this in half\nto get the maximum number of users that the SuperIMAP process can\nmanage.\n\n[FiveStreet.com](https://www.fivestreet.com) uses SuperIMAP to manages\nthousands of users and process over 1M incoming emails per week (as of\nJanuary 2015). We currently run this load on a single Heroku PX dyno,\nwith plenty of headroom. Our SuperIMAP instance serving thousands of\nusers requires just 10 database connections, uses about 3GB of RAM,\nand has a 0.50 load average. The work queue usually sits near 0, with\na latency of \u003c 0.5 seconds.\n\n#### Why Ruby?\n\nAt first glance, and from a purely technical point-of-view, Ruby is a\npoor choice for an application like SuperIMAP. SuperIMAP is highly\nconcurrent, and Ruby is bad at concurrency.\n\nSpecifically, the `imap_client` process spawns what could technically\nbe described as a \"boatload\" of threads (2 threads per connected user,\nplus a handful of other threads). Ruby threads are heavyweight, so the\ninterpreter has to burn significant resources just to create and\nschedule the threads before it can do any real work.\n\nUsing Erlang, Go, or Rust (all of which support lightweight threads\nand actor-style programming) would have made the concurrent bits of\nSuperIMAP less tricky to write, and would have required fewer\ncomputing resources, possibly allowing a single box to handle tens of\nthousands of active users.\n\nSo, why Ruby? A few reasons:\n\n+ **FiveStreet uses Ruby** - The primary goal of SuperIMAP is to power\n  a critical part of FiveStreet's application. FiveStreet.com is\n  written in Ruby. It is built and maintained by a small team of Ruby\n  engineers. Introducing a new language would force the team to spend\n  dozens of hours learning a new stack and maintaining a new\n  development environment.\n\n+ **Low barrier to entry** - A secondary goal of SuperIMAP is to become\n  a healthy open-source project. Based only on language popularity, it\n  is more likely that another team can use, troubleshoot, and contribute to\n  a Ruby-based SuperIMAP than an Erlang/Go/Rust-based SuperIMAP.\n\n+ **For us, the cost savings are small** - FiveStreet's SuperIMAP\n  cluster currently runs on three commodity servers and easily handles\n  thousands of users. It's possible that if SuperIMAP were written in\n  a different language, we could handle the load on a single machine,\n  saving us a few hundred dollars a month. Not worth changing our\n  stack for it.\n\n+ **The concurrency is not complicated** - The concurrency in SuperIMAP is\n  fairly straightforward -- one parent process, many child\n  processes. It's a little painful to solve the problem in Ruby, but\n  not impossible.\n\n+ **Ruby has mature IMAP and OAuth 2.0 libraries** - Ruby has a\n  [built-in IMAP library](http://ruby-doc.org/stdlib-2.0.0/libdoc/net/imap/rdoc/Net/IMAP.html),\n  and a\n  [widely-used OAuth 2.0 library](https://github.com/intridea/oauth2). As\n  of 2015, the IMAP and OAuth 2.0 libraries for other languages are\n  [far](https://github.com/alexcrichton/oauth2-rs)\n  [less](https://github.com/boorad/erlimap)\n  [mature](https://github.com/mxk/go-imap).\n\nSide Note: This was a deeply considered choice. I (Rusty Klophaus, the\nauthor of SuperIMAP) spent about 4 years writing Erlang\nprofessionally. It's a fascinating language.\n\n## Appendix\n\n#### Understanding OAuth 2.0\n\nSuperIMAP can authenticate to email providers using OAuth 2.0. OAuth\n2.0 can be difficult to understand. Here is how it works, in the\ncontext of a user authenticating to Gmail through SuperIMAP:\n\nOn your application:\n\n* John visits a \"Connect Your Email\" page on your website.\n* On the server side, your application uses the SuperIMAP API to fetch a special url called a `connect_url`. ([sample code](#add-and-connect-a-user-programatically))\n* John clicks a link and is redirected to the `connect_url`.\n\nOn your SuperIMAP instance:\n\n* The `connect_url` links to a SuperIMAP page.\n* The SuperIMAP page construct a special URL and redirects to Google.\n\nOn Google:\n\n* Google displays a page asking John to confirm certain permissions for your application.\n* John clicks the \"Approve\" button.\n* Google redirects John to SuperIMAP and includes a \"refresh token\" parameter.\n\nOn your SuperIMAP instance:\n\n* SuperIMAP grabs the \"refresh token\", and issues a server side request for an \"access token\".\n* SuperIMAP saves the \"access token\". This is what allows SuperIMAP to connect to Google on behalf of the user in the future.\n* SuperIMAP redirects the user back to the \"success\" page provided in step #2 above.\n\nOn your application:\n\n* Your application tells John that the connection was successful.\n\nTo get one step more complicated, OAuth is secured in a few different ways:\n\n+ Google requires your application to pre-register callback URLS for your app. Sending the user back to a non-registered URL will fail.\n+ The access token is tied to a client id and a client secret on our site. Someone needs all three to authenticate as the user.\n+ There is more too it, but that's the extent of my knowledge.\n\nIf you want more detail, here's a video tutorial: https://www.youtube.com/watch?v=tFYrq3d54Dc\n\nThe OAuth settings are configured through the Google Developer console: https://console.developers.google.com/\n\n#### Understanding IMAP\n\nOnce you are authenticated to an IMAP server, IMAP itself is a fairly\nstraightforward protocol. It consists of simple plain text commands\nand responses. The commands and responses are tagged, allowing\nmultiple commands to run in parallel.\n\nBelow is a sample IMAP session, taken directly from the [Internet Message Access Protocol RFC (3501)](https://tools.ietf.org/html/rfc3501#section-8):\n\n```\n    S:   * OK IMAP4rev1 Service Ready\n    C:   a001 login mrc secret\n    S:   a001 OK LOGIN completed\n    C:   a002 select inbox\n    S:   * 18 EXISTS\n    S:   * FLAGS (\\Answered \\Flagged \\Deleted \\Seen \\Draft)\n    S:   * 2 RECENT\n    S:   * OK [UNSEEN 17] Message 17 is the first unseen message\n    S:   * OK [UIDVALIDITY 3857529045] UIDs valid\n    S:   a002 OK [READ-WRITE] SELECT completed\n    C:   a003 fetch 12 full\n    S:   * 12 FETCH (FLAGS (\\Seen) INTERNALDATE \"17-Jul-1996 02:44:25 -0700\"\n         RFC822.SIZE 4286 ENVELOPE (\"Wed, 17 Jul 1996 02:23:25 -0700 (PDT)\"\n         \"IMAP4rev1 WG mtg summary and minutes\"\n         ((\"Terry Gray\" NIL \"gray\" \"cac.washington.edu\"))\n         ((\"Terry Gray\" NIL \"gray\" \"cac.washington.edu\"))\n         ((\"Terry Gray\" NIL \"gray\" \"cac.washington.edu\"))\n         ((NIL NIL \"imap\" \"cac.washington.edu\"))\n         ((NIL NIL \"minutes\" \"CNRI.Reston.VA.US\")\n          (\"John Klensin\" NIL \"KLENSIN\" \"MIT.EDU\")) NIL NIL\n          \"\u003cB27397-0100000@cac.washington.edu\u003e\")\n          BODY (\"TEXT\" \"PLAIN\" (\"CHARSET\" \"US-ASCII\") NIL NIL \"7BIT\" 3028 92))\n    S:    a003 OK FETCH completed\n    C:    a004 fetch 12 body[header]\n    S:    * 12 FETCH (BODY[HEADER] {342}\n    S:    Date: Wed, 17 Jul 1996 02:23:25 -0700 (PDT)\n    S:    From: Terry Gray \u003cgray@cac.washington.edu\u003e\n    S:    Subject: IMAP4rev1 WG mtg summary and minutes\n    S:    To: imap@cac.washington.edu\n    S:    cc: minutes@CNRI.Reston.VA.US, John Klensin \u003cKLENSIN@MIT.EDU\u003e\n    S:    Message-Id: \u003cB27397-0100000@cac.washington.edu\u003e\n    S:    MIME-Version: 1.0\n    S:    Content-Type: TEXT/PLAIN; CHARSET=US-ASCII\n    S:\n    S:    )\n    S:    a004 OK FETCH completed\n    C:    a005 store 12 +flags \\deleted\n    S:    * 12 FETCH (FLAGS (\\Seen \\Deleted))\n    S:    a005 OK +FLAGS completed\n    C:    a006 logout\n    S:    * BYE IMAP4rev1 server terminating connection\n    S:    a006 OK LOGOUT completed\n```\n\nSuperIMAP uses the IDLE command to wait for incoming email, defined in\nthe\n[IMAP4 Idle Command RFC (2177)](https://tools.ietf.org/html/rfc2177). The\nIMAP client sends an IDLE command to the server and awaits a\nresponse. When an IMAP connection is in IDLE mode, no other commands\nare allowed.\n\n## Development Tasks\n\nThe information below is mainly intended at developers who want to modify the SuperIMAP codebase.\n\n#### Running Unit Tests\n\nRun this once:\n\n    RAILS_ENV=test rake db:setup db:seed\n\nThen run all tests:\n\n    rake test:all\n\n#### Running Stress Tests\n\nThe stress test exercises the multi-threaded aspects of SuperIMAP, as\nwell as the error recovery code. To do this, we point the SuperIMAP IMAP\nclient code against a local IMAP server and generate a bunch of fake emails\nfor many users.\n\nAdditionally, the IMAP server generates 'chaotic' events; it will\nintentionally generate incorrect or gibberish responses. The SuperIMAP\nIMAP client code is expected to recover gracefully while using a\nminimal amount of system resources.\n\nRun this once:\n\n    RAILS_ENV=stress rake db:setup db:seed\n\nThen run the stress test:\n\n    script/stress-test\n\n#### Future Work\n\n* Configure stress test to report code coverage.\n* Make a way to \"sweep\" a user's inbox, generating webhook events for all emails.\n\n#### Contributions\n\nTo contribute to this project, please fork and file a pull\nrequest. Small patches will be accepted more quickly than large\npatches.\n\n## Changes\n\n#### Version 0.1.2\n\n* Re-organize and improve documentation in README.md.\n* Detect and handle race condition around changing UIDVALIDITY.\n* Properly escape OAuth 2.0 disconnect URL.\n* Improve usage of Rails connection pool.\n* Clean up heartbeat records during exit.\n* Synchronize access to shared objects.\n\n## License\n\nThe MIT License (MIT)\n\nCopyright (c) 2015 Rusty Klophaus / FiveStreet.com\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in\nall copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\nTHE SOFTWARE.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frustyio%2Fsuper-imap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frustyio%2Fsuper-imap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frustyio%2Fsuper-imap/lists"}