{"id":13687313,"url":"https://github.com/ascoderu/lokole","last_synced_at":"2026-02-24T23:03:06.071Z","repository":{"id":39769023,"uuid":"66427049","full_name":"ascoderu/lokole","owner":"ascoderu","description":"Source code for the Lokole project. Lokole enables communities in the Congo DRC to pool resources to access efficient communication via email at an affordable price.","archived":false,"fork":false,"pushed_at":"2023-02-16T06:27:39.000Z","size":6012,"stargazers_count":43,"open_issues_count":36,"forks_count":12,"subscribers_count":14,"default_branch":"master","last_synced_at":"2024-05-01T09:38:04.976Z","etag":null,"topics":["antd","azure","celery","connexion","dnsmasq","docker","hacktoberfest","helm","hostapd","kubernetes","libcloud","python3","raspberry-pi","reactjs","sendgrid","usb-modeswitch","wvdial"],"latest_commit_sha":null,"homepage":"https://ascoderu.ca","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ascoderu.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2016-08-24T03:45:28.000Z","updated_at":"2024-03-15T14:38:28.000Z","dependencies_parsed_at":"2024-01-14T15:21:44.175Z","dependency_job_id":"3c2e8d1c-7e94-4728-8578-1884f9ee3eee","html_url":"https://github.com/ascoderu/lokole","commit_stats":{"total_commits":1540,"total_committers":17,"mean_commits":90.58823529411765,"dds":0.5896103896103896,"last_synced_commit":"1b2a7b18f472952df06f9938ebd01a0ecd749a79"},"previous_names":[],"tags_count":58,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ascoderu%2Flokole","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ascoderu%2Flokole/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ascoderu%2Flokole/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ascoderu%2Flokole/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ascoderu","download_url":"https://codeload.github.com/ascoderu/lokole/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219859904,"owners_count":16556031,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["antd","azure","celery","connexion","dnsmasq","docker","hacktoberfest","helm","hostapd","kubernetes","libcloud","python3","raspberry-pi","reactjs","sendgrid","usb-modeswitch","wvdial"],"created_at":"2024-08-02T15:00:52.336Z","updated_at":"2025-10-28T08:31:44.592Z","avatar_url":"https://github.com/ascoderu.png","language":"Python","readme":"======\nLokole\n======\n\n.. image:: https://github.com/ascoderu/lokole/workflows/CI/badge.svg\n  :target: https://github.com/ascoderu/lokole/actions\n\n.. image:: https://img.shields.io/pypi/v/opwen_email_client.svg\n  :target: https://pypi.python.org/pypi/opwen_email_client/\n\n.. image:: https://codecov.io/gh/ascoderu/lokole/branch/master/graph/badge.svg\n  :target: https://codecov.io/gh/ascoderu/lokole\n\n------------\nWhat's this?\n------------\n\nThis repository contains the source code for the Lokole project by the\nCanadian-Congolese non-profit `Ascoderu \u003chttps://ascoderu.ca\u003e`_. The Lokole\nproject consists of two main parts: an email client and an email server.\n\nThe Lokole email client is a simple application that offers functionality like:\n\n1. Self-service creation of user accounts\n2. Read emails sent to the account\n3. Write emails including rich formatting\n4. Send attachments\n\nAll emails are stored in a local SQLite database. Once per day, the emails that\nwere written during the past 24 hours get exported from the database, stored in\na JSON file, compressed and uploaded to a location on Azure Blob Storage. The\nLokole Server picks up these JSON files, manages the actual mailboxes for the\nusers on the Lokole and sends new emails back to the Lokole by using the same\ncompressed file exchange format.\n\nThe Lokole email application is intended to run on low-spec Raspberry Pi 3\nhardware (or similar). Read the \"Production setup\" section below for further\ninformation on how to set up the client devices.\n\nThe Lokole email server has two main responsibilities:\n\n1. Receive emails from the internet that are addressed to Lokole users and\n   forward them to the appropriate Lokole device.\n2. Send new emails created by Lokole users to the rest of the internet.\n\n-------------------\nWhy is this useful?\n-------------------\n\nEmail is at the core of our modern life, letting us keep in touch with friends\nand family, connecting us to our businesses partners and fostering innovation\nthrough exchange of information.\n\nHowever, in many parts of the developing world, email access is not very\nwide-spread, usually because bandwidth costs are prohibitively high compared to\nlocal purchasing power. For example, in the Democratic Republic of the Congo\n(DRC) only 3% of the population have access to emails which leaves 75 million\npeople unconnected.\n\nThe Lokole is a project by the Canadian-Congolese non-profit `Ascoderu \u003chttps://ascoderu.ca\u003e`_\nthat aims to address this problem by tackling it from three perspectives:\n\n1. The Lokole is an email client that only uses bandwidth on a schedule. This\n   reduces the cost of service as bandwidth can now be purchased when the cost\n   is lowest. For example, in the DRC, $1 purchases only 65 MB of data during\n   peak hours. At night, however, the same amount of money buys 1 GB of data.\n\n2. The Lokole uses an efficient data exchange format plus compression so that\n   it uses minimal amounts of bandwidth, reducing the cost of service. All\n   expensive operations (e.g. creating and sending of emails with headers,\n   managing mailboxes, etc.) are performed on a server in a country where\n   bandwidth is cheap.\n\n3. The Lokole only uses bandwidth in batches. This means that the cost of\n   service can be spread over many people and higher savings from increased\n   compression ratios can be achieved. For example, individually purchasing\n   bandwidth for $1 to check emails is economically un-viable for most people\n   in the DRC. However, the same $1 can buy enough bandwidth to provide email\n   for hundreds of people via the Lokole. Spreading the cost in this way makes\n   email access sustainable for local communities.\n\n---------------\nSystem overview\n---------------\n\nTechnologies\n============\n\nBelow is a list of some of the key technologies used in the Lokole project:\n\n- `Connexion \u003chttps://jobs.zalando.com/tech/blog/crafting-effective-microservices-in-python/\u003e`_ is the web framework for the Lokole email server API.\n- `Flask \u003chttps://flask.palletsprojects.com/\u003e`_ is the web framework for the Lokole email client application.\n- `Dnsmasq \u003chttp://www.thekelleys.org.uk/dnsmasq/doc.html\u003e`_ and `hostapd \u003chttps://w1.fi/hostapd/\u003e`_ are used to set up a WiFi access point on the Lokole device via which the Lokole email client application is accessed.\n- `WvDial \u003chttps://wiki.debian.org/Wvdial\u003e`_ is used to access the internet on the Lokole device to synchronize emails with the Lokole email server.\n- `Celery \u003chttp://www.celeryproject.org/\u003e`_ is used to run background workers of the Lokole email server in `Azure ServiceBus \u003chttps://azure.microsoft.com/en-us/services/service-bus/\u003e`_ (production) or `RabbitMQ \u003chttps://www.rabbitmq.com/\u003e`_ (development). Celery is also used to run background workers and scheduled tasks on the Lokole email client application in `SQLAlchemy \u003chttps://www.sqlalchemy.org/\u003e`_.\n- `Libcloud \u003chttps://libcloud.apache.org/\u003e`_ is used to store emails in `Azure Storage \u003chttps://azure.microsoft.com/en-us/services/storage/\u003e`_ (production) or `Azurite \u003chttps://github.com/Azure/Azurite\u003e`_ (development).\n- `Sendgrid Inbound Parse \u003chttps://sendgrid.com/docs/for-developers/parsing-email/setting-up-the-inbound-parse-webhook/\u003e`_ is used to receive emails from email providers and forward them to the Lokole email server. `Sendgrid Web API v3 \u003chttps://github.com/sendgrid/sendgrid-python\u003e`_ is used to deliver emails from the Lokole email server to email providers. The MX records for Sendgrid are automatically generated via `Cloudflare API v4 \u003chttps://api.cloudflare.com/\u003e`_.\n- `Github API v4 \u003chttps://developer.github.com/v4/\u003e`_ is used to authenticate interactive calls to the Lokole email server API such as registering new clients or managing existing clients. Authorization is managed by Github team memberships on the Ascoderu organization. Management operations are exposed via the Lokole status page which is implemented in `React \u003chttps://reactjs.org/\u003e`_ with `Ant Design \u003chttps://ant.design/docs/react/introduce\u003e`_.\n- `Github Actions \u003chttps://github.com/ascoderu/lokole/actions\u003e`_ are used to verify pull requests and deploy updates to production.\n\nThe diagram below shows the technologies in the context of the system as well as their interactions:\n\n.. image:: https://user-images.githubusercontent.com/1086421/95025387-687ee480-0657-11eb-8519-4ef4c0224648.png\n  :width: 800\n  :align: center\n  :alt: Overview of technologies and interactions in the Lokole system\n  :target: https://drive.google.com/file/d/1F9LMqpoglaKWRw8HjhZ1jzPkdCMpuOur/view\n\nInteractions\n============\n\nThe key data flows and client/server interactions of the system are documented in the diagrams below.\n\n.. image:: https://static.swimlanes.io/23added12f9ab7faa03ac6d1c6bdc733.png\n  :width: 800\n  :align: center\n  :alt: Overview of the Lokole client registration flow\n  :target: https://swimlanes.io/u/SfWS0LVYu\n\n.. image:: https://static.swimlanes.io/b070c40083a3f67ede3e49fa9cd25933.png\n  :width: 800\n  :align: center\n  :alt: Overview of the Lokole client email upload flow\n  :target: https://swimlanes.io/u/hub7TEZgp\n\n.. image:: https://static.swimlanes.io/3dc4b74d377eb3094dc83fc1da9dfe84.png\n  :width: 800\n  :align: center\n  :alt: Overview of the Lokole client email download flow\n  :target: https://swimlanes.io/u/_QqT0iQx8\n\n--------------------\nData exchange format\n--------------------\n\nIn order to communicate between the Lokole cloud server and the Lokole email\napplication, a protocol based on gzipped jsonl files uploaded to Azure Blob\nStorage is used. The files contains a JSON object per line. Each JSON object\ndescribes an email, using the following schema:\n\n.. sourcecode :: json\n\n  {\n    \"sent_at\": \"yyyy-mm-dd HH:MM\",\n    \"to\": [\"email\"],\n    \"cc\": [\"email\"],\n    \"bcc\": [\"email\"],\n    \"from\": \"email\",\n    \"subject\": \"string\",\n    \"body\": \"html\",\n    \"attachments\": [{\"filename\": \"string\", \"content\": \"base64\", \"cid\": \"string\"}]\n  }\n\n-----------------\nDevelopment setup\n-----------------\n\nFirst, install the system dependencies:\n\n- `docker \u003chttps://docs.docker.com\u003e`_\n- `docker-compose \u003chttps://docs.docker.com/compose/\u003e`_\n- `git \u003chttps://git-scm.com\u003e`_\n- `make \u003chttps://www.gnu.org/software/make/\u003e`_\n\nSecond, get the source code.\n\n.. sourcecode :: sh\n\n  git clone git@github.com:ascoderu/lokole.git\n  cd lokole\n\nThird, build the project images. This will also verify your checkout by\nrunning the unit tests and other CI steps such as linting:\n\n.. sourcecode :: sh\n\n  make build\n\nYou can now run the application stack:\n\n.. sourcecode :: sh\n\n  make start logs\n\nFinding your way around the project\n===================================\n\nThere are OpenAPI specifications that document the functionality of the\napplication and provide references to the entry points into the code\n(look for the yaml files in the swagger directory). The various\nAPIs can also be easily called via the testing console that is available\nby adding /ui to the end of the API's URL. Sample workflows are shown\nin the integration tests folder and can be run via:\n\n.. sourcecode :: sh\n\n  # run the services, wait for them to start\n  make build start\n\n  # in another terminal, run the integration tests\n  # the integration tests also serve the purpose of\n  # seeding the system with some test data\n  # you can access the email service at http://localhost:8080\n  # you can access the email client at http://localhost:5000\n  # you can access the status page at http://localhost:3000\n  make integration-tests test-emails\n\n  # finally, tear down the services\n  make stop\n\nThe state of the system can be inspected via:\n\n.. sourcecode :: sh\n\n  # run the development tools and then\n  # view storage state at http://localhost:10001\n  # view database state at http://localhost:8882\n  # view queue state at http://localhost:5555\n  make start-devtools\n\nNote that by default the application is run in a fully local mode, without\nleveraging any cloud services. For most development purposes this is fine\nbut if you wish to set up the full end-to-end stack that leverages the\nsame services as we use in production, keep on reading.\n\nIntegration setup\n=================\n\nThe project uses Sendgrid, so to emulate a full production environment,\nfollow these `Sendgrid setup instructions \u003chttps://sendgrid.com/free/\u003e`_ to\ncreate a free account, authenticate your domain, and create an API key with\nat least Inbound Parse and Mail Send permissions.\n\nThe project uses Cloudflare to automate DNS management whenever new Lokole\nclients are set up. Create an account, set your domain to be managed by\nCloudflare and look up the `Cloudflare Global API Key \u003chttps://dash.cloudflare.com/profile/api-tokens\u003e`_.\n\nThe project also makes use of a number of Azure services such as Blobs,\nTables, Queues, Application Insights, and so forth. To set up all the\nrequired cloud resources programmatically, you'll need to create a service\nprincipal by following these `Service Principal instructions \u003chttps://aka.ms/create-sp\u003e`_.\nAfter you created the service principal, you can run the Docker setup script\nto initialize the required cloud resources.\n\n.. sourcecode :: sh\n\n  cat \u003e ${PWD}/secrets/sendgrid.env \u003c\u003c EOM\n  LOKOLE_SENDGRID_KEY={the sendgrid key you created earlier}\n  EOM\n\n  cat \u003e ${PWD}/secrets/cloudflare.env \u003c\u003c EOM\n  LOKOLE_CLOUDFLARE_USER={your cloudflare user account email address}\n  LOKOLE_CLOUDFLARE_KEY={your cloudflare global api key}\n  EOM\n\n  cat \u003e ${PWD}/secrets/users.env \u003c\u003c EOM\n  OPWEN_SESSION_KEY={some secret for user session management}\n  LOKOLE_REGISTRATION_USERNAME={some username for the registration endpoint}\n  LOKOLE_REGISTRATION_PASSWORD={some password for the registration endpoint}\n  EOM\n\n  docker-compose -f ./docker-compose.yml -f ./docker/docker-compose.setup.yml build setup\n  docker-compose -f ./docker-compose.yml -f ./docker/docker-compose.setup.yml run --rm \\\n    -e SP_APPID={appId field of your service principal} \\\n    -e SP_PASSWORD={password field of your service principal} \\\n    -e SP_TENANT={tenant field of your service principal} \\\n    -e SUBSCRIPTION_ID={subscription id of your service principal} \\\n    -e LOCATION={an azure location like eastus} \\\n    -e RESOURCE_GROUP_NAME={the name of the resource group to create or reuse} \\\n    -v ${PWD}/secrets:/secrets \\\n    setup ./setup.sh\n\nThe secrets to access the Azure resources created by the setup script will be\nstored in files in the :code:`secrets` directory. Other parts of the\nproject's tooling (e.g. docker-compose) depend on these files so make sure to\nnot delete them.\n\n---------------------\nProduction deployment\n---------------------\n\nTo set up a production-ready deployment of the system, follow the development\nsetup scripts described above, but additionally also pass the following\nenvironment variables to the Docker setup script:\n\n- :code:`DEPLOY_COMPUTE`: Must be set to :code:`k8s` to toggle the Kubernetes\n  deployment mode.\n\n- :code:`KUBERNETES_RESOURCE_GROUP_NAME`: The resource group into which to\n  provision the Azure Kubernetes Service cluster.\n\n- :code:`KUBERNETES_NODE_COUNT`: The number of VMs to provision into the\n  cluster. This should be an odd number and can be dynamically changed later\n  via the Azure CLI.\n\n- :code:`KUBERNETES_NODE_SKU`: The type of VMs to provision into the cluster.\n  This should be one of the supported `Linux VM sizes \u003chttps://docs.microsoft.com/en-us/azure/virtual-machines/linux/sizes\u003e`_.\n\nThe script will then provision a cluster in Azure Kubernetes Service and\ninstall the project via Helm. The secrets to connect to the provisioned\ncluster will be stored in the :code:`secrets` directory.\n\nAs an alternative to the Kubnernets deployment, a Virtual Machine may also be\nprovisioned to run the services by passing the following environment variables\nto the Docker setup script:\n\n- :code:`DEPLOY_COMPUTE`: Must be set to :code:`vm` to toggle the Virtual\n  Machine deployment mode.\n\n- :code:`VM_RESOURCE_GROUP_NAME`: The resource group into which to\n  provision the Azure Virtual Machine.\n\n- :code:`VM_SKU`: The type of VMs to provision into the cluster.\n  This should be one of the supported `Linux VM sizes \u003chttps://docs.microsoft.com/en-us/azure/virtual-machines/linux/sizes\u003e`_.\n\nThere is a `script \u003chttps://github.com/ascoderu/lokole/blob/master/install.py\u003e`_\nto set up a new Lokole email client. The script will install the email app in this\nrepository as well as standard infrastructure like nginx and gunicorn.\nThe script will also make ready peripherals like the USB modem used for data\nexchange, and set up any required background jobs such as the email\nsynchronization cron job.\n\nThe setup script assumes that you have already set up:\n\n* 3 Azure Storage Accounts, general purpose: for the cloudserver to manage its\n  queues, tables and blobs.\n* 1 Azure Storage Account, blob storage: for the cloudserver and email app to\n  exchange email packages.\n* 1 Application Insights account: to collect logs from the cloudserver and\n  monitor its operations.\n* 1 SendGrid account: to send and receive emails in the cloudserver.\n\nThe setup script is tested with hardware:\n\n* `Raspberry Pi 3 \u003chttps://www.raspberrypi.org/products/raspberry-pi-3-model-b/\u003e`_\n  running Raspbian Jessie lite\n  `v2016-05-27 \u003chttps://downloads.raspberrypi.org/raspbian_lite/images/raspbian_lite-2016-05-31/2016-05-27-raspbian-jessie-lite.zip\u003e`_,\n  `v2017-01-11 \u003chttps://downloads.raspberrypi.org/raspbian_lite/images/raspbian_lite-2017-01-10/2017-01-11-raspbian-jessie-lite.zip\u003e`_,\n  `v2017-04-10 \u003chttps://downloads.raspberrypi.org/raspbian_lite/images/raspbian_lite-2017-04-10/2017-04-10-raspbian-jessie-lite.zip\u003e`_, and\n  `v2017-11-29 \u003chttp://vx2-downloads.raspberrypi.org/raspbian_lite/images/raspbian_lite-2017-12-01/2017-11-29-raspbian-stretch-lite.zip\u003e`_.\n\n* `Orange Pi Zero \u003chttp://www.orangepi.org/orangepizero/\u003e`_\n  running `Armbian Ubuntu Xenial \u003chttps://dl.armbian.com/orangepizero/Ubuntu_xenial_default.7z\u003e`_\n\nThe setup script is also tested with USB modems:\n\n* `Huawei E303s-65 \u003chttps://www.aliexpress.com/item/4000961436981.html\u003e`_\n* `Huawei E3131 \u003chttps://www.aliexpress.com/item/32404274659.html\u003e`_\n* `Huawei MS2131i-8 \u003chttps://www.aliexpress.com/item/32964337576.html\u003e`_\n\nThe setup script installs the latest version of the email app published to PyPI.\nNew versions get automatically published to PyPI (via Travis) whenever a new\n`release \u003chttps://github.com/ascoderu/lokole/releases/new\u003e`_ is created\non Github.\n\nYou can run the script on your client device like so:\n\n.. sourcecode :: sh\n\n  curl -fsO https://raw.githubusercontent.com/ascoderu/lokole/master/install.py \u0026\u0026 \\\n  sudo python3 install.py \u003cclient-name\u003e \u003csim-type\u003e \u003csync-schedule\u003e \u003cregistration-credentials\u003e\n\n---------------------\nAdding a new language\n---------------------\n\nTo translate Lokole to a new language, install Python, `Babel \u003chttps://babel.pocoo.org/\u003e`_\nand a translation editor such as `poedit \u003chttps://poedit.net/\u003e`_. Then follow the steps below.\n\n.. sourcecode :: sh\n\n  # set this to the ISO 639-1 language code for which you are adding the translation\n  export language=ln\n\n  # generate the translation file\n  pybabel init -i babel.pot -d opwen_email_client/webapp/translations -l \"${language}\"\n\n  # fill-in the translation file\n  poedit \"opwen_email_client/webapp/translations/${language}/LC_MESSAGES/messages.po\"\n\n  # finalize the translation file\n  pybabel compile -d opwen_email_client/webapp/translations\n","funding_links":[],"categories":["docker"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fascoderu%2Flokole","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fascoderu%2Flokole","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fascoderu%2Flokole/lists"}