{"id":21114570,"url":"https://github.com/webu/dalec","last_synced_at":"2025-07-08T18:32:06.404Z","repository":{"id":77060501,"uuid":"420088740","full_name":"webu/dalec","owner":"webu","description":"  Django Aggregator of a Lot of External Contents (DALEC) is a generic app to aggregate contents from various sources and display it in a generic way. Designed to be customized.","archived":false,"fork":false,"pushed_at":"2023-11-03T11:55:28.000Z","size":330,"stargazers_count":5,"open_issues_count":3,"forks_count":6,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-06-22T09:51:26.505Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/webu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-10-22T12:23:21.000Z","updated_at":"2025-05-21T15:27:19.000Z","dependencies_parsed_at":"2023-09-22T20:24:18.848Z","dependency_job_id":"91c62a90-1c3b-4b88-837f-4308a3ba865f","html_url":"https://github.com/webu/dalec","commit_stats":null,"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"purl":"pkg:github/webu/dalec","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/webu%2Fdalec","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/webu%2Fdalec/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/webu%2Fdalec/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/webu%2Fdalec/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/webu","download_url":"https://codeload.github.com/webu/dalec/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/webu%2Fdalec/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264323974,"owners_count":23590776,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-20T02:00:42.116Z","updated_at":"2025-07-08T18:32:05.886Z","avatar_url":"https://github.com/webu.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Django Aggregator of a Lot of External Contents aka DALEC 🤖\n\n[![Stable Version](https://img.shields.io/pypi/v/dalec?color=blue)](https://pypi.org/project/dalec/)\n![](https://img.shields.io/badge/python-3.7%20to%203.11-blue)\n![](https://img.shields.io/badge/django-2.2%20to%204.2-blue)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n![](https://img.shields.io/badge/coverage-100%25-green)\n[![Checked with mypy](https://www.mypy-lang.org/static/mypy_badge.svg)](https://mypy-lang.org/)\n[![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat\u0026labelColor=ef8336)](https://pycqa.github.io/isort/)\n[![semver](https://img.shields.io/badge/semver-2.0.0-green)](https://semver.org/)\n[![Documentation Status](https://readthedocs.org/projects/dalec/badge/?version=latest)](https://dalec.readthedocs.io/en/latest/?badge=latest)\n\nDjango Aggregator of a Lot of External Contents (DALEC) is a generic app to aggregate contents from various\nexternal sources. Purposes are to manage (retrieve, clean, display…) those contents in a\ngeneric way independent of the source.\n\n\nIt's designed to be customized / extended to fit your needs.\n\n![Logo de DALEC](https://raw.githubusercontent.com/webu/dalec/main/logo.png)\n\n## Tests, QA, consistency and compatibility\n\nThis app is tested to runs with:\n\n* Django 2.2, 3.2, 4.2\n* Python 3.7, 3.9, 3.11\n\nTo ensure code quality and consistency:\n\n* Python formatted with [black](https://pypi.org/project/black/)\n* Python validated with [flake8](https://pypi.org/project/flake8/). \n* Python static types checked with [mypy](https://pypi.org/project/mypy/)\n* Python tests coverage checked with [coverage](https://pypi.org/project/coverage/) (100% tested)\n* JS tested with [jest](https://jestjs.io/fr/)\n* JS formatted with [prettier](https://prettier.io/)\n* Tests runned in local via [tox](https://pypi.org/project/tox/) and on github via [github actions workflow](https://docs.github.com/en/actions/using-workflows)\n* versionned with [semver](https://semver.org) logic\n\n## Concepts\n\nContents are categorized via :\n\n* `app`: the app which is requested to retrieve contents \n  (eg. `gitlab`)\n* `content_type`: the type of contents we want to retrieve from this app \n  (eg. `issue`, `activity`, `commit`, `merge requests`…)\n* `channel`: some apps can be requested to get a more or less filtered contents. \n  For example, in Gitlab, it's called \"scope\". You can retrieve issues from:\n  * the whole site (`channel=None` and `channel_object=None` for us)\n  * a specific group (`channel=\"group\"` and `channel_object=\"\u003cgitlab_group_id\u003e\"`)\n  * a specific project (`channel=\"project\"` and `channel_object=\"\u003cgitlab_project_id\u003e\"`)\n  * a specific author (`channel=\"author\"` and `channel_object=\"\u003cgitlab_user_id\u003e\"`)\n  * a specific assignee (`channel=\"assignee\"` and `channel_object=\"\u003cgitlab_user_id\u003e\"`)\n  * etc.\n* `channel_object`: the ID (for the app requested) of the channel's related object\n* `channel_objects`: the list of IDs (for the app requested) of the channel's related objects (eg. `\"['42','443']\"`)\n\nBefore you ask: yes, some contents can be duplicated (eg. an issue from the \"group\" channel \ncould be duplicated in the \"project\" channel). It's normal and **wanted**. Remember : the purpose\nof this app is to retrieve and display the `last N contents of something`. If the issue we are \ntalking about is in the `last N issues of project \"cybermen\"`, it does not mean it's also in \n`the last N issues of group \"dr-who-best-friends\"`. To manage different timelines for each channel,\nand keep a KISS Model, we need those duplicates.\n\n## External sources supported\n\n* 🦝 [dalec-gitlab](https://github.com/webu/dalec-gitlab/):\n  get issues and activities from a gitlab instance\n* ☁ [dalec-nextcloud](https://github.com/webu/dalec-nextcloud/):\n  get events and activities from a nextcloud instance\n* 🗣 [dalec-discourse](https://github.com/webu/dalec-discourse/):\n  get last messages or topics from a discourse instance\n* 🔗 [dalec-openproject](https://github.com/webu/dalec-openproject/):\n  get tasks from an openproject instance\n* 📅 [dalec-caldav](https://github.com/webu/dalec-caldav/): events and tasks from a caldav instance (eg: nextcloud agenda)\n\n## External sources which could be nice to support\n\n* 🌍 dalec-mediawiki: get last pages from a mediawiki instance\n* 📂 dalec-webdav: get lastmodified files from a webdav instance (eg: nextcloud files)\n* 📰 dalec-feedparser: get contents from atom and rss feeds\n* 📫 dalec-imap: get emails from imap instance\n* 🐥 dalec-mastodon: get toots from a mastodon instance\n* 📺 dalec-peertube: get last uploaded videos from a peertube instance\n* 🌵 dalec-git: get last commits, branches, tags from a git repository\n* 🐱 dalec-github: get issues, pull-requests, activity from github\n* 🐦 dalec-twitter: get tweets from twitter\n* 🎞 dalec-youtube: get last uploaded videos from youtube\n* 🎥 dalec-imdb: get last movies from imdb\n\n## Installation\n\nIf you are using Django older than 3.1, the JSONField is provided from \n[django-jsonfield-backport](https://pypi.org/project/django-jsonfield-backport/). \nFor 3.1+, you **MUST** use a DB supporting the official\n[django's `JSONField`](https://docs.djangoproject.com/fr/3.1/ref/models/fields/#jsonfield).\n\nYou **SHOULD NOT** install this app but you **SHOULD** install one (or more) of it's children \n(see [](#external-sources-supported)). eg:\n\n`pip install dalec-gitlab dalec-nextcloud`\n\nThen, add `dalec`, `dalec_prime` and dalec's children to `INSTALLED_APPS` in `settings.py`:\n\n```python\n# settings.py\n\nINSTALLED_APPS = [\n    # …\n    \"dalec\",\n    \"dalec_prime\",  # if you want to use your own Content model, don't add it\n    \"dalec_gitlab\",\n    \"dalec_nextcloud\",\n    # …\n]\n```\n\nAnd in your `urls.py`:\n\n```python\n# urls.py\n\nurl_patterns = [\n  ...\n  re_path(\"^dalec/\", include(\"dalec.urls\")),\n  ...\n]\n```\n\n## Usage\n\nEach dalec's child app will probably need some specific configuration to retrieve external contents \n(eg: token or login/password). Please refer to this dalec's child app configuration section first.\n\nNow your dalec's child app is configured, you can display it's X last contents somewhere in a \ntemplate by using the templatetag `dalec`:\n\n```django\n{% load dalec %}\n\n{% dalec app content_type [channel=None] [channel_object=None] [template=None] [ordered_by=None] %}\n\nreal examples:\n\nRetrieves last gitlab issues for a specific user:\n{% dalec \"gitlab\" \"issue\" channel=\"user\" channel_object=\"doctor-who\" %}\n\nRetrieves recent gitlab event for a group:\n{% dalec \"gitlab\" \"event\" channel=\"group\" channel_object='42' %}\n\nRetrieves recent gitlab event for a project:\n{% dalec \"gitlab\" \"event\" channel=\"project\" channel_object='443' %}\n\nRetrieves recent gitlab issues for a project:\n{% dalec \"gitlab\" \"issue\" channel=\"project\" channel_object='404' %}\n\nRetrieves recent gitlab issues for multiple projects:\n{% dalec \"gitlab\" \"issue\" channel=\"project\" channel_objects='[\"42\",\"443\"]' %}\n\nRetrieves recent gitlab issues for multiple projects and order them by descending\nissue internal ID (default is `last_update_dt`):\n{% dalec \"gitlab\" \"issue\" channel=\"project\" channel_object='42' ordered_by=\"-iid\" %}\n```\n\n### dalec_example\n\nAn example app is packaged to get a working example which does not require any extra configuration.\n\n* add `dalec_example`, `dalec_prime` and `dalec` to `INSTALLED_APPS`\n* run migrations\n* include dalec.urls inside your project's urls\n* add those fragments of code inside the template where you want to display the example:\n\n```django\n{% load dalec %}\n\u003ch1\u003eLast quarters (very usefull, isn't it ?)\u003c/h1\u003e\n{% dalec \"example\" \"quarter\" %}\n\n\u003ch1\u003eLast updated establishments of french national education\u003c/h1\u003e\n{% dalec \"example\" \"french_educ\" %}\n\n\u003ch1\u003eLast updated establishments of french national education depending of the Academy of Grenoble\u003c/h1\u003e\n{% dalec \"example\" \"french_educ\" channel=\"academy\" channel_object=\"Grenoble\" %}\n```\n\n\n## Configuration\n\nThis app have general settings which can be erased for all of it's children and sometimes by \ncontent type.\n\n* General setting format : `DALEC_SOMETHING`\n* override child version (it's app name, like gitlab for example): `DALEC_GITLAB_SOMETHING`\n* override content type version (gitlab's issues for example): `DALEC_GITLAB_ISSUE_SOMETHING`\n\n### DALEC_NB_CONTENTS_KEPT\n\n* *default*: `10`\n* per child app setting: yes\n* per child app's content type setting: yes\n\nSet the number of contents to keep by type. Oldest will be purged to keep only the last X contents \nof each channel and type.\n`0` means \"no limit\".\n\n### DALEC_AJAX_REFRESH\n\n* *default*: `True`\n* per child app setting: yes\n* per child app's content type setting: yes\n\nIf `True`, when an user display a channel contents, an ajax requests is sent to refresh content. \nIt's usefull if you do not want to use a cron task and/or need to get always the last contents.\n\n### DALEC_TTL\n\n* *default*: `900`\n* per child app setting: yes\n* per child app's content type setting: yes\n\nNumber of seconds before an ajax request sends a new query to the instance providing instance.\n\n### DALEC_CONTENT_MODEL\n\n* *default*: `\"dalec_prime.Content\"`\n* per child app setting: no\n* per child app's content type setting: no\n\nConcrete model to use to store contents. If you do not want to use the default one,\nyou should not add `dalec.prime` in `INSTALLED_APPS` to avoid to load a useless model\nand have an empty table in your data base.\n\n### DALEC_FETCH_HISTORY_MODEL\n\n* *default*: `\"dalec_prime.FetchHistory\"`\n* per child app setting: no\n* per child app's content type setting: no\n\nSame as `DALEC_CONTENT_MODEL` but for the `FetchHistory` model.\n\n### DALEC_CSS_FRAMEWORK\n\n* *default*: `None`\n* per child app setting: yes\n* per child app's content type setting: no\n\nName of the (S)CSS framework you use on your website. It changes the default templates used to \ndisplay contents. Templates are priorized in this order (`css_framework` versions only used if \nyou set a value to `DALEC_CSS_FRAMEWORK`):\n\n* `dalec/%(app)s/%(tpl)s-list.html`: only if you use a specific template, see templatetag `dalec`\n* `dalec/%(app)s/%(css_framework)s/%(content_type)s-%(channel)s-list.html`\n* `dalec/%(app)s/%(content_type)s-%(channel)s-list.html`\n* `dalec/%(app)s/%(css_framework)s/%(content_type)s-list.html`\n* `dalec/%(app)s/%(content_type)s-list.html`\n* `dalec/%(app)s/%(css_framework)s/list.html`\n* `dalec/%(app)s/list.html`\n* `dalec/default/%(css_framework)s/list.html`\n* `dalec/default/list.html` \n\nSupported valued are:\n\n* `None`: only dalec classes and data will be added. HTML elements are sementics. Templates are the default ones.\n\nFuture supported values could be:\n\n* materialize\n* bulma\n* bootstrap\n* semantic-ui\n\n## Customization\n\n### Styles\n\nNo styles are included inside dalec who must remains pure with no feelings. \nBut some SCSS framework may be supported. Please refer to `DALEC_CSS_FRAMEWORK` setting.\n\n### Templates\n\nYou should always inherit from the default templates and use defined blocks to customise it.\n\n#### List\n\nEach app can have it's own \"list\" template but if it doesn't, a default one will be used. \nIn priority order:\n\n* `dalec/\u003cchildappname\u003e/\u003ccontenttype\u003e-\u003cchannel\u003e-list.html`\n* `dalec/\u003cchildappname\u003e/\u003ccontenttype\u003e-list.html`\n* `dalec/\u003cchildappname\u003e/list.html`\n* `dalec/default/list.html`\n\n#### Item\n\nEach content types can have it's own template. Those filenames will be tried:\n\n* `dalec/\u003cchildappname\u003e/\u003ccontenttype\u003e-\u003cchannel\u003e-item.html`\n* `dalec/\u003cchildappname\u003e/\u003ccontenttype\u003e-item.html`\n* `dalec/\u003cchildappname\u003e/item.html`\n* `dalec/default/item.html`\n\nFor \"gitlab\" app, \"issue\" content type and \"project\" channel we will try :\n\n* `dalec/gitlab/issue-project-item.html`\n* `dalec/gitlab/issue-item.html`\n* `dalec/gitlab/item.html`\n* `dalec/default/item.html`\n\n### Models\n\nModel used to store contents is defined by the setting `DALEC_CONTENT_MODEL` which has the\ndefault value `\"dalec_prime.Content\"`.\nIf you want to use your own model, in your `settings.py`:\n\n* remove `dalec.prime` from `INSTALLED_APPS`\n* set the setting `DALEC_CONTENT_MODEL` with `\u003cyourapp\u003e.\u003cyourModel\u003e`\n\n## Manage a new external source\n\nIf you want to add a specific external source, you just have to extends `dalec.proxy.Proxy`\nand override it's `_fetch` method.\n\nTo create a dalec child (a proper way), you should create a new django app with the name pattern\n`dalec_\u003cyourExternalSourceUname\u003e`\n\n## NAQ (Never Asked Questions)\n\n### Why this logo is so ugly ?\n\nBecause I'm a developper\n\n### Aren't you afraid to be accused of blasphemy by the Daleks ?\n\nA Dalek which have a concept of Blasphemy can not be a sane Dalek! \nhttps://youtu.be/6ThpkjDgdvY?t=162\n\n### What's the dalec origin ?\n\nThe Dale**k**s are a fictional extraterrestrial race of mutants principally portrayed in the \nBritish science fiction television programme Doctor Who. Name (and logo) of this django app is \ndirectly related to them.\n\nThis project was made possible thanks to the \n[Open Space Maker Federation](https://www.federation-openspacemakers.com) whose goal is to open \nup the world of space infrastructure to as many people as possible.  \nThey need boards to aggregate contents from different sources (gitlab, discourse…). \nWe ([Webu](https://www.webu.coop)) didn't find any applications which fit our needs, \nso we create a new one and released it under the MIT licence.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwebu%2Fdalec","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwebu%2Fdalec","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwebu%2Fdalec/lists"}