{"id":16434565,"url":"https://github.com/dflock/seance","last_synced_at":"2026-05-10T23:57:04.800Z","repository":{"id":9968735,"uuid":"11993272","full_name":"dflock/seance","owner":"dflock","description":"Taking a little peek behind the curtain at medium.com","archived":false,"fork":false,"pushed_at":"2013-08-09T06:17:21.000Z","size":384,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-24T02:15:00.402Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"CSS","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dflock.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-08-09T04:44:25.000Z","updated_at":"2017-12-30T16:32:31.000Z","dependencies_parsed_at":"2022-09-19T05:01:50.100Z","dependency_job_id":null,"html_url":"https://github.com/dflock/seance","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dflock%2Fseance","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dflock%2Fseance/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dflock%2Fseance/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dflock%2Fseance/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dflock","download_url":"https://codeload.github.com/dflock/seance/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240746959,"owners_count":19850996,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T08:49:33.149Z","updated_at":"2026-05-10T23:57:04.772Z","avatar_url":"https://github.com/dflock.png","language":"CSS","funding_links":[],"categories":[],"sub_categories":[],"readme":"Welcome to the Séance!\n=========================\n\nTaking a little peek behind the curtain at medium.com - Just for fun.\n\nFrontend\n---------\n\nThe interesting bit! See the results here: http://dflock.github.io/seance/\n\nFeatures\n----------\n\nScrape medium.com to compile and display some data about activity on the site. It currently collects and displays the following things:\n\n- Totals: Authors, Articles, Words, Reading Time, Avg Words Per Article\n- Articles published over time, by Year, by Day of the Week\n- The first ever and most recent articles\n- Most prolific authors: By Articles Published, By Words Published, By Time to Read\n- Most Popular Words: Work Frequency by: All words, Medium Words (5-7 chars), Long Words (\u003e 7 chars), V. Long Words (\u003e 10 chars), Danke, Deutschland: (\u003e 20 chars)\n- Word Frequency by Word Length\n\nIt uses `scrapy \u003chttp://scrapy.org/\u003e`_ for the crawling, `pygreen \u003chttp://pygreen.neoname.eu/\u003e`_ to generate the static frontend page and MySQL to store everything.\n\nRunning It\n-----------\n\nDon't - it's against Medium.com's T\u0026C's. Also, it takes ages and doesn't work very well yet.\n\nIf you really want to, you'll need a vitrualenv with the stuff from ``requirements.txt`` installed in it and *lots* of patience. Here's the basic idea:\n\n- Create the MySQL database using ``/db/re-create-db.sql``\n- Change the MySQL login credentials in the following files to match your setup (yes, I know):\n  - post_process_data.py\n  - frontend/index.html\n  - seance_scrapy/seance_scrapy/pipelines.py\n\n- To start the crawl:\n\n.. code-block:: console\n\n    $ cd seance_scrapy\n    $ scrapy crawl medium\n\n- Once this finishes (I left it overnight and it didn't, so good luck - see below), run the ``post_process_data.py`` to calculate the word frequency data. you can press Ctrl + c to abort the crawl at any point.\n- Then do this to generate the frontend/report page, putting it into the ``/output`` folder:\n\n.. code-block:: console\n\n    $ cd frontend\n    $ pygreen gen ../output/\n\n\nTODO\n--------\n\nI made a mistake by relying on the scrapy sitemap crawler - either I'm using it wrong, medium.com's sitemaps overlap/duplicate themselves a lot, or it crawls *way* more stuff that it needs to. In light of that, and a desire for less magic and the ability to cleanly and efficiently re-start things where they left off, I want to rewrite the crawler part of this:\n\nRewrite crawler \u0026 break into two parts:\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n- One that crawls the medium.com sitemap.xml tree and records a list of article urls in the db, along with a last_crawled timestamp\n  - Also needs to record a list of sitemap files, along with a timestamp\n- Another that works through that list and crawls the articles, records the stats \u0026 updates the last_crawled timestamp\n\nOnce this is done, re-run the crawl and rebuild everything.\n\nOther minor issues:\n^^^^^^^^^^^^^^^^^^^^^\n\n- Add information on number of comments\n- Sort out the various unicode issues, especially in the frontend output.\n- I couldn't find anything in the article markup to specify the language the article is written in - maybe have another look?\n- The XPath for importing categories isn't working.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdflock%2Fseance","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdflock%2Fseance","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdflock%2Fseance/lists"}