{"id":16742272,"url":"https://github.com/clemsos/topoparser","last_synced_at":"2025-03-16T00:20:38.991Z","repository":{"id":34081787,"uuid":"37898991","full_name":"clemsos/topoparser","owner":"clemsos","description":"Command-line parser for topogram  ","archived":false,"fork":false,"pushed_at":"2015-06-23T13:31:36.000Z","size":152,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-22T13:19:35.406Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/clemsos.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-06-23T05:24:30.000Z","updated_at":"2020-02-28T13:14:07.000Z","dependencies_parsed_at":"2022-07-18T01:10:48.772Z","dependency_job_id":null,"html_url":"https://github.com/clemsos/topoparser","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clemsos%2Ftopoparser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clemsos%2Ftopoparser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clemsos%2Ftopoparser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clemsos%2Ftopoparser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/clemsos","download_url":"https://codeload.github.com/clemsos/topoparser/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243807431,"owners_count":20350992,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T01:23:13.748Z","updated_at":"2025-03-16T00:20:38.968Z","avatar_url":"https://github.com/clemsos.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Topoparser\n\nCommand-line parser for [Topogram](http://github.com/topogram/topogram). \n\nUse a YAML file to describe data source and workflow\n\n## Install \n\n    virtualenv venv\n    . venv/bin/activate\n    pip install -r requirements.txt\n\n## Usage\n\nUse the command line to call a YAML file containing the parsing instructions  \n\n    python topoparser.py weibo.yaml\n\nYou can save the output to json file by specficing and ```--output-dir``` params\n\n    python topoparser.py --output-dir test weibo.yaml\n\n\n## YAML command file structure\n\nThe overall workflow is separated in 3 main steps :\n\n1. **corpora** : load and describe the corpus\n2. **process** : extract information from the data\n3. **viz** : parse the data properly  (JSON output)\n\n\n#### Corpora\n\nThe corpora description follow a standard model  \n\n    corpora : \n        - content:\n            type : csv # could be a mongo or dict adapter\n            file : 'examples/sampleweibo.csv' # path of the file\n            columns:  # name of the relevant columns or fields\n                source : uid\n                text : text\n                timestamp : created_at\n                time_pattern : '%Y-%m-%d %H:%M:%S'\n                data : [ permission_denied, deleted_last_seen ]\n\nMultiple corpora can be used together (not implemented yet)\n\n#### Process\n\n\nYou first select the column you want to process from the data set -- ex . ```content.text```\n\nThe process remains on several data processors :\n\n* **regexp** : compile a regexp  -- ex. ```regexp : '@([^:：,，\\)\\(（）|\\\\\\s]+)' ``` extract @ mentions from Twitter-like corpus\n* **nlp** : extract keywords from a specific languages  -- ex. ``` nlp : zh ``` extract words from Chinese language\n* **graph** :  will add a list of elements into a graph -- ex. ``` graph : add_edges_from_nodes_list ```\n* **timeseries** : will format time information following specific time scales (second, minute, hour, day, month, year) -- ex. ``` timeseries : minute ```\n\n\nYou can use  2 different operators to link them together :\n\n**save** : will store the results of the operation \n    -- ex. extract all mentions and keep the results \n\n    - mentions : \n            regexp : '@([^:：,，\\)\\(（）|\\\\\\s]+)'\n            type : save \n\n\n**pipe** : will pass the data to the next operations ( like unix ```|``` symbol )\n    -- ex. extract all keywords from a Chinese sentence and add words  into a network\n\n    - words :\n        nlp : \"zh\"\n        type : pipe\n    - words_relationships :\n        graph: add_edges_from_nodes_list\n        type : save\n\n\n**Complete example**\n\nThis will extract hashtags, mentions and urls + create a network of words + compute a daily timeseries from the quantity of messages \n\n    process: \n        content.text :\n            - hashtags :\n                regexp: '#([^#\\s]+)#'\n                type : save\n            - mentions : \n                regexp : '@([^:：,，\\)\\(（）|\\\\\\s]+)'\n                type : save \n            - urls: \n                regexp : '\\b(([\\w-]+://?|www[.])[^\\s()\u003c\u003e]+(?:\\([\\w\\d]+\\)|([^\\p{P}\\s]|/)))'\n                type : save\n            - words :\n                nlp : \"zh\"\n                type : save\n            - words :\n                nlp : \"zh\"\n                type : pipe\n            - words_relationships :\n                graph: add_edges_from_nodes_list\n                type : save\n        content.time :\n            - timecount: \n                timeseries: day\n                type : save\n\n\n## Visualization parsers\n\nThe final step is to parse into JSON formatted properly for visualization library (like d3js)\n\nCurrently available : timeseries and network. \n\n    viz :\n        timeseries:\n            data : content.time.timecount\n    network : \n        nodes : content.text.words\n        edges : content.text.words_relationships \n\n\n## TODO\n\n* Use multiple datasets\n* Additional visualization models (map \u0026 network+map)\n* New data operators like ```fork``` for parrallel processing \n* Support for custom scripts and  operations \n\n\nProject inspired by [Datscript](https://github.com/datproject/datscript/) \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclemsos%2Ftopoparser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fclemsos%2Ftopoparser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclemsos%2Ftopoparser/lists"}