{"id":19174615,"url":"https://github.com/equalitie/open-corroborator","last_synced_at":"2025-05-07T18:20:59.178Z","repository":{"id":16704788,"uuid":"19461544","full_name":"equalitie/open-corroborator","owner":"equalitie","description":"Data analysis and fact corroboration","archived":false,"fork":false,"pushed_at":"2019-01-14T16:32:43.000Z","size":56042,"stargazers_count":7,"open_issues_count":28,"forks_count":3,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-04-20T01:33:01.182Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://equalitie.github.io/open-corroborator/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"yakyak/yakyak","license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/equalitie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-05-05T15:55:53.000Z","updated_at":"2019-11-07T05:34:35.000Z","dependencies_parsed_at":"2022-07-26T08:48:06.941Z","dependency_job_id":null,"html_url":"https://github.com/equalitie/open-corroborator","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/equalitie%2Fopen-corroborator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/equalitie%2Fopen-corroborator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/equalitie%2Fopen-corroborator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/equalitie%2Fopen-corroborator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/equalitie","download_url":"https://codeload.github.com/equalitie/open-corroborator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252931813,"owners_count":21827171,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T10:18:31.682Z","updated_at":"2025-05-07T18:20:59.146Z","avatar_url":"https://github.com/equalitie.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"corroborator\n============\n\nCase management with data validation, corroboration, omission and duplication checking.\nSee `doc/system.rst` for an overview (https://github.com/equalitie/open-corroborator/blob/updatedemo/doc/system.rst)\n\nInstallation\n============\n\nInstall virtualenv:\n\n```\n[sudo] apt-get install python-virtualenv\n```\n\nOther packages are required for modules and other build dependencies:\n\n```\n[sudo] apt-get install libmysqlclient-dev python-dev mysql-client libxslt-dev libgeos-c1 git libjpeg-dev libevent-dev\n```\n\nIf video thumbnailing is turned on (`settings.VIDEO_THUMBNAILING = True`) then `ffmpeg` is required:\n\nsee https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu (or use the Ansible deployment scripts)\n\n\n#### Install local python environment\nFrom the project root folder run:\n\n```\nvirtualenv --python=python2.7 env --no-site-packages\n```\n\n#### Install required packages\n\n```\nenv/bin/easy_install -U distribute\nenv/bin/pip install -r requirements.txt\n```\n\n#### Create and configure the (development) database\nIn mysql, create a database and user for the app:\n\n```\nCREATE DATABASE corroborator_dev CHARACTER SET utf8 COLLATE utf8_general_ci;\nCREATE USER 'django'@'localhost' identified by 'password';\nGRANT ALL ON corroborator_dev.* to 'django'@'localhost';\n```\n\nIf you change the username/password, make sure they are put in the DATABASE setting in ```corroborator/settings/dev.py```.\n\n#### Initial database sync.\nIf you have no database set up, run:\n\n```\nenv/bin/python2.7 manage.py migrate --fake-initial --settings=corroborator.settings.dev\n```\n\nTo install the minimum data fixtures (including an admin user, password\n'password'):\n\n```\nenv/bin/python2.7 manage.py loaddata corroborator_app/fixtures/admin_user.json --settings=corroborator.settings.dev \nenv/bin/python2.7 manage.py loaddata corroborator_app/fixtures/status_update.json --settings=corroborator.settings.dev \nenv/bin/python2.7 manage.py loaddata corroborator_app/fixtures/auth.json --settings=corroborator.settings.dev \n```\n\nAnd to install some sample fixtures (including 'demo' and 'demodata' users, password 'demo'):\n\n```\nenv/bin/python2.7 manage.py loaddata corroborator_app/fixtures/demo_user.json --settings=corroborator.settings.dev \nenv/bin/python2.7 manage.py loaddata corroborator_app/fixtures/crimes.json --settings=corroborator.settings.dev \nenv/bin/python2.7 manage.py loaddata corroborator_app/fixtures/locations_UA.json --settings=corroborator.settings.dev \n```\n\n##### Permissions\nThe `demo` user is a member of the chief-data-analyst, data-analyst and senior-data-analyst groups and should also be given \nthe following permissions (via the admin interface) to be able to edit any item:\n\n   * 'Can assign users via api'\n   * 'Can delete entities via api'\n   * 'Can edit entities via api'\n   * 'Can edit assigned entities via api'\n\nThe `demodata` user is a member of the data-entry group and sees a simplified data-entry interface for bulletins and actors.\n\n### Solr installation\n\nSolr needs a few packages installed before use:\n\n```\n[sudo] apt-get install openjdk-6-jdk\n```\n\nDownload and install Solr 4, e.g.\n\n```\nwget http://www.eu.apache.org/dist/lucene/solr/4.10.4/solr-4.10.4.tgz\ntar -xvf solr-4.10.4.tgz\n```\n\nThen add a new core:\n\n```\ncd solr-4.10.4/\n\nmkdir example/solr/corroborator-search\ncp -R example/solr/collection1/conf/ example/solr/corroborator-search/\necho \"name=corroborator-search\" \u003e example/solr/corroborator-search/core.properties\ncp ../conf/solr.schema.demo.xml example/solr/corroborator-search/conf/schema.xml\n```\n\nAnd start Solr\n\n```\nbin/solr start\n```\n \n#### Other configuration\n```\ncd ..\nenv/bin/python2.7 manage.py backfill_api_keys --settings=corroborator.settings.dev\nenv/bin/python2.7 manage.py rebuild_index --settings=corroborator.settings.dev\n```\n\n#### Media storage\nBy default, uploaded media is stored in MEDIA_ROOT. To store media in S3, \nset `QUEUED_STORAGE = True` and set the 3 `AWS_*` settings - see the deployment \nsteps for more details.\n\n### Running\nTo run the celery worker (to process Solr updates and any S3 updates if used)\n\n```\nenv/bin/python2.7 manage.py celery worker --events --time-limit 300 --concurrency 1 --queues celery --settings=corroborator.settings.dev\n```\n\nTo run the development server:\n\n```\nenv/bin/python2.7 manage.py runserver --settings=corroborator.settings.dev\n```\n\nTest in a browser via http://localhost:8000/ and login with 'demo', password 'demo'.\n\nDeployment\n==========\nSee https://github.com/equalitie/open-corroborator-deploy for Ansible deployment scripts.\n\n\nDevelopment\n===========\n\n### Solr\nChanges to model schema where new fields are added or removed from Bulletin, Incident or Actor\nmodels will require an update to the solr schema. To do this the schema.xml file must be updated.\nHaystack will generate a new schema.xml file with the command\n\n```\nenv/bin/python manage.py build_solr_schema --settings=corroborator.settings.dev \u003e schema.demo.xml\n```\nfrom the root dir of the project\n\nyou must then copy this file to the conf folder of your solr instance and add this line under the fields section\n```\n\u003cfield name=\"_version_\" type=\"long\" indexed=\"true\" stored=\"true\"/\u003e\n```\n\nAnd prefix any `stopwords_en.txt` references with `lang/`, e.g. \"lang/stopwords_en.txt\"\n\nThe Solr index can then be re-generated by running\n```\nenv/bin/python manage.py rebuild_index --settings=corroborator.settings.dev \n```\n\n### Javascript\nFor Javascript template compiling etc. see the github wiki. Some steps to getting things working are:\n\nInstall node and npm packages:\n\n```\n[sudo] apt-get install nodejs npm nodejs-legacy\n```\n\nInstall latest npm, grunt, karma:\n\n```\nnpm install -g grunt-cli karma\n```\n\nInstall the application packages:\n\n```\ncd static/js\nnpm install\n```\n\n(Note: this creates a static/js/node_modules folder with many subdirectories. These can impact initial load times in development\nbecause Django FileSystemFinder and Django debug-toolbar scan the static folder. Switching off the 'Static files' checkbox \nin debug-toolbar should improve things.)\n\nTo build a build.js file (used if PROD_BUILD=True, though currently built for a single language):\n\n```\ncd static/js\ngrunt build\n```\n\nTo compile handlebars templates:\n\n```\ncd static/js\ngrunt handlebars\n```\n\n### CSS\nSee the wiki (bundle, guard, compass (sass) etc), though this seemed to work:\n\n```\nsudo apt-get install bundler\n\nbundle install\n\nbundle exec guard\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fequalitie%2Fopen-corroborator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fequalitie%2Fopen-corroborator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fequalitie%2Fopen-corroborator/lists"}