{"id":20672317,"url":"https://github.com/pasiol/koski-reader","last_synced_at":"2025-09-27T02:30:39.324Z","repository":{"id":119438588,"uuid":"247711157","full_name":"pasiol/koski-reader","owner":"pasiol","description":null,"archived":false,"fork":false,"pushed_at":"2020-12-17T11:23:26.000Z","size":51,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-11-16T20:34:54.017Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pasiol.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-03-16T13:36:11.000Z","updated_at":"2022-03-02T17:15:31.000Z","dependencies_parsed_at":null,"dependency_job_id":"94211f1e-e96c-4706-8291-10564878936b","html_url":"https://github.com/pasiol/koski-reader","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pasiol%2Fkoski-reader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pasiol%2Fkoski-reader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pasiol%2Fkoski-reader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pasiol%2Fkoski-reader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pasiol","download_url":"https://codeload.github.com/pasiol/koski-reader/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234371473,"owners_count":18821592,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T20:33:35.330Z","updated_at":"2025-09-27T02:30:33.987Z","avatar_url":"https://github.com/pasiol.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# koski-reader\n\nThe REST client reads student's data from the Koski database and stores data to the MongoDB-database. File aggeregate_vocational.py extracting vocational studies to own collection. Uses docker-compose to run database and scripts.\n\n## Prerequisites\n\n* Docker\n* docker-compose\n\n[https://docs.docker.com/engine/install/]\n\nIn Mac and Windows use the Docker Desktop. Or you can use WSL on the Windows. \n\n### Ubuntu\n\nI prefer Canonical maintained packages which are robust.\n\n    sudo apt update \u0026\u0026 sudo apt install -y docker.io docker-compose\n    sudo usermod -aG docker $USER\n\n## Installation\n\n    git clone https://github.com/pasiol/koski-reader.git\n    cd koski-reader\n    docker-compose build\n\n## Usage\n\nRunning the MongoDB database and docker volumes. Keep containers and volumes in the safe environment and do not expose them outside.\n\n    docker-compose up\n\n### Useful commands\n\n    docker ps\n    docker volume ls\n    docker exec -it koskireader_database_1  /bin/bash\n\n### Running commands on the reader container\n\n    docker-compose run reader python3 reader.py --help\n    Starting koskireader_database_1 ... done\n    Usage: reader.py [OPTIONS] USERNAME PASSWORD INPUT_FILE FTYPE\n\n    Options:\n    --help  Show this message and exit.\n\n#### Running the Rest-client\n\nYou need input file which contains studyrights on the csv-file. It could be generate with the primusquery utility. Put csv input file on the root directory before docker-compose build command. Reader can update old data, so there is no need get always all studyrights only the changed ones. If you remove mongodb docker volume, you also loose all the data. You can import data to host environment with mongoexport and docker cp commands.\n\n    docker-compose run reader python3 reader.py user password studyrights.csv studyrights\n\nIf studyright exists on the database, script replacing it with updated data.\n\n    2020-12-17 07:23:11,669 - __main__ - INFO - Processing id: N.N.NNN.NNN.NN.NNNNNNNNN\n    2020-12-17 07:23:13,095 - __main__ - INFO - Trying to find oid: N.N.NNN.NNN.NN.NNNNNNNNN, result length: 1\n    2020-12-17 07:23:13,096 - __main__ - INFO - Removing oid: N.N.NNN.NNN.NN.NNNNNNNNN, count: 1\n    2020-12-17 07:23:13,100 - __main__ - INFO - Inserting to Collection(Database(MongoClient(host=['database:27017'], document_class=dict, tz_aware=False, connect=True), 'koski'), 'studyrights') oid: N.N.NNN.NNN.NN.NNNNNNNNN: \u003cpymongo.results.InsertOneResult object at 0x7f950b250d08\u003e\n\n#### Attaching to database container\n\n[https://docs.mongodb.com/manual/]\n\n    docker exec -it koskireader_database_1 /bin/bash\n    root@e9cb474b57d5:/# mongo\n    \u003e db.disableFreeMonitoring()\n    \u003eshow dbs\n    admin   0.000GB\n    koski   0.NNNGB\n    local   0.000GB\n    \u003e use koski\n    switched to db koski\n    \u003edb.getCollectionNames()\n    [ \"studyrights\" ]\n    \u003e db.studyrights.find().count()\n    NNNNNN\n    \u003e db.studyrights.find().pretty()\n    ...\n    \u003eexit\n\n#### Aggerating the vocational achievements to own collection\n\n    docker-compose run reader python3 aggregate_vocational.py\n    ...\n    2020-12-17 09:14:27,289 - __main__ - INFO - Executing aggregate pipeline succeed: Collection(Database(MongoClient(host=['database:27017'], document_class=dict, tz_aware=False, connect=True), 'koski'), 'studyrights')\n\n    \u003e show dbs\n    admin    0.000GB\n    koski    0.NNNGB\n    local    0.000GB\n    reports  0.NNNGB\n    \u003e use reports \n    switched to db reports\n    \u003e db.getCollectionNames()\n    [ \"koski_vocational_accomplishments\" ]\n    \u003edb.koski_vocational_accomplishments.find().pretty()\n    ...\n    {\n        \"_id\" : ObjectId(\"5fdb21660a5337559b369c1b\"),\n        \"arvosana\" : \"5\",\n        \"hyväksytty\" : true,\n        \"koodi\" : \"105993\",\n        \"laajuus\" : 20,\n        \"nimi\" : \"Metsätraktorin käyttö\",\n        \"oid\" : \"N.N.NNN.NNN.NN.NNNNNNNNNN\",\n        \"tutkinnonosanryhmä\" : \"Ammatilliset tutkinnon osat\",\n        \"yksikkö\" : \"osaamispistettä\"\n    }\n\n#### Copying data from the database container\n\nIf you want compare the koski data to local data, you can move it for example to the data warehouse. Here is how it can do manually. If you want do it more automatically, write your own container which reads the mongo data and transfer it to another database.\n\n[https://docs.mongodb.com/v4.2/reference/program/mongoexport/]\n\nAttach to database container.\n\n    docker exec -it koskireader_database_1 /bin/bash\n\nJSON\n\n    root@e9cb474b57d5:/# mongoexport -d reports -c koski_vocational_accomplishments --jsonArray -o ammatilliset.json\n    2020-12-17T09:49:18.557+0000    connected to: mongodb://localhost/\n    2020-12-17T09:49:19.558+0000    [####....................]  reports.koski_vocational_accomplishments  NNNNN/NNNNN  (18.2%)\n    2020-12-17T09:49:20.558+0000    [###############.........]  reports.koski_vocational_accomplishments  NNNNN/NNNNN  (63.8%)\n    2020-12-17T09:49:21.282+0000    [########################]  reports.koski_vocational_accomplishments  NNNNN/NNNNN  (100.0%)\n    2020-12-17T09:49:21.282+0000    exported NNNNN record\n\nCSV \n\n    root@e9cb474b57d5:/# mongoexport -d reports -c koski_vocational_accomplishments --type csv -o ammatilliset.csv --fields arvosana,koodi,laajuus \n    2020-12-17T09:54:08.715+0000    connected to: mongodb://localhost/\n    2020-12-17T09:54:09.284+0000    exported NNNNN records\n\nCopying file from container to the host system.\n\n    docker cp koskireader_database_1:ammatilliset.json .\n    docker cp koskireader_database_1:ammatilliset.csv .\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpasiol%2Fkoski-reader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpasiol%2Fkoski-reader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpasiol%2Fkoski-reader/lists"}