{"id":20297390,"url":"https://github.com/ekelen/memdbclone-admin","last_synced_at":"2025-03-04T05:44:29.108Z","repository":{"id":112101607,"uuid":"323498913","full_name":"ekelen/memdbclone-admin","owner":"ekelen","description":"Open-source, validated clone of data from Rutger University's Medieval and Early Modern Databank","archived":false,"fork":false,"pushed_at":"2021-01-25T18:24:56.000Z","size":30,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-14T10:14:30.611Z","etag":null,"topics":["medieval-studies","numismatics"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ekelen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-12-22T02:25:15.000Z","updated_at":"2023-11-08T20:21:50.000Z","dependencies_parsed_at":null,"dependency_job_id":"96657529-d5cd-4c47-9d7b-06461422cbfd","html_url":"https://github.com/ekelen/memdbclone-admin","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekelen%2Fmemdbclone-admin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekelen%2Fmemdbclone-admin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekelen%2Fmemdbclone-admin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekelen%2Fmemdbclone-admin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ekelen","download_url":"https://codeload.github.com/ekelen/memdbclone-admin/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241794112,"owners_count":20021189,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["medieval-studies","numismatics"],"created_at":"2024-11-14T15:47:29.579Z","updated_at":"2025-03-04T05:44:29.091Z","avatar_url":"https://github.com/ekelen.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# memdbclone-admin\n\nI find the [Medieval and Early Modern Databank](http://www2.scc.rutgers.edu/memdb/) a useful and reliable tool, but it has some misformatted data and isn't conducive for programmatic use.\n\n**This repo provides only scripts to scrape the site's results pages, clean the data, and upload them to MongoDB**. It pulls data only from [Spufford's Currency Exchanges](http://www2.scc.rutgers.edu/memdb/search_form_spuf.php) and [Rainer Metz' Prices](http://www2.scc.rutgers.edu/memdb/search_form_metzpr.php).\n\nA seperate repository is under construction to support a browser-based production UI, similar to Rutgers' with some UX improvements.\n\n## Short-term TODO:\n\n- Usage guide\n- Link simple search interface in this doc\n- Add About section for each collection\n- (Open to gql queries?)\n- Scrape other databases in the bank\n\n  - [ ] Currency Exchanges (Mueller)\n  - [ ] Currency Exchanges (Metz)\n  - [x] Currency Exchanges (Spufford)\n  - [x] Prices (Posthumus)\n  - [ ] Prices (Metz)\n  - [ ] Prices and Wages (Munro)\n    - [ ] `wages.xls`\n    - [ ] `coinage.xls`\n    - [ ] `brabant_comm.xsl`\n    - [ ] `textile_prod.xls`\n    - [ ] `vanderwee.xls`\n    - [ ] `bruges_wool.xls`\n    - [ ] `ghent_wool.xls`\n    - [ ] `ghent_wool1.xls`\n    - [ ] `flanders_basket.xls`\n    - [ ] `oxford_cloth.xls`\n    - [ ] `eng_comm.xls`\n    - [ ] `EngBasketPrices.xls` - only:\n      - [ ] Annual\n      - [ ] Wage5YrMeans\n    - [ ] `pbh_index.xls`\n    - [ ] `WagesPricesAntwerpEngl.xls`\n    - [ ] `fland_drape.xls`\n    - [ ] `ghent_cloth.xls`\n    - [ ] `ghent_drape.xls`\n    - [ ] `ghent_drape1.xls`\n    - [ ] `leiden_wool.xls`\n    - [ ] `ypres_farm.xls`\n    - [ ] `ypres_stats.xls`\n    - [ ] `ypres_drape.xls`\n    - [ ] `ypres_stall.xls`\n    - [ ] `ypres_stats1.xls`\n    - [ ] `ypres_wool.xls`\n    - [ ] `mech_woool.xls`\n    - [ ] `erasmus_income.xls`\n    - [ ] `kortrijk_linen.xls`\n\n## Schemas\n\n### Prices (Posthumus)\n\n```js\n{\n  $jsonSchema: {\n    properties: {\n      Currency: {\n        bsonType: 'string'\n      },\n      Modified: {\n        bsonType: 'date'\n      },\n      Month: {\n        bsonType: 'int'\n      },\n      Notes: {\n        bsonType: 'int'\n      },\n      Num: {\n        bsonType: 'int'\n      },\n      Price: {\n        bsonType: [\n          'int',\n          'double'\n        ]\n      },\n      ProductDutch: {\n        bsonType: 'string'\n      },\n      ProductEnglish: {\n        bsonType: 'string'\n      },\n      Volume: {\n        bsonType: 'string'\n      },\n      Year: {\n        bsonType: 'int'\n      },\n      _rawEntry: {\n        bsonType: 'string'\n      }\n    }\n  }\n}\n```\n\n### Currency Exchanges (Spufford)\n\n```js\n{\n  $jsonSchema: {\n    required: [\n      '_rawEntry',\n      'Modified',\n      'Num',\n      'Currency (To)',\n      'Currency (From)',\n      'Date (Start)',\n      'Date (End)',\n      'Amount (From)',\n      'Amount (To)',\n      'Type of Exchange',\n      'Invalid Fields'\n    ],\n    properties: {\n      Modified: {\n        bsonType: 'date'\n      },\n      Place: {\n        bsonType: 'string'\n      },\n      'Currency (From)': {\n        bsonType: 'string'\n      },\n      'Currency (To)': {\n        bsonType: 'string'\n      },\n      'Amount (From)': {\n        bsonType: [\n          'double',\n          'int',\n          'null'\n        ]\n      },\n      'Amount (To)': {\n        bsonType: [\n          'double',\n          'int',\n          'null'\n        ]\n      },\n      Num: {\n        bsonType: 'int'\n      },\n      'Date (Start)': {\n        bsonType: [\n          'date',\n          'string',\n          'null'\n        ]\n      },\n      'Date (End)': {\n        bsonType: [\n          'date',\n          'string',\n          'null'\n        ]\n      },\n      'Invalid Fields': {\n        bsonType: 'array',\n        items: {\n          bsonType: 'string',\n          'enum': [\n            'Date (End)',\n            'Date (Start)',\n            'Amount (From)',\n            'Amount (To)'\n          ]\n        }\n      }\n    }\n  }\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fekelen%2Fmemdbclone-admin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fekelen%2Fmemdbclone-admin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fekelen%2Fmemdbclone-admin/lists"}