{"id":16510601,"url":"https://github.com/wragge/recordsearch-functions","last_synced_at":"2025-08-04T03:34:12.827Z","repository":{"id":141841259,"uuid":"50706555","full_name":"wragge/recordsearch-functions","owner":"wragge","description":"Exploring functions in the National Archives of Australia's RecordSearch database","archived":false,"fork":false,"pushed_at":"2016-04-18T05:09:59.000Z","size":67,"stargazers_count":4,"open_issues_count":1,"forks_count":2,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-08T15:21:17.345Z","etag":null,"topics":["archives","code4lib","recordsearch"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wragge.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-01-30T03:21:15.000Z","updated_at":"2017-06-05T23:49:23.000Z","dependencies_parsed_at":null,"dependency_job_id":"36f40483-8f39-4791-a4ea-68ac0b5eb248","html_url":"https://github.com/wragge/recordsearch-functions","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/wragge/recordsearch-functions","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wragge%2Frecordsearch-functions","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wragge%2Frecordsearch-functions/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wragge%2Frecordsearch-functions/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wragge%2Frecordsearch-functions/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wragge","download_url":"https://codeload.github.com/wragge/recordsearch-functions/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wragge%2Frecordsearch-functions/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268644510,"owners_count":24283338,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-04T02:00:09.867Z","response_time":79,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["archives","code4lib","recordsearch"],"created_at":"2024-10-11T15:56:06.445Z","updated_at":"2025-08-04T03:34:12.819Z","avatar_url":"https://github.com/wragge.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# RecordSearch Functions\n\nSome code to harvest and play around with the functions associated with Australian government agencies in the National Archives of Australia's RecordSearch database.\n\nSee here for a [list of the functions](https://github.com/wragge/recordsearch-functions/blob/master/functions.txt) in use.\n\n## Requirements\n\nTo set up your own harvest you'll need my [RecordSearch Tools repository](https://github.com/wragge/recordsearch_tools). Just `git clone` it into your working directory. A list of [other requirements is here](https://github.com/wragge/recordsearch-functions/blob/master/requirements.txt).\n\n## Setting things up\n\nAssuming you have Python, Git, Pip and Virtualenv installed, and you know how to use the command line:\n\n* Create a new working directory with virtualenv -- `virtualenv mynewdirectory`\n* Move to the new directory -- `cd mynewdirectory`\n* Activate your environment -- `source bin/activate`\n* Clone this repo -- `git clone https://github.com/wragge/recordsearch-functions.git`\n* Move to the repo directory -- `cd recordsearch-functions`\n* Clone RecordSearch Tools -- `git clone https://github.com/wragge/recordsearch_tools.git`\n* Install other requirements -- `pip install -r requirements.txt`\n\nThe harvester is expecting to save data to a MongoDB database hosted by [MongoLab](https://mongolab.com/). You can set up a sandbox database for free. Once you've created your database, you'll need to add a user who can access it. Then just:\n\n* Copy the MongoDB URI from your database's control panel to the `credentials_blank.py` file.\n* Replace `\u003cdbuser\u003e` and `\u003cdbpassword\u003e` in the URI with your database user's details. \n* Rename `credentials_blank.py` to `credentials.py`.\n\nOf course you can change this to point to a local MongoDB instance, or anywhere else really.\n\n## Starting a harvest\n\nAssuming you're in the `recordsearch-functions` directory, just:\n\n* Start up Python, or preferably iPython -- `ipython`\n* Import the harvest module -- `import harvest_functions`\n* Set up a harvester instance, supplying the name of the function you want to harvest, eg. MIGRATION -- `harvester = harvest_functions.FunctionHarvester(function='MIGRATION')`\n* If everything's working you'll see the number of agencies that are going to be harvested, and the number of results pages that will be processed.\n* Then just start the harvest -- `harvester.start_harvest()`\n\n## Analysing the harvested data\n\nSave a summary of the agencies associated with a particular function to a CSV file:\n\n* Start up Python, or preferably iPython -- `ipython`\n* Import the analyse module -- `import analyse_functions`\n* Create a CSV file for a single function, eg. MIGRATION -- `analyse_functions.write_csv('MIGRATION')`\n* Create CSV files for all harvested functions -- `analyse_functions.write_csv()`\n\nLook in the [data](https://github.com/wragge/recordsearch-functions/tree/master/data) directory for some example CSV files.\n\nAggregate agencies associated with a particular function by their status, eg. 'Department of State', or 'Head Office':\n\n* Start up Python, or preferably iPython -- `ipython`\n* Import the analyse module -- `import analyse_functions`\n* Aggregate agencies for a single function, eg. 'INTERNAL SECURITY' -- `analyse_functions.calculate_status_totals('INTERNAL SECURITY')`\n\nThe results look something like this:\n\n```python\n[\n    {u'agency_status': u'Regional or State Office', u'total': 22},\n    {u'agency_status': u'Head Office', u'total': 10},\n    {u'agency_status': u'Local Office', u'total': 1},\n    {u'agency_status': u'Judicial Court or Tribunal', u'total': 1},\n    {u'agency_status': u'Department of State', u'total': 1}\n]\n ```\n\nRetrieve a list of agencies by function and agency status:\n\n* Start up Python, or preferably iPython -- `ipython`\n* Import the analyse module -- `import analyse_functions`\n* Retrieve agencies with a particular status, associated with a specific function -- `analyse_functions.get_agencies('MIGRATION', status='Department of State')`\n\nThe result is a list of agencies, something like this:\n\n```python\n[\n    {\n        u'agency_id': u'CA 9152',\n        u'agency_status': u'Department of State',\n        u'end_date': {\n            u'date': datetime.datetime(2013, 9, 18, 0, 0),\n            u'day': True,\n            u'month': True\n            },\n        u'function_end': {\n            u'date': datetime.datetime(2013, 9, 18, 0, 0),\n            u'day': True,\n            u'month': True\n            },\n        u'function_start': {\n            u'date': datetime.datetime(2007, 1, 30, 0, 0),\n            u'day': True,\n            u'month': True},\n        u'location': u'Australian Capital Territory',\n        u'start_date': {\n            u'date': datetime.datetime(2007, 1, 30, 0, 0),\n            u'day': True,\n            u'month': True\n            },\n        u'title': u'Department of Immigration and Citizenship, Central Office'\n    },\n    {\n        u'agency_id': u'CA 9431',\n        u'agency_status': u'Department of State',\n        u'end_date': {\n            u'date': None, \n            u'day': False, \n            u'month': False\n            },\n        u'function_end': {\n            u'date': None, \n            u'day': False, \n            u'month': False\n            },\n        u'function_start': {\n            u'date': datetime.datetime(2013, 9, 18, 0, 0),\n            u'day': True,\n            u'month': True\n            },\n        u'location': u'Australian Capital Territory',\n        u'start_date': {\n            u'date': datetime.datetime(2013, 9, 18, 0, 0),\n            u'day': True,\n            u'month': True\n            },\n        u'title': u'Department of Immigration and Border Protection, Central Office'\n    }\n]\n```\n\nPlot a list of agencies by function and agency status:\n\n* Start up Python, or preferably iPython -- `ipython`\n* Import the analyse module -- `import analyse_functions`\n* Retrieve agencies with a particular status, associated with a specific function -- `analyse_functions.plot_agencies('MIGRATION', status='Department of State')`\n\nNote that this uses [Plot.ly](http://plot.ly) to generate the charts. You'll have to sign up for a free account and install the necessary Python modules.\n\nHere's the result:\n\n[![](https://dl.dropbox.com/s/tatks4mk7qgcvzc/agencies-migration-department_of_state-2016-01-31.png)](https://plot.ly/~wragge/384/commonwealth-agencies-department-of-state-associated-with-the-function-migration/)\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwragge%2Frecordsearch-functions","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwragge%2Frecordsearch-functions","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwragge%2Frecordsearch-functions/lists"}