https://github.com/timwis/beacon-data-importer
Convert person data to a CSV format that can easily be imported to Beacon's database
https://github.com/timwis/beacon-data-importer
Last synced: 2 months ago
JSON representation
Convert person data to a CSV format that can easily be imported to Beacon's database
- Host: GitHub
- URL: https://github.com/timwis/beacon-data-importer
- Owner: timwis
- Created: 2020-04-02T16:41:04.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2021-04-20T16:48:21.000Z (over 5 years ago)
- Last Synced: 2025-01-25T07:11:55.546Z (over 1 year ago)
- Language: Python
- Size: 54.7 KB
- Stars: 1
- Watchers: 3
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Beacon data importer
Convert person data to a CSV format that can easily be imported
to Beacon's database.
## Installation
Ensure you have Python 3.7+ installed on your machine.
```
pip3 install "git+https://github.com/timwis/beacon-data-importer#egg=beacon-data-importer"
```
## Usage
```bash
Usage: beacon [OPTIONS] COMMAND [ARGS]...
Options:
--help Show this message and exit.
Commands:
prepare-calls Prepares call log records for import into a temporary...
prepare-contacts Extracts core contact fields from gds_file_path, and...
```
> Tip: To preview the data, pass it to [csvlook][csvlook] and less:
>
> ```
> beacon | csvlook --no-inference | less --chop-long-lines
> # aka:
> beacon | csvlook -I | less -S
> ```
## Authentication
We'll use the heroku CLI to interact with the database. Login to heroku by copying the URL from the following command into your browser.
```bash
heroku auth:login
```
## Contacts
Prepare the data for import using the `beacon` CLI tool.
```bash
beacon prepare-contacts gds.csv healthintent.csv > contacts.csv
```
Load `contacts.csv` into the `contacts` table, which should already be created by the application's migrations.
```bash
heroku pg:psql --app --command "\COPY contacts (nhs_number, first_name, middle_names, surname, address, postcode, telephone, mobile, date_of_birth, created_at, updated_at, gds_import_data) FROM contacts.csv DELIMITER ',' CSV HEADER"
```
## Call logs
Prepare the data for import using the `beacon` CLI tool.
```bash
beacon prepare-calls --food-needs-user USER --complex-needs-user USER --simple-needs-user USER --output-dir ./output calls.csv
```
Create the temporary loading tables.
```bash
heroku pg:psql --app --file sql/create_tmp_tables.sql
```
Load prepared files into the temporary loading tables.
```bash
heroku pg:psql --app --command "\COPY tmp_original_triage_needs (nhs_number, category, name, created_at, updated_at, completed_on) FROM original_triage_needs.csv DELIMITER ',' CSV HEADER"
heroku pg:psql --app --command "\COPY tmp_original_triage_notes (nhs_number, category, body, created_at, updated_at, import_data) FROM original_triage_notes.csv DELIMITER ',' CSV HEADER"
heroku pg:psql --app --command "\COPY tmp_identified_needs (nhs_number, category, name, created_at, updated_at, completed_on, supplemental_data, user_id) FROM food_needs.csv DELIMITER ',' CSV HEADER"
heroku pg:psql --app --command "\COPY tmp_identified_needs (nhs_number, category, name, created_at, updated_at, start_on) FROM callback_needs.csv DELIMITER ',' CSV HEADER"
heroku pg:psql --app --command "\COPY tmp_identified_needs (nhs_number, category, name, created_at, updated_at, user_id) FROM remaining_needs.csv DELIMITER ',' CSV HEADER"
heroku pg:psql --app --command "\COPY tmp_contact_profile_updates (nhs_number, additional_info, delivery_details, dietary_details, has_covid_symptoms) FROM contact_profile_updates.csv DELIMITER ',' CSV HEADER"
```
You can verify it's been loaded in via the psql tool. Use `\q` to quit.
> Note: run export PAGER="less -S" first to support horizontal scrolling.
```bash
heroku pg:psql --app
=> select * from tmp_original_triage_needs;
```
Import the data from the temporary loading tables into the application tables.
```bash
heroku pg:psql --app --file sql/import_original_triage_needs_and_notes.sql
heroku pg:psql --app --file sql/import_identified_needs.sql
heroku pg:psql --app --file sql/import_contact_profile_updates.sql
```
Remove the temporary calls table you created.
```bash
heroku pg:psql --app --command "DROP TABLE tmp_original_triage_needs, tmp_original_triage_notes, tmp_identified_needs, tmp_contact_profile_updates"
```
[csvlook]: https://csvkit.readthedocs.io/en/latest/scripts/csvlook.html