{"id":13619940,"url":"https://github.com/smklancher/ExistExportToSQL","last_synced_at":"2025-04-14T18:32:44.054Z","repository":{"id":73394832,"uuid":"228080479","full_name":"smklancher/ExistExportToSQL","owner":"smklancher","description":"Import Exist.io data into SQL Server","archived":false,"fork":false,"pushed_at":"2023-02-19T21:14:10.000Z","size":78,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-11-08T06:39:30.258Z","etag":null,"topics":["exist","quantified-self","sql","sql-server"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/smklancher.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2019-12-14T19:54:42.000Z","updated_at":"2024-06-08T21:00:19.000Z","dependencies_parsed_at":null,"dependency_job_id":"f4485326-11c6-43d3-a61b-93d604e83236","html_url":"https://github.com/smklancher/ExistExportToSQL","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smklancher%2FExistExportToSQL","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smklancher%2FExistExportToSQL/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smklancher%2FExistExportToSQL/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smklancher%2FExistExportToSQL/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/smklancher","download_url":"https://codeload.github.com/smklancher/ExistExportToSQL/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248936974,"owners_count":21186135,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["exist","quantified-self","sql","sql-server"],"created_at":"2024-08-01T21:00:50.367Z","updated_at":"2025-04-14T18:32:43.721Z","avatar_url":"https://github.com/smklancher.png","language":"C#","readme":"# ExistExportToSQL\n\nExistExportToSQL is a command line utility to create a script that lets SQL Server import the contents of a [full JSON archive](https://exist.io/account/export/) from an account on [Exist.io](https://exist.io).\n\nExist already does a great job of letting you explore relationships between different pices of data that you track, and it has an API that allows you to build whatever you'd like.  But for me, the natural way to play with data is through SQL queries.\n\n``` text\nExistExportToSQL:\n  Generates a SQL import script from a folder of a full json export from Exist.io\n\nUsage:\n  ExistExportToSQL [options]\n\nOptions:\n  --input-folder \u003cinput-folder\u003e    Folder containing Exist.io json files. Defaults to current directory.\n  --output-file \u003coutput-file\u003e      Path to output SQL script. Defaults to inputFolder\\ImportExistJson.sql\n  --version                        Display version information\n```\n\n## Details\n\nThe utility does not directly import the data, just generates a script that tells SQL Server how to import the data from the json files.  Note that a static script would not be sufficient because different people have different attributes enabled on their accounts, and use different custom tags.  The data available will depend on which attibutes are enabled and populated with data in your Exist account, and each will result in its own table.  A few additional helper tables created as well.\n\n### Helper Tables and Views\n\n* 'location_geo' enhances original table 'location' by storing coordinates in SQL Server's native GEOGRAPHY data type\n* 'sleep_end_ex' enhances original table 'sleep_end' by normalizing the 'minutes from midnight as integer' data to TIME and DATETIME\n* 'sleep_start_ex' enhances original table 'sleep_start' by normalizing the 'minutes from midday as integer' data to TIME and DATETIME\n* 'AllBoolTraits' is a view that is a union of all tables with boolean data (usually custom tags), allowing for easier queries across a set of different tags\n* 'AllIntTraits' is a view that is a union of all tables with int data whether a built-in attribute, or from manual tracking, or custom tag (1 or 0)\n* 'LastTag' is a view that shows the date of the last occurrence of each tag\n* 'TagUsePast60Days' is a view that view that shows, for any given tag and date, the use of that tag in the past 60 days\n\n## Example Queries\n\n### Mood notes and tags\n\nShow tags and mood score for each day that has a note.\n\n``` SQL\nSELECT mn.date, m.value AS Mood, mn.value AS Note, t.Tags\nFROM mood_note mn\nLEFT JOIN mood m ON m.date=mn.date\nLEFT JOIN (\n  SELECT act.date, STRING_AGG(act.name, ', ') AS Tags\n  FROM AllBoolTraits act\n  WHERE act.value\u003e0\n  GROUP BY act.date\n  ) AS t ON t.date=mn.date\nWHERE mn.value IS NOT NULL\nORDER BY mn.date DESC\n```\n\n### Last occurrence and recent use of each tag\n\nFor each tag, this shows the last date it was used, and the number of times it has been used in the past 60 days.\n\n``` SQL\nSELECT lt.name, lt.LastOccurrence, p60.TagCount AS Past60DayTagUse\nFROM LastTag AS lt\nINNER JOIN TagUsePast60Days p60 ON p60.name=lt.name \n  AND p60.date=(SELECT MAX(date) FROM TagUsePast60Days)\nORDER BY lt.LastOccurrence DESC\n```\n\n### Rolling sum of tag occurrences\n\nThis is the inertia of a custom tag: for any given day, how many times the tag was used in the preceding 60 days.\n\n``` SQL\n-- rolling sum of occurrences of a tag\nSELECT date, value, SUM(value) OVER\n  (PARTITION BY name ORDER BY date ASC ROWS 59 PRECEDING) --59 + current row\n  AS OccurrencesInPrev60Days\nFROM tag1 as c\nORDER BY date DESC\n```\n\nUsing the helper view, this is now the same as:\n\n``` SQL\nSELECT name, date, value, TagCount\nFROM TagUsePast60Days\nWHERE name='tag1'\nORDER BY date DESC\n```\n\n### Group of tags over time\n\nIf you use custom tags to track positive or negative habits, you might want to look at several tags together and to see how they have gone up or down over time.  This query shows the combined occurrences of several tags, grouped by month.\n\n``` SQL\n--sum of tags per month\nSELECT YEAR(date) AS Year, MONTH(date) AS Month, SUM(CAST(value AS INT)) AS Total\nFROM (\n  SELECT date, value FROM tag1\n  UNION ALL\n  SELECT date, value FROM tag2\n  UNION ALL\n  SELECT date, value FROM tag3\n) as VicesOrVirtues\nGROUP BY YEAR(date), MONTH(date)\nORDER BY YEAR(date) DESC, MONTH(date) DESC\n```\n\n### Steps per day by number calendar events\n\nHere is a query that shows the average and median number of steps per day grouped by number of calendar events were present in that day.  This would depend on feeding steps and calendar data into Exist.\n\n``` SQL\n--avg and median steps by number of events in calendar\nSELECT m.EventCount, m.MedianSteps, AVG(m.steps) AS AvgSteps, COUNT(*) AS DaysCount\nFROM (\n  -- Window functions like PERCENTILE_CONT cannot also be used with GROUP BY, so use a subquery to get median\n  SELECT ISNULL(events.value,0) AS EventCount, steps.value as steps,\n  PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY steps.value)\n  OVER (PARTITION BY ISNULL(events.value,0)) AS MedianSteps\n  FROM steps\n  INNER JOIN events ON events.date=steps.date\n) AS m\nGROUP BY m.EventCount, m.MedianSteps\nORDER BY m.EventCount DESC\n```\n\n### Distance from home\n\nThis query uses the location_geo helper table to show the dates you were recorded as furthest from home.  Mostly in the interest of learning how the GEOGRAPHY data type works.\n\n``` SQL\n-- replace with your home coordinates\nDECLARE @home geography = 'POINT (-117.587059 33.589965)';\n\n--Dates furthest from home  \nSELECT TOP(5) date, geo.ToString() AS point,\n  @home.STDistance(geo) * 0.001 as Kilometers,\n  @home.STDistance(geo) * 0.000621371 as Miles\nFROM [Exist].[dbo].[location_geo]\nWHERE geo.STDistance(@home) IS NOT NULL  \nORDER BY geo.STDistance(@home) DESC;  \n```\n\n## Requirements\n\n[.NET Runtime](https://dotnet.microsoft.com/download) 7.0+  \n[SQL Server 2016](https://www.microsoft.com/en-us/sql-server/sql-server-downloads)+ for the resulting script to use the [OPENJSON](https://docs.microsoft.com/en-us/sql/t-sql/functions/openjson-transact-sql?view=sql-server-ver15) command\n\n## Implementation details\n\nPart of the initial reason I did this project in 2019 was to use a number of things that I wanted to play with:\n\n* .NET Core 3.1 and C# 8.0 with excuses to touch at least some of the new things: [System.Text.Json](https://devblogs.microsoft.com/dotnet/try-the-new-system-text-json-apis/), [Nullable reference types](https://docs.microsoft.com/en-us/dotnet/csharp/whats-new/csharp-8#nullable-reference-types), [Switch Expressions](https://docs.microsoft.com/en-us/dotnet/csharp/whats-new/csharp-8#switch-expressions), [Using declarations](https://docs.microsoft.com/en-us/dotnet/csharp/whats-new/csharp-8#using-declarations).\n* [Dragonfruit](https://github.com/dotnet/command-line-api/wiki/Your-first-app-with-System.CommandLine.DragonFruit) provides nice command line functionality with very little work.\n* [Single File Publishing](https://docs.microsoft.com/en-us/dotnet/core/whats-new/dotnet-core-3-0#single-file-executables) the official .NET Core way, rather than [ILMerge](https://www.nuget.org/packages/MSBuild.ILMerge.Task/) or [Fody/Costura](https://github.com/Fody/Costura).  Although for size I still prefer to make it framework dependent rather than self contained.\n* [Github Actions](https://github.com/actions/upload-release-asset) to automate build and release.\n* Presumably cross platform code, which is to say, I only develop and test on Windows but it should work cross platform. 😉\n","funding_links":[],"categories":["C# #"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmklancher%2FExistExportToSQL","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsmklancher%2FExistExportToSQL","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmklancher%2FExistExportToSQL/lists"}