Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/smklancher/ExistExportToSQL

Import Exist.io data into SQL Server
https://github.com/smklancher/ExistExportToSQL

exist quantified-self sql sql-server

Last synced: about 16 hours ago
JSON representation

Import Exist.io data into SQL Server

Awesome Lists containing this project

README 

        
# ExistExportToSQL



ExistExportToSQL is a command line utility to create a script that lets SQL Server import the contents of a [full JSON archive](https://exist.io/account/export/) from an account on [Exist.io](https://exist.io).



Exist already does a great job of letting you explore relationships between different pices of data that you track, and it has an API that allows you to build whatever you'd like.  But for me, the natural way to play with data is through SQL queries.



``` text

ExistExportToSQL:

  Generates a SQL import script from a folder of a full json export from Exist.io



Usage:

  ExistExportToSQL [options]



Options:

  --input-folder     Folder containing Exist.io json files. Defaults to current directory.

  --output-file       Path to output SQL script. Defaults to inputFolder\ImportExistJson.sql

  --version                        Display version information

```



## Details



The utility does not directly import the data, just generates a script that tells SQL Server how to import the data from the json files.  Note that a static script would not be sufficient because different people have different attributes enabled on their accounts, and use different custom tags.  The data available will depend on which attibutes are enabled and populated with data in your Exist account, and each will result in its own table.  A few additional helper tables created as well.



### Helper Tables and Views



* 'location_geo' enhances original table 'location' by storing coordinates in SQL Server's native GEOGRAPHY data type

* 'sleep_end_ex' enhances original table 'sleep_end' by normalizing the 'minutes from midnight as integer' data to TIME and DATETIME

* 'sleep_start_ex' enhances original table 'sleep_start' by normalizing the 'minutes from midday as integer' data to TIME and DATETIME

* 'AllBoolTraits' is a view that is a union of all tables with boolean data (usually custom tags), allowing for easier queries across a set of different tags

* 'AllIntTraits' is a view that is a union of all tables with int data whether a built-in attribute, or from manual tracking, or custom tag (1 or 0)

* 'LastTag' is a view that shows the date of the last occurrence of each tag

* 'TagUsePast60Days' is a view that view that shows, for any given tag and date, the use of that tag in the past 60 days



## Example Queries



### Mood notes and tags



Show tags and mood score for each day that has a note.



``` SQL

SELECT mn.date, m.value AS Mood, mn.value AS Note, t.Tags

FROM mood_note mn

LEFT JOIN mood m ON m.date=mn.date

LEFT JOIN (

  SELECT act.date, STRING_AGG(act.name, ', ') AS Tags

  FROM AllBoolTraits act

  WHERE act.value>0

  GROUP BY act.date

  ) AS t ON t.date=mn.date

WHERE mn.value IS NOT NULL

ORDER BY mn.date DESC

```



### Last occurrence and recent use of each tag



For each tag, this shows the last date it was used, and the number of times it has been used in the past 60 days.



``` SQL

SELECT lt.name, lt.LastOccurrence, p60.TagCount AS Past60DayTagUse

FROM LastTag AS lt

INNER JOIN TagUsePast60Days p60 ON p60.name=lt.name 

  AND p60.date=(SELECT MAX(date) FROM TagUsePast60Days)

ORDER BY lt.LastOccurrence DESC

```



### Rolling sum of tag occurrences



This is the inertia of a custom tag: for any given day, how many times the tag was used in the preceding 60 days.



``` SQL

-- rolling sum of occurrences of a tag

SELECT date, value, SUM(value) OVER

  (PARTITION BY name ORDER BY date ASC ROWS 59 PRECEDING) --59 + current row

  AS OccurrencesInPrev60Days

FROM tag1 as c

ORDER BY date DESC

```



Using the helper view, this is now the same as:



``` SQL

SELECT name, date, value, TagCount

FROM TagUsePast60Days

WHERE name='tag1'

ORDER BY date DESC

```



### Group of tags over time



If you use custom tags to track positive or negative habits, you might want to look at several tags together and to see how they have gone up or down over time.  This query shows the combined occurrences of several tags, grouped by month.



``` SQL

--sum of tags per month

SELECT YEAR(date) AS Year, MONTH(date) AS Month, SUM(CAST(value AS INT)) AS Total

FROM (

  SELECT date, value FROM tag1

  UNION ALL

  SELECT date, value FROM tag2

  UNION ALL

  SELECT date, value FROM tag3

) as VicesOrVirtues

GROUP BY YEAR(date), MONTH(date)

ORDER BY YEAR(date) DESC, MONTH(date) DESC

```



### Steps per day by number calendar events



Here is a query that shows the average and median number of steps per day grouped by number of calendar events were present in that day.  This would depend on feeding steps and calendar data into Exist.



``` SQL

--avg and median steps by number of events in calendar

SELECT m.EventCount, m.MedianSteps, AVG(m.steps) AS AvgSteps, COUNT(*) AS DaysCount

FROM (

  -- Window functions like PERCENTILE_CONT cannot also be used with GROUP BY, so use a subquery to get median

  SELECT ISNULL(events.value,0) AS EventCount, steps.value as steps,

  PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY steps.value)

  OVER (PARTITION BY ISNULL(events.value,0)) AS MedianSteps

  FROM steps

  INNER JOIN events ON events.date=steps.date

) AS m

GROUP BY m.EventCount, m.MedianSteps

ORDER BY m.EventCount DESC

```



### Distance from home



This query uses the location_geo helper table to show the dates you were recorded as furthest from home.  Mostly in the interest of learning how the GEOGRAPHY data type works.



``` SQL

-- replace with your home coordinates

DECLARE @home geography = 'POINT (-117.587059 33.589965)';



--Dates furthest from home  

SELECT TOP(5) date, geo.ToString() AS point,

  @home.STDistance(geo) * 0.001 as Kilometers,

  @home.STDistance(geo) * 0.000621371 as Miles

FROM [Exist].[dbo].[location_geo]

WHERE geo.STDistance(@home) IS NOT NULL  

ORDER BY geo.STDistance(@home) DESC;  

```



## Requirements



[.NET Runtime](https://dotnet.microsoft.com/download) 7.0+  

[SQL Server 2016](https://www.microsoft.com/en-us/sql-server/sql-server-downloads)+ for the resulting script to use the [OPENJSON](https://docs.microsoft.com/en-us/sql/t-sql/functions/openjson-transact-sql?view=sql-server-ver15) command



## Implementation details



Part of the initial reason I did this project in 2019 was to use a number of things that I wanted to play with:



* .NET Core 3.1 and C# 8.0 with excuses to touch at least some of the new things: [System.Text.Json](https://devblogs.microsoft.com/dotnet/try-the-new-system-text-json-apis/), [Nullable reference types](https://docs.microsoft.com/en-us/dotnet/csharp/whats-new/csharp-8#nullable-reference-types), [Switch Expressions](https://docs.microsoft.com/en-us/dotnet/csharp/whats-new/csharp-8#switch-expressions), [Using declarations](https://docs.microsoft.com/en-us/dotnet/csharp/whats-new/csharp-8#using-declarations).

* [Dragonfruit](https://github.com/dotnet/command-line-api/wiki/Your-first-app-with-System.CommandLine.DragonFruit) provides nice command line functionality with very little work.

* [Single File Publishing](https://docs.microsoft.com/en-us/dotnet/core/whats-new/dotnet-core-3-0#single-file-executables) the official .NET Core way, rather than [ILMerge](https://www.nuget.org/packages/MSBuild.ILMerge.Task/) or [Fody/Costura](https://github.com/Fody/Costura).  Although for size I still prefer to make it framework dependent rather than self contained.

* [Github Actions](https://github.com/actions/upload-release-asset) to automate build and release.

* Presumably cross platform code, which is to say, I only develop and test on Windows but it should work cross platform. 😉