https://github.com/titaniumbones/visualization-projects
A novice experiments with data visualization
https://github.com/titaniumbones/visualization-projects
Last synced: about 1 month ago
JSON representation
A novice experiments with data visualization
- Host: GitHub
- URL: https://github.com/titaniumbones/visualization-projects
- Owner: titaniumbones
- Created: 2019-11-29T02:37:14.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2019-11-30T04:40:43.000Z (over 6 years ago)
- Last Synced: 2026-03-08T03:56:32.046Z (3 months ago)
- Language: JavaScript
- Size: 54.7 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 4
-
Metadata Files:
- Readme: README.org
Awesome Lists containing this project
README
* Visualization Test Projects
A collection of small scripts, web pages, and other files related to experiments visualizing data, mostly meteorological data relating to surfing and paddling:
- wave and wind forecasts from the GLOS surface model point forecast (for surfing). This is a single endpoint which actually returns data from two distinct models: the "input" atmospheric forcing model (maybe thi comes from US weather service?), and the "output" surface model, which includes wave height, direction, period, etc., as well as current speed & direction
- historical (i.e., up-until-the-present-moment) river gauge data, so far only from the Canadian wateroffice, mostly useful for whitewater kayaking.
So far, I'm mostly interested in single-variable time series graphs of wind/wave/flow amplitude, with wind/wave direction added in a tooltip. Trying to use color as a rough predictor of surf quality, but it's a bit lame so far.
Using all these different libraries to fool around. The rest of this file consists mostly of annotated source code for immature projects; as they get more mature, I'll fix them up.
* Installation
This is just a sandbox for now. If for some reason you want to try it out, clone this repo and then run ~npm install~ and ~bower install~ (I know, I know, it's ridiculous). Then play with the examples.
* Contributing
umm... source code is all ocntained in org-mode source blocks for now. Would be hard for most people to deal with that setup! May switch later once I start to understand these frameworks better/settle on one in particular.
* List of generated files (incomplete)
Sorry, this is maintained by hand (!)
#+begin_src web :tangle index.html
Document
#+end_src
* Experiments with curl and vis libraries
** Restclient: get some data going
#+NAME: wateroffice
#+begin_src restclient :results file :exports results :file-ext json
:params = station=02HB029&start_date=2019-11-14&end_date=2019-11-22¶m1=46¶m2=47
#
GET https://wateroffice.ec.gc.ca/services/real_time_graph/json/inline?station=02HB029&start_date=2019-11-14&end_date=2019-11-22¶m1=46¶m2=47
#+end_src
#+RESULTS: wateroffice
[[file:wateroffice.json]]
** [[https://www.chartjs.org/][Chart.js]]
*** concerns
Was somewhat daunting when I started but no longer looks all that hard:
- in dataset, can use ~{x: date, y: value}~ array elements instead of scalars (the default in most examples)
- *thresholds*
- [[https://stackoverflow.com/questions/42691873/draw-horizontal-line-on-chart-in-chart-js-on-v2][explanation of using chart.js annotation library]] to simulate the threhold/line funcitonality already working in chartist code below
- [[https://stackoverflow.com/questions/36916867/chart-js-line-different-fill-color-for-negative-point][much cooler & more powerful explanation of how you would modify the line chart type]] to change the fill gradients, by passing a threshold value. The example uses just ~0~ but looksl ike it could really be anything. [[http://jsfiddle.net/g2r2q5Lu/][cf. jsFiddle]].
- another alternative is the cool-looking [[https://github.com/bbc/chart.bands.js][chart pands plugin]](not under active development), which can adjust line type etc according to y-value. [[https://codepen.io/Tarqwyn/pen/QNzNVg][cf. codepen]].
- also look at new [[https://github.com/chartjs/Chart.js/blob/master/samples/advanced/line-gradient.html][line-gradient plugin]].
- [[https://stackoverflow.com/questions/35249498/how-to-change-line-segment-color-of-a-line-graph-in-chart-js][low-level code to redraw line segments]]; may not be trivial w/ dynamically-generated data sets. Would have to do the math first. [[https://jsfiddle.net/egamegadrive16/zjdwr4fh/][jsFiddle]].
- [[https://stackoverflow.com/questions/52120036/chartjs-line-color-between-two-points][more manual intervention from stackoverflow]] -- still looks messy!
- *tooltips*
- [[https://www.chartjs.org/samples/latest/scales/time/financial.html][this example has pretty good UI for tooltip]] -- uses ~intersect: false, mode:index~ and a callback to make the tooltips.
- [[https://www.chartjs.org/docs/latest/configuration/tooltip.html][tooltip docs are pretty extensive and interesting]].
- [[https://stackoverflow.com/questions/45159895/moving-vertical-line-when-hovering-over-the-chart-using-chart-js][cool example of how to extend line chart type to include a line that appearso n the graph on hover]] -- helps to show you where you are. Note the comments about the issue with which tooltip gets activated -- need to set options.tooltips.axis to ~x~ on initializaiton of the chart. [[https://stackoverflow.com/questions/54990176/chart-js-vertical-line-when-hovering-and-shadow-on-line/55045517#55045517][another very similar question and cool implementation]].
- *time axes*
- [[https://www.chartjs.org/samples/latest/scales/time/combo.html][timexcales using moment]]
*** code
We'll use the file [[./chart-js.html]] for code
#+begin_src css :tangle chart-js.css
main: {display:flex; flex-direction:row; height: 75vh; width:100vh;justify-content:space-around;}
section.chart-container { flex-basis:1fr;}
canvas#myChart {max-width:100vw; max-height:80vh}
#+end_src
#+begin_src web :tangle chart-js.html
Chart.js Data visualization
#+end_src
#+begin_src js :tangle chartjs-lines.js
const ctx = document.getElementById('myChart').getContext('2d');
const timeFormat = 'MM/DD - HH:mm';
let myChart;
async function loadMe () {
let dataArray = await getJSON('data/pqt-out.csv', 2)
.then( (json) => {
// console.log(json);
// console.log(processNOAAData(json, "wsp"));
return processNOAAData(json);
});
//console.log(j);
const thisSpot = abay;
myChart = new Chart(ctx, {
type: 'line',
data: {
datasets: [{
label: 'Wave Height in Meters',
data: dataArray,
backgroundColor: 'red',
borderColor: 'red',
fill: false, //true,
borderWidth: 1
}]
},
options: {
scales: {
xAxes: [{
type: 'time',
display: true,
time: {
format: timeFormat,
//round: 'hour'
}
}],
},
tooltips: {
backgroundColor: `rgba(0,0,0,0.5)`,
intersect: false,
mode: 'index',
callbacks: {
//title: (item, data) => "I am a title",
label: function(tooltipItem, myData) {
console.log("creating label");
console.log(tooltipItem);
let d = myData.datasets,
i = tooltipItem.datasetIndex,
dir = d[i].data[tooltipItem.index].direction ? `direction: ↓` : '';
arrow = ``;
console.log(d[i].data[tooltipItem.index]);
var label = myData.datasets[tooltipItem.datasetIndex].label || '';
if (label) {
label += ': ';
}
label += parseFloat(tooltipItem.value).toFixed(2);
label += dir;
console.log(label);
return label;
}
}
}
}
});
}
loadMe();
#+end_src
see [[https://www.chartjs.org/docs/latest/getting-started/usage.html][getting started guide]] and [[https://www.chartjs.org/docs/latest/getting-started/usage.html][samples page]] for example code.
** [[https://gionkunz.github.io/chartist-js/examples.html][Chartist]] 1: using river gauge data
Some basic thoughts:
- graphing wateroffice data turns out to be easy once it's available: convert unix date-time to javascript Date object & graph. Improvements might include:
- use either `target-line` or maybe better, ~threshold~ plugin to identify periods of navigability. cf. descriptions and examples on [[https://gionkunz.github.io/chartist-js/plugins.html][plugins page]].
- tooltips plugin is a little less awesome than other frameworks:
- no touch support!!
- tooltips only show up on mouseover of *points*, not of areas. This will be a real pain in these graphs.
#+begin_src css :tangle chartist-line.css
/* Use this selector to override the line style on a given series */
.ct-series-a .ct-line {
/* Set the colour of this series line */
stroke:blue;
/* Control the thikness of your lines */
stroke-width: 1px;
/* Create a dashed line with a pattern */
// stroke-dasharray: 4px 8px;
}
/* This selector overrides the points style on line charts. Points on line charts are actually just very short strokes. This allows you to customize even the point size in CSS */
.ct-series-a .ct-point {
/* Colour of your points */
stroke:red;
/* Size of your points */
stroke-width: 3px;
/* Make your points appear as squares */
/* stroke-linecap: square; */
}
.ct-target-line {
stroke: var(--no-surf);
stroke-width: 2px;
stroke-dasharray: 4px;
shape-rendering: crispEdges;
}
#+end_src
#+begin_src html :tangle chartist-line.html
My first Chartist Tests
main#main {
display: flex;
max-height: 400px;
grid-column-template: 1fr 1fr 1fr;
justify-content: space-around
}
#+end_src
#+begin_src js :tangle chartist-line.js
let woURL = `https://wateroffice.ec.gc.ca/services/real_time_graph/json/inline?`,
params = `station=02HB029&start_date=2019-11-14&end_date=2019-11-22¶m1=46¶m2=47`,
chart;
function gWO (rawData) {
let processed = rawData["46"].provisional.map( (item) => { return {x: new Date(item[0]), y: item[1]};});
console.log(processed);
return processed
}
let proxy = 'https://cors-anywhere.herokuapp.com/';
async function getJSON (url, params ) {
let target = `${url}${params}`;
return await fetch(target)
.then(function(response){console.log(response.json);return response.json();})
.then(function(json) {
// console.log(json.parse);
return json;
})
.catch(function(error){console.log(error);});
}
async function buildChart () {
const woRaw = await getJSON('wateroffice.json', '')
.then((data) => data["47"].provisional.map((item) => {
const itemObj = { x: new Date(item[0]), y: item[1] }; return itemObj
}) )
let wo = woRaw
//wo = woRaw["46"].provisional.map( (item) => { return {x: new Date(item[0]), y: item[1]};});
console.log(wo);
chart = new Chartist.Line('.ct-chart', {
series: [
{name: 'actual-data',
data: wo
}
]
}, {
showArea: true,
axisX: {
type: Chartist.FixedScaleAxis,
divisor: 5,
labelInterpolationFnc: function(value) {
return moment(value).format('dd');
}
},
targetLine: {
value: 5,
class: 'ct-target-line'
}
});
chart.on('created', function (context) {
console.log(context);
let targetLineY = projectY(context.chartRect, context.bounds, context.options.targetLine.value);
context.svg.elem('line', {
x1: context.chartRect.x1,
x2: context.chartRect.x2,
y1: targetLineY,
y2: targetLineY
}, context.options.targetLine.class);
});
}
function projectY(chartRect, bounds, value) {
return chartRect.y1 - (chartRect.height() / bounds.max * value)
}
buildChart();
#+end_src
** Plot.ly
Lots of cool features but not clear why I would need them:. [[https://plot.ly/javascript/gapminder-example/][one such example]]
** Flot
Jquery=based library. see [[https://www.flotcharts.org/flot/examples/][docs]]. Used by wateroffice for their data.
** MatricsGraphics
[[https://metricsgraphicsjs.org/][Looks somewhati nteresting. Originally a Mozilla project for website analytics]].
** Cubism
[[https://square.github.io/cubism/][also optimized for pullinganalytics from survers]], but with defaults close to what I'm looking for.
** [[https://c3js.org/][C3.js]]
meta library of D3, much easier to work with, in principle.
- [[https://c3js.org/gettingstarted.html][getting started guide]]
- [[https://c3js.org/examples.html][examples]]
- [[https://c3js.org/samples/simple_regions.html][line segments, kind of nice]]
-
** [[https://canvasjs.com/javascript-charts/json-data-api-ajax-chart/][canvasjs]] -- might be nonfree
** Windy
- Windy has a fairly rich API
- it's pretty difficult to ineract with without building a map (!)
- [[https://www.google.com/search?client=firefox-b-d&q=windy+api+get+forecast+value+at+point][google search]] (not that helpful)
- [[https://github.com/windycom/API][API examples homeplge]]
- [[https://api4.windy.com/examples/picker][activating the picker]] -- could be possible to add content to picker? don't know.
- [[https://github.com/windycom/windy-plugins/blob/master/docs/WINDY_API.md#module-plugindataloader][description of ~pluginDataLoader~ module form windy plugin API]] -- this is where you could get point-specific data
- [[https://api4.windy.com/api-key][link to API key]]
- note: the plugins can be pretty cool, need to figure out how to add them to default display!
- [[file:///home/matt/src/visualization-projects/windy-api-examples/picker/index.html][my code to try to get access to the backend API directly without instantiating the map (!)]]
* Abstract some libraries for processing NOAA/GLOS data
Originally written into Chartist 2, now living in its own tiny library
#+begin_src js :tangle js/noaaParser.js
let abay = {
"name": "Ashbridge's Bay",
"lat":123.456,
"long": -456.789,
"directions": [[0,10,"bad"], [10,30,"shoulder"], [30,150,"good"],[150,180,"shoulder"],[180,360,"bad"]],
"qualityPeaks": [ [0,"bad"], [85, "good"], [170, "bad"], [290, "bad"] ],
"minHeight": 0.85
}
//var csv is the CSV file with headers
function noaaCsvToJSON(csv, h=2){
var lines=csv.split("\n");
var result = [];
var headers=lines[h].split(",");
headers = headers.map(s => s.trim());
//console.log(headers);
for(var i= h+1 ; i x.trim());
//console.log(i + ": " + currentline);
//console.log (currentline.length)
for(var j=0;j {return response.text()} )
.then( (text) => {return noaaCsvToJSON(text, headline)} )
.catch(function(error){console.log(error);});
}
function testGood (direction, spotMeta=abay) {
let value = 'bad';
spotMeta.directions
.some( function (d) {
if ( (d[0] < direction) && ( direction < d[1]) ) {
// console.log(d);
value = d[2]; return; }
});
//console.log(value)
return value
}
function processNOAAData (raw,spotMeta=abay, yaxis=true) {
return raw.map((item) => {
item.wvd = Number(item.wvd);
item.direction = item.wvd ? (Number(item.wvd) + 180) % 360 : Number(item.wdir);
//console.log(item.wvd);
//console.log ( item.wvd ? (item.wvd + 180) : item.wdir )
//console.log( (item.wvd ? "WVD: " : "WDIR: " ) + ( item.wvd ? (item.wvd + 180) % 360 : item.wdir) );
item.quality = testGood(item.direction);
item.direction = Math.trunc(item.direction);
const itemObj = { x: new Date(item["Date String"]),
y: item.wvh || (item.wsp * 3.6 ),
// wvd: item.wvd,
// wdir: item.wdir,
// direction: (item.wvd ? ((item.wvd + 180) % 360) : item.wdir),
//direction: Math.trunc( (item.wvd ? (item.wvd + 180) % 360 : item.wdir) ),
//meta: `↑`
meta: item
};
return itemObj
})
}
#+end_src
* Design Considerations
** Tooltips
tooltips are hard to manage in more abstracted platforms like chrtist. cf. [[https://github.com/tmmdata/chartist-plugin-tooltip/issues/157][make activation area wider]] for chartist; somewhat less clumsy [[https://c3js.org/samples/tooltip_format.html][in C3, but still a bit jumpy]]; [[https://bl.ocks.org/Qizly/5a78caaf03ed96757e72][way nicer in D3 directly]], but code is dramatically more complex. [[http://bl.ocks.org/wdickerson/64535aff478e8a9fd9d9facccfef8929][another d3 example]] showing more complex HTML in tooltip. [[https://www.chartjs.org/docs/latest/configuration/tooltip.html#external-custom-tooltips][Chart.js tooltips]] are extremely flexible by comparison.
** Arrows
We can use wind direction to set the angle and color of a directional arrow in a tooltip.
- add ~↑~ to the tooltip
- add ~transform:rotate(var(--angle)deg)~ to the css for that span
- for colors, need to set a class of "good" bad" "shoulder" and set those colors in CSS
- different frameworks have very different tooltip presents, will be complex to move from one library to another
#+begin_src css :tangle arrows.css
:root {
--angle: 45deg;
}
body {
display: grid;
justify-items: space-around;
align-items: center;
margin-top: 200px;
align-content: center;
justify-content: space-around;
}
div {
color: green;
font-size: 60px;
/* standard gradient background */
background: linear-gradient(red, blue);
/* clip hackery */
-webkit-background-clip: text;
-webkit-text-fill-color: transparent;
}
div.arrow {
transform: rotate(var(--angle))
}
#+end_src
#+begin_src html :tangle arrows.html
Document
↑
#+end_src
* Designing A Wave/Wind Graph Set
Ideally, for each spot we have a json object a little like this:
#+begin_src json
{
"name": "Ashbridge's Bay",
"lat":123.456,
"long": -456.789,
"directions": [[0,50,"bad"], [50,70,"shoulder"], [70,120,"good"],[120,150,"shoulder"],[150,360,"bad"]],
"minHeight": 0.85
}
#+end_src
Where ~minHeight~ is in meters and directions are wind or wave directions in degrees from true North (lat/long are decimal degrees).
We then use ~minHeight~ to set the threshold for fill colors, and ~directions~ to color the tooltip elements & and of possible adjust fill colors above the height threshold.
** Graph Layout
Two graphs stacked on top of each other, wave height and wind speed (both scalars). Ideally they still share a single axis.
- fill color above ~minHeight~ is different from below (which is probably unfilled)
- ideally they share a single time axis
** time Axis generation
a lot easier to use moment than native data objects! Hoping to do that.
** Tooltips
Tooltips display *a directional arrow* (see "arrows" above for some HTML & CSS) along with the date-time value, and absolute magntude of the two points. *color should be used to emphasize "good", "bad" or "shoulder" status of the directional data.
** UI
It should be possible to:
- zoom the map to adjust timescale
- use a slider to navigate the map & activate tooltips (better than just a mouseover for lots of reasons, e.g. could be used to control a windy map in tandem)
- probably a few other things would be nice
** Data Sources
There are at least 2 possible data sources
- NOAA/GLOS surface model point data endpoint, currently down, but providing high-quality data sources. Available only in CSV so needs to be parsed before it's fed to a map.
- Windy data -- only possible from within a Windy plugin (!!) but provides an alternative to the GLOS model & has a longer-range forecast. cf [[https://www.chartjs.org/docs/latest/configuration/tooltip.html#external-custom-tooltips][my forum post]] and
** Processing csv to json
- [[https://www.papaparse.com/docs#config][Papaparse]] is a pretty full-featured CSV translator that maybe should prelace my siplistic code (though maybe not!).
- [[http://techslides.com/convert-csv-to-json-in-javascript][down-and-dirty csv converter]] (works for my case, though I also had to trim spaces fro mthe ends.
** Promise chaining, enables, etc.
- [[https://stackoverflow.com/questions/31264153/assign-value-from-successful-promise-resolve-to-external-variable][explanation of why vars set to ~then~ don't get the resolved value]] -- they're just the promise (drat!). Need to set it to promise.resolve instead, also need to wat for Promse.all(...) before moving on.
- [[https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all][Promise.all in MDN]]
* Chartist Plugin: Meta Line Segmenter
I want to be able to add a class to points and lines where meta property contains values that fall in various ranges. The Threshold plugin does this with a single value, and that works fine, but he method can't be trivially extended to my usecase b/c:
1. it uses values, not meta values, and
2. it draws two horizontal boxes, so there's no way to make vertically-oriented area stripes below the lines.
Here's an attempt.
#+begin_src js
function ctPointLabels(options) {
return function ctPointLabels(chart) {
var defaultOptions = {
labelClass: 'ct-label',
labelOffset: {
x: 0,
y: -10
},
textAnchor: 'middle'
};
options = Chartist.extend({}, defaultOptions, options);
if(chart instanceof Chartist.Line) {
chart.on('draw', function(data) {
if(data.type === 'point') {
data.group.elem('text', {
x: data.x + options.labelOffset.x,
y: data.y + options.labelOffset.y,
style: 'text-anchor: ' + options.textAnchor
}, options.labelClass).text(data.value);
}
});
}
}
}
#+end_src