{"id":49373943,"url":"https://github.com/simondedman/marspatauto","last_synced_at":"2026-04-28T01:10:53.884Z","repository":{"id":84206384,"uuid":"322201602","full_name":"SimonDedman/MarSpatAuto","owner":"SimonDedman","description":"Automating marine spatial date acquisition \u0026 analysis","archived":false,"fork":false,"pushed_at":"2024-06-25T01:37:09.000Z","size":3612,"stargazers_count":2,"open_issues_count":37,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-02-23T01:13:24.599Z","etag":null,"topics":["acquisition","data-science","marine-data","spatial-analysis"],"latest_commit_sha":null,"homepage":"","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SimonDedman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-12-17T06:26:06.000Z","updated_at":"2024-06-25T01:37:12.000Z","dependencies_parsed_at":"2024-06-24T23:44:25.654Z","dependency_job_id":"d50ff6ee-5ac7-4456-9df0-7a561d544569","html_url":"https://github.com/SimonDedman/MarSpatAuto","commit_stats":{"total_commits":14,"total_committers":1,"mean_commits":14.0,"dds":0.0,"last_synced_commit":"4ef22b2852f1c369abdd008725ec2106020f2df4"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/SimonDedman/MarSpatAuto","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SimonDedman%2FMarSpatAuto","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SimonDedman%2FMarSpatAuto/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SimonDedman%2FMarSpatAuto/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SimonDedman%2FMarSpatAuto/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SimonDedman","download_url":"https://codeload.github.com/SimonDedman/MarSpatAuto/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SimonDedman%2FMarSpatAuto/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32361567,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-27T20:07:02.737Z","status":"ssl_error","status_checked_at":"2026-04-27T20:07:00.910Z","response_time":128,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["acquisition","data-science","marine-data","spatial-analysis"],"created_at":"2026-04-28T01:10:48.531Z","updated_at":"2026-04-28T01:10:53.877Z","avatar_url":"https://github.com/SimonDedman.png","language":"R","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MarSpatAuto\r\nAutomating marine spatial date acquisition \u0026amp; analysis\r\n\r\n\r\n# Automating (marine) spatiotemporal data processing, acquisition and analysis\r\nSimon Dedman, 2020-11-12, simondedman@gmail.com www.simondedman.com \r\n\r\n# Background\r\nTaggers \u0026 trackers of sharks and other large marine species typically collect the same type of data, and then one person spends countless days researching which explanatory variables / correlates they should try to acquire, tries to acquire them, acquires some, possibly researches how best to format them, formats some, researches which test to use to get the result they’re interested in, uses probably one of them, occasionally the best one, researches what the results mean, and then spends days plotting and replotting figures and tables, and poring over them.\r\n\r\nThis solitary drudge work perpetuates despite our living in The Future, surrounded by powerful computers and mostly using the same two free programming languages which host innumerable freely-shared tests. Arguably this endemic tedious yeomanlike wheel-reinventing is due to the structure of science and academia, resulting in its being inherently brutally anti-collaborative and therefore highly inefficient when considered as a single sector.\r\n\r\nI propose that the majority of the ostensibly-highly-technical work that often comprises the bulk of the time spent on science, could be automated down to requiring only the barest user input, for any analytical technique that has been coded and shared. This would free up humans to do things that humans want to do (fieldwork, research) and are better at (interpreting findings into the wider contexts of ecology, policy, management, etc.).\r\n\r\nRelated article, concept contextualised within the wider future of humanity as relates to software and AI: https://medium.com/@SimonDedman/ai-and-the-future-of-marine-science-68a53dcd313d\r\n\r\n# What do we have?\r\nTag data from large marine species typically have latitude, longitude, and datetime (typically daily). They maybe have (ARGOS) satellite strength (codes) or similar spatial error metrics, fish size, and other sensor readings like depth, internal and external temperature, light level, DO2. They might have additional metadata parameters like the ID of different individuals in the case of data contain multiple tracks, and similar metadata grouping columns like sex, stock, etc.\r\n\r\n# Which correlates?\r\nSome are obvious: temperature, ocean depth. Most others require increasing levels of experience / background in the field (of marine spatio-temporal analysis) to know about. This is a barrier to entry. \r\nCorrelate research \u0026 gathering is typically a substantial part of a project, and is a drudge.\r\nToo few correlates can result in low explanatory power or, worse, acceptable explanatory power and a conclusion that misses key drivers of the studied system.\r\nSimple spatiotemporal projects are a staple of early career science work but a failure to complete exhaustive assessment of plausibly pertinent variables is chronic.\r\n\r\nSo: deciding which correlates you’re even interested in is a problem.\r\nBut it’s a solvable problem: one thorough literature review would probably find 99% of variables that folks have used. My hope is that community members will simply share their variables and code.\r\n\r\n# From where?\r\nErddap, rerddap, rextracto, libraries of databases, and extraction tools, already exist.\r\nBut also, my voluminous bookmarks of data sources. NASA, Planet, local agencies (Florida mangroves), NGOs (Allen coral atlas), Rebecca Degagne’s team.\r\nThere are many large data holders, some of whom are trying to index all marine spatial data in one place (ERDDAP, Rebecca). [Warning: ensure this project isn’t Reinventing the wheel]\r\nWe’re not aiming to replace or compete with these database libraries nor their extractions tools, where they exist, rather we’re looking to leverage them into an overarching system which removes the work of sourcing the appropriate correlates for your XY(Z)T data, given that there can only be one publicly-available highest-resolution source for any XYZT point.\r\n\r\n# What format?\r\nBecause the ‘Big Ones’ are obvious (temp: °C; salinity: ppm) it’s easy to stride from acquisition to analysis without thinking about processing. How do you account for date? Yearday? So this variable  shrinks to 1/365th its size on New Year’s Day? That’s some hangover. Month? A way of dividing a year by lunar cycles using a system that isn’t exactly tied to lunar cycles? And for the lunar signal - % of full? Cycle position in radians?\r\nAgain, these are researchable answers. But, for every one of potentially tens of variables. That you already researched and took ages acquiring.\r\nVery likely, we as a community can agree on a best format and that’ll be that. If there are genuine alternatives, fine, we can provide both formats, document that, and allow the user to choose the alternative(s) if they so wish.\r\n\r\n# What tests?\r\nSo you provided your lat lon \u0026 datetime and now you have a whole slew of extra columns of data. Hoorah! Now what? Literature review for “best” technique to answer a specific question that was arbitrarily conceived in the original project hypothesis that led to the data being gathered. Technique chosen is likely a compromise of commonality of use, ease of acquisition, ease of understanding, ease of use, and ease of understanding of results. A single-technique analysis approach is likely. Two occasionally. \u003e3 rare.\r\nThe aim, therefore, is to analyse the data using the best version of every applicable (s/t) analysis technique.\r\nInclusive of pre-testing for S/T autocorrelation of original data \u0026 autocorrelation of new correlates.\r\n\r\n# What do the results mean?\r\nSo you ran all these tests and R spat out a whole slew of opaque values and coefficients, across hundreds of screen-height scrolls. Hoorah? Now what? Literature review for the meaning of each of these. Presume that information on how to interpret all elements of a test result exists (they don’t always). Presume that available information warns about risks from certain data types and/or formats (they don’t usually). Re-run the test but this time assign the output to an object so you can use it. Spend days formatting plots and tables so you can display and share said results.\r\nALL of this is formulaic and can be codified. It’s a time-well that disproportionately swallows earlier career researchers, until they level up their background knowledge and (gg)plotting skills.\r\n\r\nInstead:\r\n\r\nContextualised meaning of results, using simple text code in R (if P\u003c0.2 then “this result is significant”), plain English interpretation, with references and quoted text blocks of key bits from those papers as appropriate.\r\nAll individual results plotted and saved as images into folders with appropriate names, and accompanying text files for suggested captions. R Markdown (\u0026 pdf, html) reports created for each test.\r\n\r\nAnd even then there’s drudge that can be automated. It’s great that we can produce hundreds of plots with a few lines of code. But you now have to analyse and common-sense-interpret hundreds of plots, scribble notes for each, decide which are important and/or interesting, consider their interactions, coalesce this into a physically/biologically/ecologically-contextualised story. Again, much of this is formulaic, even if it may not feel that way initially.\r\nFirstly, simple plot analysis. If that correlate is contributing nearly zero explanatory power, ignore it. Easy enough right? Tell me you’ve never looked at those plots. Tell me you haven’t thought “that’s quite interesting” even though you know the caveat is that this correlate explains 0.001% of the variance. You can’t get those brain cycles back.\r\n\r\nHarder stuff that’s still easy: is it a linear relationship? Curvilinear? Which way is it going? “More of X results in less of Y”. “Y has a domed relationship with X which peaks when X is 4”. Are there thresholds? “Y is notably higher when X \u003e 2.79”.\r\nTake those, order them by variable importance. Couch paragraphs in those terms: “Salinity explains 7% of the variance so its relationship with bull shark abundance is of relatively low but not insignificant value.” Etc.\r\nSecondary interpretations can be similarly codified (community input/legwork will be key here); temperature and depth have a known relationship so if both are present then a default paragraph can be included, followed by a dynamic text chunk which relates to the interaction strength in the results, the importance of each to the variance, etc.\r\n\r\nTertiary (ecological / ecosystemic) results can be codified, given more work. “The data are predominantly from these depths and distances from shore, suggesting this is a coastal pelagic species. Known important drivers for this ecosystem group are A B and C, which are all found to be important in this analysis”. [This element of the results could be community collaborative, with results submittable to a shared database that could be queried for trends, feeding back to the interpretations for others. Generate results interactions pairplot matrix for all correlates, to populate with default text blocks \u0026 dynamic text code].\r\n\r\nOverall results for all tests and pre-checks synthesised into a single report (markdown, html, pdf) with tables, figures, and text.\r\n\r\n# How will the package be organised?\r\nModular, component R scripts formatted to accept the standardised input data.\r\n\r\nOrder:\r\n\r\n0. set overall theme for all plots \u0026 reports which can be changed. E.g. per journal. Like CSL files. Colour-blind friendly. Viridis2.\r\n\r\n1. Check input data – are positions all marine (if this is marine only), datetime in order, no duplicates, (anything else)? Plot basic stuff. Save in standardised folder structure. Ingest samples, clean, report.\r\n\r\n2. Test input data for autocorrelation, other tests? Outliers, residual analysis, random forest imputation, etc. Save report in folder.\r\n\r\n3. Computer system check: all packages installed \u0026 loaded, get system details to provide time estimates (later). Additional (numbered) report.\r\n\r\n4. Correlate acquisition. Collectively agree defaults – most but maybe not all? And which variants e.g. satellite ChlA: daily, 3day, 8day, month, etc. User options to change these. Ability to ingest user’s own correlates where not universally available. Save output df and make table of variables used with resolution (\u0026 other data details), sources \u0026 references (user-ingested data to be accompanied with these details in text file, csv, bibtex, etc).\r\n\r\n5. Correlate formatting (do as part of above?)\r\n \r\n6. acquisition: embarassingly parallel: all rows of XYZT, for all variables.\r\nacq.SST.source()\r\nhave people dump their scripts of functions into subfolders on github by var e.g. “acq.SST”. Regardless of how messy they are.\r\nIdeally have a compressed boolean/logical 4D raster for the data they have (i.e. 4D space of data provision they occupy). Get text info from the data source, then convert.\r\nFor saving outputs: Add an index column to the samples df and save acquired data with just have an index column, i.e. 1c * Nr instead of 4c * Nr. Stitching scripts later can then join by Index only. Save as high compression R files.\r\n\r\n7. Run tests. User options of which to run, default all? Likely default most. Function to deselect tests if it's mathematically unreasonable/illegel to run them on these data for whatever reason. Not sure what the reason would be given the point is the data are all tracking? Or any XYZT i.e. could be acoustic. Pre-check of estimated time per-test \u0026 total. Option to restart. Save results (data, plots, etc.?) to per-test subfolders. Script format: probably 1 control script which calls individual tests scripts and final report script. Tests scripts run the tests (duh) and also do the plotting, reporting, file saving. Report script stitches together individual subfolder reports into sections, including initial data plots \u0026 status, processing done, autocorrelation results, correlates used table. See https://github.com/tidymodels/tidymodels \u0026 https://www.tidymodels.org/ \u0026 https://www.tidymodels.org/learn/ and build around that format.\r\nUse tidymodels where possible: individual scripts to analyse, generate results including errors, model evaluation, plots, save model objects for others to plot differently. Scrape acquisitions folders for available variables. Specify groups \u0026 XYZT subsets here.\r\n\r\n7.5 Results meta-analysis (meta-results?): Do this here as part of each anl.SST.source() script, OR separately after, in case cross-variable insights allow increased reduction of results number? Maybe both: within-var reductions here, then\r\n\r\n8. Across-var results meta-analysis \u0026 reduction, insights generation.\r\nSynthesising Huge Results Stack: \r\nIf analysing the same types of data, the results outputs will have the same biological meaning.\r\nThose numerical results (e.g. linearity index = 0.8) can thus have simple text lines (e.g. \"this is a straight path indicative of migration\").\r\nExpanding this, whole suites of results can be summarised e.g. \"the path is STRAIGHT and LONG. The home range is LARGE; compared to the JUVENILE stage, the ADULT home range is LARGER. Etc\".\r\nExpanding THIS, multiple groups (e.g. adult/juvenile above) can have comparison summaries. \"LEMON SHARKS have smaller home ranges than BULL SHARKS\".\r\nThis can all be collated into a report.\r\nWhich bits are the most important? Differences I suppose. So it could highlight differences by sorting outcomes by differential.\r\nAND THEN, potentially there could be a way to evaluate which results components are most informative to specific conclusions. I'm thinking PCA here maybe.\r\nResults shape evaluation. If you have line plots of variables' relationships, those can probably be decoded with some relatively simple algorithmic rules. Is the relationship positive/negative, is it strong, is it linear, curvilinear, threshold, threshold where. Convert all that to numbers and again give the plain text meaning.\r\nIf you have lots of results which mean the same thing (plots metrics are similar), that should be easy to show e.g. with K-means clustering. Then you could sort \u0026 prioritise the 'best' of those similar results.\r\n\r\n9. Free on Github \u0026 CRAN.\r\n\r\n# How do we start?\r\nGithub project. Also some kind of community location, email list?\r\nAcquisition: have people dump their scripts of functions into subfolders on github by var e.g. “acq.SST”. Regardless of how messy they are.\r\nIdeally have a compressed boolean/logical 4D raster for the data they have (i.e. 4D space of data provision they occupy). Get text info from the data source, then convert. Script to automatically A-union-B 4D join all 4D data presence rasters per variable subfolder.\r\nTests: List of tests. And existing packages. And code experts for those packages who’ll be prepared to assist in writing the automation script.\r\nPer acquisition: have people dump their scripts there. Might just be few-line functions. But should confirm to report output formatting.\r\n\r\n# List Of Tests\r\nHabitat comparison: if you have a vector of lat lon datetimes, and get the accompanying environmental variables, you can compare the habitat subset against a larger habitat subset (e.g. our study area vs whole known species range) or total global habitat (i.e. ignoring the species data).\r\n\r\nElectivity: follows from above\r\n\r\nCore use area, minimum 50% 75% 90% kernel use density polygons, including coastline removal.\r\n\r\nKmeans XYT behaviour bins, resident vs migration\r\n\r\nImportance \u0026 relationship of environmental predictors to spatial abundance, gbm.auto etc\r\n\r\n# Ok so what do *I* do?\r\nGo to issues, think of an explanatory variable you use. Is there an issue for that? No? Create one. Go one better and find the highest res publically available dataset for that. If nothing else, create the issue. Or create an issue to chat about any element of this. Cheers!\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimondedman%2Fmarspatauto","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsimondedman%2Fmarspatauto","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimondedman%2Fmarspatauto/lists"}