{"id":43832939,"url":"https://github.com/openthc/data","last_synced_at":"2026-02-06T03:33:30.079Z","repository":{"id":63536914,"uuid":"235596672","full_name":"openthc/data","owner":"openthc","description":"Cannabis Data Analysis Portal","archived":false,"fork":false,"pushed_at":"2025-04-15T22:35:50.000Z","size":353,"stargazers_count":9,"open_issues_count":0,"forks_count":2,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-15T23:26:34.355Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openthc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-01-22T14:58:13.000Z","updated_at":"2025-04-15T22:35:54.000Z","dependencies_parsed_at":"2024-04-18T03:40:56.111Z","dependency_job_id":"1b12e30a-a8db-4bf0-9629-5a724cf0eee3","html_url":"https://github.com/openthc/data","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/openthc/data","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openthc%2Fdata","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openthc%2Fdata/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openthc%2Fdata/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openthc%2Fdata/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openthc","download_url":"https://codeload.github.com/openthc/data/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openthc%2Fdata/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29148265,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-06T02:39:25.012Z","status":"ssl_error","status_checked_at":"2026-02-06T02:37:22.784Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-02-06T03:33:29.192Z","updated_at":"2026-02-06T03:33:30.039Z","avatar_url":"https://github.com/openthc.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Data\n\nAn Open Data Portal for Cannabis and tools for Converting and Importing said data.\n\nThis tool is designed to import data from BioTrack, GrowFlow, LeafData or METRC into a common schema.\nThen you can run reports on this data-set using a front-end tool like [Metabase](https://www.metabase.com/)\n or [Apache Superset](https://superset.apache.org/).\n\n\n## Converting XLS\n\nIf the source data is in XLS, first convert it into CSV.\nThen those files can be processed into JSON.\n\n```shell\n./bin/cli.php convert \\\n\t--source-type XLS \\\n\t--source-file ./path-to-some-file.xls \\\n\t--output-path ./path-to-directory/\n```\n\nIf the XLS files worksheet headers are not conforming to the OpenTHC/CSV specification edit them before conversion.\nIt may also be useful to remove columns that are not significant.\n\nOther data may also need to be cleaned up, setting the proper product weights.\nRemoving rows that duplicate (eg Plants in Inventory and in a Plants worksheet).\n\n\n## Converting CSV\n\nOpenTHC has defined some \"standard\" CSV file Headers to use for data importing.\nTypically, one can use existing software exports, update their header column names and then convert/import.\nThe names are not case sensitive.\n\n```\nSection_GUID\nSection_Name\nSection_Type\nVariety_GUID\nVariety_Name\nVariety_Type\nProduct_GUID\nProduct_Name\nProduct_Type\nPackage_Unit_Weight\nPackage_Unit_Volume\nInventory_GUID\nInventory_Name\nInventory_Unit_Count_Current | Inventory_Unit_Count | Unit_Count\nInventory_Unit_Count_Initial | Unit_Count_Initial\nCrop_GUID\nCrop_Name\nCrop_Unit_Count_Current | Crop_Unit_Count | Unit_Count\nCrop_Unit_Count_Initial | Unit_Count_Initial\nPlant_GUID\nPlant_Name\n```\n\nCrop and Plant are the same thing; we have both names to help thing about their usage uniquely.\nA Crop is an identifier for one or more plants (or a whole field) but Plant explicitly means just a single plant.\n\nThen just convert the objects out of the source document.\n\n```shell\n/bin/cli.php convert \\\n\t--output-path output-data/convert-example/\n\t--source-file=./source-data/Inventory.csv --source-type=product-csv\n\n/bin/cli.php convert \\\n\t--output-path output-data/convert-example/\n\t--source-file=./source-data/Inventory.csv --source-type=section-csv\n\n/bin/cli.php convert \\\n\t--output-path output-data/convert-example/\n\t--source-file=./source-data/Inventory.csv --source-type=variety-csv\n\n/bin/cli.php convert \\\n\t--output-path output-data/convert-example/\n\t--source-file=./source-data/Inventory.csv --source-type=inventory-csv\n```\n\n\n## Loading BioTrack\n\nWe can load BioTrack data from either the API, CSV or SQL data.\n\n```shell\n./bin/import.php \\\n\t--source-type=BioTrack-API \\\n\t--source=$SOURCE_URI\n\t--object=crop \\\n\t--output=OUTPUT_PATH\n```\n\n\n## Loading Akerna/LeafData/MJFreeway/MJPlatform Data\n\nGet the ZIP files or Table Dumps from LeafData.\nPut the LeafData CSV/Zip files in some origin directory (`./source-data/leafdata`).\nExtract and prepare them as necessary.\nSymlink those files into `./source-data/$NAME.csv`, using the name required by the import script.\nThen remove the origin file, leaving an orphan symlink to track what's been completed.\nAnd clean up when all the way done.\n\nSee `./bin/leafdata/extract.sh` for an automated process.\n\n\n## Loading CCRS\n\nGet the ZIP file dumps from the LCB and store them in something like `./source-data-YYYY-MM-DD`.\nUse the `box-download.php` helper.\nThen, from that source-data directory run `../bin/ccrs/import.php [OBJECT] | OUTPUT-[OBJECT].txt`\n\n\n## Loading GrowFlow\n\nThe scripts in lib/Import/GrowFlow can import from the `CSV` exports or from collected `HAR` files.\n\n```shell\n./bin/import.php \\\n\t--source=source-data/More_Vegging_Plants_export.csv  \\\n\t--source-type=GrowFlow-CSV \\\n\t--object=crop \\\n\t--output=OUTPUT_PATH\n```\n\n\n## Loading Cultivera\n\nThe scripts in lib/Import/Cultivera can import from the `HAR` or `XLSX` exports.\n\n```shell\n./bin/import.php \\\n\t--source=source-data/Inventory.xlsx  \\\n\t--source-type=Cultivera-XLSX \\\n\t--object=inventory \\\n\t--output=OUTPUT_PATH\n```\n\n\n### BOM, UTF-16\n\nSometimes the files are like this.\nSome of the files are UTF-16-LE.\nOur pre-processing scripts will normalise them.\n\n* https://www.dave-baker.com/2017/10/03/converting-a-utf-16-csv-to-utf-8-in-php/\n* http://www.craiglotter.co.za/2010/03/07/how-to-convert-an-utf-16-file-to-an-utf-8-file-using-php/\n\n\n### iconv\n\n* Use iconv -f UTF-16LE -t UTF-8\niconv -f UTF-16LE -t UTF-8 \u003cfilename\u003e -o \u003cnew-filename\u003e\n\n\n### Extracting\n\nThe files may be delivered as `zip`, and internally they are labeled as `csv` files.\nThey may use a comma, they may use a TAB.\n\n\n----\n\n# Data Information\n\nProduct Categories:\n\tAll the Standard Numbers\n\t+Joints, BHO\n\nGroup by Sizes\n\tBut find the Common Sizes/Size Groups First!\n\nFacet: License\nFacet: Product\nFacet: Variety\n\n## Retail Detail\n\n\n# Data!\n\nThe data scripts make TSV files from the SQL.\nEach TSV answers one question built around these \"base facets\"\n\n  * Category: Plants, Wholesale, Retail, QA\n  * An Inventory Type (5, 13, 28, etc)\n  * A Date Delta - Daily, Weekly, Monthly, Q, Y\n\nThe answers are \"business questions\" - like price per gram or volume\nDetails are in the ./data-*.php scripts, the parameter is 'a'\n\nEach Data category is answered by a PHP script ./data-*.php\nThose scripts are called by the wrapper ./data.php, like this:\n\n\t./data.php wholesale d=w t=28 a=ppg-avg\n\nThat will in-turn execute code in the ./data-wholesale.php and pass the parameters in as $_GET values\n\nThose $_GET values are used to control logic that will assemble the SQL in the form of\n\n\tSELECT (value-expression)\n\tFROM (necessary-tables)\n\t\tJOIN (may-need-one-or-more-of-these)\n\tWHERE (desired-constructed-filter)\n\t\tAND (additional-conditions-we-discover-to-remove-crap-data)\n\nAnd then iterate the Date Delta spewing the results into a TSV file\nActually, the PHP scripts output to STDOUT which is redirected to capture the output.\n\n## Shell Wrappers\n\nThe shell wrappers (data-*.sh) are designed to iterate over the \"base facets\"\n\n\tforeach Date\n\t\tforeach Type\n\t\t\tforeach OtherThing\n\t\t\t\t./data.php [category] d=Date t=Type a=ppg-avg \\\n\t\t\t\t\t\u003e ./webroot/pub/data/[predictable-path-here-somehow]\n\n\nWe kind of know these to be the base queries for the data that people want.\nSo we can pre-build these.\n\n## Time to First Harvest\n## Time from Harvest to Cure\n\n##\nStore-level summary by inventory type of what they paid for X and what\nthey charged for X where X is the # of units or grams sold during the\ntime period in question carry level of detail that allows drilldown to\na specific processor or producer\n\nThis should enable a bunch of stuff I'll add to the document later today\n\n## Markup by County\n\nChocolate vs Cookies\nRegions\nStrains on Sales\n\n\n## Business Intelligence Tools\n\n* [eBay TSV Tools](https://github.com/eBay/tsv-utils)\n* [Poli](https://news.ycombinator.com/item?id=20507592)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenthc%2Fdata","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenthc%2Fdata","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenthc%2Fdata/lists"}