{"id":32375952,"url":"https://github.com/zignig/substrate","last_synced_at":"2025-10-24T22:51:58.621Z","repository":{"id":3735521,"uuid":"4809383","full_name":"zignig/substrate","owner":"zignig","description":"open hardware spooler","archived":false,"fork":false,"pushed_at":"2014-07-20T01:17:32.000Z","size":576,"stargazers_count":8,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2023-03-11T00:53:03.504Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"McCouman/Wikiunity-Podcast-iFrame-Tag","license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zignig.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-06-27T14:34:19.000Z","updated_at":"2017-04-30T07:34:32.000Z","dependencies_parsed_at":"2022-07-09T00:30:19.937Z","dependency_job_id":null,"html_url":"https://github.com/zignig/substrate","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"purl":"pkg:github/zignig/substrate","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zignig%2Fsubstrate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zignig%2Fsubstrate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zignig%2Fsubstrate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zignig%2Fsubstrate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zignig","download_url":"https://codeload.github.com/zignig/substrate/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zignig%2Fsubstrate/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280878267,"owners_count":26406643,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-24T02:00:06.418Z","response_time":73,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-24T22:49:35.975Z","updated_at":"2025-10-24T22:51:58.616Z","avatar_url":"https://github.com/zignig.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Substrate\n=========\n\nOpen hardware spooler\n\nI needed a place to store and process and ever increasing number of Open Hardware design files. This is the result.\n\n#Sofware\nsubstrate is written in python 2.7\n\n##Servers\n\n1. [couchdb](http://couchdb.apache.org) for document storage and map/reduce views\n2. [rabbitmq](http://rabbitmq.com) for spooling\n3. [redis](http://redis.io) for FAST local storage/cache and data editing\n\n##Clients\n\n1. [blender](http://blender.org) rendering and editing\n2. [openscad](http://openscad.org) build and assemble\n\n##Basic Use##\n\nI have tried to build a spooler and processor based on some networked servers for distribution so small clients can do a single job \nand not have to worry about local access to any services.\n\nSo once you have installed the above servers you need to set up some stuff to get it running. \n\nfirst install the following python modules.\n\n\npip install pika redis couchdbkit couchapp\n\nNow you have all the python tools you need.\n\n1. find the couch database that you want to process\n2. couchapp push http://user:pass@couch.server/database\n3. check that the design/robot file design file has been created \n\nIn this directory there is a file called current , this is the document that builds the substrate ( ie mod it for your stuff )\n\nIt looks kind of like this:\n\n\n\t\tcurrent = {\n\t\t\t\"robot_version\": 0.1, \n\t\t\t\"redis_query\": {\n\t\t\t\t\"status\": \"robot/status\", \n\t\t\t\t\"name\": \"robot/name\", \n\t\t\t\t\"author\": \"robot/author\", \n\t\t\t\t\"process\": \"robot/process\", \n\t\t\t\t\"tags\": \"robot/tag\", \n\t\t\t\t\"thingi_id\": \"robot/thingi_id\", \n\t\t\t\t\"derivative\": \"robot/derivative\"\n\t\t\t}, \n\t\t\t\"status_states\": {\n\t\t\t\t\"new\": \"pending\", \n\t\t\t\t\"running\": \"finished\", \n\t\t\t\t\"pending\": \"running\"\n\t\t\t}, \n\t\t\t\"ttl\": 3600, \n\t\t\t\"_rev\": \"82-7cccca067b884c747cb870c67b62d18e\", \n\t\t\t\"mime_routing\": {\n\t\t\t\t\"text/scad\": \"scad\", \n\t\t\t\t\"application/sla\": \"stl\"\n\t\t\t}, \n\t\t\t\"broker\": [\n\t\t\t\t\"gateway.terra\"\n\t\t\t], \n\t\t\t\"couch\": [\n\t\t\t\t\"http://gateway.terra:5984/\"\n\t\t\t], \n\t\t\t\"mime_queues\": {\n\t\t\t\t\"text/scad\": [\n\t\t\t\t\t\"render_to_stl\", \n\t\t\t\t\t\"extract\"\n\t\t\t\t], \n\t\t\t\t\"application/sla\": [\n\t\t\t\t\t\"thumbnail\", \n\t\t\t\t\t\"large\", \n\t\t\t\t\t\"rotational\", \n\t\t\t\t\t\"slice\"\n\t\t\t\t]\n\t\t\t}, \n\t\t\t\"status_queue\": {\n\t\t\t\t\"robot/status\": {\n\t\t\t\t\t\"new\": \"initialize\", \n\t\t\t\t\t\"finished\": \"finished\", \n\t\t\t\t\t\"pending\": \"pending\"\n\t\t\t\t}, \n\t\t\t\t\"robot/process\": {\n\t\t\t\t\t\"download\": \"download\", \n\t\t\t\t\t\"author\": \"author\"\n\t\t\t\t}\n\t\t\t}, \n\t\t\t\"databases\": [\n\t\t\t\t\"incoming\"\n\t\t\t], \n\t\t\t\"initialize\": \"robot/robot_status\", \n\t\t\t\"_id\": \"current\", \n\t\t\t\"queues\": [\n\t\t\t\t\"stl\", \n\t\t\t\t\"scad\", \n\t\t\t\t\"incoming\", \n\t\t\t\t\"new\", \n\t\t\t\t\"assemble\", \n\t\t\t\t\"recycle\", \n\t\t\t\t\"finished\", \n\t\t\t\t\"pending\", \n\t\t\t\t\"classify\", \n\t\t\t\t\"initialize\", \n\t\t\t\t\"changes\", \n\t\t\t\t\"out\", \n\t\t\t\t\"error\", \n\t\t\t\t\"fetch\", \n\t\t\t\t\"slice\"\n\t\t\t], \n\t\t\t\"ttl_long\": 86400\n\t\t}\n\nLooks kind of long and complicated doesn't it , HMM. yes it does. \n\ncouch , database and broker are the important varibles here.\n\nIf these are pointing to your local servers they you should be nearly ready to cook.\n\ncd into the \\_attachement folder and run:\n\n./substrate\\_bootstrap.py  ( CURRENTLY BROKEN )\n\nIf you want to attach to the bl3dr multiverse , this will create a replica of the \\_entire\\_ incoming database (for now).\n\nPLEASE :)\n\notherwise upload the \"current\" document into a couchdb and point config.json at it giving you a local substrate\n\n#Usage\n\n## Browsing\n\nrun ./console.py in the \\_attachments directory\n\nThis should show you a bunch of stuff and give you a python console.\n\ncouchdb queries ( ie the robot design ) are mapped into redis with the following stanza:\n\n    \"redis_query\": {\n        \"status\": \"robot/status\",\n        \"name\": \"robot/name\",\n        \"author\": \"robot/author\",\n        \"process\": \"robot/process\",\n        \"tags\": \"robot/tag\",\n        \"thingi_id\": \"robot/thingi_id\",\n        \"derivative\": \"robot/derivative\"\n    },\n\n-- The console is tab completed --\n\ncq.author() will give you a list of the most prolific authors.\n\n\u003e\u003ecq.author('zignig')\n\nwill list the uuids of the documents that I have.\n\n\u003e\u003ecq.load(cq.author('zignig'))\n\nwill load all of the couchdb docs ( not including attachments ) into RAM so access is fast.\nthese keys have a TTL and will disappear in good time (TTL in current).\n\n\u003e\u003ecq.id('1637b134-d1f2-4181-8e21-3863e3325400') \n\nwill yield \n\n\ndoc = {\n    \"_id\": \"1637b134-d1f2-4181-8e21-3863e3325400\", \n    \"_rev\": \"5-b2c142c6d573de5b91376e069a5f52fd\", \n    \"author\": \"zignig\", \n    \"name\": \"YAESU FT-857 head mount\", \n    \"processed\": true, \n    \"robot_status\": \"done\", \n    \"tags\": [\n        \"3D\", \n        \"makerbot\", \n        \"useful\"\n    ], \n    \"thingi_derivative\": [], \n    \"thingi_download\": [\n        [\n            \"/download:3202\", \n            \"bot-left.stl\"\n        ], \n        [\n            \"/download:3203\", \n            \"bot-right.stl\"\n        ], \n        [\n            \"/download:3204\", \n            \"joiner.stl\"\n        ], \n        [\n            \"/download:3205\", \n            \"top-left.stl\"\n        ], \n        [\n            \"/download:3206\", \n            \"top-right.stl\"\n        ], \n        [\n            \"/download:3207\", \n            \"FT857-mount.blend\"\n        ]\n    ], \n    \"thingi_id\": 1251, \n    \"thumb_url\": \"http://thingiverse-production.s3.amazonaws.com/renders/a4/26/c3/f5/c6/FT857mount_thumb_large.jpg\", \n    \"type\": \"thingiverse\", \n    \"url\": \"http://www.thingiverse.com/thing:1251\"\n}\n\nas a json object\n\nIf you want to save a edited json file you can do it a couple of ways\n\n\u003e\u003ecq.save(doc,id) \n\nwill write the changes back to redis and add it to a dirty set\n\n\u003e\u003ecq.sync() \n\nwill  try so save back to the couchdb or fail and keep it dirty.\n\nOR \n\n\u003e\u003ecq.write(doc,id)\n\nwill write to the couchdb and delete the key even on a failed write.\n\n#Spooling \n\nFor this bit you will need three consoles open\n\n1. ./console.py so you can manipulate the queues and data.\n2. ./spooler.py so changes in redis and couchdb are constantly spooled in\n3. ./combined.py that starts a bunch of threads with individual spool munchers\n\neach of the scripts can be run multiple times with exception of the spooler. \n\nso running ./stl.py locally or remotely ( with the python libs installed ) will grab process and run files out of the substrate.\n\nThis should take all the docs in the database and put them through a state machine defined by the \n\n    \"status_queue\": {\n        \"robot/status\": {\n            \"new\": \"initialize\",\n            \"finished\": \"finished\",\n            \"pending\": \"pending\"\n        },\n        \"robot/process\": {\n            \"download\": \"download\",\n            \"author\": \"author\"\n        }\n\nThe changes.py script will grab items out of the rabbitmq changes queue and check the robot\\_status of each document and send to the next spool based on the current value.\n\nEach of these states has their own spool and script that is bound to it. That script contains a callback that is used to take a rabbitmq message and modify a couchdb-\\\u003eredis document and do somthing to it.\n\nhave a read of the code :) \n\nno really.\n\n#Attachments\n\nThe initialize script will also take attachments on the documents and route the mime types based on the mime\\_routing stanza\n\n    \"mime_routing\": {\n        \"text/scad\": \"scad\",\n        \"application/sla\": \"stl\"\n    },\n\nThese are spooled into scad and stl queus on rabbitmq. The are in turn consumed by the ./stl.py and ./scad.py scripts.\n\nIf the attachments do not have a mapping they are spooled to the classify spool , this script needs to be written ( hint , hint ) :)\n\n#Thoughts \n\nUsing some of the nifty network servers ( redis , rabbitmq and couchdb ) I have managed to process heaps of documents. \n\nI have set up http://bl3dr.iriscouch.com/incoming as a test bed database if you want to add more files to it please email me \n\ntigger at interthingy dot com\n\nAnd I will add you as an editor and with bidirectional replicas we can build somthing really big.\n\nzignig.\n\nP.S. way past bed time.\n\nout PSSSSSSSHT.\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzignig%2Fsubstrate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzignig%2Fsubstrate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzignig%2Fsubstrate/lists"}