{"id":16847643,"url":"https://github.com/fabiosantoscode/crpy","last_synced_at":"2026-03-18T22:36:12.485Z","repository":{"id":19488337,"uuid":"22734233","full_name":"fabiosantoscode/crpy","owner":"fabiosantoscode","description":"CrowdProcess Python API Client","archived":false,"fork":false,"pushed_at":"2014-05-21T16:01:20.000Z","size":379,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-17T22:44:56.826Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fabiosantoscode.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-08-07T19:41:26.000Z","updated_at":"2023-08-23T12:28:37.000Z","dependencies_parsed_at":"2022-08-21T08:11:20.110Z","dependency_job_id":null,"html_url":"https://github.com/fabiosantoscode/crpy","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabiosantoscode%2Fcrpy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabiosantoscode%2Fcrpy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabiosantoscode%2Fcrpy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabiosantoscode%2Fcrpy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fabiosantoscode","download_url":"https://codeload.github.com/fabiosantoscode/crpy/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244173553,"owners_count":20410300,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T13:08:37.343Z","updated_at":"2026-01-03T15:55:19.710Z","avatar_url":"https://github.com/fabiosantoscode.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# CrowdProcess API Client for Python\n\nThis is a client for [CrowdProcess](https://crowdprocess.com/)'s [REST API](https://crowdprocess.com/rest).\n\nIt works in python 2.7 and 3.4+.\n\n## Installing\n\n\tpip install crowdprocess\n\nor\n\n\teasy_install crowdprocess\n\n## Usage example\n\n```python\n\u003e\u003e\u003e from crowdprocess import CrowdProcess\n\u003e\u003e\u003e crp = CrowdProcess('username', 'password')\n\n\u003e\u003e\u003e x2 = crp.job('function Run (d) { return d*2; }')\n\u003e\u003e\u003e results = x2(range(5)).results\n\u003e\u003e\u003e list(results)\n[0, 2, 4, 6, 8, 10] # comes in a random order\n```\n\n## More detailed use\n\n### Importing and instanciating\n```python\n\u003e\u003e\u003e from crowdprocess import CrowdProcess\n\u003e\u003e\u003e crp = CrowdProcess('username@email.com', 'password')\n```\n\nTo get those credentials you must [register](https://crowdprocess.com/register) with CrowdProcess.\n\nYou can also instanciate it with a token instead of a username and password:\n\n```python\n\u003e\u003e\u003e crp = CrowdProcess(token='3c46d593-5435-47c5-92aa-1613ade978c2')\n```\n\n### Jobs\n\n#### Creating a job\n\nWith the `CrowdProcess` class instanciated above,\n\n```python\n\u003e\u003e\u003e program='function Run (d) { return d }'\n\u003e\u003e\u003e job = crp.job(program)\n\u003e\u003e\u003e job.id\n'3c46d593-5435-47c5-92aa-1613ade978c2'\n```\n\nInvoking `crp.job` with the `program` parameter automatically creates a job in CrowdProcess and returns an instanciated `Job`. \n\nAfter you get a `job.id`, you can use it to get a `Job` again, without creating it:\n\n```python\n\u003e\u003e\u003e job = crp.job(id='3c46d593-5435-47c5-92aa-1613ade978c2')\n```\n\n#### Listing jobs\n\n```python\n\u003e\u003e\u003e crp.list_jobs()\n[{u'status': u'active', u'failed': 0, u'bid': 1, u'created': u'2014-05-14T10:07:52.747503Z', u'modified': u'2014-05-14T10:07:53.716147Z', u'browserHours': 137, u'finished': 1000, u'lastResult': u'2014-05-14T10:07:59.06Z', u'total': 1000, u'id': u'3c46d593-5435-47c5-92aa-1613ade978c2'}]\n```\n\nPrettier:\n\n```python\n\u003e\u003e\u003e jobs = crp.list_jobs()\n\u003e\u003e\u003e print(json.dumps(jobs, sort_keys=True, indent=2))\n[\n  {\n    \"bid\": 1, \n    \"browserHours\": 137, \n    \"created\": \"2014-05-14T10:07:52.747503Z\", \n    \"failed\": 0, \n    \"finished\": 1000, \n    \"id\": \"3c46d593-5435-47c5-92aa-1613ade978c2\", \n    \"lastResult\": \"2014-05-14T10:07:59.06Z\", \n    \"modified\": \"2014-05-14T10:07:53.716147Z\", \n    \"status\": \"active\", \n    \"total\": 1000\n  }\n]\n```\n\n#### Deleting a job\n\n```python\n\u003e\u003e\u003e job = crp.job(id='3c46d593-5435-47c5-92aa-1613ade978c2')\n\u003e\u003e\u003e job.delete()\n```\n\n#### Deleting all jobs\n\n```python\n\u003e\u003e\u003e crp.delete_jobs()\n```\n\n## Tasks and Results\n\nAfter creating a job, you're all set to send it tasks and get back results.\n\n`tasks` can be any iterable object, `results` will be a generator:\n\n```python\n\u003e\u003e\u003e job = crp.job('function Run (d) { return Math.pow(d, 2); }')\n\u003e\u003e\u003e tasks = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\n\u003e\u003e\u003e results = job(tasks).results\n\u003e\u003e\u003e list(results)\n[49, 64, 16, 25, 9, 36, 4, 81, 0, 1]\n```\n\nwhich would be the same as,\n\n```python\n\u003e\u003e\u003e job = crp.job('function Run (d) { return Math.pow(d, 2); }')\n\u003e\u003e\u003e list(job(range(10)).results)\n[49, 64, 16, 25, 9, 36, 4, 81, 0, 1]\n```\n\nwhich would also be the same as,\n\n```python\n\u003e\u003e\u003e job = crp.job('function Run (d) { return Math.pow(d, 2); }')\n\u003e\u003e\u003e def tasks():\n...     for i in range(10):\n...             yield i\n... \n\u003e\u003e\u003e list(job(tasks).results)\n[25, 64, 49, 16, 36, 9, 0, 81, 1, 4]\n```\n\nNotice that the results never come in order.\n\n### Pro tip: you can use the results of one job as tasks of another job\n\n```python\n\u003e\u003e\u003e multiply = crp.job('function Run (d) { return d*2 }')\n\u003e\u003e\u003e divide = crp.job('function Run (d) { return d/2 }')\n\u003e\u003e\u003e numbers = range(10)\n\u003e\u003e\u003e multiplied = multiply(numbers).results\n\u003e\u003e\u003e divided = divide(multiplied).results\n\u003e\u003e\u003e list(divided)\n[7, 2, 6, 1, 5, 9, 8, 4, 3, 0]\n```\n\n### Don't forget about the errors\n\nSometimes your tasks will throw uncaught exceptions that you should know about, and you can get them the same way you get results:\n\n```python\n\u003e\u003e\u003e job = crp.job('function Run (d) { if (d === 4) { throw new Error(\"oh no, \"+d) } return d; }')\n\u003e\u003e\u003e tasks = range(10)\n\u003e\u003e\u003e errors = job(tasks).errors\n\u003e\u003e\u003e for error in errors:\n...     print error\n```\n\n## Tasks and Results, lower level\n\n### Submitting tasks\n\nOnce again, tasks may be any iterable:\n\n```python\n\u003e\u003e\u003e multiply = crp.job('function Run (d) { return d*2 }')\n\u003e\u003e\u003e multiply.submit_tasks(range(10))\n```\n\n### Getting results\n\n```python\n\u003e\u003e\u003e results = multiply.get_results()\n\u003e\u003e\u003e list(results)\n[18, 8, 10, 4, 6, 16, 14, 0, 2, 12]\n```\n\nThis delivers all the job's computed results at the moment, but you should in fact get every result as soon as it's computed, in a stream:\n\n### Streaming results\n\nYou can also iterate through every result as soon as it comes in:\n\n```python\n\u003e\u003e\u003e expected_results = 10\n\u003e\u003e\u003e results = multiply.get_results_stream()\n\u003e\u003e\u003e for result in results:\n...     print(result)\n...\t\texpected_results -= 1\n...\t\tif expected_results == 0:\n...\t\t\tbreak\n```\n\nThe stream does not know if or when a result might be computed and delivered, so you must count how many results you still expect to break the loop.\n\nTo use this properly you should start listening for streaming results before sending tasks, probably a separate thread:\n\n```python\n\u003e\u003e\u003e import threading\n\u003e\u003e\u003e job = crp.job(\"function Run(d) { return d; }\")\n\u003e\u003e\u003e def get_results():\n...\t\texpected_results = 10\n...     for result in job.get_results_stream():\n...             print(result)\n...\t\t\t\texpected_results -= 1\n...\t\t\t\tif expected_results == 0:\n...\t\t\t\t\tbreak\n... \n\u003e\u003e\u003e t = threading.Thread(target=get_results)\n\u003e\u003e\u003e t.start()\n\u003e\u003e\u003e job.submit_tasks(range(10))\n\u003e\u003e\u003e 7\n9\n6\n2\n3\n8\n1\n4\n0\n5\n\u003e\u003e\u003e t.join()\n```\n\nSometimes your tasks will have uncaught exceptions and those will cause a result to not be delivered, so you must account for those as well to decrease your expected_results counter.\n\n### Errors and streaming errors\n\nSometimes your tasks throw uncaught exceptions, and you should get them:\n\n```python\n\u003e\u003e\u003e program = \"\"\"\n... function Run (d) {\n...     if (d === 4) {\n...             throw new Error(\"oops, it's \"+d);\n...     } else {\n...             return d;\n...     }\n... }\n... \"\"\"\n\u003e\u003e\u003e job = crp.job(program)\n\u003e\u003e\u003e job.submit_tasks(range(10))\n\u003e\u003e\u003e list(job.get_results())\n[1, 6, 9, 8, 5, 7, 2, 3, 0] # oh no, 4 is missing...\n\u003e\u003e\u003e list(job.get_errors())\n[{u'message': u\"oops, it's 4\", u'type': u'program', u'name': u'Error', u'stack': u'Run@blob:9a4029f7-fff7-4da8-b552-92507e341749:5\\n[2]\u003c/\u003c/self.onmessage@blob:9a4029f7-fff7-4da8-b552-92507e341749:9\\n'}]\n\u003e\u003e\u003e print(json.dumps(list(job.get_errors()), sort_keys=True, indent=2)) # prettier\n[\n  {\n    \"message\": \"oops, it's 4\", \n    \"name\": \"Error\", \n    \"stack\": \"Run@blob:9a4029f7-fff7-4da8-b552-92507e341749:5\\n[2]\u003c/\u003c/self.onmessage@blob:9a4029f7-fff7-4da8-b552-92507e341749:9\\n\", \n    \"type\": \"program\"\n  }\n]\n```\n\nThe same way you get streaming results, you can (and should) get streaming errors:\n\n```python\n\u003e\u003e\u003e errors = multiply.get_errors_stream()\n\u003e\u003e\u003e for error in errors:\n...     print(error)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffabiosantoscode%2Fcrpy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffabiosantoscode%2Fcrpy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffabiosantoscode%2Fcrpy/lists"}