{"id":18765369,"url":"https://github.com/terrycojones/jsongrep","last_synced_at":"2025-09-01T18:32:00.858Z","repository":{"id":28391162,"uuid":"31905407","full_name":"terrycojones/jsongrep","owner":"terrycojones","description":"Python for extracting pieces of JSON objects","archived":false,"fork":false,"pushed_at":"2015-03-09T15:19:48.000Z","size":140,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-11-07T18:52:32.505Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/terrycojones.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-03-09T15:12:14.000Z","updated_at":"2024-05-31T03:08:49.000Z","dependencies_parsed_at":"2022-09-03T14:41:25.625Z","dependency_job_id":null,"html_url":"https://github.com/terrycojones/jsongrep","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terrycojones%2Fjsongrep","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terrycojones%2Fjsongrep/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terrycojones%2Fjsongrep/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terrycojones%2Fjsongrep/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/terrycojones","download_url":"https://codeload.github.com/terrycojones/jsongrep/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":231705070,"owners_count":18413948,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T18:33:48.714Z","updated_at":"2024-12-29T06:08:28.983Z","avatar_url":"https://github.com/terrycojones.png","language":"Python","funding_links":[],"categories":["\u003ca name=\"data-management-json\"\u003e\u003c/a\u003eData management - JSON/YAML/etc."],"sub_categories":[],"readme":"# jsongrep\n\nPython for extracting pieces of JSON objects\n\nLots of APIs these days return JSON objects. I love JSON, but reading a raw\nJSON dump can be awkward. You can print the whole thing using pprint in\nPython (or your favorite language), but what if you want to grep out and\nprint only parts of the JSON? I was thinking tonight that it would be easy\nand useful to write a recursive script to do that. It took about half an hour\nto arrive at this solution:\n\n    #!/usr/bin/env python\n\n    import sys\n    import re\n    import json\n    from pprint import pprint\n\n    def jsongrep(d, patterns):\n        try:\n            pattern = patterns.pop(0)\n        except IndexError:\n            pprint(d)\n        else:\n            if isinstance(d, dict):\n                keys = filter(pattern.match, d.keys())\n            elif isinstance(d, list):\n                keys = map(int,\n                           filter(pattern.match,\n                                  ['%d' % i for i in range(len(d))]))\n            else:\n                if pattern.match(str(d)):\n                    pprint(d)\n                return\n            for item in (d[key] for key in keys):\n                jsongrep(item, patterns[:])\n\n    if __name__ == '__main__':\n        try:\n            j = json.loads(sys.stdin.read())\n        except ValueError, e:\n            print \u003e\u003esys.stderr, 'Could not load JSON object from stdin.'\n            sys.exit(1)\n\n        jsongrep(j, map(re.compile, sys.argv[1:]))\n\nUsage is really simple. Let's look at a couple of easy examples from the command line:\n\n    $ echo '{\"hey\" : \"you\"}' | jsongrep.py hey\n    u'you'\n\n`jsongrep.py` has matched the \"hey\" key in the JSON object and printed its\nvalue. Let's do the same thing with a 2-level JSON object:\n\n    $ echo '{\"hey\" : { \"there\" : \"you\"}}' | jsongrep.py hey\n    {u'there': u'you'}\n\nAgain, we see the entire object corresponding to the \"hey\" key. We can add\nanother argument to drill down into the object\n\n    $ echo '{\"hey\" : { \"there\" : \"you\"}}' | jsongrep.py hey there\n    u'you'\n\nAs you might hope, you can use a regular expression for an argument:\n\n    $ echo '{\"hey\" : { \"there\" : \"you\"}}' | jsongrep.py 'h.*' '.*'\n    u'you'\n\nwhich in this case could have been given more concisely as\n\n    $ echo '{\"hey\" : { \"there\" : \"you\"}}' | jsongrep.py h .\n    u'you'\n\nSo you can drill down into nested dictionaries quite easily. When\n`jsongrep.py` runs out of patterns it just prints whatever's left. A special\ncase of this is if you give no patterns at all, you get the whole JSON\nobject:\n\n    $ echo '{\"hey\" : { \"there\" : \"you\"}}' | jsongrep.py\n    {u'hey': {u'there': u'you'}}\n\nThe regex patterns you pass on the command line are being matched against\nthe keys of JSON objects (Python dicts). If `jsongrep.py` runs into a list,\nit will instead match against the list indices like so:\n\n    $ echo '{\"hey\" : { \"joe\" : [\"blow\", \"xxx\" ]}}' | jsongrep.py hey joe 1\n    u'xxx'\n\nYou can see we've pulled out just the first list element after matching\n\"hey\" and \"joe\". So `jsongrep.py` regex args can be used to navigate your way\nthrough both JSON objects and lists.\n\nNow let's do something more interesting.\n\nTwitter's API can give you JSON, and the JSON is pretty chunky. For\nexample, if I get my first 100 followers with this command:\n\n    $ curl 'http://api.twitter.com/1/statuses/followers.json?screen_name=terrycojones'\n\nthere's 164Kb of output (try it and see). What if I just want the Twitter\nuser names of the people who follow me? Looking at the JSON, I can see it\nstarts with:\n\n    [{\"profile_background_color\":\"131516\",\"description\":null\n\nHmm… looks like it's a list of dictionaries. Let's print just the first dictionary in the list:\n\n    $ curl 'http://api.twitter.com/1/statuses/followers.json?screen_name=terrycojones' |\n    jsongrep.py 0\n\nwhich starts out:\n\n    {u'contributors_enabled': False,\n     u'created_at': u'Wed Jul 19 00:29:58 +0000 2006',\n     u'description': None,\n     u'favourites_count': 0,\n     u'follow_request_sent': False,\n     u'followers_count': 178,\n     u'following': False,\n     u'friends_count': 67,\n     u'geo_enabled': False,\n     u'id': 2471,\n     u'id_str': u'2471',\n     u'lang': u'en',\n     u'listed_count': 3,\n     u'location': None,\n     u'name': u'Roy',\n     u'notifications': False,\n     u'profile_background_color': u'131516',\n     u'profile_background_image_url': u'http://s.twimg.com/a/1288470193/images/themes/theme14/bg.gif',\n     u'profile_background_tile': True,\n     u'profile_image_url': u'http://a3.twimg.com/profile_images/194788727/roy_with_phone_normal.jpg',\n     u'profile_link_color': u'009999',\n     u'profile_sidebar_border_color': u'eeeeee',\n     u'profile_sidebar_fill_color': u'efefef',\n     u'profile_text_color': u'333333',\n     u'profile_use_background_image': True,\n     u'protected': False,\n     u'screen_name': u'wasroykosuge',\n\nand you can see a \"screen_name\" key in there which looks like what we want. Let's see the first few:\n\n    $ curl 'http://api.twitter.com/1/statuses/followers.json?screen_name=terrycojones' |\n    jsongrep.py . screen_name | head\n    u'wasroykosuge'\n    u'Piiiu_piiiu'\n    u'350'\n    u'KardioFit'\n    u'jrecursive'\n    u'doodlesockingad'\n    u'revinprogress'\n    u'cloudjobs'\n    u'PointGcomics'\n    u'lucasbuchala'\n\nFinally, here's an example using FluidDB‘s new /values HTTP call. I'll ask FluidDB for all \nobjects matching the query has `unionsquareventures.com/portfolio` and from those matching\nobjects I'll pull back the value of the FluidDB tag named `fluiddb/about`.\nThe result is JSON that starts out like this:\n\n    {\"results\": { \"id\" : {\"93989942-b519-49b4-87de-ac834e6a6161\": {\"fluiddb/about\": {\"value\": \"http://www.outside.in\"}}\n\nYou can see there's a 5-level deep nesting of JSON objects. I just want the \"value\" key on all matching objects. Easy:\n\n    $ curl 'http://fluiddb.fluidinfo.com/values?query=has%20unionsquareventures.com/portfolio\u0026tag=fluiddb/about' |\n    jsongrep.py results . . fluiddb/about value | sort\n    u'http://amee.cc'\n    u'http://getglue.com'\n    u'http://stackoverflow.com'\n    u'http://tumblr.com'\n    u'http://www.10gen.com'\n    u'http://www.boxee.tv'\n    u'http://www.buglabs.net'\n    u'http://www.clickable.com'\n    u'http://www.cv.im'\n    u'http://www.disqus.com'\n    u'http://www.etsy.com'\n    u'http://www.flurry.com'\n    u'http://www.foursquare.com'\n    u'http://www.heyzap.com'\n    u'http://www.indeed.com'\n    u'http://www.meetup.com'\n    u'http://www.oddcast.com'\n    u'http://www.outside.in'\n    u'http://www.returnpath.net'\n    u'http://www.shapeways.com'\n    u'http://www.simulmedia.com'\n    u'http://www.targetspot.com'\n    u'http://www.tastylabs.com'\n    u'http://www.tracked.com'\n    u'http://www.twilio.com'\n    u'http://www.twitter.com'\n    u'http://www.workmarket.com'\n    u'http://www.zemanta.com'\n    u'http://zynga.com'\n\nAnd there you have it, a sorted list of all Union Square Ventures portfolio companies,\nfrom the command line, as listed here.\n\n`jsongrep.py` will also try to match on things that are not objects or lists if it\nruns into them, so we can refine this list a little. E.g.,\n\n    $ curl 'http://fluiddb.fluidinfo.com/values?query=has%20unionsquareventures.com/portfolio\u0026tag=fluiddb/about' |\n    jsongrep.py results . . fluiddb/about value '.*ee' | sort\n    u'http://www.meetup.com'\n    u'http://amee.cc'\n    u'http://www.indeed.com'\n    u'http://www.boxee.tv'\n\nbeing the USV companies with \"ee\" somewhere in their URL. Or, for some advanced regex fun,\nthe USV companies whose URLs don't end in \".com\":\n\n    $ curl 'http://fluiddb.fluidinfo.com/values?query=has%20unionsquareventures.com/portfolio\u0026tag=fluiddb/about' |\n    jsongrep.py results . . fluiddb/about value '.*(?\u003c!\\.com)$'\n    u'http://www.outside.in'\n    u'http://amee.cc'\n    u'http://www.cv.im'\n    u'http://www.buglabs.net'\n    u'http://www.returnpath.net'\n    u'http://www.boxee.tv'\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fterrycojones%2Fjsongrep","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fterrycojones%2Fjsongrep","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fterrycojones%2Fjsongrep/lists"}