{"id":17043497,"url":"https://github.com/rhsimplex/commoncrawljob","last_synced_at":"2025-03-23T02:23:44.443Z","repository":{"id":77394891,"uuid":"55368846","full_name":"rhsimplex/CommonCrawlJob","owner":"rhsimplex","description":"Extract data from common crawl using elastic map reduce","archived":false,"fork":false,"pushed_at":"2016-04-03T21:05:34.000Z","size":125,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-28T05:27:25.533Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rhsimplex.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-04-03T21:02:39.000Z","updated_at":"2023-01-03T17:56:50.000Z","dependencies_parsed_at":null,"dependency_job_id":"b004bb93-8b62-4d3a-bc3e-644abd078a91","html_url":"https://github.com/rhsimplex/CommonCrawlJob","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rhsimplex%2FCommonCrawlJob","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rhsimplex%2FCommonCrawlJob/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rhsimplex%2FCommonCrawlJob/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rhsimplex%2FCommonCrawlJob/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rhsimplex","download_url":"https://codeload.github.com/rhsimplex/CommonCrawlJob/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245045691,"owners_count":20552079,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-14T09:29:41.427Z","updated_at":"2025-03-23T02:23:44.409Z","avatar_url":"https://github.com/rhsimplex.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Common Crawl Data Extraction\n============================\n\nExtract data from common crawl using elastic map reduce\n\n    Note: This project uses Python 2.7.11\n\nCommonCrawlJob is a framework which wraps the ``MRJob`` hadoop library for streaming\nanalytics over internet scale data.\n\nFor more information on using `MRJob`_ framework.\n\nSetup\n-----\n\nTo develop locally, you will need to install the ``mrjob`` Hadoop\nstreaming framework library, and the ``boto`` library for accessing amazon cloud\npublic dataset resources.\n\nUse pip to install these libraries.\n\n.. code:: sh\n\n    $ pip install CommonCrawlJob\n\nGetting Started\n---------------\n\nTo first get started, we are going to create a Google Analytics extractor. We will go from start to\nfinish in creating a Common Crawl extractor that uses regular expression capture groups to extract\ngoogle analytics tracker id's.\n\nFirst let's create a file ``GoogleAnalytics.py``.\n\n.. code:: sh\n\n   $ touch GoogleAnalytics.py\n\nUsing a text editor, write to this file\n\n.. code:: python\n\n    import re\n\n    from ccjob import CommonCrawl\n\n    class GATagJob(CommonCrawl):\n\n        def process_record(self, body):\n            # Regular Expression for Google Analytics Tracker\n            pat = re.compile(r\"[\\\"\\']UA-(\\d+)-(\\d)+[\\'\\\"]\")\n\n            for match in pat.finditer(body):\n                if match:\n                    yield match.groups()[0]\n\n            self.increment_counter('commoncrawl', 'processed_document', 1)\n\n\n    if __name__ == '__main__':\n        GATagJob.run()\n\nOur ``GATagJob`` class has one method ``process_record`` taking in one argument containing\nthe body of a HTML file and yields the results matching our regular expression.\n\nAll common crawl jobs will generally obey this pattern.\n\nTesting Locally\n---------------\n\nRun the Google Analytics extractor locally to test your script.\n\n.. code:: sh\n\n    $ python GoogleAnalytics.py -r local \u003c(tail -n 1 data/latest.txt)\n\n\nRegion Configuration\n--------------------\n\nFor best performance, you should launch the cluster in the same region\nas your data. Currently data from `aws-publicdatasets`_ are stored in\n``us-east-1``, which is where you want to point your EMR cluster.\n\nCommon Crawl Region\n-------------------\n:S3: US Standard\n:EMR: US East (N. Virginia)\n:API: ``us-east-1``\n\nCreate an Amazon EC2 Key Pair and PEM File\n------------------------------------------\n\nAmazon EMR uses an Amazon Elastic Compute Cloud (Amazon EC2) key pair\nto ensure that you alone have access to the instances that you launch.\n\nThe PEM file associated with this key pair is required to ssh directly to the master node of the cluster.\n\nTo create an Amazon EC2 key pair:\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n1. Go to the Amazon EC2 console\n2. In the Navigation pane, click Key Pairs\n3. On the Key Pairs page, click Create Key Pair\n4. In the Create Key Pair dialog box, enter a name for your key pair, such as, mykeypair\n5. Click Create\n6. Save the resulting PEM file in a safe location\n\nConfiguring ``mrjob.conf``\n--------------------------\n\nMake sure to download an EC2 Key Pair ``pem`` file for your map reduce\njob and add it to the ``ec2_key_pair`` and ``ec2_key_pair_file``\nvariables.\n\nMake sure that the ``PEM`` file has permissions set properly by running\n\n.. code:: sh\n\n    $ chown 600 $MY_PEM_FILE\n\nDownload the latest version of python to send to your EMR instances.\n\n.. code:: sh\n\n   $ wget https://www.python.org/ftp/python/2.7.11/Python-2.7.11.tgz\n\nCreate a ``mrjob.conf`` file to set up your configuration parameters to match\nthat of AWS.\n\nThere is a default configuration template located at ``mrjob.conf.template`` that you can use.\n\n.. code:: yaml\n\n    runners:\n      emr:\n        aws_region: 'us-east-1'\n        aws_access_key_id: \u003cYour AWS_ACCESS_KEY_ID\u003e\n        aws_secret_access_key: \u003cYour AWS_SECRET_ACCESS_KEY\u003e\n        cmdenv:\n            AWS_ACCESS_KEY_ID: \u003cYour AWS_ACCESS_KEY_ID\u003e\n            AWS_SECRET_ACCESS_KEY: \u003cYour AWS_SECRET_ACCESS_KEY\u003e\n        ec2_key_pair: \u003cPath to your PEM file\u003e\n        ec2_key_pair_file: \u003cName of the Key\u003e\n        ssh_tunnel_to_job_tracker: true\n        ec2_instance_type: 'm1.xlarge'\n        ec2_master_instance_type: 'm1.xlarge'\n        emr_tags:\n            name: '\u003cYour Project Name\u003e'\n        num_ec2_instances: 12\n        ami_version: '2.4.10'\n        python_bin: python2.7\n        interpreter: python2.7\n        bootstrap_action:\n            - s3://elasticmapreduce/bootstrap-actions/install-ganglia\n        upload_files:\n            - CommonCrawl.py\n        bootstrap:\n            - tar xfz Python-2.7.11.tgz#\n            - cd Python-2.7.11\n            - ./configure \u0026\u0026 make \u0026\u0026 sudo make install\n            - sudo python2.7 get-pip.py#\n            - sudo pip2 install --upgrade pip setuptools wheel\n            - sudo pip2 install -r requirements.txt#\n\nRun on Amazon Elastic MapReduce\n-------------------------------\n\nFirst copy the ``mrjob.conf.template`` into ``mrjob.conf``\n\nNote: \u003e Make sure to fill out the necessary AWS credentials with your\ninformation\n\n.. code:: sh\n\n    $ python GoogleAnalytics.py -r emr \\\n                                --conf-path=\"mrjob.conf\" \\\n                                --output-dir=\"s3n://$S3_OUTPUT_BUCKET\" \\\n                               data/arcindex.txt\n\n\n.. _MRJob: https://pythonhosted.org/mrjob/\n.. _aws-publicdatasets: https://aws.amazon.com/public-data-sets/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frhsimplex%2Fcommoncrawljob","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frhsimplex%2Fcommoncrawljob","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frhsimplex%2Fcommoncrawljob/lists"}