{"id":18783919,"url":"https://github.com/sparkpost/httpdump","last_synced_at":"2025-07-30T23:02:29.904Z","repository":{"id":57484559,"uuid":"44572381","full_name":"SparkPost/httpdump","owner":"SparkPost","description":"Store HTTP request data, and provide a simple framework for post-processing in parallel.","archived":false,"fork":false,"pushed_at":"2016-11-02T15:29:11.000Z","size":25,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-05T23:04:49.876Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SparkPost.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-10-20T00:34:36.000Z","updated_at":"2020-08-19T03:55:06.000Z","dependencies_parsed_at":"2022-08-26T14:22:42.290Z","dependency_job_id":null,"html_url":"https://github.com/SparkPost/httpdump","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/SparkPost/httpdump","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SparkPost%2Fhttpdump","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SparkPost%2Fhttpdump/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SparkPost%2Fhttpdump/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SparkPost%2Fhttpdump/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SparkPost","download_url":"https://codeload.github.com/SparkPost/httpdump/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SparkPost%2Fhttpdump/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267959602,"owners_count":24172431,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-30T02:00:09.044Z","response_time":70,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T20:41:10.367Z","updated_at":"2025-07-30T23:02:29.855Z","avatar_url":"https://github.com/SparkPost.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# httpdump\n\nStore HTTP request data, and provide a simple framework for post-processing in parallel.\n\n### Example code\n\nThe program in the `output/loggly` directory is provided as an example implementation.\n\n### Command-line parameters\n\nThe example program accepts two command line parameters and starts up an HTTP server on the specified port.\n\n**-port** (default 80) listen for http requests on this port  \n**-batch-interval** (default 10) how often to process stored requests, in seconds  \n\n### Environment variables\n\n**LOGGLY_TOKEN**  \nYour Loggly API key. This uuid tells Loggly what account should store the incoming data. This is required, and there is no default.\n\n**POSTGRESQL_DB**  \nDatabase within local PostgreSQL instance where event data will be stored. Defaults to the database named after your user, which may not have been automatically created.\n\n**POSTGRESQL_SCHEMA**  \nSchema within local PostgreSQL database where event data will be stored. Defaults to `request_dump`, and will be automatically created if it doesn't exist.\n\n**POSTGRESQL_USER**  \nUser to connect to PostgreSQL as. This defaults to the OS-level username, which will usually have had an account automatically created.\n\n**POSTGRESQL_PASS**  \nPassword for user. This is usually not required for connections to a local db. See [pg_hba.conf] (http://www.postgresql.org/docs/9.4/static/auth-pg-hba-conf.html \"host based auth\") on your system to enable/disable password-less logins, if the defaults aren't working for you.\n\n### Example\n\nHere's a server example, which will listen for incoming HTTP requests on port `12345`, store the requests in `postgres.request_dump` (database.schema) in PostgreSQL. Replace `LOGGLY_TOKEN` with the relevant API key.\n\n```\n$ export LOGGLY_TOKEN=ffffffff-ffff-ffff-ffff-ffffffffffff\n$ go build output/loggly/loggly.go\n$ POSTGRESQL_DB=postgres ./loggly -port 12345\n```\n\nAnd here's an example of `POST`ing to that server using `cURL`, sending the contents of the file `test.json`:\n\n```\n$ echo '{\"abc\":123,\"def\":456}' \u003e ./test.json\n$ curl -XPOST -H 'Content-Type: application/json' --data @test.json http://127.0.0.1:12345/\n```\n\nOnce you've `POST`ed some data, wait a couple seconds (max 10, by default), and you'll see something like this for a successful batch upload:\n\n```\n2015/10/27 11:23:40 loggly.go:62: Sent 854 bytes with status 200 OK\n```\n\nIf there's an error, you'll see a dump of your request and Loggly's response. It will look something like this error, which is what shows up when a Loggly API key isn't set, and the local error checking is disabled:\n\n```\n2015/10/27 11:17:32 loggly.go:57: POST /bulk/tag/bulk/ HTTP/1.1\nHost: logs-01.loggly.com\nUser-Agent: Go-http-client/1.1\nContent-Length: 854\nAccept-Encoding: gzip\n\nPOST / HTTP/1.1\\nHost: 127.0.0.1:12345\\nAccept: */*\\nContent-Type: application/json\\nUser-Agent: curl/7.43.0\\n\\n{ \"ip_address\": \"12.34.56.78\", \"template_version\": \"2\", \"recv_method\": \"rest\", \"transmission_id\": \"30110032678944751\", \"rcpt_to\": \"circ-test-template@nowhere.sinkhole.msyscloud.com\", \"num_retries\": \"0\", \"delv_method\": \"esmtp\", \"campaign_id\": \"msys-test:api_transmissions_post_template\", \"rcpt_meta\": { \"binding\": \"test\" }, \"timestamp\": \"2015-10-22T22:25:04.000-07:00\", \"friendly_from\": \"circonus-probe@test.sinkhole.msyscloud.com\", \"message_id\": \"0001308903569a26297f\", \"template_id\": \"circonus-probe\", \"msg_size\": \"2452\", \"binding_group\": \"test\", \"queue_time\": \"162\", \"routing_domain\": \"nowhere.sinkhole.msyscloud.com\", \"msg_from\": \"bounces-template@test.sinkhole.msyscloud.com\", \"type\": \"delivery\", \"binding\": \"test21109\", \"rcpt_tags\": []},\n\nHTTP/1.1 403 Forbidden\nContent-Length: 15\nConnection: keep-alive\nContent-Type: text/html\nDate: Tue, 27 Oct 2015 17:17:32 GMT\nServer: nginx/1.1.19\n\nInvalid API key\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsparkpost%2Fhttpdump","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsparkpost%2Fhttpdump","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsparkpost%2Fhttpdump/lists"}