{"id":27627185,"url":"https://github.com/treasure-data/embulk-input-marketo","last_synced_at":"2025-04-23T13:53:26.028Z","repository":{"id":34082280,"uuid":"37900806","full_name":"treasure-data/embulk-input-marketo","owner":"treasure-data","description":"Embulk Input Plugin for Marketo","archived":false,"fork":false,"pushed_at":"2025-04-21T11:10:05.000Z","size":746,"stargazers_count":7,"open_issues_count":3,"forks_count":10,"subscribers_count":87,"default_branch":"master","last_synced_at":"2025-04-21T12:25:18.406Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://docs.treasuredata.com/articles/data-connector-marketo","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/treasure-data.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2015-06-23T06:14:16.000Z","updated_at":"2025-04-21T11:10:03.000Z","dependencies_parsed_at":"2024-01-18T04:37:58.416Z","dependency_job_id":"dbfcf156-907d-49ee-805a-57ee04afd35d","html_url":"https://github.com/treasure-data/embulk-input-marketo","commit_stats":null,"previous_names":[],"tags_count":41,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treasure-data%2Fembulk-input-marketo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treasure-data%2Fembulk-input-marketo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treasure-data%2Fembulk-input-marketo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treasure-data%2Fembulk-input-marketo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/treasure-data","download_url":"https://codeload.github.com/treasure-data/embulk-input-marketo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250056432,"owners_count":21367560,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-23T13:53:25.378Z","updated_at":"2025-04-23T13:53:26.026Z","avatar_url":"https://github.com/treasure-data.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/treasure-data/embulk-input-marketo.svg?branch=master)](https://travis-ci.org/treasure-data/embulk-input-marketo)\n[![Code Climate](https://codeclimate.com/github/treasure-data/embulk-input-marketo/badges/gpa.svg)](https://codeclimate.com/github/treasure-data/embulk-input-marketo)\n[![Test Coverage](https://codeclimate.com/github/treasure-data/embulk-input-marketo/badges/coverage.svg)](https://codeclimate.com/github/treasure-data/embulk-input-marketo/coverage)\n[![Gem Version](https://badge.fury.io/rb/embulk-input-marketo.svg)](http://badge.fury.io/rb/embulk-input-marketo)\n\n# Marketo input plugin for Embulk\n\nembulk-input-marketo is the gem preparing Embulk input plugins for [Marketo](http://www.marketo.com/).\n\n- Lead(lead)\n- Activity log(activity)\n- Lead by list(all_lead_with_list_id)\n- Lead by program(all_lead_with_program_id)\n- Campaign(campaign)\n- Assets Programs (program)\n- Program Members (program_members)\n\nThis plugin uses Marketo REST API.\n\n## Overview\n\nRequired Embulk version \u003e= 0.8.33 (since 0.6.0).\n\n* **Plugin type**: input\n* **Resume supported**: no\n* **Cleanup supported**: no\n* **Guess supported**: no\n\n## Install\n\n```\n$ embulk gem install embulk-input-marketo\n```\n\n## Configuration\n\n### API\n\nBelow parameters are shown in \"Admin\" \u003e \"Web Services\" page in Marketo.\n\n### Base configuration parameter\n\nAll target have this configuration parameters\n\n| name                             | required | default value | description                                                                                                                      |\n|----------------------------------|----------|---------------|----------------------------------------------------------------------------------------------------------------------------------|\n| **target**                       | true     |               | Marketo targets                                                                                                                  |\n| **account_id**                   | true     |               | Marketo Muchkin id                                                                                                               |\n| **client_id**                    | true     |               | Marketo REST client id                                                                                                           |\n| **client_secret**                | true     |               | Marketo REST client secret                                                                                                       |\n| **marketo_limit_interval_milis** | false    | 20            | Marketo have limitation of 100 calls per 20 second. If REST API calls are failed they will wait this amount of time before retry |\n| **batch_size**                   | false    | 300           | Token paging batch size. Some REST API support batch                                                                             |\n| **max_return**                   | false    | 200           | Max return for Endpoint that use offset paging                                                                                   |\n| **partner_api_key**              | false    |               | Set Marketo Partner API Key see: http://developers.marketo.com/support/Marketo_LaunchPoint_Technology_Partner_API_Key.pdf        |\n\n### Bulk extract target configuration parameter (Lead and Activity)\n\nAll bulk extract target use this configuration parameter\n\n| name                        | required | default value | description                                                                                                                   |\n|-----------------------------|----------|---------------|-------------------------------------------------------------------------------------------------------------------------------|\n| **from_date**               | true     |               | Import data since this date. Example: 2017-10-11T06:43:24+00:00                                                               |\n| **fetch_days**              | false    | 1             | Amount of days to fetch since from_date                                                                                       |\n| **polling_interval_second** | false    | 60            | Amount of time to wait between pooling job status in second                                                                   |\n| **bulk_job_timeout_second** | false    | 3600          | Amount of time to wait for bulk job to complete in second                                                                     |\n| **incremental**             | false    | true          | If incremental is set to true, next run will have from_date set to the previous to_date(calculated by from_date + fetch_days) |\n| **incremental_column**      | false    | createdAt     | Column use to filter from_date and to_date                                                                                    |\n\n\nCsv parser use this configuration parameter\n\n| name                        | required  | default value | description                                                                                                                                                                                |\n|-----------------------------|-----------|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| **escape**                  | false     | \\\"            | Escape character for bulk extract                                                                                                                                                          |\n| **quote**                   | false     | \\\"            | Quote character for bulk extract                                                                                                                                                           |\n| **quotes_in_quoted_fields** | false     | NONE          | Specify how to deal with irregular unescaped quote characters in quoted fields. Supported values: NONE, ACCEPT_ONLY_RFC4180_ESCAPED, ACCEPT_STRAY_QUOTES_ASSUMING_NO_DELIMITERS_IN_FIELDS. |\n\n### Lead\n\nLead target extract all Marketo leads, it use Marketo bulk extract feature. Configuration include bulk extract configuration. \n\n`target: lead` \n\nConfiguration:\n\n| name                | required | default value | description                                                                                                                                               |\n|---------------------|----------|---------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|\n| **use_updated_at**  | false    | false         | Support filter with `updateAt` column, but not all Marketo Account have the feature to filter by updatedAt, updatedAt don't support incremental ingestion |\n| **included_fields** | false    | null         | List of lead fields to included in export request sent to Marketo, can be used to reduce the size of BulkExtract file                                     |\n\nSchema type: Dynamic via describe lead endpoint.\n\nIncremental support: yes\n\nRange ingestion: yes\n\n### Activity\n\nActivity target extract all Marketo  activity log. Configuration include all bulk extract configuration\n\n`target: activity`\n\nSchema type: Static schema\n\nIncremental support: yes\n\nRange ingestion: yes\n\nFilter by specific activity type ids: yes. See [#95](https://github.com/treasure-data/embulk-input-marketo/issues/95)\n\n### Campaign\n\nCampaign extract all campaign data from Marketo\n\n`target: campaign`\n\nSchema type: Static schema\n\nIncremental support: no\n\nRange ingestion: no\n\n### Lead by list\n\nExtract all Lead data including lead's list id\n\n`target: all_lead_with_list_id`\n\nConfiguration:\n\n| name                | required | default value | description                                                                                                     |\n|---------------------|----------|---------------|-----------------------------------------------------------------------------------------------------------------|\n| **included_fields** | false    | null          | List of lead fields to included in export request sent to Marketo, can be used to reduce request, response size |\n| **list_ids**        | false    | null          | Import Leads by specified Lists_ID. If not specified will import all Leads by all List IDs                      |\n\nSchema type: Dynamic via describe leads. Schema will have 1 addition column name listId that contain the id of the list the lead belong to\n\nIncremental support: no\n\nRange ingestion: no\n\n### Lead by program\n\nExtract all Lead data including lead's program id\n\n`target: all_lead_with_program_id`\n\nConfiguration:\n\n| name                | required | default value | description                                                                                                           |\n|---------------------|----------|---------------|-----------------------------------------------------------------------------------------------------------------------|\n| **included_fields** | false    | null          | List of lead fields to included in export request sent to Marketo, can be used to reduce request, response size       |\n| **program_ids**     | false    | null          | Import Members by specified Program_ID (comma-separated). If not specified will import all Members by all Program IDs |\n\nSchema type: Dynamic via describe leads. Schema will have 1 addition column name listId that contain the id of the list the lead belong to\n\nIncremental support: no\n\nRange ingestion: no\n\n### Assets programs\n\nGet Assets Programs by Query Tag type, Date range or all if no query by specified.\n\n`target: program`\n\nConfiguration:\n\n| name                        | required | default value | description                                                                                                                                                  |\n|-----------------------------|----------|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| **query_by**                | false    |   null        | Get assets programs by query, supported values `date_range`, `tag_type` leave unset to fetch all programs                                                    |\n| **earliest_updated_at**     | false    |   null        | Required if query by `date_range` is selected. Exclude programs prior to this date. Must be valid ISO-8601 string                                            |\n| **latest_updated_at**       | false    |   null        | Required if query by `date_range` is selected. Exclude programs after this date. Must be valid ISO-8601 string                                               |\n| **filter_type**             | false    |   null        | Optional value send with query by `date_range` is selected to filter out the result from Marketo. Supported values `id`, `programId`, `folderId`, `workspace`|\n| **filter_values**           | false    |   null        | Set the values associated with `filter_type`                                                                                                                 |\n| **tag_type**                | false    |   null        | Required if query by `tag_type` is selected. Type of program tag                                                                                             |\n| **tag_value**               | false    |   null        | Required if query by `tag_type` is selected. Value of the tag                                                                                                |\n| **report_duration**         | false    |   null        | Amount of milliseconds to fetch from `earliest_updated_at`. If `incremental = true` this value will automatically calculated for the first run by `latest_updated_at` - `earliest_updated_at` |\n| **incremental**             | false    |   true        | If incremental is set to true, next run will have `earliest_updated_at` set to the previous `latest_updated_at` + `report_duration`. Incremental import only support by query `date_range`     |\n\nSchema type: Static schema\n\nIncremental support: yes (Query by `date_range` only)\n\nRange ingestion: yes\n\n`target: all_lead_with_program_id`\n\nConfiguration:\n\n| name                                  | required | default value | description                                                                                                           |\n|---------------------------------------|----------|---------------|-----------------------------------------------------------------------------------------------------------------------|\n| **custom_object_api_name**            | true     | null          | The API name of the custom object                                                                                     |\n| **custom_object_fields**              | false    | null          | Comma separated API name of fields of the custom object (Optional)                                                    |\n| **custom_object_filter_type**         | true     | null          | Field to search on Valid values are: dedupeFields, idFields, and any field defined in searchableFields attribute of Describe endpoint. Default is dedupeFields |\n| **custom_object_filter_values**       | false    | null          | Comma-separated list of field values to match.                                                                        |\n| **custom_object_filter_from_value**   | false    | null          | Filter Marketo Custom Object has value greater than this value                                                        |\n| **custom_object_filter_to_value**     | false    | null          | Filter Marketo Custom Object has value smaller than this value. If not set, only records that have value greater than \"From Value\" will be returned. Job will stop if no record found in 300 consecutive value. |\n\nSchema type: dynamic schema\nIncremental support: no \n\n### Program Members configuration parameter\n\nGet Members by Program Ids or All Program.\n\n| name                | required | default value | description                                                                                                           |\n|---------------------|----------|---------------|-----------------------------------------------------------------------------------------------------------------------|\n| **program_ids**     | false    | null          | Import Members by specified Program_ID (comma-separated). If not specified will import all Members by all Program IDs |\n\n\n## Example\n\nFor lead, you have `partial-config.yml` like below:\n\n```yaml\nin:\n  type: marketo\n  target: lead\n  account_id: ACCOUNT_ID\n  client_id: CLIENT_ID\n  client_secret: CLIENT_SECRET\n  from_date: 2017-09-01\n  fetch_days: 1\nout:\n  type: stdout\n```\n\nYou can run `embulk guess partial-config.yml -o lead-config.yml` and got `lead-config.yml`. `lead-config.yml` includes a schema for Lead.\n\nNext, you can run `embulk preview lead-config.yml` for preview and `embulk run lead-config.yml` for run.\n\nExample of Assets Programs config\n```yaml\nin:\n  account_id: ACCOUNT_ID\n  client_id: CLIENT_ID\n  client_secret: CLIENT_SECRET\n  target: program\n  type: marketo\n  query_by: date_range\n  filter_type: folderId\n  filter_values: \n   - 2598\n   - 1001\n  earliest_updated_at: 2018-08-20T00:00:00.000Z\n  latest_updated_at: 2018-08-31T00:00:00.000Z\n  incremental: true\n  ```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftreasure-data%2Fembulk-input-marketo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftreasure-data%2Fembulk-input-marketo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftreasure-data%2Fembulk-input-marketo/lists"}