{"id":24458613,"url":"https://github.com/headwirecom/franklin-importer-tools","last_synced_at":"2025-03-14T08:43:45.341Z","repository":{"id":187972167,"uuid":"677888734","full_name":"headwirecom/franklin-importer-tools","owner":"headwirecom","description":"Content migration tools for AEM Franklin","archived":false,"fork":false,"pushed_at":"2024-01-25T18:12:23.000Z","size":68,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-01-21T03:14:55.676Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/headwirecom.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-08-13T01:37:34.000Z","updated_at":"2023-08-13T01:37:39.000Z","dependencies_parsed_at":"2024-01-25T19:45:09.408Z","dependency_job_id":null,"html_url":"https://github.com/headwirecom/franklin-importer-tools","commit_stats":null,"previous_names":["headwirecom/franklin-importer-tools"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/headwirecom%2Ffranklin-importer-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/headwirecom%2Ffranklin-importer-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/headwirecom%2Ffranklin-importer-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/headwirecom%2Ffranklin-importer-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/headwirecom","download_url":"https://codeload.github.com/headwirecom/franklin-importer-tools/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243551234,"owners_count":20309290,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-21T03:15:11.349Z","updated_at":"2025-03-14T08:43:45.306Z","avatar_url":"https://github.com/headwirecom.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# franklin-importer-tools\nContent migration tools for AEM Franklin\n\n## Getting Started\n\nClone this project and run `npm install`.\n\n### Supported Operations - Get Help \n\nGet URLs (from sitemap) to import:\n```\n./index.js urls --help\n```\n\nDocument import:\n```\n./index.js import --help\n```\n\nGoogle Drive upload:\n```\n./index.js upload --help\n```\n\nPublishing and Indexing (via Franklin Admin API)\n```\n./index.js publish --help\n```   \n\n## Get URLs\n\n```\n./index.js urls --help\n```\n\nExample - get a list of URLs and save to a file as JSON:\n```\n./index.js urls -s https://www.golfdigest.com/sitemaps/sitemap_golfdigest_index.xml -o urls.json\n```\n\nExample - project specific url mapping\n```\n./index.js urls -s https://www.golfdigest.com/sitemaps/sitemap_golfdigest_index.xml --mappingScript longurlmapping.js\n```\n\nExample - using lastmod date\n```\n./index.js urls -s https://www.golfdigest.com/sitemaps/sitemap_golfdigest_index.xml -t 2023-10-01T09:00:00-00:00\n```\n\n### URL mapping script example - extracting long path from data attribute\n```javascript\nimport fetch from '@adobe/node-fetch-retry';\nimport { JSDOM } from 'jsdom';\n\nasync function parseHTML(text) {\n    const jsdom = new JSDOM(text);\n    return jsdom.window.document;\n}\n\nasync function fetchDocument(url) {\n    const resp = await fetch(url);\n    if (resp.ok) {\n      const text = await resp.text();\n      return await parseHTML(text);\n    } else {\n      console.log(`Unable to fetch ${url}. Response status: ${resp.status}`);\n    }\n}\n  \nasync function fetchLongPath(url) {\n    // console.log(`fetching long path from ${url}`);\n    const doc = await fetchDocument(url);\n    if (doc) {\n        return doc.body.getAttribute('data-page-path');\n    }\n}\n\nconst map = async (orgUrl, params) =\u003e {\n    const longPath = await fetchLongPath(orgUrl); \n    if (longPath) {\n        try {\n            const url = new URL(orgUrl);\n            return `${url.protocol}//${url.hostname}${longPath}`\n        } catch (err) {\n            console.log(`Unable to get long path for ${orgUrl}. ${err.message}`);\n        }\n    }\n    return orgUrl;\n}\n\nexport {\n    map\n}\n```\n\n## Import Usage and Examples\n\n```\n./index.js import --help\n```\n\n### To run an import\nSimple example:\n```\n./index.js import --urls data/test-urls.json --ts scripts/import.mjs\n```\nChange `data/test-urls.json` to a path of the file containing JSON array of URLs to import. Change `scripts/import.js` to project specifc transformation script. \nSee [Franklin Importer UI](https://github.com/adobe/helix-importer-ui) project and [Importer Guidelines](https://github.com/adobe/helix-importer-ui/blob/main/importer-guidelines.md)\nfor more information on transformation script.\n\nBy default imported documents and import report will be stored under `docs` subfolder. You can change this as follows.\n```\n./index.js import --urls data/test-urls.json --ts scripts/import.mjs --target /myImportedDocs\n```\n\nYou can pass parameters to transformation script (params argument in your transformation script method).\n```\n./index.js import --urls data/test-urls.json --ts scripts/import.mjs --target /myImportedDocs --params data/params.json\n```\n\nNote that both ```--urls``` and ```--params``` except a valid JSON string as input insead of a file path or URL (JSON array for ```--urls```).\n```\n./index.js import --urls '[\"https://www.golfdigest.com/story/british-open-2023-sleepers\"]' --ts scripts/import.mjs --target /myImportedDocs --params '{ \"param1\": \"banana\", \"param2\": \"apple\" }'\n```\n\n#### Notes on transformation script\nNote that in order to be loaded by Node module system the transformation script file has a `.mjs` extension. \nAlso, for compatibility with both this CLI tool and [Franklin Importer UI](https://github.com/adobe/helix-importer-ui) runing in the browser it needs to have both exports bellow.\n```javascript\n// export compatible with node\nexport {\n  preprocess,\n  transform\n}\n\n// export compatible with browser but breaks with node\nexport default {\n  preprocess: preprocess,\n  transform: transform\n}\n```\n\n## Google Drive Authentication (credentials.json)\n\nThis applies to upload and publish commands which require access to files and folders on Google Drive. \n\n### Create credentials.json file \n\n1. Go to the [Google Developers Console](https://console.cloud.google.com).\n2. Create a new project or select an existing one.\n3. Enable the Google Drive API for your project.\n4. Create credentials (OAuth 2.0 client ID) for your project.\n5. Download the JSON credentials file, and save it as ```credentials.json``` in your local file system.\n\n### Login\n\nUse  ```--credentials``` or ```-c``` argument to pass  ```credentials.json``` location to ```upload``` or ```publish``` commands. The first time you run one of these commands the program will open a browser window with Google Authentication prompt. Login as Google user that has access to Google Drive folder where your Franklin project content will be stored. Authentication token will saved in ```token.json``` file and used authenticate next time you run one of these commands. Delete token.json if you need to refresh the token or login as different user. \n\n## Upload Usage and Examples\n\nGoogle Drive upload:\n\n```\n./index.js upload --help\n```\n\nUpload to an empty folder on google drive:\n```\n./index.js upload -t 1HyaaV7_cFS4O0rHm2Zk2KChUOjjjj -s ./myImportedDocs -c credentials.json\n```\n\nUpload to a folder on google drive and overwrite files with the same name:\n```\n./index.js upload -t 1HyaaV7_cFS4O0rHm2Zk2KChUOjjjj -s ./myImportedDocs -c credentials.json -mode overwrite\n```\n\nUpload to a folder on google drive and overwrite files with the same name only if a local copy is newer:\n```\n./index.js upload -t 1HyaaV7_cFS4O0rHm2Zk2KChUOjjjj -s ./myImportedDocs -c credentials.json -mode overwriteOlder\n```\n\nUpload to a folder on google drive and overwrite files with the same name only if a local copy is newer:\n```\n./index.js upload -t 1HyaaV7_cFS4O0rHm2Zk2KChUOjjjj -s ./myImportedDocs -c credentials.json -mode overwriteOlder\n```\n\nUpload to a folder on google drive only if the file with the same name does not exist on google drive:\n```\n./index.js upload -t 1HyaaV7_cFS4O0rHm2Zk2KChUOjjjj -s ./myImportedDocs -c credentials.json -mode keepRemote\n```\n\nEstimate total number of files and total size of the upload:\n```\n./index.js upload -t 1HyaaV7_cFS4O0rHm2Zk2KChUOjjjj -s ./myImportedDocs -c credentials.json -mode scanonly\n```\n\nEstimate total number of files and total size of the upload, and print current file listing from Google Drive:\n```\n./index.js upload -t 1HyaaV7_cFS4O0rHm2Zk2KChUOjjjj -s ./myImportedDocs -c credentials.json -mode scanonly -p\n```\n\nStart a folder listener and upload files when they are created/modified:\n```\n./index.js upload -t 1HyaaV7_cFS4O0rHm2Zk2KChUOjjjj -s ./myImportedDocs -c credentials.json -mode overwrite -l\n```\n\n## Publishing Usage and Examples\n\n```\n./index.js publish --help\n``` \n\nPublish to preview\n```\n./index.js publish -t 1HyaaV7_cFS4O0rHm2Zk2KChUOjjjj --op preview -h main--helix-sportsmagazine--headwirecom.hlx\n``` \n\nPublish live\n```\n./index.js publish -t 1HyaaV7_cFS4O0rHm2Zk2KChUOjjjj --op live -h main--helix-sportsmagazine--headwirecom.hlx\n``` \n\nIndex\n```\n./index.js publish -t 1HyaaV7_cFS4O0rHm2Zk2KChUOjjjj --op index -h main--helix-sportsmagazine--headwirecom.hlx\n``` \n\nGet current index\n```\n./index.js publish -t 1HyaaV7_cFS4O0rHm2Zk2KChUOjjjj --op index -m GET -h main--helix-sportsmagazine--headwirecom.hlx\n``` ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fheadwirecom%2Ffranklin-importer-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fheadwirecom%2Ffranklin-importer-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fheadwirecom%2Ffranklin-importer-tools/lists"}