{"id":26375527,"url":"https://github.com/consento-org/dat-ssg","last_synced_at":"2026-01-03T22:12:31.468Z","repository":{"id":123732507,"uuid":"244580573","full_name":"consento-org/dat-ssg","owner":"consento-org","description":"A command line tool that uses configuration in a folder to download websites with wget and publishes the content.","archived":false,"fork":false,"pushed_at":"2020-04-06T07:28:26.000Z","size":164,"stargazers_count":8,"open_issues_count":3,"forks_count":0,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-04-22T09:15:26.416Z","etag":null,"topics":["cli","dat","static-site"],"latest_commit_sha":null,"homepage":null,"language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/consento-org.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null},"funding":{"open_collective":"consento"}},"created_at":"2020-03-03T08:24:58.000Z","updated_at":"2024-04-26T10:22:08.127Z","dependencies_parsed_at":null,"dependency_job_id":"bc86f76b-f968-45c4-89fe-be0c27d38443","html_url":"https://github.com/consento-org/dat-ssg","commit_stats":{"total_commits":75,"total_committers":1,"mean_commits":75.0,"dds":0.0,"last_synced_commit":"4f071d391061186f61f17e8d4ec121757a5648bf"},"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/consento-org%2Fdat-ssg","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/consento-org%2Fdat-ssg/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/consento-org%2Fdat-ssg/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/consento-org%2Fdat-ssg/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/consento-org","download_url":"https://codeload.github.com/consento-org/dat-ssg/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243867121,"owners_count":20360702,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","dat","static-site"],"created_at":"2025-03-17T02:17:39.993Z","updated_at":"2026-01-03T22:12:26.411Z","avatar_url":"https://github.com/consento-org.png","language":"HTML","readme":"# dat-ssg\n\n`dat-ssg` is a command line tool that uses configuration in a folder to download websites with `wget` and publish the content to:\n\n- [`netlify`](https://www.netlify.com/)\n- [`dat`](https://dat.foundation/)\n- ... (your Pull Request)\n\n## Why?\n\nPrior to `dat-ssg`, to host a website on dat you can either edit it by hand or setup a static site generator and a CI system\nto do the publishing. This, however, prevents the use of any cms. Be it [Ghost](https://ghost.org/), [Wordpress](https://wordpress.com/)\n, [Drupal](https://www.drupal.com/) or any other dynamic publishing system. With `dat-ssg` you can finally start using \nthose systems and host the site to be accessible through `dat`.\n\nIt is also quite useful if you want to use cheaper static-site hosting that deploys well in a global cdn.\n\n## Introduction\n\nFor `dat-ssg` you need to have two folders: the _\"configuration-folder\" and the _\"work-folder\"_.\n\n- The _configuration-folder_ can have a list of configuration files, each specifying a download process of files.\n- The _work-folder_ will be holding a folder for every file in the configuration folder and the `.out` and `.error`\n    files for the process of each configuration.\n\n`dat-ssg` needs [`wget`](https://www.gnu.org/software/wget/) and [`git`](https://git-scm.com/) available in the command line.\n`wget` is used to download the site, `git` is used to only trigger a publication if the site has changed.\n\n`dat-ssg` will rewrite the downloaded files (e.g. the meta tags and links) to look correct when hosted on the target.\n\n## Installation\n\n`dat-ssg` is a node application and can be run quickly using\n\n```sh\n$ npx dat-ssg --work-folder ./work --configuration-folder ./config\n```\n\nThe process will look for any changes in the configuruation folder and start a process for each file.\n\n## Configuration files\n\nFor configuration you can simply add a file with the ending `.js` to the configuration folder.\n\nThe file needs to export a javascript object looking like this:\n\n```javascript\nmodule.exports = {\n  title: \"My Site\", // Title to be used in the process, e.g. for the dat config file\n  description: \"This is my homepage\", // Description used for the process, e.g. for the dat config file\n  domain: \"mydomain.com\", // domain to download data from\n  https: true, // The domain support https\n  newDomain: \"https://targetdomain.org\", // domain of the new site (needed for content-rewrites)\n  /*\n   * wget will download all files that are found at the domain's root but there may be extra roots to look for content.\n   * You need to specify all the roots \n   */\n  roots: [\n    \"/\",\n    \"/b.html\",\n    \"/c.html\"\n  ],\n  notFound: '/404/', // Adds '/404/ to the roots and adds 404 configuration to the deployments (netlify, dat,...)\n  // Interval to look for updates\n  update: 1000 * 60 * 20, // Update every 10 minutes\n  // The process will use following user name to make commits\n  git: {\n    email: \"my@domain.com\",\n    name: \"The Name\"\n  },\n  // If you want to publish the page to netlify you need to specify this property, else you can skip it!\n  netlify: {\n    siteId: \"aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa\", // Id, provided in the netlify setup\n    production: false, // Publish as production = true, else = false\n    /*\n     * Auth token as provided through the netlify user interface, can be also loaded through `process.env`\n     */\n    authToken: \"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\n    config: `# Optional configuration, see https://docs.netlify.com/configure-builds/file-based-configuration/#sample-file`\n  },\n  dat: {\n    // Time to push after changes, in which a hosting service can connect and download the latest update.\n    pushTime: 1000 * 60 * 2 // 2 minutes\n  }\n}\n```\n\nEvery time you change the file, the process will be restarted and a deploy will be triggered.\n\n## Caching notes\n\nSome CMS add query parameters to javascript or css links, e.g.: `assets/main/css/main.css?v=25be6de7e6`\nThese files will be transformed with wget to: `assets/main/css/main.css@v=25be6de7e6.css`.\n\nYou may want to specify the cache settings in `netlify.config` to cache the files \"forever\":\n\n```toml\n[[headers]]\n  for = \"/*@v=*\"\n  [headers.values]\n    cache-control = \"public, max-age=31536000\"\n```\n\n## DAT notes\n\nThe first time a configuration is fetched will create a new dat in the workfolder. This new dat\nwill have a new link, so every time you delete the folder, you will need to update replication\nlinks and the links you used in your dat-dns settings.\n\n## Funding\n\n\u003ca href=\"https://ec.europa.eu/info/\" target=\"_blank\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/consento-org/mobile/master/assets/icon/eu@2x.png\" width=\"66\"\u003e\u003c/a\u003e\n\u003ca href=\"https://www.ngi.eu/about/ledger/\" target=\"_blank\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/consento-org/mobile/master/assets/icon/ngi-ledger@2x.png\" width=\"189\"\u003e\u003c/a\u003e\n\nThis project has received funding from the European Union’s [Horizon 2020](https://ec.europa.eu/programmes/horizon2020/) research and innovation programme within the framework of the [LEDGER Project](https://www.ngi.eu/about/ledger/) funded under grant agreement [No825268](https://cordis.europa.eu/project/id/825268).\n\n## License\n\n[MIT](./LICENSE)\n\n","funding_links":["https://opencollective.com/consento"],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconsento-org%2Fdat-ssg","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fconsento-org%2Fdat-ssg","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconsento-org%2Fdat-ssg/lists"}