{"id":20492131,"url":"https://github.com/zotonic/mod_import_anymeta","last_synced_at":"2025-04-13T17:03:41.553Z","repository":{"id":2798125,"uuid":"3798567","full_name":"zotonic/mod_import_anymeta","owner":"zotonic","description":"Import an Anymeta site into Zotonic","archived":false,"fork":false,"pushed_at":"2017-02-22T14:15:58.000Z","size":334,"stargazers_count":3,"open_issues_count":2,"forks_count":4,"subscribers_count":12,"default_branch":"master","last_synced_at":"2024-03-26T09:58:51.184Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Erlang","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zotonic.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-03-22T14:30:19.000Z","updated_at":"2024-03-26T09:58:51.187Z","dependencies_parsed_at":"2022-09-04T10:03:50.014Z","dependency_job_id":null,"html_url":"https://github.com/zotonic/mod_import_anymeta","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zotonic%2Fmod_import_anymeta","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zotonic%2Fmod_import_anymeta/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zotonic%2Fmod_import_anymeta/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zotonic%2Fmod_import_anymeta/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zotonic","download_url":"https://codeload.github.com/zotonic/mod_import_anymeta/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224819098,"owners_count":17375218,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-15T17:27:47.944Z","updated_at":"2024-11-15T17:27:48.785Z","avatar_url":"https://github.com/zotonic.png","language":"Erlang","funding_links":[],"categories":[],"sub_categories":[],"readme":"Zotonic - Import data from an anyMeta site\n==========================================\n\n[Zotonic](http://zotonic.com/) is a CMS.\nIt is written in Erlang.\nIt is extensible with modules.\nIt is pretty fast.\n\nThis is a module for Zotonic.\nIt can import data from anyMeta sites.\n\nanyMeta is an information management system made by [Mediamatic Lab](http://www.mediamatic.nl/)\n\nAll data is fetched using the anyMeta thing API: http://example.com/thing/12345?format=json Where 12345 is the id of the to be exported page.\n\nWhen importing you can supply a range of ids to be imported. If only the start of the range is filled in then the import will continue till it finds 100 consecutive not founds, at which point it will stop.\n\nIt is safe to re-run the import, as the module keeps a mapping of Anymeta UUID to Zotonic resource id.\n\nOptionally you dan supply the sysadmin password of the anyMeta site to import non-public data.\n\n\nDatabase\n--------\n\nThe module keeps an administration of all imported pages.\n\nThis administration is in the table `import_anymeta` and contains a mapping from the anyMeta uuid and thing-id to the local resource id, including a timestamp when the import was done.\n\n\nAdmin templates and menu\n------------------------\n\nThe import module overrules and adds some extra templates to the admin:\n\n * A menu item *Import from anyMeta* in the **Modules** menu\n * The basic form (title, subtitle etc.) is changed so that an extra field `chapeau` is added.\n * An extra sidebar template with information about the conversion and a link to the page on the original anyMeta site\n\n\nOld URLs and references to the previous site\n--------------------------------------------\n\nAfter importing a site, you can enable `mod_import_anymeta_dispatch` (https://github.com/zotonic/mod_import_anymeta_dispatch). The dispatch module adds a dispatch handler to map old Anymeta specific URLs to the newly imported resources.\n\n\nLimitations\n-----------\n\nThis module has the following limitations:\n\n * anyMeta `listpublish` and `listedit` are only partially converted, a manual correction will be necessary.\n * An empty placeholder resource will be created for some unconverted *things*, these placeholders can be deleted later.\n * Menu’s need to be re-done, as the menu structure of Zotonic and anyMeta is totally different.\n\n\nTest cases\n----------\n\nThis module comes with a number of test cases. For this purpose, JSON files with Anymeta JSON exports are shipped with this module in the `testdata/` directory. To run the importer on these files and then run the tests from zotonic, assuming your Zotonic site is called `mysite`, do the following:\n\n    import_anymeta_tests:test_full(z_context:new(mysite)).\n\nYou'll see some output of the importer scrolling by and a message at the end telling you that all assertions have passed:\n\n    15:13:57.087 [info] All tests ok.\n\n\nConversion callbacks\n--------------------\n\nThe *import_anymeta* module uses three notifications to help in the conversion:\n\n * A *first* notification to map an anyMeta kind/type combination into a Zotonic category:\n   `{import_anymeta_kind_type, AnymetaKind, AnymetaTypeList}` This expects a return of `{ok, CategoryAtom}` or `undefined`\n\n * A *fold* to convert an anyMeta thing to a Zotonic resource: `import_anymeta_fields` with a property list as the accumulator.\n   A first default mapping of anyMeta fields to Zotonic properties has been done already, the handlers can do subsequent mapping\n   or translation of values.\n\n * A *first* to map a predicate: `{import_anymeta_map_predicate, Predicate}` Where `Predicate` is an atom, either already mapped for \n   some standard predicates, or a lower-cased atom representation of the original predicate’s name.  A handler should either\n   return `undefined` to keep the original predicate name, or an atom with the name of the predicate to be used.\n\nBelow are two example modules from live projects.\n\n\nExample 1: Women on Waves\n-------------------------\n\nThis is the conversion used by the Women on Waves web site:\n\n    %% @doc Women on Waves main site module\n    %% @copyright 2012 Marc Worrell\n\n    -module(womenonwaves).\n\n    -mod_prio(50).\n    -mod_title(\"Women on Waves\").\n    -mod_description(\"Web site for Women on Waves.\").\n    -mod_schema(3).\n\n    -export([\n        observe_import_anymeta_kind_type/2,\n        observe_import_anymeta_map_predicate/2,\n\n        manage_schema/2\n    ]).\n\n    -include_lib(\"zotonic.hrl\").\n\n    observe_import_anymeta_kind_type({import_anymeta_kind_type, \u003c\u003c\"ATTACHMENT\"\u003e\u003e, [\u003c\u003c\"PRESS_COVERAGE\"\u003e\u003e,\u003c\u003c\"CARTOON\"\u003e\u003e|_]}, _Context) -\u003e\n        {ok, cartoon};\n    observe_import_anymeta_kind_type({import_anymeta_kind_type, \u003c\u003c\"ATTACHMENT\"\u003e\u003e, [\u003c\u003c\"CARTOON\"\u003e\u003e|_]}, _Context) -\u003e\n        {ok, cartoon};\n    observe_import_anymeta_kind_type({import_anymeta_kind_type, \u003c\u003c\"ATTACHMENT\"\u003e\u003e, [\u003c\u003c\"PRESS_COVERAGE\"\u003e\u003e|\u003c\u003c\"OPINION\"\u003e\u003e]}, _Context) -\u003e\n        {ok, press_image};\n    observe_import_anymeta_kind_type({import_anymeta_kind_type, \u003c\u003c\"ATTACHMENT\"\u003e\u003e, [\u003c\u003c\"PRESS_COVERAGE\"\u003e\u003e|_]}, _Context) -\u003e\n        {ok, press_image};\n    observe_import_anymeta_kind_type({import_anymeta_kind_type, \u003c\u003c\"ATTACHMENT\"\u003e\u003e, [\u003c\u003c\"PRESS_COVERAGE_E\"\u003e\u003e|_]}, _Context) -\u003e\n        {ok, press_image};\n    observe_import_anymeta_kind_type({import_anymeta_kind_type, \u003c\u003c\"ATTACHMENT\"\u003e\u003e, [\u003c\u003c\"CAMPAIGNS_DIARY_POLAND\"\u003e\u003e|_]}, _Context) -\u003e\n        {ok, campaign_image};\n    observe_import_anymeta_kind_type({import_anymeta_kind_type, \u003c\u003c\"ARTICLE\"\u003e\u003e, [\u003c\u003c\"PRESSRELEASE\"\u003e\u003e|_]}, _Context) -\u003e\n        {ok, press_release};\n    observe_import_anymeta_kind_type({import_anymeta_kind_type, \u003c\u003c\"ARTICLE\"\u003e\u003e, [\u003c\u003c\"PRESS_RELEASE\"\u003e\u003e|_]}, _Context) -\u003e\n        {ok, press_release};\n    observe_import_anymeta_kind_type({import_anymeta_kind_type, \u003c\u003c\"ARTICLE\"\u003e\u003e, [\u003c\u003c\"PRESS\"\u003e\u003e|_]}, _Context) -\u003e\n        {ok, press};\n    observe_import_anymeta_kind_type({import_anymeta_kind_type, \u003c\u003c\"ARTICLE\"\u003e\u003e, [\u003c\u003c\"ABOUT_US\"\u003e\u003e|_]}, _Context) -\u003e\n        {ok, about_us};\n    observe_import_anymeta_kind_type({import_anymeta_kind_type, \u003c\u003c\"ARTICLE\"\u003e\u003e, [\u003c\u003c\"CAMPAIGNS\"\u003e\u003e,\u003c\u003c\"DIARY\"\u003e\u003e]}, _Context) -\u003e\n        {ok, campaign_diary};\n    observe_import_anymeta_kind_type({import_anymeta_kind_type, \u003c\u003c\"ARTICLE\"\u003e\u003e, [\u003c\u003c\"DIARY\"\u003e\u003e]}, _Context) -\u003e\n        {ok, diary};\n    observe_import_anymeta_kind_type({import_anymeta_kind_type, \u003c\u003c\"ARTICLE\"\u003e\u003e, [\u003c\u003c\"CAMPAIGNS\"\u003e\u003e|_]}, _Context) -\u003e\n        {ok, campaigns};\n    observe_import_anymeta_kind_type({import_anymeta_kind_type, \u003c\u003c\"ARTICLE\"\u003e\u003e, [\u003c\u003c\"REALITY\"\u003e\u003e|_]}, _Context) -\u003e\n        {ok, reality};\n    observe_import_anymeta_kind_type({import_anymeta_kind_type, \u003c\u003c\"ARTICLE\"\u003e\u003e, [\u003c\u003c\"WE_HELP\"\u003e\u003e|_]}, _Context) -\u003e\n        {ok, reality};\n    observe_import_anymeta_kind_type({import_anymeta_kind_type, \u003c\u003c\"ARTICLE\"\u003e\u003e, [\u003c\u003c\"PRESS_COVERAGE\"\u003e\u003e|_]}, _Context) -\u003e\n        {ok, media_press};\n    observe_import_anymeta_kind_type({import_anymeta_kind_type, _Kind, _TypesSymbolic}, _Context) -\u003e\n        ?DEBUG({_Kind, _TypesSymbolic}),\n        undefined.\n\n\n    observe_import_anymeta_map_predicate({import_anymeta_map_predicate, diary_portugal}, _Context) -\u003e\n        related;\n    observe_import_anymeta_map_predicate({import_anymeta_map_predicate, set_campaigns}, _Context) -\u003e\n        hascampaigncollection;\n    observe_import_anymeta_map_predicate({import_anymeta_map_predicate, set_reality}, _Context) -\u003e\n        hasrealitycollection;\n    observe_import_anymeta_map_predicate({import_anymeta_map_predicate, P}, _Context) -\u003e\n        P.\n\n    manage_schema(install, _Context) -\u003e\n        #datamodel{\n            categories = [\n                  {anymeta_type, categorization, [{title, \u003c\u003c\"Anymeta Type/Kind\"\u003e\u003e}] },\n                  {press_image, image, [{title, \u003c\u003c\"Press Image\"\u003e\u003e}]},\n                  {campaign_image, image, [{title, \u003c\u003c\"Campaign Image\"\u003e\u003e}]},\n                  {cartoon, image, [{title, \u003c\u003c\"Cartoon\"\u003e\u003e}]}\n            ],\n            predicates=[\n                {hasfeatured, [{title, \u003c\u003c\"Has Featured\"\u003e\u003e}], []}\n            ],\n            resources=[\n                    {page_home,\n                     text,\n                     [{title, \u003c\u003c\"Home\"\u003e\u003e},\n                      {summary, \u003c\u003c\"Women on Waves is a Dutch non-profit organization concerned with women’s human rights. Its mission is to prevent unwanted pregnancy and unsafe abortions throughout the world.\"\u003e\u003e},\n                      {page_path, \u003c\u003c\"/\"\u003e\u003e}\n                     ]\n                    }\n            ]};\n    manage_schema({upgrade, 2}, _Context) -\u003e #datamodel{categories=[{cartoon, image, [{title, \u003c\u003c\"Cartoon\"\u003e\u003e}]}]};\n    manage_schema({upgrade, 3}, _Context) -\u003e #datamodel{categories=[{campaign_image, image, [{title, \u003c\u003c\"Campaign Image\"\u003e\u003e}]}]}.\n\n\n\n\n\n\nExample 2: Women on Web\n-----------------------\n\nThis is the conversion used by the Women on Web web site:\n\n    -module(womenonweb).\n\n    -mod_prio(50).\n    -mod_title(\"Women on Web\").\n    -mod_description(\"Web site for Women on Web.\").\n\n    -export([\n        observe_import_anymeta_kind_type/2,\n        observe_import_anymeta_fields/3\n    ]).\n\n    -include(\"zotonic.hrl\").\n\n    observe_import_anymeta_kind_type({import_anymeta_kind_type, \u003c\u003c\"ATTACHMENT\"\u003e\u003e, [\u003c\u003c\"PORTRAIT\"\u003e\u003e|_]}, _Context) -\u003e\n        {ok, portrait};\n    observe_import_anymeta_kind_type({import_anymeta_kind_type, \u003c\u003c\"LOCATION\"\u003e\u003e, [\u003c\u003c\"COUNTRY\"\u003e\u003e|_]}, _Context) -\u003e\n        {ok, country};\n    observe_import_anymeta_kind_type({import_anymeta_kind_type, \u003c\u003c\"KEYWORD\"\u003e\u003e, [X|_]}, _Context) -\u003e\n        {ok, z_convert:to_atom(z_string:to_lower(X))};\n    observe_import_anymeta_kind_type({import_anymeta_kind_type, \u003c\u003c\"ARTICLE\"\u003e\u003e, Ts}, _Context) -\u003e\n        case lists:member(\u003c\u003c\"QUESTION\"\u003e\u003e, Ts) of\n            true -\u003e {ok, consult_question};\n            false -\u003e\n                case lists:member(\u003c\u003c\"CONSULT\"\u003e\u003e, Ts) of\n                    true -\u003e {ok, qanda};\n                    false -\u003e undefined\n                end\n        end;\n    observe_import_anymeta_kind_type({import_anymeta_kind_type, _Kind, _TypesSymbolic}, _Context) -\u003e\n        ?DEBUG({_Kind, _TypesSymbolic}),\n        undefined.\n\n\n    %% @doc Map some block texts in the portraits to normal properties.\n    observe_import_anymeta_fields(import_anymeta_fields, Props, _Context) -\u003e\n        case proplists:get_value(category, Props) of\n            portrait -\u003e\n                Bs = proplists:get_value(blocks, Props, []),\n                [\n                    {name_first,   take_one(qb([\u003c\u003c\"name_first\"\u003e\u003e], Bs))},\n                    {name_surname, take_one(qb([\u003c\u003c\"name_last\"\u003e\u003e], Bs))},\n                    {q_feeling,    qb([\u003c\u003c\"wiz_keyword_q_feeling\"\u003e\u003e, \u003c\u003c\"q_feeling\"\u003e\u003e], Bs)},\n                    {q_how,        qb([\u003c\u003c\"wiz_keyword_q_how\"\u003e\u003e, \u003c\u003c\"q_how\"\u003e\u003e], Bs)},\n                    {q_situation,  qb([\u003c\u003c\"wiz_keyword_q_situation\"\u003e\u003e, \u003c\u003c\"q_situation\"\u003e\u003e], Bs)},\n                    {q_status,     qb([\u003c\u003c\"wiz_keyword_q_status\"\u003e\u003e, \u003c\u003c\"q_status\"\u003e\u003e], Bs)},\n                    {q_where,      qb([\u003c\u003c\"wiz_keyword_q_where\"\u003e\u003e, \u003c\u003c\"q_where\"\u003e\u003e], Bs)},\n                    {blocks, []}\n                    | proplists:delete(blocks, Props)\n                ];\n            _ -\u003e\n                Props\n        end.\n    \n    qb(_, []) -\u003e\n        undefined;\n    qb(P, [B|Bs]) -\u003e\n        Name = proplists:get_value(name, B),\n        case lists:member(Name, P) of\n            true -\u003e\n                case proplists:get_value(body, B) of\n                    undefined -\u003e undefined;\n                    null -\u003e undefined;\n                    X when is_binary(X) -\u003e z_string:trim(z_html:strip(X));\n                    L when is_list(L) -\u003e z_string:trim(z_html:strip(iolist_to_binary(L)));\n                    {trans, Ts} -\u003e {trans, [{Iso,z_string:trim(z_html:strip(X))} || {Iso,X} \u003c- Ts]}\n                end;\n            false -\u003e\n                qb(P, Bs)\n        end.\n    \n    take_one(undefined) -\u003e undefined;\n    take_one(null) -\u003e undefined;\n    take_one(B) when is_binary(B) -\u003e B;\n    take_one(L) when is_list(L) -\u003e L;\n    take_one({trans, []}) -\u003e undefined;\n    take_one({trans, [{_,B}|_]}) -\u003e B.\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzotonic%2Fmod_import_anymeta","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzotonic%2Fmod_import_anymeta","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzotonic%2Fmod_import_anymeta/lists"}