{"id":16573449,"url":"https://github.com/roald87/infosyscrawler","last_synced_at":"2025-03-05T15:22:09.755Z","repository":{"id":107062311,"uuid":"453620488","full_name":"Roald87/InfoSysCrawler","owner":"Roald87","description":"Crawler for Beckhoff's InfoSys","archived":false,"fork":false,"pushed_at":"2023-04-23T19:16:26.000Z","size":4473,"stargazers_count":5,"open_issues_count":2,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-16T03:50:24.482Z","etag":null,"topics":["beckhoff","beckhoff-twincat","beckhoff-twincat-hmi","beckhoff-twincat-plc","industrial-automation","plc","plc-programming","twincat-ads","twincat3","twincathmi"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Roald87.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":null,"patreon":null,"open_collective":null,"ko_fi":"roald87","tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":null}},"created_at":"2022-01-30T07:56:38.000Z","updated_at":"2024-05-31T20:36:38.000Z","dependencies_parsed_at":null,"dependency_job_id":"9494a970-4d77-4481-a7fa-10170dac220b","html_url":"https://github.com/Roald87/InfoSysCrawler","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Roald87%2FInfoSysCrawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Roald87%2FInfoSysCrawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Roald87%2FInfoSysCrawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Roald87%2FInfoSysCrawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Roald87","download_url":"https://codeload.github.com/Roald87/InfoSysCrawler/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242049057,"owners_count":20063503,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beckhoff","beckhoff-twincat","beckhoff-twincat-hmi","beckhoff-twincat-plc","industrial-automation","plc","plc-programming","twincat-ads","twincat3","twincathmi"],"created_at":"2024-10-11T21:42:13.476Z","updated_at":"2025-03-05T15:22:09.721Z","avatar_url":"https://github.com/Roald87.png","language":"HTML","readme":"# InfoSysCrawler\n\nCrawler for [Beckhoff's InfoSys](https://infosys.beckhoff.com/index_en.htm).\n\n## How it works\n\nThe crawler goes through the InfoSys menu and finds all subfolders and pages. It is not necessary to start at a top folder, but you can also start at a lower folder. Optionally some folders/pages can be ignored.\n\n### Current application\n\nThis crawler can be used to walk through InfoSys pages and look for a TwinCAT version number. The version number is often found on the bottom of a page, as shown below.\n\n![](img/tc_version.png)\n\nThe version numbers are used for the [unofficial TwinCAT changelog](https://tcchanges.cookncode.com).\n\n## How to use it\n\n1. Build the project using `dotnet build`.\n2. Open a FSharp interactive console.\n3. Import the dll using `#r \"InfoSysCrawler/bin/Debug/net6.0/InfoSysCrawler.dll\"`, where you should change the path of the dll according to where it is saved on your local system.\n4. Then you can crawl a subfolder as follows:\n    ```fsharp\n    open InfoSysCrawler.SiteMap\n    open InfoSysCrawler.Request\n\n    let url = (Url \"https://infosys.beckhoff.com/english/menu/menu.php?id=8644252870837316006\") // TC1xxx - TwinCAT 3 base // 5 s\n    // Add folder names you do not want to into\n    let ignoreFolders = [\"Foreword\"; \"Installation\"; \"Samples\"]\n    // Add pages which you do not want to look for a twincat version number\n    let ignorePages = [\"Overview\"; \"Search\"]\n    // Start crawling\n    let menu = traverseMenu ignoreFolders ignorePages url\n\n    // Output you should see\n    TC1000 | TwinCAT 3 ADS\n    TC1100 | TwinCAT 3 I/O\n    TC1200 | TwinCAT 3 PLC\n    TC1210 | TwinCAT 3 PLC/C++\n    TC1220 | TwinCAT 3 PLC/C++/MATLAB®/Simulink®\n    TC1250 | TwinCAT 3 PLC/NC PTP 10\n    TC1260 | TwinCAT 3 PLC/NC PTP 10/NC I\n    TC1270 | TwinCAT 3 PLC/NC PTP 10/NC I/CNC\n    TC1275 | TwinCAT 3 PLC/NC PTP 10/NC I/CNC E\n    TC1300 | TwinCAT 3 C++\n    TC1320 | TwinCAT 3 C++/MATLAB®/Simulink®\n    ```\n5. You can save the crawled data with `menu |\u003e saveAsJson \"menu.json\"`\n6. You can later open the data again with\n    ```fsharp\n    #r \"InfoSysCrawler/FSharp.Json.dll\"\n\n    open FSharp.Json\n\n    let deserialized = Json.deserialize\u003cNode list\u003e (File.ReadAllText(\"menu.json\"))\n    ```\n\n## Current dumps\n\nIn `data/` you will find a few folders which are crawled and saved as a json file. Currently the following folders were crawled where the folders \"Foreword\", \"Installation\" and \"Samples\" and the pages \"Overview\" and \"Search\" are ignored. On ignored pages/folders the crawler doesn't try to find a TwinCAT version number. Data was last updated on 8 October 2022.\n\n- `data/plc_libraries.json`: TwinCAT 3/TE1000 XAE/PLC/PLC Libraries\n- `data/texxxx.json`: TwinCAT 3/TExxxx | TwinCAT 3 Engineering\n- `data/tc1xxx.json`: TwinCAT 3/TC1xxx | TwinCAT 3 Base\n- `data/tfxxxx.json`: TwinCAT 3/TFxxxx | TwinCAT 3 Functions\n","funding_links":["https://ko-fi.com/roald87"],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Froald87%2Finfosyscrawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Froald87%2Finfosyscrawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Froald87%2Finfosyscrawler/lists"}