https://github.com/xyzesteban/selective-link-checker

This program recursively scans a website and outputs all found sub-pages and their broken links to two different CSV files
https://github.com/xyzesteban/selective-link-checker

broken-links data-mining nodejs toolkit

Last synced: 4 months ago
JSON representation

This program recursively scans a website and outputs all found sub-pages and their broken links to two different CSV files

• Host: GitHub
• URL: https://github.com/xyzesteban/selective-link-checker
• Owner: xyzesteban
• Archived: true
• Created: 2020-07-08T00:53:17.000Z (over 4 years ago)
• Default Branch: master
• Last Pushed: 2022-03-27T10:48:04.000Z (over 2 years ago)
• Last Synced: 2024-08-14T03:35:35.653Z (4 months ago)
• Topics: broken-links, data-mining, nodejs, toolkit
• Language: JavaScript
• Homepage:
• Size: 24.4 KB
• Stars: 2
• Watchers: 1
• Forks: 1
• Open Issues: 1
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• project-awesome - xyzesteban/selective-link-checker - This program recursively scans a website and outputs all found sub-pages and their broken links to two different CSV files (JavaScript)

README

        
# selective-link-checker

This program recursively scans a website and outputs all found sub-pages and their broken links into two different CSV files.

Introduction

This project is a Javascript + NodeJS console script to find broken links in a website and its sub-pages and output them to a CSV file. I worked on this project as part of my internship with Harvard T.H. Chan School of Public Health, in order to find outdated links and information. The script uses Steven Vachon's broken-link-checker as a dependency to scrape and navigate URLs and to add links to the queue.


Instructions

You do not need an IDE in order to use this application, but you do need to install NodeJS and npm. You can find the latest installers here Once you have installed NodeJS and npm, follow the next steps:



  


Download or clone this repository into a folder on your computer.

  


Navigate to the folder that contains the repository's files.

  


Open a terminal on that folder:

  


    
Windows: Shift + Click on the file explorer and select "Open Command Prompt window here" or "Open PowerShell window here".

    
macOS: Head into System Preferences and select Keyboard > Shortcuts > Services. Find "New Terminal at Folder" in the settings and click the box. Right click on the folder and select "Open Terminal".

  

  

  
On the terminal, type npm ci and press Enter to install dependencies. Do not use npm install as that will edit the package-lock.json file and install the wrong dependencies



  
On the terminal, type npm run slc and press Enter to start the application.

  


Type or paste the starting URL for the website you would like to check, e.g. https://www.korg.com/us/products/software/kc_triton/ and press Enter. NOTE: Only website URLs that contain this original URL get added to the queue.

  


Type excluded keywords and press Enter after each keyword. NOTE: When checking if a link/sub-page is broken, the program will look for these keywords on the URL as a string and will not enqueue URLs that contain them.

  
After the last keyword, press Enter a second time to start the search.

  
Once the search is completed, Type or paste the filepath for the CSV files to be exported. NOTE: This should include both the path and the filename and should end in '.csv'; e.g. C:\Users\xyzes\Documents\BrokenLinks.csv. The first file will contain all the broken links found per sub-page, and the second file will contain all the websites that were in the queue during the search.



Roadmap



  
 

    
A user-friendly web interface to interact with the application.

    


WordPress integration to output information about title, authors, editors, and timelines.