https://github.com/lschmid83/wikipedia-extractor

Wikipedia Extractor is a lightweight C# library which can be used to extract XML page data from a Wikipedia data dump.
https://github.com/lschmid83/wikipedia-extractor

data-dump integration-testing page-title regex search-algorithm search-index unit-testing wikipedia xml

Last synced: 6 days ago
JSON representation

Wikipedia Extractor is a lightweight C# library which can be used to extract XML page data from a Wikipedia data dump.

• Host: GitHub
• URL: https://github.com/lschmid83/wikipedia-extractor
• Owner: lschmid83
• Created: 2023-08-26T13:12:48.000Z (about 1 year ago)
• Default Branch: master
• Last Pushed: 2024-03-30T15:15:14.000Z (7 months ago)
• Last Synced: 2024-10-06T07:16:30.232Z (about 1 month ago)
• Topics: data-dump, integration-testing, page-title, regex, search-algorithm, search-index, unit-testing, wikipedia, xml
• Language: C#
• Homepage: https://www.nuget.org/packages/WikipediaExtractor
• Size: 32.2 KB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Wikipedia Extractor

Wikipedia Extractor is a lightweight C# library which can be used to extract XML page data from a Wikipedia data dump. It makes use of the index file included with the compressed data dump to find the position of the page and quickly retrieve it from the archive. It was developed using Visual Studio 2022.

The current URL for the data dumps is https://dumps.wikimedia.org/enwiki/ you will need to download both files and extract the index but not the dump and enter the correct paths for the library to find the files. 

The test project can be run without using the data dump as all of the index and page contents are created in memory.

This library does not parse the XML page elements instead it just returns an object containing the XML. There are other projects on GitHub for parsing the XML.

Here are some screenshots of the library running:





# Example

```cs

var pageTitles = new List

{

	"Software development",

	"Microsoft Visual Studio",

	"JavaScript"

};

using (var indexSearcher = new PageIndexSearcher(@"c:\enwiki-20190701-pages-articles-multistream-index.txt"))

{

	var pageIndexItems = indexSearcher.Search(pageTitles);

	foreach (PageIndexItem pii in pageIndexItems)

	{

		Console.WriteLine(pii.PageId + ": " + pii.PageTitle);

	}

	using (var dataDumpReader = new DataDumpReader(@"c:\enwiki-20190701-pages-articles-multistream.xml.bz2"))

	{

		var results = dataDumpReader.Search(pageIndexItems);

		foreach (var result in results) 

		{ 

			Console.WriteLine(result.Name + ": " + result.Value);               

		}

	}

}