{"id":16356080,"url":"https://github.com/joehowarth/grep-jh","last_synced_at":"2025-07-13T01:36:38.898Z","repository":{"id":100830103,"uuid":"78356701","full_name":"JoeHowarth/Grep-JH","owner":"JoeHowarth","description":null,"archived":false,"fork":false,"pushed_at":"2017-01-09T00:59:35.000Z","size":1352,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-12-29T06:11:39.437Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JoeHowarth.png","metadata":{"files":{"readme":"README","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-01-08T16:47:22.000Z","updated_at":"2017-01-08T16:47:49.000Z","dependencies_parsed_at":"2023-06-10T07:15:37.104Z","dependency_job_id":null,"html_url":"https://github.com/JoeHowarth/Grep-JH","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JoeHowarth%2FGrep-JH","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JoeHowarth%2FGrep-JH/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JoeHowarth%2FGrep-JH/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JoeHowarth%2FGrep-JH/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JoeHowarth","download_url":"https://codeload.github.com/JoeHowarth/Grep-JH/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239665489,"owners_count":19676945,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T01:42:38.027Z","updated_at":"2025-02-19T13:27:50.676Z","avatar_url":"https://github.com/JoeHowarth.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"\nJoseph Howarth - Efficient Directory Searching with Boyer-Moore\n\t\t Comp 15, Project 2 - Herp de Gerp\n\nPurpose\n```````\n\nDesign and Architecture Overview\n``````````````````````\n\t\tMy initial approach to the project was similar to many others: \n\tcreate an index of lines from the inputted files and use a hash table  \n\tto store lists of which lines and files containing each word. This \n\twould enable constant time lookup and very quick printing, however the\n\ttime to process every word in every file and then store the entire \n\tcontents of each file line-by-line in ram seemed too inefficient to be\n\tpractical.\n\t\tI investigated using other data structures, notable binary \n\tsearch trees and tries, these had the same fundamental limitations\n\tof very significant pre-processing and ram requirements \u003e= to the\n\tsize of the indexed directory. The direct alternative of searching\n\tthe entire directory each search naively would certainly take too\n\tlong as well.\n\t\tAfter researching several alternate methods to achieve more\n\treasonable indexing times and O(1) ram usage (relative to directory\n\tsize), the boyer-moore algorithm appeared to solve the issue. By \n\tcomparing the pattern to hay at the left, if there is a mis-match\n\tthe algorithm can shift nearly the entire length of the pattern \n\tinstead of only by 1 char. This means with a pattern of length 10,\n\thay of length 1000 and a match located at position 500, BM might\n\tonly have to perform 50-100 comparisons instead of \u003e500. (For more\n\tinformation, see the linked article in acknowledgements)\n\n\tArchitecture:\n\t\t\n\t\tThe search tool built around Boyer-Moore consists of four\n\tmain parts:\n\t\t\tA.) main.cpp\n\t\t\t\tOrchestrates command loop, arguments,\n\t\t\t\tother classes\n\t\t\t\n\t\t\tB.) SearchClass\n\t\t\t\tInitialized on pattern, contains \n\t\t\t\tpre-processing for BM, BMSearch itself\n\t\t\t\tand the master Search() function that\n\t\t\t\torchestrates each search per query cycle\n\t\n\t\t\tC.) File_inf \n\t\t\t\tClass containing data and functions for\n\t\t\t\teach file.\n\t\t\t\t3 main jobs:\n\t\t\t\t\t\n\t\t\t\t\t1.) Open file to mmap\n\t\t\t\t\t\n\t\t\t\t\t2.) Find and store positions \n\t\t\t\t\t     of new lines and store \n\t\t\t\t\t     positions of found matches\n\t\t\t\t\t\n\t\t\t\t\t3.) Print \n\t\t\t\t\t      Path + line_num + line content\n\t\t\t\t\t    for each match\n\t\t\t\n\t\t\tD.) FSTreeTraverse\n\t\t\t\tGiven a valid directory, creates list of\n\t\t\t\tFile_inf objects in Files vector\n\t\t\t\tset complete Path for each file\n\t\n\tIntersting Design Choices:\n\t\t\n\t\tIn order to reduce overhead due to copying to and from buffers\n\t\trepeatedly, as well as for ease of use with BM, each text file\n\t\tis mmap-ed from disk directly to ram based off its filesize\n\t\tand set to a char*. This means that it can be read just like\n\t\ta c_string without needing to copy data through buffers too\n\t\tmany times.\n\n\t\tWhile designing the project, one of the biggest time sinks \n\t\tappeared to be file i/o, so I wrote several aspects in such\n\t\ta way to make adding asynchronous io or other concurrency\n\t\tmethods easier. Most notably, the file opening process is\n\t\tdone by looping through the Files vector, mmap-ing each\n\t\tand placing a pointer to the File_inf object in the\n\t\tthe SearchClass's queue to be processed. This means that \n\t\tthe SearchClass will only process opened files, and its loop \n\t\tcould be active in a different thread/processing while \n\t\tasynchrous io occurs. Unfortunately, due to time constraints\n\t\tI haven't implemented that optimization yet, though I \n\t\twould like to tinker over the next weeks.\n\n\t\tInterestingly, due to the nature of BM, searching for newline\n\t\tcharacters actually takes longer than lengthier patters \n\t\tbecause it actually has to compare each character. So in\n\t\torder to efficiently calculate the line numbers of matches\n\t\tand print the corresponding lines, the position of\n\t\tnew lines (only \\n) in the file (ie. the indeces in the \n\t\tconst char[]) are placed into a vector. \n\t\t\tFrom this, because they were found by a forward \n\t\tpass and so are in order a binary search is performed \n\t\ton these \"known_lines\" with the position of a match. This \n\t\treturns the index of the match, which corresponds to \n\t\tline number - 1. \n\t\t\tTo print, a loop is run on the file mmap, printing\n\t\teach char from the returned index - 1 to index. \n\t\t\n\t\tThe case insensitivity was achieved by setting if statments\n\t\tin the BM-preprocessing stage and search stage. In preBM,\n\t\tthe offset table was set both for tolower and toupper, \n\t\tpreserving whatever non-alpha char was input, but correctly\n\t\toffsetting both lower and upper case.\n\t\t\tThe BMSearch function with case insensitivity simply\n\t\tuses strncasecmp instead of strncmp. This solution only works\n\t\tdefinitively on Posix systems, but it can be easily upgraded\n\t\tin the future. The entire function loop was copied because\n\t\tcomparing the boolean num_files * char_in_file * ave_comp \n\t\tresults in a large number of checks that can be avoided by\n\t\tonly checking once at the beginning. \n\n\t\tA downside of boyer-moore is that it recognizes patterns, but\n\t\tnot words naturally. Looking at the_gerp, it does not count\n\t\tmatches followed by punctuation, but does ignore whitespace.\n\t\tTo mimick this functionality, each match is checked to see\n\t\tif it is followed and preceded by whitespace (or it is the \n\t\tfirst word in a file, nasty corner case!)\n\n\n\t\tJust to reiterate a point again, I did initially consider\n\t\tusing hash tables (and briefly ties), however based on \n\t\tperformance, I used the Boyer-Moore approach instead.\n\nFiles Provided\n``````````````\n\tREADME -- documentation\n\tMakefile -- compiles and provides \n\tDirNode.h -- used in FSTree\n\tDirNode.o  \n\tFile_inf.cpp -- described in architecture, stands for file_info \n\tFile_inf.h \n\tFSTree.h -- creates directory tree\n\tFSTree.o \n\tFSTreeTraversal.cpp -- traverses directory tree, see architecture\n\tFSTreeTraversal.h \n\tmain.cpp --ochestrates project, see architecture\n\tSearch.cpp --updated each query cycle with patter, see architecture\n\tSearch.h\n\nCompiling and Running\n`````````````````````\n\ttype make to compile\n\n\tto run ./gerp [valid directory]\n\n\to AnyString\n\t\tSearches directory for string\n\n\to \"@i AnyString\" or \"@insensitive AnyString\"\n\t\tperforms case insensitive search\n\n\to \"@q\" or \"@quit\": \n\t\tThese commands will completely quit the program, and print \n\t\t\"Goodbye! Thank you and have a nice day.\" This statement \n\t\tshould be followed by a new line.\n\nTesting\n```````\n\t\tThe FSTreeTraversal was tested through cout statements in \n\tmain, various directory names would be passed in and the correct \n\toutput would be compared. \n\n\t\tThe Boyer-Moore algorithm was tested originally in its own\n\tfile with various short strings and patterns. It was then integrated\n\tinto the SearchClass\n\n\t\tThe File_inf and Search classes were tested incrementally,\n\tfirst with only simple opening and closing of files, testing queues\n\tetc. Then once preliminary bugs were resolved, the project was tested\n\tonly printing the location of matches, then matches and known newlines\n\tand finally the full print functionallity was added and tested\n\t\tOnce this worked, the case insensitive feature was added and\n\ttested, followed by whitespace detection (ie if looking for the, there\n\tdoes NOT count).\n\t\t\n\n\nAknowledgements\n```````````````\n\tInformation on the Boyer-Moore algorithm from various web sources\n\tand Stack Exchange, most notably this page \n\t\thttp://www-igm.univ-mlv.fr/~lecroq/string/node14.html\n\tThe version of Boyer-Moore implemented currently is \n\tPratt-Boyer-Moore without the Good-Suffix rule adapted from Jerry\n\tCoffins example\n\t\n\tExtensive use of cplusplus.com looking up how to use new tools\n\tsuch as mmap/munmap, fstat, etc. as well as interfaces for STL\n\tclasses and miscellaneous extra information\n\n\n\n\t\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoehowarth%2Fgrep-jh","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoehowarth%2Fgrep-jh","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoehowarth%2Fgrep-jh/lists"}