{"id":23543924,"url":"https://github.com/winter000boy/keyword-extraction","last_synced_at":"2026-04-13T21:03:59.426Z","repository":{"id":269439240,"uuid":"907420982","full_name":"winter000boy/Keyword-Extraction","owner":"winter000boy","description":"This repository contains a Jupyter Notebook for performing keyword extraction from a dataset of NIPS papers. The notebook demonstrates data preprocessing, including removing HTML tags and special characters, tokenizing text, removing stopwords, and stemming words. It then applies TF-IDF to extract keywords.","archived":false,"fork":false,"pushed_at":"2024-12-24T06:16:26.000Z","size":311,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-17T09:45:14.437Z","etag":null,"topics":["artificial-intelligence","data-science","deep-learning","neural-network","nltk-python","pandas","python3","scikitlearn-machine-learning"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/winter000boy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"authors.csv","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-23T14:42:07.000Z","updated_at":"2024-12-24T06:18:30.000Z","dependencies_parsed_at":null,"dependency_job_id":"69f56660-aa9e-4ebb-af15-f53480f35e6a","html_url":"https://github.com/winter000boy/Keyword-Extraction","commit_stats":null,"previous_names":["winter000boy/keyword-extraction"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/winter000boy%2FKeyword-Extraction","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/winter000boy%2FKeyword-Extraction/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/winter000boy%2FKeyword-Extraction/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/winter000boy%2FKeyword-Extraction/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/winter000boy","download_url":"https://codeload.github.com/winter000boy/Keyword-Extraction/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254304533,"owners_count":22048421,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","data-science","deep-learning","neural-network","nltk-python","pandas","python3","scikitlearn-machine-learning"],"created_at":"2024-12-26T07:12:12.678Z","updated_at":"2026-04-13T21:03:59.396Z","avatar_url":"https://github.com/winter000boy.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Keyword Extraction\n\nThis repository contains a Jupyter Notebook for performing keyword extraction from a dataset of NIPS papers. The notebook demonstrates data preprocessing, including removing HTML tags and special characters, tokenizing text, removing stopwords, and stemming words. It then applies TF-IDF to extract keywords.\n\n## Installation\n\nTo run the notebook, you need to install the required libraries. Use the following commands to set up your environment:\n\n```bash\npip install pandas\npip install kaggle\npip install nltk\npip install scikit-learn\n```\n\n## Usage\n\n1. **Download the Dataset**:\n   - Ensure you have a `kaggle.json` file with your Kaggle API credentials.\n   - Upload the `kaggle.json` file in the notebook.\n\n```python\nfrom google.colab import files\nfiles.upload()  # Select your kaggle.json file\n\n!mkdir -p ~/.kaggle\n!mv kaggle.json ~/.kaggle/\n!chmod 600 ~/.kaggle/kaggle.json\n\n!pip install kaggle\n!kaggle datasets download -d benhamner/nips-papers\n\nimport zipfile\nwith zipfile.ZipFile(\"nips-papers.zip\", 'r') as zip_ref:\n    zip_ref.extractall(\"nips-papers\")\n```\n\n2. **Load the Data**:\n   - Load the dataset into a Pandas DataFrame.\n\n```python\nimport pandas as pd\ndf = pd.read_csv(\"/content/nips-papers/papers.csv\")\ndf.head()\n```\n\n3. **Process the Text**:\n   - Preprocess the text data by converting to lowercase, removing HTML tags and special characters, tokenizing, removing stopwords, and stemming.\n\n```python\nimport re\nimport nltk\nfrom nltk.corpus import stopwords\nfrom nltk.stem.porter import PorterStemmer\n\nnltk.download('stopwords')\nnltk.download('punkt_tab')\nStop_Words = set(stopwords.words('english'))\n\n# Define additional stopwords\nnew_words = [\"fig\", \"figure\", \"sample\", \"using\", \"image\", \"show\", \"result\", \"large\", \"also\", \"one\", \"two\", \"three\", \"four\", \"five\", \"six\", \"seven\", \"eight\", \"nine\"]\nStop_Words = list(Stop_Words.union(new_words))\n\ndef processing_text(txt):\n    txt = txt.lower()\n    txt = re.sub(r'\u003c.*?\u003e', ' ', txt)\n    txt = re.sub(r'[^a-zA-Z]', ' ', txt)\n    txt = nltk.word_tokenize(txt)\n    txt = [word for word in txt if word not in Stop_Words]\n    txt = [word for word in txt if len(word) \u003e 3]\n    stemming = PorterStemmer()\n    txt = [stemming.stem(word) for word in txt]\n    return txt\n\ndocs = df['paper_text'].apply(lambda x: processing_text(x))\n```\n\n4. **Apply TF-IDF**:\n   - Use CountVectorizer and TfidfTransformer from scikit-learn to compute TF-IDF scores for the processed text.\n\n```python\nfrom sklearn.feature_extraction.text import CountVectorizer\nfrom sklearn.feature_extraction.text import TfidfTransformer\n\ncv = CountVectorizer(max_df=95, max_features=5000, ngram_range=(1, 3))\nword_count_vectors = cv.fit_transform(docs)\n\ntfidf_transformer = TfidfTransformer(smooth_idf=True, use_idf=True)\ntfidf_transformer.fit(word_count_vectors)\n```\n\n## Keywords\n\n- Keyword Extraction\n- Text Processing\n- Data Preprocessing\n- TF-IDF\n- Natural Language Processing (NLP)\n\n## Libraries\n\n- `pandas`\n- `kaggle`\n- `nltk`\n- `scikit-learn`\n\n## Dataset\n\nThe dataset used in this project is from Kaggle: [NIPS Papers](https://www.kaggle.com/datasets/benhamner/nips-papers).\n\n## License\n\nThe dataset is licensed under ODbL-1.0.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwinter000boy%2Fkeyword-extraction","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwinter000boy%2Fkeyword-extraction","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwinter000boy%2Fkeyword-extraction/lists"}