{"id":16282871,"url":"https://github.com/asaficontact/stack_classifier_project","last_synced_at":"2026-04-08T21:31:16.969Z","repository":{"id":197723448,"uuid":"209146521","full_name":"asaficontact/stack_classifier_project","owner":"asaficontact","description":"We classified Stack Overflow Python questions from 2008-2016 with Natural Language Processing and Deep Learning. Using Regular Expressions, we removed HTML tags and punctuation. We also utilized spaCy to tokenize, lemmatize and remove stop words. Using Keras, we built a 4 layered artificial neural network with a 20% dropout rate using relu and softmax activation functions. We also utilized the adam optimizer and categorical cross-entropy loss function which classified 11 tags 88% successfully.","archived":false,"fork":false,"pushed_at":"2022-12-05T17:41:27.000Z","size":173,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-08T18:24:15.059Z","etag":null,"topics":["cross-entropy-loss","deep-learning","deep-neural-networks","keras","lemmatization","neural-networks","object-oriented-programming","pandas","python3","regular-expressions","relu","sklearn","spacy","spacy-nlp","stackoverflow","tfidf","tokenization"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/asaficontact.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-09-17T20:05:00.000Z","updated_at":"2022-12-05T17:41:32.000Z","dependencies_parsed_at":null,"dependency_job_id":"1c219059-ae52-462d-b0d6-87404b5d53f4","html_url":"https://github.com/asaficontact/stack_classifier_project","commit_stats":null,"previous_names":["asaficontact/stack_classifier_project"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/asaficontact/stack_classifier_project","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asaficontact%2Fstack_classifier_project","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asaficontact%2Fstack_classifier_project/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asaficontact%2Fstack_classifier_project/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asaficontact%2Fstack_classifier_project/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/asaficontact","download_url":"https://codeload.github.com/asaficontact/stack_classifier_project/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asaficontact%2Fstack_classifier_project/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31575391,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-08T14:31:17.711Z","status":"ssl_error","status_checked_at":"2026-04-08T14:31:17.202Z","response_time":54,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cross-entropy-loss","deep-learning","deep-neural-networks","keras","lemmatization","neural-networks","object-oriented-programming","pandas","python3","regular-expressions","relu","sklearn","spacy","spacy-nlp","stackoverflow","tfidf","tokenization"],"created_at":"2024-10-10T19:11:53.500Z","updated_at":"2026-04-08T21:31:16.954Z","avatar_url":"https://github.com/asaficontact.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n## NOTE  \n#### Original files for the project exceed GitHub's maximum file size limit. Please contact me at asaficontact@gmail.com to access complete code and data files for the project.\n\n\n## Initial Setup\n\n\n```python\nfrom imports import *\n```\n\n    Using TensorFlow backend.\n\n\n\n```python\n#Import sc class which contains all the functions for operating the stackClassifier\nfrom stackClassifier import sc\n```\n\n## Python Stack Overflow Questions Tags being Classified\n\n#### 1: pandas\n#### 2: list\n#### 3: numpy\n#### 4: matplotlib\n#### 5: regex\n#### 6: dictionary\n#### 7: string\n#### 8: mysql\n#### 9: json\n#### 10: csv\n#### 11: beautifulsoup\n\n\n## Initialize Stack Classifier\n\n\n```python\n# We have already preprocessed the dataframe and saved it as a pickle file called 'processed_df' for faster execution\n# If the preprocessed dataframe doesnot exist set load = False inorder to create a new one\nstack_classifier = sc(load = True)\n```\n\n#### Total number of observations\n\n\n```python\nlen(stack_classifier.df)\n```\n\n\n\n\n    112661\n\n\n\n#### Visualizing distribution of Tags in the dataset\n\n\n```python\nax = sns.catplot(x='Tag', kind=\"count\", palette=\"ch:.25\", data=stack_classifier.df)\nax.fig.set_size_inches(18,8)\n```\n\n\n![png](README_files/main_12_0.png)\n\n\n## Load Model\n\nThe best performing neural network with 512 nodes in first hidden layer and 64 nodes in the second hidden layer has already been created and saved as 'tag_classifier.h5' to save executation time. You can train a new neural network by calling the 'train_neural_network' function.\n\n\n```python\nmodel = stack_classifier.load_neural_network('tag_classifier')\n```\n\n    WARNING: Logging before flag parsing goes to stderr.\n    W0917 16:23:20.661309 4429637056 deprecation_wrapper.py:119] From /anaconda3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n    \n    W0917 16:23:20.674825 4429637056 deprecation_wrapper.py:119] From /anaconda3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:4115: The name tf.random_normal is deprecated. Please use tf.random.normal instead.\n    \n    W0917 16:23:20.684943 4429637056 deprecation_wrapper.py:119] From /anaconda3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:131: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n    \n    W0917 16:23:20.685856 4429637056 deprecation_wrapper.py:119] From /anaconda3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:133: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.\n    \n    W0917 16:23:20.692325 4429637056 deprecation.py:506] From /anaconda3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n    Instructions for updating:\n    Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n    W0917 16:23:21.306606 4429637056 deprecation_wrapper.py:119] From /anaconda3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n    \n    W0917 16:23:22.659685 4429637056 deprecation_wrapper.py:119] From /anaconda3/lib/python3.7/site-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n    \n    W0917 16:23:22.730759 4429637056 deprecation.py:323] From /anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.\u003clocals\u003e.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n    Instructions for updating:\n    Use tf.where in 2.0, which has the same broadcast rule as np.where\n\n\n## Evaluate Model\n\n\n```python\n#Get confusion matrix, classification report and accuracy score\ncm, c_report, a_report = stack_classifier.evaluate_model_df(model, c_matrix = True, c_report= True, a_score= True, load = True)\n```\n\n#### Classficiation Report\n\n\n```python\nprint(c_report)\n```\n\n                   precision    recall  f1-score   support\n    \n    beautifulsoup       0.91      0.94      0.92      1750\n              csv       0.87      0.86      0.86      1946\n       dictionary       0.88      0.83      0.85      3008\n             json       0.89      0.91      0.90      2109\n             list       0.82      0.84      0.83      4457\n       matplotlib       0.94      0.95      0.94      4244\n            mysql       0.91      0.93      0.92      2288\n            numpy       0.89      0.90      0.90      4344\n           pandas       0.94      0.94      0.94      6722\n            regex       0.89      0.89      0.89      3910\n           string       0.73      0.69      0.71      2401\n    \n        micro avg       0.89      0.89      0.89     37179\n        macro avg       0.88      0.88      0.88     37179\n     weighted avg       0.89      0.89      0.89     37179\n    \n\n\n#### Accuracy Score\n\n\n```python\nprint(f'The Accuracy Score is: {a_report}')\n```\n\n    The Accuracy Score is: 0.8868985179805804\n\n\n#### Confusion Matrix\n\n\n```python\nclasses = list(stack_classifier.df.Tag.value_counts().to_dict().keys())\n\ntitle = 'Normalized confusion matrix'\n\ncm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n\nfig, ax = plt.subplots(figsize = (20,10))\nim = ax.imshow(cm, interpolation='nearest', cmap=plt.cm.Blues)\nax.figure.colorbar(im, ax=ax)\n# We want to show all ticks...\nax.set(xticks=np.arange(cm.shape[1]),\n   yticks=np.arange(cm.shape[0]),\n   # ... and label them with the respective list entries\n   xticklabels=classes, yticklabels=classes,\n   title=title,\n   ylabel='True label',\n   xlabel='Predicted label')\n\n# Rotate the tick labels and set their alignment.\nplt.setp(ax.get_xticklabels(), rotation=45, ha=\"right\",\n     rotation_mode=\"anchor\")\n\n# Loop over data dimensions and create text annotations.\nfmt = '.2f'\nthresh = cm.max() / 2.\nfor i in range(cm.shape[0]):\n    for j in range(cm.shape[1]):\n        ax.text(j, i, format(cm[i, j], fmt),\n                ha=\"center\", va=\"center\",\n                color=\"white\" if cm[i, j] \u003e thresh else \"black\")\nfig.tight_layout()\n```\n\n\n![png](README_files/main_23_0.png)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fasaficontact%2Fstack_classifier_project","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fasaficontact%2Fstack_classifier_project","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fasaficontact%2Fstack_classifier_project/lists"}