{"id":14065785,"url":"https://github.com/DeDeDeDer/Personal_Projects","last_synced_at":"2025-07-29T21:33:34.741Z","repository":{"id":112237993,"uuid":"146153302","full_name":"DeDeDeDer/Personal_Projects","owner":"DeDeDeDer","description":"This holds all my personal data-related project's (Automation, Modelling, Analysis)","archived":false,"fork":false,"pushed_at":"2019-08-24T00:33:52.000Z","size":20539,"stargazers_count":7,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-08-14T07:09:41.513Z","etag":null,"topics":["actuarial-science","actuarial-statistics","claims-reserving","datascience","datascraping","excelvba","exploratory-data-analysis","feature-engineering","insurance-claims","modelling-framework","predictive-modeling","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DeDeDeDer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2018-08-26T05:13:33.000Z","updated_at":"2023-03-30T16:31:42.000Z","dependencies_parsed_at":"2023-05-23T04:45:51.320Z","dependency_job_id":null,"html_url":"https://github.com/DeDeDeDer/Personal_Projects","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DeDeDeDer%2FPersonal_Projects","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DeDeDeDer%2FPersonal_Projects/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DeDeDeDer%2FPersonal_Projects/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DeDeDeDer%2FPersonal_Projects/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DeDeDeDer","download_url":"https://codeload.github.com/DeDeDeDer/Personal_Projects/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228052620,"owners_count":17862103,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["actuarial-science","actuarial-statistics","claims-reserving","datascience","datascraping","excelvba","exploratory-data-analysis","feature-engineering","insurance-claims","modelling-framework","predictive-modeling","python3"],"created_at":"2024-08-13T07:04:43.431Z","updated_at":"2024-12-04T05:30:59.520Z","avatar_url":"https://github.com/DeDeDeDer.png","language":"Python","readme":"# Personal_Projects\nThis GitHub holds all my personal project's that I have worked on as a past time. \nProject's are mainly focused on Data Science, Insurance Pricing \u0026 Reserving fields.\n\u003cbr\u003eA mapping of these are laid out below.\n\n# Mapping\n| Scraped Data Analysis | Articles | Simulators \u0026 Kernels |\n| ------ | ------ | ------ |\n| \u003ca href=\"https://github.com/DeDeDeDer/Personal_Projects/tree/master/Scraped%20Data%20Analysis\"\u003ePython Scripts\u003c/a\u003e | \u003ca href=\"https://medium.com/@DRicky.Ch29\"\u003eArticle Profile\u003c/a\u003e | \u003ca href=\"https://github.com/DeDeDeDer/Personal_Projects/tree/master/Simulator%20Kernels\"\u003ePython Scripts\u003c/a\u003e |\n| \u003ca href=\"https://medium.com/@DRicky.Ch29/geo-visualisation-singapore-housing-sales-market-python-web-scraping-c58525e9eaa\"\u003eGeo-Visual SG Housing Prices\u003c/a\u003e \u003cbr /\u003e ![ScreenShot](images/Geo_Property.png) | \u003ca href=\"https://medium.com/@DRicky.Ch29/at-its-simplest-predictive-modelling-b3c0c0b0716d\"\u003ePredictive Modelling\u003c/a\u003e \u003cbr /\u003e ![ScreenShot](images/MindM_Modelling.png)| \u003ca href=\"https://medium.com/@DRicky.Ch29/inflation-adjusted-chain-ladder-iacl-with-only-python-pandas-module-512914d9a1d\"\u003eClaims Simulator\u003c/a\u003e \u003cbr /\u003e ![ScreenShot](images/Line_ClaimSim.png)\n| \u003ca href=\"https://medium.com/@DRicky.Ch29/geo-visualisation-singapore-housing-sales-market-python-web-scraping-c58525e9eaa\"\u003eBox-Plot SG Housing Prices\u003c/a\u003e \u003cbr /\u003e ![ScreenShot](images/Box_Property.png) | \u003ca href=\"https://medium.com/@DRicky.Ch29/web-scraping-pdf-tables-data-cleaning-part-1-cb6d8d47a6de\"\u003eWeb-Scraping Workflow\u003c/a\u003e \u003cbr /\u003e ![ScreenShot](images/Flow_Scrape.png)| \u003ca href=\"https://github.com/DeDeDeDer/Personal_Projects/tree/master/Simulator%20Kernels/FOREX_ML\"\u003eFOREX \u0026 ML Algorithms\u003c/a\u003e \u003cbr /\u003e ![](images/Bubb_MLA_2.gif)\n| \u003ca href=\"https://medium.com/@DRicky.Ch29/visualising-14yrs-of-general-insurance-data-python-web-scraping-part-3-12e89a8c8249\"\u003ePrivate Insurance 14-Years\u003c/a\u003e \u003cbr /\u003e ![](images/Bubb_PteInsur.gif) |  \u003ca href=\"https://medium.com/@DRicky.Ch29/machine-learning-algorithms-on-forex-high-level-test-bd71c90aee95\"\u003eFOREX ML Algorithms Workflow\u003c/a\u003e \u003cbr /\u003e![ScreenShot](images/Flow_MLA.png) | \u003ca href=\"\"\u003ePending Stock Screener\u003c/a\u003e \u003cbr /\u003e\n| \u003ca href=\"https://medium.com/@DRicky.Ch29/visualising-14yrs-of-general-insurance-data-python-web-scraping-part-3-12e89a8c8249\"\u003ePublic Insurance 14-Years\u003c/a\u003e \u003cbr /\u003e ![](images/Bubb_PubInsur.gif) |  \u003cbr /\u003e | \u003cbr /\u003e\n\n\n# For more info..\n![ScreenShot](/Pictures/MapBackground_4.png)\n\u003cbr\u003e\n\u003cbr\u003e\n\n\n\u003e # **Insurance (Pricing) \u0026 Data Science**\n\n\n\u003cbr\u003e\n\n# **What is Predictive Modelling?**\n\n\u003cbr\u003e\nIt is simply the framework to integrate past data \u0026 statistics to predict \nfuture outcomes or project liabilities. There are 4 main techniques;\nBayesian, Decision Trees, Support Vector Machines \u0026 Neural Networks.\nMy project's utilizes mainly Bayesian \u0026 Decision Tree techniques.\nHence, focused primarily on linear regression models.\n\u003cbr\u003e\n\n\n.\n\u003cbr\u003e\n## [At Its Simplest, Predictive Modelling](https://medium.com/@DRicky.Ch29/at-its-simplest-predictive-modelling-b3c0c0b0716d)\n\u003cbr\u003e\n\n![ScreenShot](/Pictures/IntroModel_1.png)\n\nAn article publication aimed at explaining concepts to:\n\u003cbr\u003e1. Generalised structure to Predictive Modelling\n\u003cbr\u003e2. Alternative interpretations to various statistical model metrics\n\u003cbr\u003e\n\u003cbr\u003eThe article follows the generalized framework of:\n\u003cbr\u003e\n\u003col\u003e\u003cins\u003eData preparation\u003c/ins\u003e\u003c/ol\u003e\n\u003col\u003e- Preliminary data analysis, executing 4-Tier's of data cleaning. (Correct, Complete, Create, Convert)\u003c/ol\u003e\n\n\u003col\u003e\u003cins\u003eExploratory Data Analysis\u003c/ins\u003e\u003c/ol\u003e\n\u003col\u003e- Uni- Bi- \u0026 Multi- Analysis\u003c/ol\u003e\n\n\u003col\u003e\u003cins\u003eModel Preparation\u003c/ins\u003e\u003c/ol\u003e\n\u003col\u003e- Data stratified Train/Test splits, Hyper parameter tuning, parameter evaluation metrics.\u003c/ol\u003e\n\u003col\u003e- Feature Engineering (Quantity \u0026 Quality), Feature evaluation metrics\u003c/ol\u003e\n\n\u003col\u003e\u003cins\u003ePredictive Modelling (Classification Problem)\u003c/ins\u003e\u003c/ol\u003e\n\u003col\u003e- Ensembles (Hard \u0026 Soft Voting)\u003c/ol\u003e\n\n\u003cbr\u003e\n\u003ca href=\"https://medium.com/@DRicky.Ch29/at-its-simplest-predictive-modelling-b3c0c0b0716d\"\u003e\u003cstrong\u003eClick To View\u003c/strong\u003e\u003c/a\u003e\n\u003cbr\u003e\n\n\n\n\u003cbr\u003e\n\n# **What is Web Scraping?**\n\n\u003cbr\u003e\nIn short, it is simply the automated process of extracting data from the web. \nSubsequently, cleaning any irregularities \u0026 conducting Exploratory Data Analysis\nto spot Trends \u0026 Patterns.\n\u003cbr\u003e\n\n.\n\u003cbr\u003e\n## Python Web Scraping PDF \u0026 Data Cleaning (Part 1) \n[Article](https://medium.com/@DRicky.Ch29/web-scraping-pdf-tables-data-cleaning-part-1-cb6d8d47a6de)\nor\n[Python Code](https://github.com/DeDeDeDer/Personal_Projects/blob/master/Web%20Scraping%20(Data%20Science%20%26%20Insurance%20Pricing)/Web_Scrap_Insurance_Returns.py)\n\u003cbr\u003e\n\n![ScreenShot](/Pictures/WebScrapPart1.png)\n\nA Python Kernel written to automate repetitive clicking of 1,228c URLs \u0026\nconverting 1,000c PDF Tables into CSV to compile data.\n\n\u003cbr\u003eContents:\n\u003col\u003e1. Collate online source code URLs \u0026 sub-page URLs\u003c/ol\u003e\n\u003col\u003e2. Download online data via URLs\u003c/ol\u003e\n\u003col\u003e3. Convert \u0026 Neaten PDF Table into CSV\u003c/ol\u003e\n\u003col\u003e4. Compile all CSV Tables\u003c/ol\u003e\n\n\u003cbr\u003e\n\u003ca href=\"https://medium.com/@DRicky.Ch29/web-scraping-pdf-tables-data-cleaning-part-1-cb6d8d47a6de\"\u003e\u003cstrong\u003eClick To View\u003c/strong\u003e\u003c/a\u003e\n\u003cbr\u003e\n\n\n\n.\n\u003cbr\u003e\n## [Python Web Scraping Data Analysis Motor Insurance (Part 2)](https://medium.com/@DRicky.Ch29/python-web-scraping-data-analysis-motor-insurance-part-2-4cd7162ba644)\n\u003cbr\u003e\n\n![ScreenShot](/Pictures/WebScrapPart2.png)\n\nAfter extracting Annual Insurance Data Returns in the Part 1 series, we proceed to\nanalyze the data.\n\n\u003cbr\u003eContents:\n\u003col\u003e\u003cins\u003ePatterns\u003c/ins\u003e\u003c/ol\u003e\n\u003col\u003e1. Benchmark Range of ROC on Expense \u0026 Loss Ratios\u003c/ol\u003e\n\n\u003col\u003e\u003cins\u003eTrends\u003c/ins\u003e\u003c/ol\u003e\n\u003col\u003e2. Growing reinsurance ceded abroad beyond the ASEAN region\u003c/ol\u003e\n\u003col\u003e3. Declining averages for Earned Premiums \u0026 Claims Incurred (with falling inflation rates)\u003c/ol\u003e\n\u003col\u003e4. Average ROC, Expense \u0026 Loss Ratios\u003c/ol\u003e\n\n\n\u003cbr\u003e\n\u003ca href=\"https://medium.com/@DRicky.Ch29/python-web-scraping-data-analysis-motor-insurance-part-2-4cd7162ba644\"\u003e\u003cstrong\u003eClick To View\u003c/strong\u003e\u003c/a\u003e\n\u003cbr\u003e\n\n\n\n\u003cbr\u003e\n\n# **What is Exploratory Data Analysis?**\n\n\u003cbr\u003e\nIt is simply the analyzing of data sets to summarize characteristics \u0026 patterns. \nThese include Uni- Bi- \u0026 Multi- Variate Analysis. Often discovering underlying\nrelationships that conventional models overlook.\n\u003cbr\u003e\n\n.\n\u003cbr\u003e\n## [EDA \u0026 Feature Engineering Focused](https://www.kaggle.com/derrickchua29/feature-engineering-eda-focused/notebook)\n\u003cbr\u003e\n\n![ScreenShot](/Pictures/EDA_article_1.png)\n\n\n\u003cbr\u003e\u003cins\u003eEDA Summary\u003c/ins\u003e\n\n\u003cbr\u003e1. Those who have had past experience of financial distress (target variable):\n\u003cbr\u003e\u003eMade lesser loans or exceed deadlines\n\u003cbr\u003e\u003eTend to have lesser dependents \u0026 debt ratio \u0026 net worth\n\u003cbr\u003e\u003eAs expected are of lower-tier income, But lower debt ratio\n\u003cbr\u003e\n\n\u003cbr\u003e2. Ignoring mortality and time value of money (i.e.Annuities)\n\u003cbr\u003e\u003eDebt ratio \u0026 Net worth shows gaussian distribution against age\n\u003cbr\u003e\n\n\u003cbr\u003e3. Those who had acts of debt delinquency (Made loans or exceed deadlines)\n\u003cbr\u003e\u003eTend to be from the higher-tier income or Retired\n\u003cbr\u003e\n\n\u003cbr\u003e4. Others\n\u003cbr\u003e\u003eThe higher the income, the higher the debt ratio\n\u003cbr\u003e\u003eThe higher the income, the lower the dependents\n\n\n\u003cbr\u003e\n\u003ca href=\"https://www.kaggle.com/derrickchua29/feature-engineering-eda-focused/notebook\"\u003e\u003cstrong\u003eClick To View\u003c/strong\u003e\u003c/a\u003e\n\u003cbr\u003e\n\n\n\n# **What is General Linear Modelling?**\n\n\u003cbr\u003e\nIt is simply applying the fundamental straight line concept of a Y = mx + C. \nIn other words, the idea that variable relationships are 1-dimensional (positive\nor negative).\n\u003cbr\u003e\n\n\n.\u003cbr\u003e\n## [Ensemble Models Comparison Techniques](https://www.kaggle.com/derrickchua29/ensemble-models-comparison-techniques)\n\u003cbr\u003e\n\n![ScreenShot](/Pictures/GLM_article_1.png)\n\n\n\u003cbr\u003eA Python Kernel aimed to:\n\u003col\u003e1. Get a better understanding of the simplified predictive modelling framework\u003c/ol\u003e\n\u003col\u003e2. Grasp the logic behind different coding methods \u0026 concise techniques used\u003c/ol\u003e\n\u003col\u003e3. Comparisons between different models\u003c/ol\u003e\n\u003cbr\u003e\n\u003col\u003e\u003cins\u003eCoding Techniques :\u003c/ins\u003e\u003c/ol\u003e\n\u003col\u003eA.List comprehensions\u003c/ol\u003e\n\u003col\u003eB.Samples to reduce computational cost\u003c/ol\u003e\n\u003col\u003eC.Concise 'def' functions that can be used repetitively\u003c/ol\u003e\n\u003col\u003eD.Pivoting using groupby\u003c/ol\u003e\n\u003col\u003eE.When \u0026 How to convert and reshape dictionary’s into lists or dataframes\u003c/ol\u003e\n\u003col\u003eF.Quickly split dataframe columns\u003c/ol\u003e\n\u003col\u003eH.Loop Sub-plots\u003c/ol\u003e\n\u003col\u003eI.Quick Lambda formulae functions\u003c/ol\u003e\n\u003col\u003eJ.Quick looping print or DataFrame conversion of summative scores\u003c/ol\u003e\n\u003col\u003eK.Order plot components\u003c/ol\u003e\n\u003col\u003eL.Create \u0026 Plot Bulk Ensemble comparative results\u003c/ol\u003e\n\n\n\u003cbr\u003e\n\u003ca href=\"https://www.kaggle.com/derrickchua29/ensemble-models-comparison-techniques\"\u003e\u003cstrong\u003eClick To View\u003c/strong\u003e\u003c/a\u003e\n\u003cbr\u003e\n\n\u003cbr\u003e\n\u003cbr\u003e\n\u003cbr\u003e\n\n\n\u003e # **Insurance (Reserving)**\n\n\n\u003cbr\u003e\n\n# **Claim Simulations**\n\n\u003cbr\u003e\nIn short, this projects contains a Python Kernel to automate the probabilistic  \nclaims simulation process for actuarial reserving calculations. \n\u003cbr\u003e\nReserving Method Used: Inflation Adjusted Chain Ladder\n\n.\n\u003cbr\u003e\n## Claims Simulation \n[Article](https://medium.com/@DRicky.Ch29/inflation-adjusted-chain-ladder-iacl-with-only-python-pandas-module-512914d9a1d)\nor\n[Python Code Guide](https://www.kaggle.com/derrickchua29/simulating-claim-data-iacl-calculation)\nor\n[Python Code v2](https://github.com/DeDeDeDer/Personal_Projects/blob/master/Claims%20Simulation%20(Insurance%20Reserving)/Claims_Simulator.py)\n\u003cbr\u003e\n\n![ScreenShot](/Pictures/ClaimsSimu_article_1.png)\n\n\u003cbr\u003ePresent: Simulation supports Claim Numbers (Poisson, Negative Binomial) \u0026 Amounts (Gaussian, LogNormal).\n\u003cbr\u003eOngoing: \n\u003cbr\u003e1. Support Bornhuetter-Ferguson Method (BF).\n\n\u003cbr\u003eContents:\n\u003col\u003e0. Assumptions\u003c/ol\u003e\n\u003col\u003e1. Development-Year lags\u003c/ol\u003e\n\u003col\u003e2. Incremental \u0026 Cumulative claim amounts\u003c/ol\u003e\n\u003col\u003e3. Uplift past inflation for incremental amounts \u0026 Derive cumulative\u003c/ol\u003e\n\u003col\u003e4. Individual Loss Development Factors (LDFs)\u003c/ol\u003e\n\u003col\u003e5. Raw preliminary view of triangle\u003c/ol\u003e\n\u003col\u003e6. Establish predicted lag years data frame\u003c/ol\u003e\n\u003col\u003e7. Impute latest cumulative amounts\u003c/ol\u003e\n\u003col\u003e8. Simple Mean \u0026 Volume Weighted LDFs \u0026 5/3 Year Averages \u0026 Select\u003c/ol\u003e\n\u003col\u003e9. Predict future cumulative amounts\u003c/ol\u003e\n\u003col\u003e10. Calculate incremental amounts\u003c/ol\u003e\n\u003col\u003e11. Project future inflation for incremental amounts\u003c/ol\u003e\n\u003col\u003e12. Reserve summation\u003c/ol\u003e\n\n\n\u003cbr\u003e\n\u003ca href=\"https://medium.com/@DRicky.Ch29/web-scraping-pdf-tables-data-cleaning-part-1-cb6d8d47a6de\"\u003e\u003cstrong\u003eClick To View\u003c/strong\u003e\u003c/a\u003e\n\u003cbr\u003e\n\n\n\u003cbr\u003e\n\u003cbr\u003e\n\u003cbr\u003e\n\n\u003e # **Microsoft Package**\n\n\n\u003cbr\u003e\n\n# **Microsoft Package**\n\n\u003cbr\u003e\nPrior to learning Python coding language, I had to refine the basics. \nSince Excel \u0026 VBA are broadly deemed essential skill-sets, I thought \nI build some personal models. Ideas are inspired whilst at my work \nplacement tenure at a  consultancy company. The main objective was to \nease manual \u0026 repetitive tasking's.\n\u003cbr\u003e\n\n.\n\u003cbr\u003e\n## Word Documentations \n[Spreadsheet](https://www.dropbox.com/s/b4cgvhjui2mj0qq/Bulk%20MailMerge%20v2.0.xlsm?dl=0)\nor\n[Excel VBA Code](https://www.dropbox.com/s/b4cgvhjui2mj0qq/Bulk%20MailMerge%20v2.0.xlsm?dl=0)\n\u003cbr\u003e\n\n![ScreenShot](/Pictures/WordExcelLogo_1.png)\n\n\u003cbr\u003e\n\n\u003cbr\u003eA reproducible Excel VBA programme that automates bulk simultaneous word \ndocument mail merges. Data entry checks (file exists etc.) \u0026 cleaning (excess \nspaces, invalid file directory ...) are done by the coding as well. This code \ndoes NOT use the standard mail merge function that operates ONLY on 1-single \ndocument. Instead allows running on mass word documentations.\n\n\u003cbr\u003eInspiration:\n\u003cbr\u003eWhilst assisting my previous employer to prepare clients for the European \nGeneral Data Protection Regulations (GDPR) privacy documentations, I created \nthis programme to streamline over 30hours of manual work.\n\u003cbr\u003e\n\n.\n\u003cbr\u003e\n## Outlook Communications \n[Spreadsheet](https://www.dropbox.com/s/o50up79cttwyfa3/Bulk%20Emailing%20v2.0.xlsm?dl=0)\nor\n[Excel VBA Code](https://www.dropbox.com/s/o50up79cttwyfa3/Bulk%20Emailing%20v2.0.xlsm?dl=0)\n\u003cbr\u003e\n\n![ScreenShot](/Pictures/OutlookExcelLogo_1.png)\n\n\u003cbr\u003e\n\n\u003cbr\u003eA reproducible Excel VBA programme that automates multiple simultaneous email \ncommunications if recipients receive overlapping/same attachments or spreadsheet \ntables.\n\n\u003cbr\u003eInspiration:\n\u003cbr\u003eA responsibility of mine at a previous company involved weekly roll-forward \nprojection updates. I found this repetitive \u0026 build this model to automate the \njob. It mitigated manual human input errors \u0026 eased the job handing over\nprocess.\n\u003cbr\u003e\n\n\n\u003cbr\u003e\n\n\n\n\n\n\n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDeDeDeDer%2FPersonal_Projects","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FDeDeDeDer%2FPersonal_Projects","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDeDeDeDer%2FPersonal_Projects/lists"}