{"id":19703606,"url":"https://github.com/phantominsights/baby-names-analysis","last_synced_at":"2025-04-05T09:08:47.890Z","repository":{"id":36456009,"uuid":"197634754","full_name":"PhantomInsights/baby-names-analysis","owner":"PhantomInsights","description":"Data ETL \u0026 Analysis on the dataset 'Baby Names from Social Security Card Applications - National Data'.","archived":false,"fork":false,"pushed_at":"2021-11-14T22:41:28.000Z","size":9733,"stargazers_count":565,"open_issues_count":2,"forks_count":67,"subscribers_count":28,"default_branch":"master","last_synced_at":"2025-03-29T08:08:23.458Z","etag":null,"topics":["eda","etl","matplotlib","numpy","pandas","python-requests","python3","seaborn"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PhantomInsights.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"github":"agentphantom","patreon":"agentphantom"}},"created_at":"2019-07-18T18:06:25.000Z","updated_at":"2025-01-26T02:05:31.000Z","dependencies_parsed_at":"2022-08-17T23:40:39.506Z","dependency_job_id":null,"html_url":"https://github.com/PhantomInsights/baby-names-analysis","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PhantomInsights%2Fbaby-names-analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PhantomInsights%2Fbaby-names-analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PhantomInsights%2Fbaby-names-analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PhantomInsights%2Fbaby-names-analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PhantomInsights","download_url":"https://codeload.github.com/PhantomInsights/baby-names-analysis/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247312081,"owners_count":20918344,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["eda","etl","matplotlib","numpy","pandas","python-requests","python3","seaborn"],"created_at":"2024-11-11T21:18:23.836Z","updated_at":"2025-04-05T09:08:47.850Z","avatar_url":"https://github.com/PhantomInsights.png","language":"Python","funding_links":["https://github.com/sponsors/agentphantom","https://patreon.com/agentphantom","https://www.patreon.com/bePatron?u=20521425"],"categories":[],"sub_categories":[],"readme":"# Data ETL \u0026 Analysis Workflow\n\nIn this repository I will document the complete process of gathering data, transforming it and presenting it with tables and plots.\n\nThe dataset used in this project is the [Baby Names from Social Security Card Applications - National Data](https://catalog.data.gov/dataset/baby-names-from-social-security-card-applications-national-level-data) which includes records from the year 1880 to 2020.\n\n## Requirements\n\nYou will only require Python 3 and the following libraries:\n\n* Requests - Used to download the dataset.\n* pandas - Used for performing Data Analysis.\n* NumPy - Used for fast matrix operations.\n* Matplotlib - Used to create plots.\n* seaborn - Used to prettify Matplotlib plots.\n\n## Downloading and Transforming (ETL)\n\nThe first thing to do is to download the zip file containing all the data. To do so we will only require the `Requests` module.\n\n```python\nimport requests\n\nurl = \"https://www.ssa.gov/oact/babynames/names.zip\"\n\nwith requests.get(url) as response:\n\n    with open(\"names.zip\", \"wb\") as temp_file:\n        temp_file.write(response.content)\n```\n\nWith only those few lines of code we downloaded the zip file and saved it to our computer, the next step is to extract the files and compute their contents.\n\nThe zip file contains txt files that represent each year from 1880 to 2020 and one readme pdf file.\n\nWhat we are going to do now is to only read into memory the txt files, process their contents and save them into a new csv file.\n\nThis new csv file will be our dataset and will include 4 fields (year, name, gender and count).\n\n```python\nimport csv\nfrom zipfile import ZipFile\n\n# This list will hold all our data. We initialize it with the header row.\ndata_list = [[\"year\", \"name\", \"gender\", \"count\"]]\n\n# We first read the zip file using a zipfile.ZipFile object.\nwith ZipFile(\"names.zip\") as temp_zip:\n\n    # Then we read the file list.\n    for file_name in temp_zip.namelist():\n\n        # We will only process .txt files.\n        if \".txt\" in file_name:\n\n            # Now we read the current file from the zip file.\n            with temp_zip.open(file_name) as temp_file:\n\n                # The file is opened as binary, we decode it using utf-8 so it can be manipulated as a string.\n                for line in temp_file.read().decode(\"utf-8\").splitlines():\n\n                    # We prepare our desired data fields and add them to the data list.\n                    line_chunks = line.split(\",\")\n                    year = file_name[3:7]\n                    name = line_chunks[0]\n                    gender = line_chunks[1]\n                    count = line_chunks[2]\n\n                    data_list.append([year, name, gender, count])\n\n# We save the data list into a csv file.\ncsv.writer(open(\"data.csv\", \"w\", newline=\"\",\n                encoding=\"utf-8\")).writerows(data_list)\n```\n\nFirst we start by declaring `data_list` which will be a 2-dimensional Array (list of lists). The purpose of this is so it can be later saved into a `csv` file using the `csv.writer.writerows()` method. I prefer to use `writerows()` instead of `writerow()` since it is faster as it does it in bulk instead of one row at a time.\n\nThen we open the zip file into memory using a `zipfile.ZipFile()` object. Afterwards we iterate over all the files inside it and only read those that contain actual data.\n\nSince they are read in binary form we decode them using `utf-8` so we can use String manipulation on them.\n\nThe txt files content looks like the following example:\n\n```\nAshley,F,9403\nElla,F,9352\nSarah,F,9045\nGrace,F,9036\n```\n\nWe use the string `split(\",\")` method to separate each line by the comma. Then we will model each data field into their respective variable and append it to the `data_list`.\n\nThis newly created csv file can be imported into any `SQL` database or other statistical tools or programming languages. We will continue to use `Python` for this project.\n\n## Data Exploration (EDA)\n\nNow that we have a csv file containing almost two million rows it's time to get some insights from it. For this we will use the `pandas` library.\n\nThe first thing to do is to import the libraries using the recommended method.\n\n```python\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\n```\n\nAfterwards we will read the csv file using the `pandas.read_csv()` method.\n\n```python\ndf = pd.read_csv(\"data.csv\")\n```\n\nSince our dataset is very simple we won't require to set any special flags or extra parameters, the default ones are good enough.\n\nLet's start with the classic `df.head()` and `df.tail()` methods which are very handy for taking a sneak peek at the dataset.\n\n```python\n# Head (First 5 rows)\ndf.head()\n```\n\n| | year | name | gender | count |\n| --- | --- | --- | --- | --- |\n| 0 | 1880 | Mary | F | 7065 |\n| 1 | 1880 | Anna | F | 2604 |\n| 2 | 1880 | Emma | F | 2003 |\n| 3 | 1880 | Elizabeth | F | 1939 |\n| 4 | 1880 | Minnie | F | 1746 |\n\n```python\n# Tail (Last 5 rows)\ndf.tail()\n```\n\n| | year | name | gender | count |\n| --- | --- | --- | --- | --- |\n| 1957041 | 2018 | Zylas | M | 5 |\n| 1957042 | 2018 | Zyran | M | 5 |\n| 1957043 | 2018 | Zyrie | M | 5 |\n| 1957044 | 2018 | Zyron | M | 5 |\n| 1957045 | 2018 | Zzyzx | M | 5 |\n\nThese samples tells us several things:\n\n* There are 4 columns (year, name, gender and count).\n* There are 1,957,046 rows.\n* Rows are sorted by year.\n* Female records are shown before male ones.\n* At least 5 parents in 2018 named their son 'Zzyzx'.\n\n*Note: The dataset only includes names which have at least 5 records, this is for privacy reasons.*\n\nLet's get more interesting insights.\n\n### Unique Names\n\nThere are 98,400 unique names in the dataset. From those, 41,475 are male names, 67,698 are female ones and 10,773 are gender neutral.\n\n```python\n# Unique names either gender.\ndf[\"name\"].nunique()\n\n# Unique names for male.\ndf[df[\"gender\"] == \"M\"][\"name\"].nunique()\n\n# Unique names for female.\ndf[df[\"gender\"] == \"F\"][\"name\"].nunique()\n\n# Unique names for gender neutral.\nboth_df = df.pivot_table(index=\"name\", columns=\"gender\", values=\"count\", aggfunc=np.sum).dropna()\nboth_df.index.nunique()\n```\n\n### Top 10 Male and Female Names\n\nTo get the top 10 most used male and female names we are going to first filter the `dataframe` by gender.\n\nOnce we have a gender specific `dataframe` we wiil select only 2 fields, name and count. From there we will use the `groupby()` method on the name field and aggregate the results using a `sum()`.\n\nFinally, we will sort the values on the count field in descending order and use the `head(10)` method to get the top 10 results.\n\n```python\n# Step by step approach, the one-liners can be found below their respective tables.\ndf = df[df[\"gender\"] == \"M\"]\ndf = df[[\"name\", \"count\"]]\ndf = df.groupby(\"name\")\ndf = df.sum()\ndf = df.sort_values(\"count\", ascending=False)\ndf.head(10)\n```\n\n| Name (Male) | Total Count |\n| --- | --- |\n| James | 5,164,280 |\n| John | 5,124,817 |\n| Robert | 4,820,129 |\n| Michael | 4,362,731 |\n| William | 4,117,369 |\n| David | 3,621,322 |\n| Joseph | 2,613,304 |\n| Richard | 2,565,301 |\n| Charles | 2,392,779 |\n| Thomas | 2,311,849 |\n\n ```python\n df[df[\"gender\"] == \"M\"][[\"name\", \"count\"]].groupby(\"name\").sum().sort_values(\"count\", ascending=False).head(10)\n ```\n\n| Name (Female) | Total Count |\n| --- | --- |\n| Mary | 4,125,675 |\n| Elizabeth | 1,638,349 |\n| Patricia | 1,572,016 |\n| Jennifer | 1,467,207 |\n| Linda | 1,452,668 |\n| Barbara | 1,434,397 |\n| Margaret | 1,248,985 |\n| Susan | 1,121,703 |\n| Dorothy | 1,107,635 |\n| Sarah | 1,077,746 |\n\n ```python\n df[df[\"gender\"] == \"F\"][[\"name\", \"count\"]].groupby(\"name\").sum().sort_values(\"count\", ascending=False).head(10)\n ```\n\n### Top 20 Gender Neutral Names\n\nThis one was a bit challenging, first we need to pivot the `dataframe` so the names are the index, the genders will be the columns and the sum of all counts (per name, per gender) will be our values.\n\nWe are going to do this in small steps. First we pivot the table and drop the rows where the value is 0. This means rows where names are not present in either male or female categories.\n\n```python\ndf.pivot_table(index=\"name\", columns=\"gender\", values=\"count\", aggfunc=np.sum).dropna()\n```\n\nThis will output us something like the following table:\n\n| gender | F | M |\n| --- | --- | --- |\n| name | | |\n| Aaden | 5.0 | 4828.0 |\n| Aadi | 16.0 | 851.0 |\n| Aadyn | 16.0 | 516.0 |\n| Aalijah | 149.0 | 212.0 |\n| Aaliyah | 87442.0 | 96.0 |\n\nWith the data in this shape we now know how many records each name has per gender.\n\nNow we will only take into account those names that atleast have 50,000 records for each gender.\n\n```python\ndf = df[(df[\"M\"] \u003e= 50000) \u0026 (df[\"F\"] \u003e= 50000)]\ndf.head(20)\n```\n\nFinally, we request the top 20 results using the `head(20)` method.\n\n*Note: Originally I was going to only take the top 10 names but I pushed it to 20 since there are only 20 names that meet our criteria.*\n\n| Name | Female Records | Male Records |\n| --- | --- | --- |\n| Alexis | 338,333 | 63,604 |\n| Angel | 95,710 | 231,800 |\n| Avery | 125,883 | 55,646 |\n| Casey | 76,312 | 110,635 |\n| Dana | 191,812 | 53,098 |\n| Jackie | 90,705 | 78,494 |\n| Jamie | 268,102 | 85,631 |\n| Jessie | 167,462 | 110,212 |\n| Jordan | 131,004 | 374,513 |\n| Kelly | 471,502 | 81,652 |\n| Lee | 62,143 | 231,130 |\n| Leslie | 267,081 | 112,726 |\n| Lynn | 181,904 | 52,268 |\n| Marion | 188,391 | 72,075 |\n| Riley | 106,901 | 94,278 |\n| Shannon | 295,024 | 51,999 |\n| Taylor | 320,446 | 110,390 |\n| Terry | 96,895 | 422,916 |\n| Tracy | 250,853 | 61,223 |\n| Willie | 146,156 | 448,946 |\n\n\n ### Highest and Lowest Years\n\nNow we will know which years had the highest and lowest amount of records by gender and combined.\n\n\n\n```python\nboth_df = df.groupby(\"year\").sum()\nmale_df = df[df[\"gender\"] == \"M\"].groupby(\"year\").sum()\nfemale_df = df[df[\"gender\"] == \"F\"].groupby(\"year\").sum()\n\n# Combined Min (count and year)\nboth_df.min()[\"count\"]\nboth_df.idxmin()[\"count\"]\n\n# Male Min (count and year)\nmale_df.min()[\"count\"]\nmale_df.idxmin()[\"count\"]\n\n# Female Min (count and year)\nfemale_df.min()[\"count\"]\nfemale_df.idxmin()[\"count\"]\n\n# Combined Max (count and year)\nboth_df.max()[\"count\"]\nboth_df.idxmax()[\"count\"]\n\n# Male Max (count and year)\nmale_df.max()[\"count\"]\nmale_df.idxmax()[\"count\"]\n\n# Female Max (count and year)\nfemale_df.max()[\"count\"]\nfemale_df.idxmax()[\"count\"]\n```\n\n| Gender and Attribute | Total Count | Year |\n| --- | --- | --- |\n| Both Min | 192,696 | 1881 |\n| Male Min | 100,743 | 1881 |\n| Female Min | 90,994 | 1880 |\n| Both Max | 4,200,022 | 1957 |\n| Male Max | 2,155,711 | 1957 |\n| Female Max | 2,044,311 | 1957 |\n\nThe year 1881 got the lowest records on the dataset, while the year 1957 got the highest records.\n\nSo far we got several interesting insights, it's time to create some pretty plots.\n\n## Plotting the Data\n\nFor creating the plots we will use `seaborn` and `matplotlib`, the reason for this is that seaborn applies some subtle yet nice looking effects to the plots.\n\nIn this project we are only going to use line plots, which are very helpful for displaying how a value changes over time.\n\nThe first thing to do is to set some custom colors that will apply globally to each plot.\n\n\n```python\n# Those parameters generate plots with a mauve color.\nsns.set(style=\"ticks\",\n        rc={\n            \"figure.figsize\": [12, 7],\n            \"text.color\": \"white\",\n            \"axes.labelcolor\": \"white\",\n            \"axes.edgecolor\": \"white\",\n            \"xtick.color\": \"white\",\n            \"ytick.color\": \"white\",\n            \"axes.facecolor\": \"#443941\",\n            \"figure.facecolor\": \"#443941\"}\n        )\n```\n\nWith our style declared we are ready to plot our data.\n\n*Note: The next code blocks are more advanced than the previous ones. I also make heavy use of one-liners for efficiency reasons, but don't worry, I will explain what each line does.*\n\n### Counts by Year\n\nOur first plot will consist on how the number of records has moved from 1880 to 2020.\n\nFirst, we create new dataframes for male, female and combined.\n\n```python\nboth_df = df.groupby(\"year\").sum()\nmale_df = df[df[\"gender\"] == \"M\"].groupby(\"year\").sum()\nfemale_df = df[df[\"gender\"] == \"F\"].groupby(\"year\").sum()\n```\n\nWe plot our dataframes directly. The x-axis will be the index and the y-axis will be the total counts.\n\n```python\nplt.plot(both_df, label=\"Both\", color=\"yellow\")\nplt.plot(male_df, label=\"Male\", color=\"lightblue\")\nplt.plot(female_df, label=\"Female\", color=\"pink\")\n```\n\nWe make our yticks in steps of 500,000. First we format the numbers for the labels. Then we use the actual numbers as the steps.\n\n```python\nyticks_labels = [\"{:,}\".format(i) for i in range(0, 4500000+1, 500000)]\nplt.yticks(np.arange(0, 4500000+1, 500000), yticks_labels)\n```\n\nWe add the final customizations.\n\n```python\nplt.legend()\nplt.grid(False)\nplt.xlabel(\"Year\")\nplt.ylabel(\"Records Count\")\nplt.title(\"Records per Year\")\nplt.show()\n```\n\n![Counts by Year](./figs/total_by_year.png)\n\n### Most Popular Names Growth\n\nFor our next plot we will observe how the all-time most popular names have grown over the years.\n\nFirst, we merge values from male and female and pivot the table so the names are our index and the years are our columns. We also fill missing values with zeroes.\n\n```python\npivoted_df = df.pivot_table(index=\"name\", columns=\"year\", values=\"count\", aggfunc=np.sum).fillna(0)\n```\n\nThen we calculate the percentage of each name by year.\n\n```python\npercentage_df = pivoted_df / pivoted_df.sum() * 100\n```\nWe add a new column to store the cumulative percentages sum.\n\n```python\npercentage_df[\"total\"] = percentage_df.sum(axis=1)\n```\n\nWe sort the dataframe to check which are the top values and slice it. After that we drop the `total` column since it won't be used anymore.\n\n```python\nsorted_df = percentage_df.sort_values(by=\"total\", ascending=False).drop(\"total\", axis=1)[0:10]\n```\n\nWe flip the axes so we can plot the data more easily.\n\n```python\ntransposed_df = sorted_df.transpose()\n```\n\nWe plot each name individually by using the column name as the label and Y-axis.\n\n```python\nfor name in transposed_df.columns.tolist():\n    plt.plot(transposed_df.index, pivoted_df[name], label=name)\n```\n\nWe set our yticks in steps of 0.5%.\n\n```python\nyticks_labels = [\"{}%\".format(i) for i in np.arange(0, 5.5, 0.5)]\nplt.yticks(np.arange(0, 5.5, 0.5), yticks_labels)\n```\n\nWe add the final customizations.\n\n```python\nplt.legend()\nplt.grid(False)\nplt.xlabel(\"Year\")\nplt.ylabel(\"Percentage by Year\")\nplt.title(\"Top 10 Names Growth\")\nplt.show()\n```\n\n![Most Popular Growth](./figs/most_popular_growth.png)\n\n### Top 10 Trending Names\n\nFor our last plot we will get the top 10 trending names in the last 10 years (2008-2018).\n\nThis one is very similar to the previous one, the first thing to do is to remove all records that are older than 2008.\n\n```python\nfiltered_df = df[df[\"year\"] \u003e= 2008]\n```\n\nThen we merge values from male and female and pivot the table so the names are our index and the years are our columns. We also fill missing values with zeroes.\n\n```python\npivoted_df = filtered_df.pivot_table(index=\"name\", columns=\"year\", values=\"count\", aggfunc=np.sum).fillna(0)\n```\n\nThen we calculate the percentage of each name by year.\n\n```python\npercentage_df = pivoted_df / pivoted_df.sum() * 100\n```\n\nWe add a new column to store the cumulative percentages sum.\n\n```python\npercentage_df[\"total\"] = percentage_df.sum(axis=1)\n```\n\nWe sort the dataframe to check which are the top values and slice it. After that we drop the `total` column since it won't be used anymore.\n\n```python\nsorted_df = percentage_df.sort_values(\"total\", ascending=False).drop(\"total\", axis=1)[0:10]\n```\n\nWe flip the axes so we can plot the dataframe more easily.\n\n```python\ntransposed_df = sorted_df.transpose()\n```\n\nWe plot each name individually by using the column name as the label and Y-axis.\n\n```python\nfor name in transposed_df.columns.tolist():\n    plt.plot(transposed_df.index, transposed_df[name], label=name)\n```\n\nWe set our yticks in steps of 0.05%.\n\n```python\nyticks_labels = [\"{:.2f}%\".format(i) for i in np.arange(0.3, 0.7, 0.05)]\nplt.yticks(np.arange(0.3, 0.7, 0.05), yticks_labels)\n```\n\nWe set our xticks in steps of 1, from 2008 to 2018.\n\n```python\nxticks_labels = [\"{}\".format(i) for i in range(2008, 2618+1, 1)]\nplt.xticks(np.arange(2008, 2018+1, 1), xticks_labels)\n```\n\nWe add the final customizations.\n\n```python\nplt.legend()\nplt.grid(False)\nplt.xlabel(\"Year\")\nplt.ylabel(\"Percentage by Year\")\nplt.title(\"Top 10 Trending Names\")\nplt.show()\n```\n\n![Trending Names](./figs/trending_names.png)\n\n## Conclusion\n\nI wanted to work on this project for quite some time and I finally got the time to do it. Years ago I worked with this dataset in another project but I always wanted to get the hidden insights from it.\n\nI hope you had fun reading it and you have learned some new skills.\n\nAs always, if you have any questions feel free to open an issue and I will try to answer them in a timely manner.\n\n[![Become a Patron!](https://c5.patreon.com/external/logo/become_a_patron_button.png)](https://www.patreon.com/bePatron?u=20521425)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphantominsights%2Fbaby-names-analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fphantominsights%2Fbaby-names-analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphantominsights%2Fbaby-names-analysis/lists"}