{"id":15173916,"url":"https://github.com/mariadb-corporation/dev-example-columnstore-developer-survey-data","last_synced_at":"2026-01-27T04:32:47.561Z","repository":{"id":66164588,"uuid":"399599747","full_name":"mariadb-corporation/dev-example-columnstore-developer-survey-data","owner":"mariadb-corporation","description":"This repository provides data and information that will enable you to transform, import and analyze the raw Stack Overflow Annual Developer Survey data with MariaDB ColumnStore.","archived":false,"fork":false,"pushed_at":"2022-01-10T10:53:43.000Z","size":9123,"stargazers_count":0,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-30T12:22:18.130Z","etag":null,"topics":["dbaas","jupyter-notebook","jupyterlab","mariadb","mariadb-columnstore","mariadb-database","mariadb-server","mariadb-skysql","pandas","pandas-dataframe","plotly","python","python3","rdbms","rdbms-import","sql"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mariadb-corporation.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-08-24T20:43:10.000Z","updated_at":"2022-01-10T10:53:46.000Z","dependencies_parsed_at":"2023-04-05T11:17:07.698Z","dependency_job_id":null,"html_url":"https://github.com/mariadb-corporation/dev-example-columnstore-developer-survey-data","commit_stats":{"total_commits":5,"total_committers":2,"mean_commits":2.5,"dds":"0.19999999999999996","last_synced_commit":"d94ca65752d348c9631f56be0f4af23b9e0e191a"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mariadb-corporation/dev-example-columnstore-developer-survey-data","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mariadb-corporation%2Fdev-example-columnstore-developer-survey-data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mariadb-corporation%2Fdev-example-columnstore-developer-survey-data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mariadb-corporation%2Fdev-example-columnstore-developer-survey-data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mariadb-corporation%2Fdev-example-columnstore-developer-survey-data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mariadb-corporation","download_url":"https://codeload.github.com/mariadb-corporation/dev-example-columnstore-developer-survey-data/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mariadb-corporation%2Fdev-example-columnstore-developer-survey-data/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28802063,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-27T03:44:14.111Z","status":"ssl_error","status_checked_at":"2026-01-27T03:43:33.507Z","response_time":168,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dbaas","jupyter-notebook","jupyterlab","mariadb","mariadb-columnstore","mariadb-database","mariadb-server","mariadb-skysql","pandas","pandas-dataframe","plotly","python","python3","rdbms","rdbms-import","sql"],"created_at":"2024-09-27T11:04:44.836Z","updated_at":"2026-01-27T04:32:47.544Z","avatar_url":"https://github.com/mariadb-corporation.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Analyzing Stack Overflow Annual Developer Survey Data with MariaDB\n\n\n⚠️ **[UNMAINTAINED]** This repository has been moved and is currently maintained [here](https://github.com/mariadb-developers/columnstore-developer-survey-data). ⚠️\n\n\u003cbr /\u003e\n\nThis repository provides data and information that will enable you to transform, import and analyze the [raw Stack Overflow Annual Developer Survey data](https://insights.stackoverflow.com/survey) with [MariaDB](https://mariadb.com) [ColumnStore](https://mariadb.com/docs/features/mariadb-columnstore/). \n\nStarting at [Step 2, Parse and transform the data,](#parse-transform) this repository will walk you through the process of preparing, importing and, ultimately, being able to analyze the [raw Stack Overflow Annual Developer Survey data for 2020](https://drive.google.com/file/d/1dfGerWeWkcyQ9GX9x20rdSGj7WtEpzBB/view) (which is included in the [developer_survey_2020](developer_survey_2020) folder). \n\n**Note:** This repository will be updated to include the raw 2021 data once it becomes available [here](https://insights.stackoverflow.com/survey).\n\n# Table of Contents\n1. [Requirements](#requirements)\n2. [Parse and transform the data](#parse-transform)\n3. [Prepare the database](#prepare-db)\n    1. [Docker Container](#docker)\n    2. [MariaDB SkySQL](#skysql)\n4. [Create the schema](#schema)\n5. [Import the data](#import)\n6. [Analyze the data](#analyze)\n    1. [SQL](#docker)\n    2. [MariaDB SkySQL](#skysql)\n7. [Support and contribution](#support-contribution)\n8. [License](#license)\n\n## Requirements \u003ca name=\"requirements\"\u003e\u003c/a\u003e\n\n* [MariaDB Client](https://mariadb.com/products/skysql/docs/clients/), used to connect to MariaDB database instances.\n* [Python v. 3+](https://www.python.org/downloads/)\n\n## Parsing and transforming the data \u003ca name=\"parse-transform\"\u003e\u003c/a\u003e\n\nIn this sample you will use the Python script file, [parse_and_transform.py](parse_and_transform.py), to parse and transform the [Stack Overflow Annual Survey data for 2020](/developer_survey_2020).\n\nExecuting [parse_and_transform.py](parse_and_transform.py) will parse through the [survey results data] and split each row, which contains all the responses from an single survey respondent, into multiple rows, one per response. The result of turns ~66k rows of data into ~5 million rows. \n\nTo execute the [parse_and_transform.py](parse_and_transform.py) you need to perform the following steps.\n\n1. Open a new terminal window at this location.\n\n2. Create a new Python virtual environment.\n\n```bash\n$ python3 -m venv venv\n```\n\n3. Activate the virtual environment.\n\n```bash \n$ . venv/bin/activate\n```\n\n4. Install the pandas Python package, which will be use for data manipulation within [parse_and_transform.py](parse_and_transform.py).\n\n```bash\n$ pip install pandas\n```\n\n5. And, finally, execute the [parse_and_transform.py](parse_and_transform.py) script!\n\n```bash\n$ python3 parse_and_transform.py\n```\n\n## Preparing the database \u003ca name=\"import\"\u003e\u003c/a\u003e\n\nTo be able to store and analyze the survey data you're going to need a place to put it. MariaDB to the rescue! Below includes instructions on setting up a local database, using the [official MariaDB Docker image](https://hub.docker.com/_/mariadb), or [MariaDB SkySQL](https://mariadb.com/products/skysql/), the ultimate MariaDB database in the cloud. \n\n### Docker Container \u003ca name=\"docker\"\u003e\u003c/a\u003e\n\nRunning a single instance (container) of MariaDB ColumnStore is incredibly simple using the MariaDB Community Server ColumnStore image. \n\nCheck out the instructions [here](https://hub.docker.com/r/mariadb/columnstore).\n\n### MariaDB SkySQL \u003ca name=\"skysql\"\u003e\u003c/a\u003e\n\n[SkySQL](https://mariadb.com/products/skysql/) is the first and only database-as-a-service (DBaaS) to bring the full power of MariaDB Platform to the cloud, including its support for transactional, analytical and hybrid workloads. Built on Kubernetes, and optimized for cloud infrastructure and services, SkySQL combines ease of use and self-service with enterprise reliability and world-class support – everything needed to safely run mission-critical databases in the cloud, and with enterprise governance.\n\n[Get started with SkySQL!](https://mariadb.com/products/skysql/#get-started)\n\n\u003cp align=\"center\" spacing=\"10\"\u003e\n    \u003ckbd\u003e\n        \u003cimg src=\"media/skysql.png\" /\u003e\n    \u003c/kbd\u003e\n\u003c/p\u003e\n\n**IMPORTANT:** Once you've registered for MariaDB SkySQL you will need to create a new analytics service so that you can take advantage of the MariaDB columnar storage engine, [ColumnStore](https://mariadb.com/docs/features/mariadb-columnstore/). For more information on how to do this check out [this walk-through](https://mariadb.com/resources/blog/getting-started-with-mariadb-skysql-for-analytics/), or check out this short video on [launching a new SkySQL service](https://www.youtube.com/watch?v=jr9oPHALmr4) - don't worry it only takes a couple of minutes! \n\n## Create the schema \u003ca name=\"schema\"\u003e\u003c/a\u003e\n\nThe survey result data contained in newly created **answers.csv** file will need to be imported to MariaDB. To accomodate that you will need to create a new database, `survey_data`, that contains a single table, `answers`.\n\nTo create the new database and table you can either copy and execute the following code within a database client of your choice.\n\n```sql\nDROP DATABASE IF EXISTS survey_data;\nCREATE DATABASE survey_data;\n\nCREATE TABLE answers (\n    respondent_id INT unsigned NOT NULL, \n    question_id VARCHAR(25) NOT NULL,\n    answer VARCHAR(65) NOT NULL\n) ENGINE=ColumnStore DEFAULT CHARSET=utf8;\n```\n\nor use the MariaDB Client to execute the [schema.sql](schema.sql) script contained within this repository.\n\nFor example: \n\n_Locally_\n```bash\n$ mariadb --host 127.0.0.1 --user root -pPassword123! \u003c schema.sql\n```\n\n_SkySQL_\n```bash\nmariadb --host analytics-1.mdb0001265.db.skysql.net --port 5001 --user DB00004537 -p --ssl-ca ~/Downloads/skysql_chain.pem \u003c schema.sql\n```\n\n**Note:** Remember to update the command above with your database location, user and SSL information accordingly!\n\n## Import the data \u003ca name=\"import\"\u003e\u003c/a\u003e\n\nAfter you've created the new schema, you can import the answers.csv data using the MariaDB Client. \n\nFor example:\n\n_Locally_\n```bash\nmariadb --host 127.0.0.1 --port 3306 --user root -pPassword123! -e \"LOAD DATA LOCAL INFILE 'answers.csv' INTO TABLE answers FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\\\"' LINES TERMINATED BY '\\n'\" survey_data\n```\n\n_SkySQL_\n```bash\nmariadb --host analytics-1.mdb0001265.db.skysql.net --port 5001 --user DB00004537 -p --ssl-ca ~/Downloads/skysql_chain.pem -e \"LOAD DATA LOCAL INFILE 'answers.csv' INTO TABLE answers FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\\\"' LINES TERMINATED BY '\\n'\" survey_data\n```\n\n**Note:** Remember to update the command above with your database location, user and SSL information accordingly!\n\n## Analyze the data \u003ca name=\"analyze\"\u003e\u003c/a\u003e\n\nOnce the data has been successfully imported into MariaDB there are many ways that you can use the data!\n\n### SQL \u003ca name=\"skysql\"\u003e\u003c/a\u003e\n\nYou can use a database client to execute SQL queries directly on the results data contained in the `answers` table. \n\nFor exmple, using the MariaDB Client:\n\n1. Start by connecting to your MariaDB database instance. \n\n```bash\n$ mariadb --host 127.0.0.1 --user root -pPassword123!\n```\n\n2. Execute the following query to `SELECT` the top 10 programming langauges that have been used by respondents that have also used MariaDB.\n\n```sql\nSELECT\n\tanswer, COUNT(answer) AS respondent_count\nFROM\n\tsurvey_data.answers\nWHERE \n\tquestion_id = \"LanguageWorkedWith\" AND \n\trespondent_id IN (SELECT respondent_id FROM answers WHERE question_id = \"DatabaseWorkedWith\" AND answer = \"MariaDB\")\nGROUP BY\n\tanswer\nORDER BY\n\tCOUNT(answer) DESC\nLIMIT 10;\n```\n\n```\n+-----------------------+------------------+\n| answer                | respondent_count |\n+-----------------------+------------------+\n| JavaScript            |             6878 |\n| HTML/CSS              |             6597 |\n| SQL                   |             6239 |\n| PHP                   |             5149 |\n| Python                |             4204 |\n| Java                  |             4028 |\n| Bash/Shell/PowerShell |             3746 |\n| TypeScript            |             2558 |\n| C#                    |             2396 |\n| C++                   |             2277 |\n+-----------------------+------------------+\n```\n\n### Python \u0026 Jupyter Lab \u003ca name=\"skysql\"\u003e\u003c/a\u003e\n\nYou can also use modern data analysis and visualization tools like [Jupyter Lab](https://jupyter.org/), in combination with [MariaDB Connector/Python](https://mariadb.com/docs/clients/mariadb-connectors/connector-python/) and [Python](https://www.python.org/) libraries like [Plotly](https://plotly.com/) and [Pandas](https://pandas.pydata.org/). \n\nFor more information on how you can do this please check out the following resources:\n\n* [Using Data Analysis and Visualization with MariaDB Connector/Python](https://github.com/mariadb-corporation/dev-example-connector-python/blob/master/samples/analysis/README.md) (GitHub Repository)\n\n* [Deep dive: Taking advantage of MariaDB Connector for Python](https://go.mariadb.com/21Q4-WBN-GLBL-OSSG-Python-Connector-2021-08-26_Registration-LP.html?_ga=2.247064967.2067136692.1629673971-817046405.1628296957\u0026_gac=1.61026014.1629298886.CjwKCAjw3_KIBhA2EiwAaAAlitUQQTxd3HGd6jjzQw3h2pLtya8fOTTptfZol-3Fh96QbVwm4Ek9VxoC-McQAvD_BwE) (Webinar) \n\n## Support and Contribution \u003ca name=\"support-contribution\"\u003e\u003c/a\u003e\n\nPlease feel free to submit PR's, issues or requests to this project project or projects within the [official MariaDB Corporation GitHub organization](https://github.com/mariadb-corporation).\n\nIf you have any other questions, comments, or looking for more information on MariaDB please check out:\n\n* [MariaDB Developer Hub](https://mariadb.com/developers)\n* [MariaDB Community Slack](https://r.mariadb.com/join-community-slack)\n\nOr reach out to us diretly via:\n\n* [developers@mariadb.com](mailto:developers@mariadb.com)\n* [MariaDB Twitter](https://twitter.com/mariadb)\n\n## License \u003ca name=\"license\"\u003e\u003c/a\u003e\n[![License](https://img.shields.io/badge/License-MIT-blue.svg?style=plastic)](https://opensource.org/licenses/MIT)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmariadb-corporation%2Fdev-example-columnstore-developer-survey-data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmariadb-corporation%2Fdev-example-columnstore-developer-survey-data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmariadb-corporation%2Fdev-example-columnstore-developer-survey-data/lists"}