{"id":13468881,"url":"https://github.com/tirthajyoti/pydbgen","last_synced_at":"2025-04-05T04:13:40.013Z","repository":{"id":49772844,"uuid":"124701797","full_name":"tirthajyoti/pydbgen","owner":"tirthajyoti","description":"Random dataframe and database table generator","archived":false,"fork":false,"pushed_at":"2021-06-09T23:32:03.000Z","size":703,"stargazers_count":309,"open_issues_count":4,"forks_count":58,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-03-29T03:11:36.085Z","etag":null,"topics":["data-generation","data-science","database","fake-data","generator","pandas-dataframe","python","random-generation","sqlite","sqlite3","synthetic-data","synthetic-dataset-generation"],"latest_commit_sha":null,"homepage":"https://pydbgen.readthedocs.io/en/latest/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tirthajyoti.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-03-10T21:57:58.000Z","updated_at":"2025-03-14T17:14:03.000Z","dependencies_parsed_at":"2022-08-03T02:15:41.027Z","dependency_job_id":null,"html_url":"https://github.com/tirthajyoti/pydbgen","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tirthajyoti%2Fpydbgen","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tirthajyoti%2Fpydbgen/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tirthajyoti%2Fpydbgen/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tirthajyoti%2Fpydbgen/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tirthajyoti","download_url":"https://codeload.github.com/tirthajyoti/pydbgen/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247284951,"owners_count":20913704,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-generation","data-science","database","fake-data","generator","pandas-dataframe","python","random-generation","sqlite","sqlite3","synthetic-data","synthetic-dataset-generation"],"created_at":"2024-07-31T15:01:21.153Z","updated_at":"2025-04-05T04:13:39.983Z","avatar_url":"https://github.com/tirthajyoti.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"************************************\nRandom database/dataframe generator\n************************************\n\nAuthored and maintained by `Dr. Tirthajyoti Sarkar, Fremont, USA \u003chttps://www.linkedin.com/in/tirthajyoti-sarkar-2127aa7/\u003e`_\n\nIntroduction\n=============\nOften, beginners in SQL or data science struggle with the matter of easy access to a large sample database file (**.DB** or **.sqlite**) for practicing SQL commands. **Would it not be great to have a simple tool or library to generate a large database with multiple tables, filled with data of one's own choice?**\n\nAfter all, databases break every now and then and it is safest to practice with a randomly generated one :-)\n\n.. image:: https://imgs.xkcd.com/comics/exploits_of_a_mom.png\n\n\nWhile it is easy to generate random numbers or simple words for Pandas or dataframe operation learning, it is often **non-trivial to generate full data tables with meaningful yet random entries of most commonly encountered fields in the world of database**, such as \n\n- name, \n- age,\n- birthday, \n- credit card number, \n- SSN, \n- email id, \n- physical address, \n- company name, \n- job title,\n\nThis Python package generates a random database ``TABLE`` (or a Pandas dataframe, or an Excel file) based on user's choice of data types (database fields). User can specify the number of samples needed. One can also designate a **\"PRIMARY KEY\"** for the database table. Finally, the ``TABLE`` is inserted into a new or existing database file of user's choice.\n\n.. image:: https://raw.githubusercontent.com/tirthajyoti/pydbgen/master/images/Top_image_1.png\n\nDependency and Acknowledgement\n===============================\nAt its core, ``pydbgen`` uses ``Faker`` as the default random data generating engine for most of the data types. Original function is written for few data types such as ``realistic email`` and ``license plate``. Also the default phone number generated by ``Faker`` is free-format and does not correspond to US 10 digit format. Therefore, a ``simple phone number`` data type is introduced in ``pydbgen``. The original contribution of ``pydbgen`` is to take the single data-generating function from ``Faker`` and use it cleverly to generate Pandas data series or dataframe or SQLite database tables as per the specification of the user.\nHere is the link if you want to look up more about ``Faker`` package,\n\n`Faker Documentation Home \u003chttps://faker.readthedocs.io/en/latest/index.html\u003e`_\n\nInstallation\n=============\n(On Linux and Windows) You can use ``pip`` to install ``pydbgen``::\n\n\tpip install pydbgen\n\n(On Mac OS), first install pip, ::\n\t\n\tcurl https://bootstrap.pypa.io/get-pip.py -o get-pip.py\n\tpython get-pip.py\n\nThen proceed as above.\n\nUsage\n=========\nCurrent version (1.0.0) of ``pydbgen`` comes with the following primary methods,\n\n* ``gen_data_series()``\n* ``gen_dataframe()``\n* ``gen_table()``\n* ``gen_excel()``\n\nThe ``gen_table()`` method allows you to build a database with as many tables as you want, filled with random data and fields of your choice. But first, you have to create an object of ``pydb`` class::\n\n\tmyDB = pydbgen.pydb()\n\n``gen_data_series()``\n----------------------\nReturns a `Pandas series object \u003chttps://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.html\u003e`_ with the desired number of entries and data type. Data types available:\n \n* Name, country, city, real (US) cities, US state, zipcode, latitude, longitude\n* Month, weekday, year, time, date\n* Personal email, official email, SSN \n* Company, Job title, phone number, license plate\n        \nPhone number can be of two types: \n\n* ``phone_number_simple`` generates 10 digit US number in xxx-xxx-xxxx format\n* ``phone_number_full`` may generate an international number with different format\n\n**Code example**::\n\t\n\tse=myDB.gen_data_series(data_type='date')\n\tprint(se)\n\n\t0    1995-08-09\n\t1    2001-08-01\n\t2    1980-06-26\n\t3    2018-02-18\n\t4    1972-10-12\n\t5    1983-11-12\n\t6    1975-09-04\n\t7    1970-11-01\n\t8    1978-03-23\n\t9    1976-06-03\n\tdtype: object\n\n``gen_dataframe()``\n---------------------\nGenerates a `Pandas dataframe \u003chttps://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html\u003e`_ filled with random entries. User can specify the number of rows and data type of the fields/columns. \n\n* Name, country, city, real (US) cities, US state, zipcode, latitude, longitude\n* Month, weekday, year, time, date\n* Personal email, official email, SSN \n* Company, Job title, phone number, license plate\nCustomization choices are following:\n\n- ``real_email``: If ``True`` and if a person's name is also included in the fields, a realistic email will be generated corresponding to the name of the person. For example, ``Tirtha Sarkar`` name with this choice enabled, will generate emails like ``TSarkar21@gmail.com`` or ``Sarkar.Tirtha@att.net``.\n- ``real_city``: If ``True``, a real US city's name will be picked up from a list (included as a text data file with the installation package). Otherwise, a fictitious city name will be generated.\n- ``phone_simple``: If ``True``, a 10 digit US number in the format xxx-xxx-xxxx will be generated. Otherwise, an international number with different format may be returned.\n\n**Code example**::\n\n\ttestdf=myDB.gen_dataframe(\n\t25,fields=['name','city','phone',\n\t'license_plate','email'],\n\treal_email=True,phone_simple=True\n\t)\n\n``gen_table()``\n------------------\nAttempts to create a table in a database (.db) file using Python's built-in ``SQLite`` engine. User can specify various data types to be included as database table fields. \n\nAll data types (fields) in the SQLite table will be of VARCHAR type. Data types available:\n        \n* Name, country, city, real (US) cities, US state, zipcode, latitude, longitude\n* Month, weekday, year, time, date\n* Personal email, official email, SSN \n* Company, Job title, phone number, license plate\n\nCustomization choices are following:\n\n- ``real_email``: If ``True`` and if a person's name is also included in the fields, a realistic email will be generated corresponding to the name of the person. For example, ``Tirtha Sarkar`` name with this choice enabled, will generate emails like ``TSarkar21@gmail.com`` or ``Sarkar.Tirtha@att.net``.\n- ``real_city``: If ``True``, a real US city's name will be picked up from a list (included as a text data file with the installation package). Otherwise, a fictitious city name will be generated.\n- ``phone_simple``: If ``True``, a 10 digit US number in the format xxx-xxx-xxxx will be generated. Otherwise, an international number with different format may be returned.\n- ``db_file``: Name of the database where the ``TABLE`` will be created or updated. Default database name will be chosen if not specified by user.\n- ``table_name``: Name of the table, to be chosen by user. Default table name will be chosen if not specified by user.\n- ``primarykey``: User can choose a PRIMARY KEY from among the various fields. If nothing specified, the first data field will be made PRIMARY KEY. If user chooses a field, which is not in the specified list, an error will be thrown and no table will be generated.\n\n **Code example**::\n\n\tmyDB.gen_table(\n\t20,fields=['name','city','job_title','phone','company','email'],\n        db_file='TestDB.db',table_name='People',\n\tprimarykey='name',real_city=False\n\t)\n\n``gen_excel()``\n------------------\nAttempts to create an Excel file using Pandas excel_writer function. User can specify various data types to be included. All data types (fields) in the Excel file will be of text type. Data types available:\n        \n* Name, country, city, real (US) cities, US state, zipcode, latitude, longitude\n* Month, weekday, year, time, date\n* Personal email, official email, SSN \n* Company, Job title, phone number, license plate\nCustomization choices are following:\n\n- ``real_email``: If ``True`` and if a person's name is also included in the fields, a realistic email will be generated corresponding to the name of the person. For example, ``Tirtha Sarkar`` name with this choice enabled, will generate emails like ``TSarkar21@gmail.com`` or ``Sarkar.Tirtha@att.net``.\n- ``real_city``: If ``True``, a real US city's name will be picked up from a list (included as a text data file with the installation package). Otherwise, a fictitious city name will be generated.\n- ``phone_simple``: If ``True``, a 10 digit US number in the format xxx-xxx-xxxx will be generated. Otherwise, an international number with different format may be returned.\n- ``filename``: Name of the Excel file to be created or updated. Default file name will be chosen if not specified by user.\n\n**Code example**::\n\n\tmyDB.gen_excel(15,fields=['name','year','email','license_plate'],\n\t\tfilename='TestExcel.xlsx',real_email=True)\n\nOther auxiliary methods available\n----------------------------------\nFew other auxiliary functions available in this package.\n\n* **Realistic email** with a given name as seed::\n\t\n\tfor _ in range(10):\n    \tprint(myDB.realistic_email('Tirtha Sarkar'))\n\n    \tSarkar.Tirtha59@zoho.com\n    \tSarkar.Tirtha@hotmail.com\n    \tSarkar.Tirtha81@yandex.com\n    \tTSarkar@mail.com\n    \tTSarkar65@yahoo.com\n    \tTirtha.S36@mail.com\n    \tTirtha_S@yandex.com\n    \tTirtha.S@aol.com\n    \tSarkar.Tirtha@mail.com\n    \tTirtha.Sarkar81@comcast.net\n\n* **License plate** in few different style (1,2, or 3)::\n\t\n\tfor _ in range(10):\n\t    print(myDB.license_plate())\n\n\t    1OAG936\n\t    LTZ-6460\n\t    ODQ-846\n\t    8KNW713\n\t    MFX-8256\n\t    6WMH396\n\t    OQX-2780\n\t    OOD-124\n\t    RXY-8865\n\t    JZV-3326\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftirthajyoti%2Fpydbgen","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftirthajyoti%2Fpydbgen","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftirthajyoti%2Fpydbgen/lists"}