{"id":25016804,"url":"https://github.com/abdullahalzubaer/some-useful-functions","last_synced_at":"2025-07-05T03:06:53.288Z","repository":{"id":140988555,"uuid":"392848369","full_name":"abdullahalzubaer/Some-Useful-Functions","owner":"abdullahalzubaer","description":"A small collection of some useful functions.","archived":false,"fork":false,"pushed_at":"2025-02-28T09:53:37.000Z","size":301,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-30T07:43:23.203Z","etag":null,"topics":["functions-python","useful-functions"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/abdullahalzubaer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-08-04T23:25:09.000Z","updated_at":"2025-02-28T09:53:40.000Z","dependencies_parsed_at":"2023-11-12T10:25:51.029Z","dependency_job_id":"0b5ce258-4edd-4a07-9eee-2065b2482ee3","html_url":"https://github.com/abdullahalzubaer/Some-Useful-Functions","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/abdullahalzubaer/Some-Useful-Functions","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abdullahalzubaer%2FSome-Useful-Functions","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abdullahalzubaer%2FSome-Useful-Functions/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abdullahalzubaer%2FSome-Useful-Functions/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abdullahalzubaer%2FSome-Useful-Functions/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/abdullahalzubaer","download_url":"https://codeload.github.com/abdullahalzubaer/Some-Useful-Functions/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abdullahalzubaer%2FSome-Useful-Functions/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263674197,"owners_count":23494531,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["functions-python","useful-functions"],"created_at":"2025-02-05T09:54:33.156Z","updated_at":"2025-07-05T03:06:53.242Z","avatar_url":"https://github.com/abdullahalzubaer.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"##  A small collection of some useful functions that come in handy from time to time (in progress).\n\n\nSometimes I write some code that might be part of a bigger project or some simple idea that comes to my mind and then I code them! There is no one single theme for this repo, but things that I found interesting and thought its good to keep track of them, or else\n\n\n```\n \"All those moments will be lost in time, like tears in rain.\"\n - Blade Runner, 1982 \n```\n\n---\n\n#### Read Json file\n\n```python\n\nimport json\n\n# Open the JSON file in read mode\nwith open('file.json', 'rb') as file:\n\n    # Load the contents of the file into a variable\n    data = json.load(file\n\n```\n\n\n#### Save dictionary in a pickle file and read\n\n```python\n\n# Write\n\nimport pickle\n\n# dictionary object to be saved\nmy_dict = {'key1': 'value1', 'key2': 'value2', 'key3': 'value3'}\n\n# open file in binary write mode\nwith open('my_dict.pkl', 'wb') as f:\n    # dump dictionary object to file\n    pickle.dump(my_dict, f)\n    \n    \n# Read\n\nimport pickle\nwith open(\"name-of-file.pkl\", \"rb\") as f:\n    file = pickle.load(f)\n\nprint(file)\n```\n\n#### Write a dictionary object as json\n\n```python\nwith open(\"file.json\", \"w\") as f:\n    json.dump(file, f, indent=4)\n```\n\n#### save a dataframe locally\n\n```python\n\ndf.to_csv(\"FILE_NAME.csv\", index=False)\n````\n\n#### Compare two dataframes and get the difference\n\n\n\n```python\ndef compare_dataframes(df1, df2, columns):\n    # Compare element-wise differences\n    differences = df1[columns].compare(df2[columns])\n    print(\"Differences:\\n\", differences)\n\n    # Check data types\n    print(\"\\nData Types DF1:\\n\", df1[columns].dtypes)\n    print(\"\\nData Types DF2:\\n\", df2[columns].dtypes)\n\n    # Check indices\n    print(\"\\nIndices DF1:\\n\", df1[columns].index)\n    print(\"\\nIndices DF2:\\n\", df2[columns].index)\n\n    # Check individual columns for equality\n    for col in columns:\n        equal = df1[col].equals(df2[col])\n        print(f\"\\nColumn {col} equality: {equal}\")\n\n# Call the function to compare DataFrames\ncompare_dataframes(temp_pv2, pv2_biased_transformed_ranks, ['mean', 'variance', 'std_dev'])\n\n```\n\n\n#### Creating data for cross validation \n\n```python\n\n# Reference: https://stackoverflow.com/questions/61512087/use-kfolds-to-split-dataframe-i-want-the-rows-to-be-split-but-the-columns-are-g\n\nimport pandas as pd\nfrom sklearn.model_selection import KFold\n\nX = [[ 0.87, -1.34,  0.31, 1],\n     [-2.79, -0.02, -0.85, 2],\n     [-1.34, -0.48, -2.55, 3],\n     [ 1.92,  1.48,  0.65, 4],\n     [ 1.92,  1.48,  0.65, 5],\n     [ 1.92,  1.48,  0.65, 6],\n     [ 1.92,  1.48,  0.65, 7],\n     [ 1.92,  1.48,  0.65, 8],\n     [ 1.92,  1.48,  0.65, 9],\n     [ 1.92,  1.48,  0.65, 10]]\n\nfinalDF = pd.DataFrame(X, columns=['col1', 'col2', 'col3', 'Target'])\n\nprint(\"=====Complete df======\")\nprint(finalDF)\n\nfolds = KFold(n_splits=5)\nscores = []\nfold = 0\n\n# you have to remove the target columsn by giving its column name.\n\nfor trainIndex, testIndex in folds.split(finalDF.drop(['Target'], axis=1)):\n    fold += 1\n    print(f\"=========FOLD: {fold} starting ==============\")\n    xTrain = finalDF.loc[trainIndex, :]\n    xTest = finalDF.loc[testIndex, :]\n    print(xTrain.shape, xTest.shape)\n    print(xTrain)\n    print(xTest)\n\n```\n\n#### Splitting a df into train and test split only\n\n:heavy_exclamation_mark: When you are saving the df using pickle then you need to have the same \npandas version to unpickle it. For example I was saving a df in jupyterhub that is from the uni passau\nand then I tried to unpickle it locally, it did not work because the pandas version were not same.\n\n```python\n\n'''\nSplitting the arguments_df that has everything to train test split\n'''\n \n \n# Reference: https://stackoverflow.com/a/42932524/12946268\n\nimport pandas as pd\nimport pickle\nfrom sklearn.model_selection import train_test_split\n\n# Load the original DataFrame from CSV\ndf = pd.read_csv(\"COMPLETE_DATAFRAME.csv\")\n\n\n# Split into train (60%), validation (20%), and test (20%) sets\ntrain_val_df, test_df = train_test_split(df, test_size=0.2, random_state=42)\ntrain_df, val_df = train_test_split(train_val_df, test_size=0.25, random_state=42)\n\nwith open(f\"train_echr_42.pkl\", 'wb') as f: pickle.dump(train_df, f)\nwith open(f\"val_echr_42.pkl\", 'wb') as f: pickle.dump(val_df, f)\nwith open(f\"test_df_echr_42.pkl\", 'wb') as f: pickle.dump(test_df, f)\n\n'''\nATTENTION! DO NOT USE to_csv method from pandas it saves a dataframe to csv\nfile which is inconsistent with the original dataframe ( I have faced this issue, when\nI was reading the csv file that was saved from to_csv method, I was getting null values\neven tho in the original dataframe there was no null values)\n\nUPDATE\n29.NOV.2023\n\nThe inconsistency was due to the nature of the dataset that has carraige return inside it '\\r'\nsomething like that. So if your dataset does not have any carraige return then you can\nuse to_csv happily\n'''\n\n# To read\n\nwith open('test_df_echr_42.pkl', 'rb') as f:\n    test = pickle.load(f)\n\n'''\n# Save the split DataFrames to CSV files\n\n# DO NOT USE IT EVER\ntrain_df.to_csv(\"train_echr_42.csv\", index=False)\nval_df.to_csv(\"validation_echr_42.csv\", index=False)\ntest_df.to_csv(\"test_echr_42.csv\", index=False)\n'''\n```\n\n\n#### Get current directory\n\n```python\nimport os\nimport datetime\n\ndef print_current_directory():\n    current_directory = os.getcwd()\n    time_now = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')\n    print(f\"Current directory as of {time_now}:\\n\")\n    print(f\"{current_directory}\")\nprint_current_directory()\n```\n\n\n### Print in a nice way with datetime and current working directory\n\n```python\ndef print_in_box(message: str) -\u003e None:\n    \"\"\"\n    Print a given message along with the current directory and timestamp in a box, separated by a horizontal line.\n\n    Parameters:\n    message (str): The message to be printed in the box.\n    \"\"\"\n    # Get current directory and timestamp\n    current_directory = os.getcwd()\n    time_now = datetime.now().strftime('%d-%b-%Y %H:%M:%S')\n\n    # Prepare the directory and time information\n    dir_info = f\"Current directory as of {time_now}:\\n{current_directory}\"\n\n    # Combine the custom message with the directory information, separated by a line\n    combined_message = message + \"\\n\\n\" + \"-\" * len(max(message.split('\\n'), key=len)) + \"\\n\" + dir_info\n\n    # Split the combined message into lines\n    lines = combined_message.split('\\n')\n    # Find the length of the longest line\n    max_length = max(len(line) for line in lines)\n    # Create the top and bottom borders of the box\n    top_border = \"+\" + \"-\" * (max_length + 2) + \"+\"\n    bottom_border = top_border\n\n    # Print the box with the combined message\n    print(top_border)\n    for line in lines:\n        # Pad each line to the length of the longest line\n        padded_line = line + ' ' * (max_length - len(line))\n        print(\"| \" + padded_line + \" |\")\n    print(bottom_border)\n\n```\n\n### Print in a nice way but simpler version\n\n```python\ndef print_in_box_simple(message: str) -\u003e None:\n    \"\"\"\n    Print a given message in a box.\n\n    Parameters:\n    message (str): The message to be printed in the box.\n    \"\"\"\n\n    # Split the message into lines\n    lines = message.split('\\n')\n    # Find the length of the longest line\n    max_length = max(len(line) for line in lines)\n    # Create the top and bottom borders of the box\n    top_border = \"+\" + \"-\" * (max_length + 2) + \"+\"\n    bottom_border = top_border\n\n    # Print the box with the message\n    print(top_border)\n    for line in lines:\n        # Pad each line to the length of the longest line\n        padded_line = line + ' ' * (max_length - len(line))\n        print(\"| \" + padded_line + \" |\")\n    print(bottom_border)\n\n```\n\n### Comparing two dataframe if they are identical or not\n\n```python\ndef compare_dataframes(df1, df2):\n    # Check if the shape of both dataframes is the same\n    if df1.shape != df2.shape:\n        return False, \"Dataframes are not identical: Different shapes.\"\n\n    # Check if columns and their order are the same\n    if not df1.columns.equals(df2.columns):\n        return False, \"Dataframes are not identical: Different columns or column order.\"\n\n    # Check if index and their order are the same\n    if not df1.index.equals(df2.index):\n        return False, \"Dataframes are not identical: Different index or index order.\"\n\n    # Check if the content of dataframes is the same\n    if not df1.equals(df2):\n        return False, \"Dataframes are not identical: Different content.\"\n\n    return True, \"Dataframes are identical.\"\n\n```\n\n### Conversion to three decimal place with widgets\n\n```python\nimport ipywidgets as widgets\nfrom IPython.display import display\n\ndef round_to_three_decimals(number):\n    rounded_number = round(number, 3)\n    return rounded_number\n\ndef on_click(btn):\n    number = float(input_text.value)\n    output_label.value = f\"{round_to_three_decimals(number)}\"\n\ninput_text = widgets.FloatText(description=\"Number:\")\nsubmit_btn = widgets.Button(description=\"Convert\")\nsubmit_btn.on_click(on_click)\noutput_label = widgets.Label()\n\ndisplay(input_text, submit_btn, output_label)\n\n```\n### Save pandas dataframe in the current directory as csv and xlsx\n```python\nfrom datetime import datetime\ndef save_dataframe(df, filename):\n    \"\"\"\n    Saves the given DataFrame to both CSV and Excel formats in the specified directory.\n\n    Args:\n    df (pandas.DataFrame): The DataFrame to save.\n    filename (str): The base filename without extension to use for saving the files.\n\n    Returns:\n    None\n    \"\"\"\n    # Get the directory from the filename\n    directory = os.path.dirname(filename)\n    \n    # Check if the directory exists\n    if not os.path.exists(directory):\n        os.makedirs(directory)\n        print(f\"Directory created: {directory}\")\n    else:\n        print(f\"Directory already exists: {directory}\")\n    \n    # Define file paths\n    csv_file = f\"{filename}.csv\"\n    excel_file = f\"{filename}.xlsx\"\n    \n    # Save as CSV\n    df.to_csv(csv_file, index=False)\n    print(f\"DataFrame saved as CSV in {csv_file}\")\n    \n    # Save as Excel\n    df.to_excel(excel_file, index=False, engine='openpyxl')\n    print(f\"DataFrame saved as Excel in {excel_file}\")\n    time_now = datetime.now().strftime('%d-%b-%Y %H:%M:%S')\n    print(f\"Saved on {time_now}:\\n\")\n\n# Example usage\n# Assuming df_data is your DataFrame and you want to save it as 'data'\nsave_dataframe(df_data, './data/after-clean/complete_data')\n```\n\n\n### toggle pandas display settings\n```python\ndef toggle_pandas_display_settings(mode='full'):\n    \"\"\"\n    Toggle the display settings of pandas DataFrame.\n\n    Parameters:\n    - mode: 'full' to display DataFrames without truncation, 'default' to reset to default settings.\n    \n    Example:\n    \n    # To turn on full display:\n    toggle_pandas_display_settings('full')\n\n    # To reset to default settings:\n    toggle_pandas_display_settings('default')\n\n    \"\"\"\n    if mode == 'full':\n        # Set to display DataFrames without truncation\n        pd.set_option('display.max_rows', None)\n        pd.set_option('display.max_columns', None)\n        pd.set_option('display.max_colwidth', None)  # For pandas versions 1.0 and later\n        # pd.set_option('display.max_colwidth', -1)  # Uncomment for pandas versions before 1.0\n        print(\"Pandas display settings set to full display mode.\")\n    elif mode == 'default':\n        # Reset to pandas default display settings\n        pd.reset_option('display.max_rows')\n        pd.reset_option('display.max_columns')\n        pd.reset_option('display.max_colwidth')\n        print(\"Pandas display settings reset to default.\")\n    else:\n        print(\"Invalid mode. Please choose 'full' or 'default'.\")\n```\n\n### Reads specified sheets from an Excel file using pandas.\n```python\ndef read_excel_sheets(file_path, sheets=None, return_type='single'):\n    \"\"\"\n    Reads specified sheets from an Excel file using pandas.\n\n    :param file_path: str, path to the Excel file.\n    :param sheets: str, int, or list, names or indices of the sheets to read.\n    :param return_type: str, 'single' to return a single DataFrame (if one sheet is specified),\n                        'dict' to return a dictionary of DataFrames (if multiple sheets are specified).\n    :return: DataFrame or dict of DataFrames depending on return_type and sheets.\n    \"\"\"\n    # Read the sheets based on the provided 'sheets' argument\n    time_now = datetime.now().strftime('%d-%b-%Y %H:%M:%S')\n    try:\n        data = pd.read_excel(file_path, sheet_name=sheets)\n        print(f\"{file_path} was read on {time_now}:\\n\")\n    except Exception as e:\n        print(f\"Failed to read the file: {e}\")\n        return None\n\n    # If multiple sheets are read into a dictionary\n    if isinstance(data, dict):\n        if return_type == 'single':\n            # If user wants a single DataFrame but multiple sheets were requested, raise an error\n            raise ValueError(\"Multiple sheets found but 'single' DataFrame requested. Specify correct 'return_type'.\")\n        return data\n    else:\n        if return_type == 'dict':\n            # If user expects a dictionary but only one sheet was read, adjust the return structure\n            print(f\"{file_path} was read on {time_now}:\\n\")\n            return {sheets: data}\n        return data\n# Example usage\ndata = read_excel_sheets(file_path='Data_complete_Can_GPT_Replace_Human_Examiners.xlsx',\n                         sheets='Robustness \u0026 Extensions')\ndata.head(6)\n\n```\n\n\n### save dictionary as json\n```python\ndef save_dict_as_json(d, filename):\n    \"\"\"\n    Saves a dictionary as a JSON file, but only if the file does not already exist.\n\n    Parameters:\n    d (dict): The dictionary to save.\n    filename (str): The path and name of the file to save the dictionary to.\n\n    Raises:\n    FileExistsError: If a file with the specified name alre exists.\n    \"\"\"\n\n    # Check if the file already exists\n    if os.path.exists(filename):\n        raise FileExistsError(f\"File '{filename}' already exists.\")\n\n    # Create the directory if it does not exist\n    os.makedirs(os.path.dirname(filename), exist_ok=True)\n\n    # Save the dictionary as a JSON file\n    with open(filename, \"w\") as file:\n        json.dump(d, file, indent=4)\n\n    # print_in_box(f\"Result saved successfully at\\n{filename}\")\n```\n\n\n### Read excel sheets\n\n```python\ndef read_excel_sheets(file_path, sheets=None, return_type=\"single\"):\n    \"\"\"\n    Reads specified sheets from an Excel file using pandas.\n\n    :param file_path: str, path to the Excel file.\n    :param sheets: str, int, or list, names or indices of the sheets to read.\n    :param return_type: str, 'single' to return a single DataFrame (if one sheet is specified),\n                        'dict' to return a dictionary of DataFrames (if multiple sheets are specified).\n    :return: DataFrame or dict of DataFrames depending on return_type and sheets.\n    \"\"\"\n    # Read the sheets based on the provided 'sheets' argument\n    try:\n        data = pd.read_excel(file_path, sheet_name=sheets)\n    except Exception as e:\n        print(f\"Failed to read the file: {e}\")\n        return None\n\n    # If multiple sheets are read into a dictionary\n    if isinstance(data, dict):\n        if return_type == \"single\":\n            # If user wants a single DataFrame but multiple sheets were requested, raise an error\n            raise ValueError(\n                \"Multiple sheets found but 'single' DataFrame requested. Specify correct 'return_type'.\"\n            )\n        return data\n    else:\n        if return_type == \"dict\":\n            # If user expects a dictionary but only one sheet was read, adjust the return structure\n            return {sheets: data}\n        return data\n```\n### drop columns from padnas\n\n```python\ndef drop_columns_from(df, start_column):\n    \"\"\"\n    Drop all columns from the specified start_column to the end of the DataFrame (inclusive).\n\n    Parameters:\n    df (pd.DataFrame): The DataFrame from which to drop columns.\n    start_column (str): The column name from which to start dropping.\n\n    Returns:\n    pd.DataFrame: A DataFrame with the specified columns removed.\n    \"\"\"\n    # Get the index of the start column\n    start_index = df.columns.get_loc(start_column)\n\n    # Get the column names to drop from start_index to the end\n    columns_to_drop = df.columns[start_index:]\n\n    # Drop the columns\n    df = df.drop(columns=columns_to_drop)\n    \n    return df\n```\n\n### read csv/excel with fallback\n\n```python\ndef read_file_with_fallback(primary_file, fallback_file):\n    \"\"\"\n    Reads a file (CSV or Excel) into a DataFrame. If the primary file does not exist, it reads the fallback file.\n\n    Parameters:\n    primary_file (str): The path to the primary file (CSV or Excel).\n    fallback_file (str): The path to the fallback file (CSV or Excel).\n\n    Returns:\n    pandas.DataFrame: DataFrame created from the read file.\n    \"\"\"\n    # Determine which file to read\n    file_to_read = primary_file if os.path.exists(primary_file) else fallback_file\n    \n    # Print message indicating which file is being read\n    print(f\"Reading from {'primary' if file_to_read == primary_file else 'fallback'} file: {file_to_read}\")\n\n    # Check file extension and read accordingly\n    if file_to_read.endswith('.csv'):\n        df = pd.read_csv(file_to_read)\n    elif file_to_read.endswith('.xlsx'):\n        df = pd.read_excel(file_to_read)\n    else:\n        raise ValueError(\"Unsupported file format. Only .csv and .xlsx are supported.\")\n\n    return df\n```\n\n### Convert to three decimal place\n\n```python\nimport ipywidgets as widgets\nfrom IPython.display import display\n\ndef round_to_three_decimals(number):\n    rounded_number = round(number, 3)\n    return rounded_number\n\ndef on_click(btn):\n    number = float(input_text.value)\n    output_label.value = f\"{round_to_three_decimals(number)}\"\n\ninput_text = widgets.FloatText(description=\"Number:\")\nsubmit_btn = widgets.Button(description=\"Convert\")\nsubmit_btn.on_click(on_click)\noutput_label = widgets.Label()\n\ndisplay(input_text, submit_btn, output_label)\n****\n```\n\n### select gpu device\n\n```python\ndef select_gpu_device(device_id):\n\n    \"\"\"\n    Note Please: Will be used when I am using huggingface parser and all the code in a script\n    \n    Selects a GPU device if available and prints information about all available GPUs.\n\n    Args:\n    device_id (int): The ID of the GPU device to use.\n\n    Returns:\n    str: The selected device, either a specific GPU or the CPU if no GPU is available.\n    \"\"\"\n    # Check available GPUs and print their names\n    \n    gpu_count = torch.cuda.device_count()\n    '''\n    print(\"Available GPUs:\", gpu_count)\n    for i in range(gpu_count):\n        print(f\"GPU {i}: {torch.cuda.get_device_name(i)}\")\n    '''\n    # Choose a specific GPU based on device_id or fallback to CPU if GPUs are unavailable\n    device = f\"cuda:{device_id}\" if torch.cuda.is_available() and device_id \u003c gpu_count else \"cpu\"\n    # print_in_box(f\"Using device: {device}\")\n    \n    return device\n```\n\n### Check for empty list and repeated elements in a list of a specified column of a pandas DataFrame\n\n```python\ndef find_repeated_and_empty_elements(df, column_name):\n    \"\"\"\n    This function checks each row in a specified column of a DataFrame for two conditions:\n    1. If the list is empty.\n    2. If there are repeated elements in the list.\n    \n    Parameters:\n    df (pd.DataFrame): The DataFrame to be checked.\n    column_name (str): The name of the column containing the string representations of lists.\n    \n    Outputs:\n    Prints the row index and details if an empty list or repeated elements are found.\n    \"\"\"\n    def get_repeated_elements(lst):\n        \"\"\"\n        Helper function to find repeated elements in a list.\n        \n        Parameters:\n        lst (list): The list to check for repeated elements.\n        \n        Returns:\n        list: A list of elements that are repeated.\n        \"\"\"\n        counter = Counter(lst)\n        return [item for item, count in counter.items() if count \u003e 1]\n    # Iterate through each row in the DataFrame\n    for index, row in df.iterrows():\n        # Convert the string representation of the list to an actual list\n        score = ast.literal_eval(row[column_name])\n        \n        # Check for empty list\n        if not score:\n            print(f\"Row {index} has an empty list.\")\n        \n        # Check for repeated elements\n        repeated_elements = get_repeated_elements(score)\n        if repeated_elements:\n            print(f\"Row {index} has repeated elements: {repeated_elements} in list {score}\")\n\n# Call the function with the DataFrame and the column name\nfind_repeated_and_empty_elements(df=df_llama3_rank_data,\n                                 column_name='prompt_v1_rank_assessment_llama3_model_ranks')\n\n\n```\n\n\n### Check if two column from two different dataframe are identical or not\n\n```python\n\ndef are_columns_identical(df1, col1, df2, col2):\n    \"\"\"\n    Check if two columns from two different DataFrames are identical.\n    \n    Parameters:\n    df1 (pd.DataFrame): The first DataFrame.\n    col1 (str): The column name from the first DataFrame.\n    df2 (pd.DataFrame): The second DataFrame.\n    col2 (str): The column name from the second DataFrame.\n    \n    Returns:\n    bool: True if the columns are identical, False otherwise.\n    \"\"\"\n    # Check if the columns exist in their respective DataFrames\n    if col1 not in df1.columns or col2 not in df2.columns:\n        raise ValueError(f\"Column not found in DataFrame: {col1} in df1 or {col2} in df2\")\n    \n    # Check if the lengths of the columns are the same\n    if len(df1[col1]) != len(df2[col2]):\n        return False\n    \n    # Check if all elements in the columns are the same\n    return df1[col1].equals(df2[col2])\n```\n\n### Check if n number of columns from two different dataframe are identical or not\n\n\u003e The order must be the same as it is given to cols1 and cols2\n\n```python\ndef are_multiple_columns_identical(df1, cols1, df2, cols2):\n    \"\"\"\n    Check if multiple columns from two different DataFrames are identical.\n    \n    Parameters:\n    df1 (pd.DataFrame): The first DataFrame.\n    cols1 (list of str): The column names from the first DataFrame.\n    df2 (pd.DataFrame): The second DataFrame.\n    cols2 (list of str): The column names from the second DataFrame.\n    \n    Returns:\n    bool: True if all specified columns are identical, False otherwise.\n    \"\"\"\n    # Check if the lengths of the column lists are the same\n    if len(cols1) != len(cols2):\n        raise ValueError(\"The number of columns to compare must be the same.\")\n    \n    # Iterate through each pair of columns and check for equality\n    for col1, col2 in zip(cols1, cols2):\n        # Check if the columns exist in their respective DataFrames\n        if col1 not in df1.columns or col2 not in df2.columns:\n            raise ValueError(f\"Column not found in DataFrame: {col1} in df1 or {col2} in df2\")\n        \n        # Check if the lengths of the columns are the same\n        if len(df1[col1]) != len(df2[col2]):\n            return False\n        \n        # Check if all elements in the columns are the same\n        if not df1[col1].equals(df2[col2]):\n            return False\n    \n    return True\n```\n\n### ASCII-Banner\n\n```bash\n\n\u003e\u003e=======================\u003c\u003c\n||                       ||\n||                       ||\n||                       ||\n||   _  _   _     ___ _  ||\n||  /  |_) |_  /\\  | |_  ||\n||  \\_ | \\ |_ /--\\ | |_  ||\n||                       ||\n||                       ||\n||                       ||\n\u003e\u003e=======================\u003c\u003c\n\n\u003e\u003e===================================================\u003c\u003c\n|| Developer: Abdullah Al Zubaer                     ||   \n|| Email: abdullahal.zubaer@uni-passau.de            ||\n|| Institution: University of Passau                 ||\n|| Project Page: https://www.uni-passau.de/deepwrite ||   \n\u003e\u003e===================================================\u003c\u003c\n\n```\n### Return a subset of the df with specified column names\n\n```python\n\ndef subset_dataframe(df: pd.DataFrame, columns: list) -\u003e pd.DataFrame:\n    \"\"\"\n    Returns a subset of the DataFrame with only the specified columns.\n\n    Parameters:\n    df (pd.DataFrame): The original DataFrame.\n    columns (list): A list of column names to include in the subset DataFrame.\n\n    Returns:\n    pd.DataFrame: A subset DataFrame containing only the specified columns, copied to ensure independence from the original DataFrame.\n    Example:\n    \n    data = {\n        'A': [1, 2, 3],\n        'B': [4, 5, 6],\n        'C': [7, 8, 9]\n    }\n    df = pd.DataFrame(data)\n    \n    # List of columns to subset\n    columns_to_keep = ['A', 'C']\n    \n    # Get the subset DataFrame\n    subset_df = subset_dataframe(df, columns_to_keep)\n    \"\"\"\n    if not all(column in df.columns for column in columns):\n        raise ValueError(\"One or more columns not found in the DataFrame\")\n    \n    return df[columns].copy()\n```\n#### Filter Dataframe \n\n```python\nimport pandas as pd\n\ndef filter_dataframe(df, column_name, match_value, selected_columns=None):\n    \"\"\"\n    Filters the DataFrame based on the given column name and matching value.\n    \n    Parameters:\n    df (pd.DataFrame): The DataFrame to filter.\n    column_name (str): The name of the column to filter by.\n    match_value (str or int): The value to match in the specified column.\n    selected_columns (list of str, optional): A list of column names to select after filtering. \n                                              If None, all columns are returned.\n    \n    Returns:\n    pd.DataFrame: The filtered DataFrame.\n    \"\"\"\n    # Filter the DataFrame\n    filtered_df = df[df[column_name] == match_value]\n    \n    # If selected_columns is provided, select those columns\n    if selected_columns is not None:\n        filtered_df = filtered_df[selected_columns]\n    \n    return filtered_df\n\n# Example usage:\n# Assuming df is your DataFrame\n# filtered_df = filter_dataframe(df, 'Task', 1)\n# or with specific columns:\n# filtered_df = filter_dataframe(df, 'Task', 1, ['Column1', 'Column2'])\n# This will filter the DataFrame where the \"Status\" column equals \"Completed\" and return only the \"Task\" and \"Assignee\" columns.\n\n```\n#### Find and prinot NaN Rows in pandas dataframe\n\n```python\n\ndef find_and_print_nan_rows(dataframe, column_name):\n    if column_name not in dataframe.columns:\n        print(f\"Column '{column_name}' does not exist in the DataFrame.\")\n        return\n    \n    nan_rows = dataframe[dataframe[column_name].isna()]\n    \n    if nan_rows.empty:\n        print(f\"No NaN values found in column '{column_name}'.\")\n    else:\n        print(f\"Indices of rows with NaN values in column '{column_name}':\")\n        print(nan_rows.index.tolist())\n        \n# find_and_print_nan_rows(data, 'Answer')\n```\n\n#### Insert new line after every n words\n\n```python\ndef insert_newline_every_n_words(text, n=12):\n    # Split the text into words\n    words = text.split()\n    \n    # Create a list to store lines\n    lines = []\n    \n    # Loop over the words in chunks of size n\n    for i in range(0, len(words), n):\n        # Join the words in the chunk into a line and append to lines list\n        lines.append(' '.join(words[i:i + n]))\n    \n    # Join all the lines with newlines\n    result = '\\n'.join(lines)\n    \n    return result\n\n## Example usage:\n# text = \"[INSERT_YOUR_LONG TEXT]\"\n#formatted_text = insert_newline_every_n_words(text, 11)\n#print(formatted_text)\n```\n#### Reorder words randomly from a string and return\n\n```python\ndef reorder_words_randomly(text: str, n: int) -\u003e str :\n    \n    \"\"\"\n    Reorder n words in the given text randomly and return the modified sentence.\n    \n    Args:\n        text (str): The input sentence to be modified.\n        n (int): The number of words to shuffle.\n    \n    Returns:\n        str: The sentence with n words shuffled.\n    \"\"\"\n\n    # Split the text into individual words\n    words = text.split()\n    total_words = len(words)\n    \n    # If n is greater than total words in the text, set n to total words\n    n = min(n, total_words)\n    \n    # If n is 0 or the total words are less than 2, return the original text\n    if n == 0 or total_words \u003c 2:\n        return text\n    \n    # Keep shuffling until the new order is different from the original\n    while True:\n        # Extract n random indices without replacement\n        indices = random.sample(range(total_words), n)\n        \n        # Extract the words at those indices\n        words_to_shuffle = [words[i] for i in indices]\n        \n        # Shuffle the extracted words\n        random.shuffle(words_to_shuffle)\n        \n        # Put the shuffled words back into their original positions\n        shuffled_words = words[:]\n        for idx, word_idx in enumerate(indices):\n            shuffled_words[word_idx] = words_to_shuffle[idx]\n        \n        # If the shuffled words are different from the original, return them\n        if shuffled_words != words:\n            return ' '.join(shuffled_words)\n'''\n# Example usage\ntext = \"one two three four five six seven eight nine ten\"\ni=0\nwhile i\u003c10:\n    i+=1\n    shuffled_sentence = reorder_words_randomly(text, 2)\n    print(shuffled_sentence)\n'''\n```\n\n##### Compare the values of multiple columns from two DataFrames, ignoring column names\n\n```python\ndef compare_columns_by_values(df1, cols1: list[str], df2, cols2: list[str], reset_index=True) -\u003e bool:\n    \"\"\"\n    Compare the values of multiple columns from two DataFrames, ignoring column names.\n    The order of the columns must be the same in both DataFrames i.e.\n    cols1[0] is compared to cols2[0], cols1[1] is compared to cols2[1], and so on.\n\n    Args:\n        df1 (pd.DataFrame): First DataFrame.\n        cols1 (list[str]): Columns to compare from the first DataFrame.\n        df2 (pd.DataFrame): Second DataFrame.\n        cols2 (list[str]): Columns to compare from the second DataFrame.\n        reset_index (bool): Whether to reset the index before comparison. Default is True.\n\n    Returns:\n        bool: True if the selected columns are equal in both DataFrames, False otherwise.\n\n    Example:\n    df1 = pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]})\n    df2 = pd.DataFrame({'x': [1, 2, 3], 'y': [4, 5, 6]})\n\n    result = compare_columns_by_values(df1, ['a', 'b'], df2, ['x', 'y'])\n    print(result)  # True\n\n    # Order of the columns matters and it returns False\n    result = compare_columns_by_values(df1, ['b', 'a'], df2, ['x', 'y'])\n    print(result)  # False\n    \"\"\"\n    # Extract subsets of the DataFrames\n    subset1 = df1[cols1]\n    subset2 = df2[cols2]\n    \n    # Optionally reset the index\n    if reset_index:\n        subset1 = subset1.reset_index(drop=True)\n        subset2 = subset2.reset_index(drop=True)\n    \n    # Compare values\n    return (subset1.values == subset2.values).all() and subset1.to_numpy().tolist() == subset2.to_numpy().tolist()\n\n```\n\n\n#### Check if the extraction function is working properly for columns contaiing the LLMs output\n\n```python\n'''\nIf there are errors then the regular expression needs modification\n\nYou need to know which columns you want to check the function with - then you pass the columns as a list of strings\n'''\n\n\ndef extract_score(value) -\u003e float:\n    \"\"\"Extracts the score from a string value formatted as 'Punktzahl: \u003cnumber\u003e'. Returns None if no score is found.\"\"\"\n\n    match = re.search(r'Punktzahl:\\s*(\\d+(?:\\.\\d+)?)', str(value))\n\n    return float(match.group(1)) if match else 'ERROR_COULD_NOT_EXTRACT'\n\ndef apply_extract_score_and_report(df, columns_to_check, extract_score_func):\n    \"\"\"\n    Applies the extract_score function to specified columns in the dataframe and tracks errors.\n    Generates a summary report of the results.\n\n    Parameters:\n    - df (pandas.DataFrame): The dataframe containing the data.\n    - columns_to_check (list of str): The list of column names to process.\n    - extract_score_func (callable): The function used to extract the score.\n\n    Returns:\n    - dict: A summary report containing total errors and details per column.\n    - int: Total number of errors found across all columns.\n    \"\"\"\n    total_errors = 0\n    summary_report = {}\n\n    for col in columns_to_check:\n        if col not in df.columns:\n            print(f\"NOT FOUND: Column '{col}' not found in dataframe. Skipping...\")\n            summary_report[col] = {\n                'errors_found': False,\n                'error_count': 0,\n                'error_rows': [],\n                'status': 'Column not found'\n            }\n            continue\n\n        print(f\"Processing column: {col}\")\n        extracted_col_name = f\"{col}_Extracted\"\n\n        # Apply the function\n        df[extracted_col_name] = df[col].apply(lambda x: extract_score_func(x))\n\n        # Identify problematic rows\n        errors = df[df[extracted_col_name] == 'ERROR_COULD_NOT_EXTRACT']\n        error_count = len(errors)\n        total_errors += error_count\n\n        # Store error details in the summary report\n        summary_report[col] = {\n            'errors_found': error_count \u003e 0,\n            'error_count': error_count,\n            'error_rows': errors.index.tolist(),\n            'status': 'Processed with errors' if error_count \u003e 0 else 'Processed successfully'\n        }\n\n        # Uncomment the following lines if you want to see the errors\n        # if error_count \u003e 0:\n        #     print(f\"\\nIssues found in column '{col}':\")\n        #     for idx, row in errors.iterrows():\n        #         print(f\"Row {idx}: Text - {row[col]}\")\n\n    if total_errors == 0:\n        print(\"\\nNo errors found in any column.\")\n    else:\n        print(\"\\nErrors were found. See details above.\")\n        print(f\"Total errors found: {total_errors}\")\n\n    # Return the summary report (empty or not)\n    return summary_report, total_errors\n\n#CHANGEHERE#\nmodels = [\"gemma2\", \"llama3.1\", 'llama3', 'mistral']\n\n#CHANGEHERE#\ncolumns_to_check = [\n    f\"prompt_v1_original_macro_eco_de_complete_response_{model}_iteration_{i}\"\n    for model in models for i in range(1, 11)\n]\n\n# Call the function\nsummary_report, total_errors = apply_extract_score_and_report(df=df_to_score,\n                                                              columns_to_check=columns_to_check,\n                                                              extract_score_func=extract_score)\n\n# Print the summary report only if errors were found\nif total_errors \u003e 0:\n    print(\"\\nSummary Report:\")\n    for col, details in summary_report.items():\n        if details['errors_found']:\n            print(f\"Column: {col}\")\n            print(f\"  Status: {details['status']}\")\n            print(f\"  Errors Found: {details['errors_found']}\")\n            print(f\"  Error Count: {details['error_count']}\")\n            print(f\"  Error Rows: {details['error_rows']}\")\nelse:\n    print(\"\\nNo errors to report.\")\n\n```\n#### Filtering pandas dataframe based on specified string present in the df\n```python\nimport pandas as pd\n\n# Example dataframe\ndata = {\n    \"name\": [\"Alice\", \"Bob\", \"Charlie\"],\n    \"age\": [25, 30, 35],\n    \"score_math\": [90, 85, 80],\n    \"score_english\": [88, 92, 78],\n    \"total_score\": [178, 177, 158],\n}\n\ndf = pd.DataFrame(data)\n\n# Use 'filter' with 'regex' to select columns matching the regex pattern\nregex_pattern = r\"score\"  # Matches any column containing \"score\"\nfiltered_df = df.filter(regex=regex_pattern, axis=1)\n\nprint(filtered_df)\n```\n\n####  Compare two strings using SequenceMatcher and print the differences.\n```python\nfrom difflib import SequenceMatcher\n\ndef compare_strings(str1, str2):\n    \"\"\"\n    Compare two strings using SequenceMatcher and print the differences.\n\n    Args:\n        str1 (str): The first string to compare.\n        str2 (str): The second string to compare.\n    \"\"\"\n    matcher = SequenceMatcher(None, str1, str2)\n    for tag, i1, i2, j1, j2 in matcher.get_opcodes():\n        print(f\"{tag}: '{str1[i1:i2]}' vs '{str2[j1:j2]}'\")\n\n# Example usage:\n# compare_strings(\"string1\", \"string2\")\n```\n#### Check any string if present in any column in DF\n```python\ndef check_string_in_any_column(df, string_to_check):\n    \"\"\"\n    Check if a given string is present in any column name of a DataFrame.\n\n    Parameters:\n    df (pd.DataFrame): The DataFrame to check.\n    string_to_check (str): The substring to look for in column names.\n\n    Returns:\n    bool: True if the string is present in any column name, False otherwise.\n    \"\"\"\n    # Filter and print columns containing the substring\n    matching_columns = [col for col in df.columns if string_to_check in col]\n    \n    if matching_columns:\n        print(f\"Columns containing '{string_to_check}': {matching_columns}\")\n        print(f\"Total columns found: {len(matching_columns)}\")\n        return True\n    else:\n        print(f\"No columns contain '{string_to_check}'.\")\n        return False\n\n# Example usage\n# string_to_check = \"temp_0.7_topP_0.8_topK_20\"\n# print(check_string_in_any_column(df_to_analyze, string_to_check))\n```\n   \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabdullahalzubaer%2Fsome-useful-functions","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabdullahalzubaer%2Fsome-useful-functions","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabdullahalzubaer%2Fsome-useful-functions/lists"}