{"id":19362428,"url":"https://github.com/steciuk/med-declat-1","last_synced_at":"2026-04-10T00:53:43.041Z","repository":{"id":78909781,"uuid":"569724397","full_name":"steciuk/MED-dEclat-1","owner":"steciuk","description":"Implements dECLAT and ECLAT algorithms to discover frequent itemsets from social media posts. Includes data retrieval, preprocessing, algorithmic implementation, and result visualization.","archived":false,"fork":false,"pushed_at":"2024-05-21T17:39:28.000Z","size":3212,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-24T12:50:14.235Z","etag":null,"topics":["d3js","data-mining","declat","eclat","frequent-itemsets","memory-complexity","praw-reddit","python","react","reddit-api","social-network-analysis"],"latest_commit_sha":null,"homepage":"https://declat.steciuk.dev/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/steciuk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-23T13:25:45.000Z","updated_at":"2024-05-21T17:39:32.000Z","dependencies_parsed_at":null,"dependency_job_id":"f910b294-cd7e-43b0-9f8c-ab134a444ac5","html_url":"https://github.com/steciuk/MED-dEclat-1","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/steciuk/MED-dEclat-1","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/steciuk%2FMED-dEclat-1","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/steciuk%2FMED-dEclat-1/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/steciuk%2FMED-dEclat-1/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/steciuk%2FMED-dEclat-1/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/steciuk","download_url":"https://codeload.github.com/steciuk/MED-dEclat-1/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/steciuk%2FMED-dEclat-1/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272233486,"owners_count":24896824,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-26T02:00:07.904Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["d3js","data-mining","declat","eclat","frequent-itemsets","memory-complexity","praw-reddit","python","react","reddit-api","social-network-analysis"],"created_at":"2024-11-10T07:28:19.795Z","updated_at":"2025-12-30T21:36:24.286Z","avatar_url":"https://github.com/steciuk.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Eclat \u0026 dEclat\n\nProject for the Data Mining Methods course at Warsaw University of Technology, during the fall semester of 2022.\n\n**Grade:** 10/10\n\nThe scope of work was divided among the authors as follows:\n- [Stage 0 (29.11.2022)](#stage-0-29112022) - Adam Steciuk, Piotr Chojnowski\n- [Stage 1 (13.12.2022)](#stage-1-13122022) - Adam Steciuk\n- [Stage 2](#stage-2) - Piotr Chojnowski\n\nThis repository contains the results of the 0th and 1st stages of the project.\n\n## Table of contents\n- [Eclat \\\u0026 dEclat](#eclat--declat)\n  - [Table of contents](#table-of-contents)\n  - [Stage 0 (29.11.2022)](#stage-0-29112022)\n    - [Problem description](#problem-description)\n    - [Determination of the Dataset](#determination-of-the-dataset)\n      - [Preprocessing Steps](#preprocessing-steps)\n      - [Implementation Concept](#implementation-concept)\n  - [Stage 1 (13.12.2022)](#stage-1-13122022)\n    - [Technical Specification](#technical-specification)\n    - [Project repository](#project-repository)\n    - [Data Retrieval and Preparation](#data-retrieval-and-preparation)\n      - [Unit Tests](#unit-tests)\n    - [Eclat and dEclat](#eclat-and-declat)\n      - [Unit Tests](#unit-tests-1)\n    - [Visualization of Results](#visualization-of-results)\n      - [Technological Assumptions](#technological-assumptions)\n      - [Functionality](#functionality)\n    - [Experiments](#experiments)\n      - [Definition of Research Data](#definition-of-research-data)\n      - [Plan of Experiments](#plan-of-experiments)\n      - [Conducting the Experiments](#conducting-the-experiments)\n      - [Visualization and Analysis of Results](#visualization-and-analysis-of-results)\n      - [Experiment Summary and Possible Improvements](#experiment-summary-and-possible-improvements)\n\n## Stage 0 (29.11.2022)\n### Problem description\ndEclat is an algorithm used for discovering frequent itemsets.\n\nAlgorithm pseudocode:\n\n![dEclat algorithm pseudocode](doc/images/pseudocode.png)\n\nThe input data for the algorithm is a set of all classes P. Frequent itemsets are generated by processing subsets of all unique pairs from the input set. The subsets of each pair are combined, and then a differential list is determined. If a given list is frequent (i.e., it meets the support threshold specified by the user), it is added to the frequent set at that level. The process is repeated until all frequent itemsets for the given input classes are generated.\n\nFor memory management, it is required to keep differential lists in memory for at most two consecutive levels. (We can remove differential lists for a given level once all frequent itemsets for the next level have been determined).\n\nConsider the following example:\n\n![Creation of differential lists example](doc/images/diff_list_example.png)\n\nAnalyzing the example above, set $A$ contains all elements that do not occur in the combined set of all elements ($(\\{1,2,3,4,5,6\\} \\setminus \\{1,3,4,5\\} = \\{2,6\\})$). Set $AD$ contains all elements occurring in $D$ that do not occur in $A$. In the attached example, `min_sup` has been set at $50%$, so set $AD$ is not frequent ($|d(A)| - |d(AD)| \u003c 3$, where $4-2=2$).\n\n### Determination of the Dataset\nThe input dataset will consist of titles from English posts on Reddit. With this dataset in mind, the algorithm can be utilized to identify sets of frequently used words specific to a particular subreddit. If a subreddit lacks distinctive vocabulary, the algorithm will likely yield the most commonly used English words, often found on stop lists. A minor adjustment to the token creation algorithm, such as filtering out stop words, could significantly alter the program's outcomes. With the right dataset, such modifications could transform dEclat into a useful heuristic for NLP tasks like Named Entity Recognition (NER).\n\n#### Preprocessing Steps\nThe retrieved posts will undergo the following preprocessing steps:\n- Removal of all punctuation marks and digits from the input texts.\n- Conversion of all uppercase letters to lowercase.\n- Tokenization - dividing the text into meaningful \"tokens\" (typically words).\n- Stemming - converting words in the input texts to their base morphological variants (e.g., `likes`, `liked`, `likely`, `liking` will be converted to `like`).\n- Filtering out word repetitions for each post.\n- Conversion of each word into a unique identifier.\n\nFor the example input data:\n\nOriginal Texts:\n- `\"You like him, don't you?\"`\n- `\"Likely, I will be getting 2 more for you.\"`\n\nPreprocessed Texts:\n- `\"You like him dont you\"`\n- `\"Likely I will be getting more for you\"`\n\nTokenized Texts:\n- `[\"you\", \"like\", \"him\", \"dont\", \"you\"]`\n- `[\"likely\", \"i\", \"will\", \"be\", \"getting\", \"more\", \"for\", \"you\"]`\n\nStemmed Texts:\n- `[\"you\", \"like\", \"him\", \"dont\", \"you\"]`\n- `[\"like\", \"i\", \"will\", \"be\", \"get\", \"more\", \"for\", \"you\"]`\n\nFiltered Texts:\n- `[\"you\", \"like\", \"him\", \"dont\"]`\n- `[\"like\", \"i\", \"will\", \"be\", \"get\", \"more\", \"for\", \"you\"]`\n\nTexts Converted to Unique Identifiers:\n- `[1, 2, 3, 4]`\n- `[2, 5, 6, 7, 8, 9, 10, 1]`\n\n#### Implementation Concept\nThe data will be retrieved using a console program created in Python. The program will accept the following arguments:\n- The name of the subreddit from which titles will be retrieved.\n- The type of posts to retrieve (new, popular, controversial, etc.).\n- The time period from which titles will be retrieved (last day, week, month, etc.).\n- The directory in which retrieved posts will be saved.\n- The number of posts to retrieve.\n\nThe posts will be fetched from the Reddit API using the [praw](https://praw.readthedocs.io/en/latest/getting_started/quick_start.html) library, tokenized, and stemmed using the [nltk](https://pypi.org/project/nltk/) library. The remaining processing will be done using the [pandas](https://pandas.pydata.org/) library. The final results will be saved in JSON format in the directory specified upon program execution.\n\n## Stage 1 (13.12.2022)\n### Technical Specification\n- The project will be developed using [Visual Studio Code](https://code.visualstudio.com/).\n- [Git](https://git-scm.com/) version control system and [GitHub](https://github.com/) service will be utilized.\n- Python 3 will be the primary programming language for the project.\n- Unit tests will be implemented using the [pytest](https://docs.pytest.org/en/7.2.x/) framework.\n- The [black](https://github.com/psf/black) autoformatter will ensure consistent source code style following PEP8 requirements.\n- Type correctness will be maintained using the [mypy](http://mypy-lang.org/) library.\n- The [isort](https://pycqa.github.io/isort/) library will be employed for automatic sorting of imported module declarations.\n\n### Project repository\nThe repository containing results of [stage 1](#stage-1-13122022) of the project can be found at: [https://github.com/steciuk/MED-dEclat-1](https://github.com/steciuk/MED-dEclat-1).\n\n### Data Retrieval and Preparation\nA script called [get_reddit.py](get_reddit.py) has been created, which, following the concept presented in section [Determination of the Dataset](#determination-of-the-dataset), determines the dataset by fetching titles from the specified subreddit.\n\nTo ensure the script functions correctly, you need to create a file named `reddit_secrets.py` in the project root directory and place your Reddit API keys in it. These keys can be generated by following the [instruction](https://docs.aws.amazon.com/solutions/latest/discovering-hot-topics-using-machine-learning/retrieve-and-manage-api-credentials-for-reddit-api-authentication.html).\n\nThe `reddit_secrets.py` file should contain the following lines:\n\n```\nCLIENT_ID = \"xxxxxxxxxxxxxxxx\"\nCLIENT_SECRET = \"xxxxxxxxxxxxxxxxxxxxxxxxxxxxx\"\n```\n\nThe tool's console interface is built using the [click](https://click.palletsprojects.com/) library.\n\nHelp message, describing the tool's usage:\n\n```\nOptions:\n  -s, --subreddit TEXT            Subreddit to scrape  [required]\n  -n, --num_posts INTEGER RANGE   Number of posts to scrape  [default: 100; x\u003e=1]\n  -l, --listing [hot|new|top|controversial]\n                                  Listing to use  [default: top]\n  -t, --time_filter [day|week|month|year|all]\n                                  Time filter. Used only for top and\n                                  controversial  [default: all]\n  -d, --directory PATH            Directory to save the data  [default: data]\n  --help                          Show this message and exit.\n```\n\nIf the program is invoked with invalid arguments, it will terminate and display an appropriate error message.\n\nThe program generates three files and saves them to the following location:\n\n```\n./\u003cdirectory\u003e/\u003csubreddit\u003e_\u003cnum_posts\u003e_\u003clisting\u003e_\u003ctime_filter\u003e_\u003ccurrent_datetime\u003e\n```\nWhere:\n- `\u003ccurrent_datetime\u003e` is a timestamp representing the script's start time in the format `YYYYMMDD_HHMMSS`.\n- The remaining placeholders correspond to the script's input parameters.\n\nThe generated files are:\n\n- `metadata.json`: Contains information about the program's input parameters.\n- `data.json`: Contains the extracted post titles and their corresponding token IDs.\n- `tokens_map.json`: Maps token IDs to tokens.\n\nFor the invocation:\n\n```bash\nget_reddit -s machinelearning -n 5 -l top -t year\n```\n\nThe following files are generated in the specified format:\n\n- `metadata.json`:\n  \n```json\n{\n  \"subreddit\": \"machinelearning\",\n  \"listing\": \"top\",\n  \"time_filter\": \"year\",\n  \"num_posts\": 5\n}\n```\n\n- `data.json`\n\n```json\n{\n  \"title\":{\n    \"0\":\"I made a robot that punishes me if it detects that if I am procrastinating on my assignments [P]\",\n    \"1\":\"[R] Speech-to-speech translation for a real-world unwritten language\",\n    \"2\":\"[P] I made an AI twitter bot that draws people\\u2019s dream jobs for them.\",\n    \"3\":\"[D] Types of Machine Learning Papers\",\n    \"4\":\"[P] WebtoonMe Project: Selfie to Webtoon style\"\n  },\n  \"tokens\":{\n    \"0\":[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15],\n    \"1\":[16, 17, 18, 19, 2, 20, 21, 22],\n    \"2\":[15, 0, 1, 23, 24, 25, 26, 4, 27, 28, 29, 30, 19, 31],\n    \"3\":[32, 33, 34, 35, 36, 37],\n    \"4\":[15, 38, 39, 40, 41, 42, 43]\n  }\n}\n```\n\n- `tokens_map.json`:\n\n```json\n{\n  \"token\":{\n    \"0\":\"i\",\n    \"1\":\"made\",\n    \"2\":\"a\",\n    \"3\":\"robot\",\n    \"4\":\"that\",\n    \"5\":\"punish\",\n    \"6\":\"me\",\n    \"7\":\"if\",\n    \"8\":\"it\",\n    \"9\":\"detect\",\n    \"10\":\"am\",\n    \"11\":\"procrastin\",\n    \"12\":\"on\",\n    ... \n    \"43\":\"style\"\n  }\n}\n```\n\n#### Unit Tests\nThe `test_get_reddit.py` file contains unit tests that verify the correct functionality of critical parts of the script.\n\n```\n...\nPASSED test/test_get_reddit.py::test_remove_non_alpha\nPASSED test/test_get_reddit.py::test_remove_duplicates_in_rows\nPASSED test/test_get_reddit.py::test_create_token_ids\n...\n```\n\n### Eclat and dEclat\nThe `build_tree.py` script is a console application that implements the Eclat and dEclat algorithms for frequent itemset mining. It utilizes the click library for its command-line interface.\n\nHelp message, describing the tool's usage:\n\n```\nOptions:\n  -d, --directory PATH            Directory to load the data from  [required]\n  -s, --support INTEGER RANGE     Minimum support for frequent itemsets [x\u003e=1; required]\n  -a, --algorithm [declat|eclat]  Algorithm to run  [default: declat]\n  --help  \n```\n\nThe script requires the following arguments:\n- the path to the directory containing the `data.json` and `tokens_map.json` files generated by the script described in the previous section,\n- the minimum support threshold for frequent itemsets,\n- the algorithm to run (`eclat` or `declat`). The default algorithm is `declat`.\n\nProviding incorrect program arguments will result in the termination of the program and the display of an appropriate message.\n\nThe program validates the format (and partially the content) of the retrieved files. If any issues are encountered, an appropriate message is displayed.\n\nThe output of the algorithm is a single JSON file (`declat.json` for the Declat algorithm or `eclat.json` for the dEclat algorithm). The file is automatically saved in the same directory from which the input files were retrieved.\n\nThe output file contains information about the minimum support selected by the user and a tree structure of objects (nodes) with properties defined by the class:\n\n```python\nclass TreeNode:\n    tokens_ids: list[int]\n    tokens: list[str]\n    support: int\n    id_set: set[int]\n    children: list[TreeNode]\n```\n\nWhere:\n\n- `tokens_ids` - a list of token identifiers that make up the itemset.\n- `tokens` - a list of decoded tokens.\n- `support` - the support value of the itemset.\n- `id_set` - in the case of Eclat, this is the tidlist containing the identifiers of the transactions where the itemset is found. In the case of dEclat, it is the difflist containing the identifiers of transactions where the itemset is not found relative to the parent.\n- `children` - the children of the given node.\n\nFor the invocation:\n\n```bash\nbuild_tree -d data/test_500_top_year_20221209_201531 -s 4 -a declat\n```\n\nThe `declat.json` with the following structure is generated:\n\n```json\n{\n  \"min_support\": 4,\n  \"tree\": {\n    \"tokens_ids\": [],\n    \"tokens\": [],\n    \"support\": 500,\n    \"id_set\": [],\n    \"children\": [\n      {\n        \"tokens_ids\": [3],\n        \"tokens\": [\"test\"],\n        \"support\": 206,\n        \"id_set\": [0, 1, 3, 4 ... 496, 497, 498, 499],\n        \"children\": [\n          {\n            \"tokens_ids\": [3, 4],\n            \"tokens\": [\"test\", \"if\"],\n            \"support\": 11,\n            \"id_set\": [6, 10, 16, 17 ... 487, 488, 489, 491, 495],\n            \"children\": [\n              {\n                \"tokens_ids\": [3, 4, 9],\n                \"tokens\": [\"test\", \"if\", \"post\"],\n                \"support\": 5,\n                \"id_set\": [483, 232, 209, 20, 87, 479],\n                \"children\": []\n              },\n              {\n                \"tokens_ids\": [3, 4, 78],\n                \"tokens\": [\"test\", \"if\", \"see\"],\n                \"support\": 5,\n                \"id_set\": [2, 483, 294, 424, 209, 216],\n                \"children\": []\n              },\n              ...\n            ]\n          },\n          ...\n        ]\n      },\n      ...\n    ]\n  }\n}\n``` \n\nFor the invocation with the Eclat algorithm, a similar file is generated.\n\n#### Unit Tests\nThe `test_build_tree.py` file contains unit tests that verify the correct functionality of critical parts of the script.\n\n```\nPASSED test/test_build_tree.py::test_load_data_missing_files\nPASSED test/test_build_tree.py::test_load_data\nPASSED test/test_build_tree.py::test_get_id_sets_map\nPASSED test/test_build_tree.py::test_TreeNode\nPASSED test/test_build_tree.py::test_build_declat_root\nPASSED test/test_build_tree.py::test_build_declat_tree\nPASSED test/test_build_tree.py::test_get_tid_sets_map\nPASSED test/test_build_tree.py::test_build_eclat_root\nPASSED test/test_build_tree.py::test_build_eclat_tree\n```\n\n### Visualization of Results\nDue to the difficulty in finding a suitable tool to visualize the structure of the resulting trees, I decided to create my own web application for this purpose.\n\n#### Technological Assumptions\n- The application was built using the [React](https://pl.reactjs.org) library.\n- [TypeScript](https://www.typescriptlang.org) was the programming language used in the project.\n- [React D3 Tree](https://github.com/bkrem/react-d3-tree) library was utilized for rendering tree structures.\n- [AG Grid](https://www.ag-grid.com/) library was used for displaying tabular structures.\n- The application was deployed using the [Firebase](https://firebase.google.com/) hosting service.\n- [ESlint](https://eslint.org) linter was configured in the project to ensure syntactic consistency and help identify errors in the source code in advance.\n- Automatic source code formatting settings were configured in the project using the [Prettier](https://prettier.io) extension. This solution helped maintain consistency in the coding style throughout the project.\n\n#### Functionality\n- The application has been deployed and is accessible at [https://declat.steciuk.dev/](https://declat.steciuk.dev/).\n- The source code is located in the [Stage 1 project repository](https://github.com/steciuk/MED-dEclat-1), in the [/tools/visualizer subdirectory](/tools/visualizer/).\n\nUI of the application after loading the page:\n\n![UI of the application after loading the page](doc/images/ui.png)\n\nUpon pressing the `UPLOAD FILES` button, the application allows uploading files that are results of the programs described in [Data Retrieval and Preparation](#data-retrieval-and-preparation) and [Eclat and dEclat](#eclat-and-declat).\n\nUpon uploading, independent validation of the contents of each file is performed. An error in one file does not interrupt the processing of the others. Detection of an error results in displaying an appropriate message on the screen.\n\nExample of an error message displayed when the data format is incorrect:\n\n![Error message displayed when the data format is incorrect](doc/images/incorrect_format.png)\n\nAfter uploading the `metadata.json` file, in the top right corner of the screen, information about the origin and size of the data is displayed, along with a button that directly navigates the user to the subreddit page from which the data was retrieved.\n\nAfter uploading the `declat.json` and `eclat.json` files the respective resulting trees are displayed in the \"Declat tree\" and \"Eclat tree\" panels. The view in these panels can be freely scrolled, zoomed in, and zoomed out.\n\nExample of a tree structure and the metadata displayed in the application:\n\n![Example of a tree structure and the metadata displayed in the application](doc/images/trees.png)\n\nExample node details of dEclat and Eclat trees, respectively:\n\n\u003cdiv style=\"display: flex; justify-content: space-between;\"\u003e\n  \u003cimg src=\"doc/images/declat_node.png\" alt=\"dEclat node details\" width=\"45%\"\u003e\n  \u003cimg src=\"doc/images/eclat_node.png\" alt=\"Eclat node details\" width=\"45%\"\u003e\n\u003c/div\u003e\n\nIn the \"Tokens Map\" panel, once the `tokens_map.json` file is uploaded, you'll see the mapping of identifiers to all defined tokens.\n\nIn the \"Data\" panel, all the retrieved titles along with their assigned token identifiers are displayed. Both tables allow you to filter and sort their contents from the header.\n\n\"Tokens Map\" and \"Data\" panels:\n\n![Tokens Map and Data panels](doc/images/tokens.png)\n\n### Experiments\n#### Definition of Research Data\n\nThe experiments will be conducted using the 500 most popular posts from the subreddits [funny](https://www.reddit.com/r/funny/) and [AMA](https://www.reddit.com/r/AMA/). The \"funny\" subreddit was chosen due to the wide variety of titles. The \"AMA\" subreddit was selected because it has a relatively large number of distinctive, recurring words in the titles.\n\nSample post titles from the \"funny\" subreddit:\n\n- \"My cab driver tonight was so excited to share with me that he’d made the cover of the calendar. I told him I’d help let the world see.\"\n- \"Gas station worker takes precautionary measures after customer refused to put out his cigarette.\"\n- \"The Denver Broncos have the entire town of ‘South Park’ in the stands for today’s NFL game.\"\n\nSample post titles from the \"AMA\" subreddit:\n\n- \"I'm going to do an AMA that will only have me, so myself, AMA!\"\n- \"I turned 100 today and have built a LEGO set. AMA.\"\n- \"I’m a US citizen aboard the Diamond Princess quarantined off the coast of Yokohama, Japan. AMA!\"\n\n#### Plan of Experiments\n\nFor each dataset, dEclat and Eclat trees will be constructed for each integer value of minimum support in the range [4, 199]. Then, memory complexity will be compared based on four statistics regarding the transaction identifier sets in the tree nodes (tid-lists for Eclat and dif-lists for dEclat):\n\n- The average size of the sets (the most significant measure for the research),\n- The median size of the sets,\n- The size of the smallest set,\n- The size of the largest set.\n\nThe root of the tree, representing the empty set, will be excluded from the analyzed statistics, as it is stored only for visualization purposes and does not provide useful information for the actual results of the algorithms.\n\n#### Conducting the Experiments\nThe experiments were conducted using a Jupyter notebook [experiments.ipynb](experiments.ipynb).\n\n#### Visualization and Analysis of Results\nThe results were visualized using the [Matplotlib](https://matplotlib.org/) library. Each chart includes the dataset label and the exact date the data was retrieved, following the convention described in the [Data Retrieval and Preparation](#data-retrieval-and-preparation) section.\n\n![Average length of AMA](doc/images/exp1.png)\n\n![Average length of funny](doc/images/exp2.png)\n\nFrom the perspective of memory complexity analysis, these two charts are the most informative. Since the number of nodes in the trees is the same, the average size of the identifier lists shows the actual difference in memory usage.\n\nOn the chart for the \"funny\" dataset, a sudden drop to zero can be observed. This happens because the dataset, due to its high diversity, does not contain any itemsets with such high support.\n\nAs predicted, dEclat does not perform well with sparse datasets. However, for the \"AMA\" dataset, it can be seen that for supports greater than 150, it starts to perform more efficiently than its predecessor. This is because the dataset contains several very frequently occurring tokens (\"ama\", \"I\", \"am\"). For supports exceeding 150, only itemsets composed of these most popular tokens are processed, effectively \"densifying\" the dataset in the deeper layers of the tree.\n\n![Biggest length for AMA](doc/images/exp3.png)\n\n![Biggest length for funny](doc/images/exp4.png)\n\nAs predicted, the charts showing the size of the largest identifier lists for trees with a given support show that for the Eclat algorithm, this size remains constant, while for dEclat, it decreases as the minimum support increases. This happens because, in the Eclat algorithm, the longest lists are for the most frequent single-item itemsets at the first level of the tree. For dEclat, longer lists occur for itemsets that are much less frequent than their parents.\n\nThe other charts do not provide significant value for the analysis but are available for review in the project repository.\n\n#### Experiment Summary and Possible Improvements\nThe dEclat algorithm was developed as an extension of the Eclat algorithm to reduce the memory required when determining tree nodes for datasets with relatively few individual tokens but a high density of their occurrences.\n\nFor the datasets defined in the project instructions (posts from social media forums), the token density is low, which means that the dEclat algorithm does not provide benefits.\n\nIt seems that the best solution for datasets with the described characteristics would be some kind of hybrid algorithm. This assumption is supported by the work of Tuan A. Trieu and Yoshitoshi Kunieda, [\"An Improvement for dEclat Algorithm\"](https://dl.acm.org/doi/pdf/10.1145/2184751.2184818).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsteciuk%2Fmed-declat-1","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsteciuk%2Fmed-declat-1","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsteciuk%2Fmed-declat-1/lists"}