{"id":18553432,"url":"https://github.com/hpcc-systems/datapatterns","last_synced_at":"2026-02-05T00:38:29.117Z","repository":{"id":39991298,"uuid":"69577394","full_name":"hpcc-systems/DataPatterns","owner":"hpcc-systems","description":"HPCC Systems ECL bundle that provides some basic data profiling and research tools to an ECL programmer","archived":false,"fork":false,"pushed_at":"2026-01-27T15:56:08.000Z","size":512,"stargazers_count":5,"open_issues_count":6,"forks_count":5,"subscribers_count":18,"default_branch":"master","last_synced_at":"2026-01-28T02:22:16.819Z","etag":null,"topics":["data-profiling","ecl-bundle","hpcc-platform","hpcc-systems"],"latest_commit_sha":null,"homepage":"","language":"ECL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hpcc-systems.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-09-29T14:48:25.000Z","updated_at":"2026-01-27T15:59:21.000Z","dependencies_parsed_at":"2023-02-01T09:31:33.883Z","dependency_job_id":"9fa9ca8d-a207-4aa4-99a3-8d36d6c50b4a","html_url":"https://github.com/hpcc-systems/DataPatterns","commit_stats":null,"previous_names":[],"tags_count":31,"template":false,"template_full_name":null,"purl":"pkg:github/hpcc-systems/DataPatterns","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hpcc-systems%2FDataPatterns","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hpcc-systems%2FDataPatterns/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hpcc-systems%2FDataPatterns/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hpcc-systems%2FDataPatterns/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hpcc-systems","download_url":"https://codeload.github.com/hpcc-systems/DataPatterns/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hpcc-systems%2FDataPatterns/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29103422,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-04T22:44:52.815Z","status":"ssl_error","status_checked_at":"2026-02-04T22:44:16.428Z","response_time":62,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-profiling","ecl-bundle","hpcc-platform","hpcc-systems"],"created_at":"2024-11-06T21:17:03.711Z","updated_at":"2026-02-05T00:38:29.107Z","avatar_url":"https://github.com/hpcc-systems.png","language":"ECL","funding_links":[],"categories":[],"sub_categories":[],"readme":"### DataPatterns\n\nDataPatterns is an ECL bundle that provides data profiling and\nresearch tools to an ECL programmer.\n\n### Table of Contents\n\n  * [Installation](#installation)\n  * [Release Notes](#release_notes)\n  * [Profile()](#profile)\n    * [Summary Report with Graphs](#summary_report_with_graphs)\n  * [NormalizeProfileResults()](#normalizeprofileresults)\n  * [BestRecordStructure()](#bestrecordstructure)\n  * [Cardinality()](#cardinality)\n  * [Data Validation Submodule](#validation)\n    * [Validate()](#validation_validate)\n    * [Fix()](#validation_fix)\n  * [Benford()](#benford)\n  * [Profile() Testing](#testing)\n\n\u003ca name=\"installation\"\u003e\u003c/a\u003e\n### Installation\n\n**Note:**  `Profile()`, `BestRecordStructure()` and `Benford()` are\nnow included in the HPCC Systems platform!  They have been added to the ECL\nStandard Library (within `Std.DataPatterns`) and `Profile()` has also been\nintegrated within ECL Watch so you can create a profile from a saved logical file\nusing only a web browser.  Note that the Std library version of `Profile()` will\ncreate a visualization of the results only when executed from ECL Watch;\nvisualizations will not be generated if `Std.DataPatterns.Profile()` is\ncalled from ECL code.  If that is important to you, install this bundle\nversion instead (they coexist peacefully).\n\nThis code is installed as an ECL Bundle.  Complete instructions for managing ECL\nBundles can be found in [The ECL IDE and HPCC Client\nTools](https://cdn.hpccsystems.com/releases/CE-Candidate-9.4.2/docs/EN_US/TheECLIDEandHPCCClientTools_EN_US-9.4.2-1.pdf)\ndocumentation.\n\nUse the ecl command line tool to install this bundle:\n\n    ecl bundle install https://github.com/hpcc-systems/DataPatterns.git\n\nYou may have to either navigate to the client tools bin directory before\nexecuting the command, or use the full path to the ecl tool.\n\nAfter installation, all of the code here becomes available after you import it:\n\n```ECL\nIMPORT DataPatterns;\n```\n\nNote that is possible to use this code without installing it as a bundle.  To do\nso, simply make it available within your IDE and just ignore the Bundle.ecl\nfile. With the Windows IDE, the DataPatterns directory must not be a top-level\nitem in your repository list; it needs to be installed one level below the top\nlevel, such as within your \"My Files\" folder.\n\n\u003ca name=\"release_notes\"\u003e\u003c/a\u003e\n### Release Notes\n\u003cdetails\u003e\n\u003csummary\u003eClick to expand\u003c/summary\u003e\n\n|Version|Notes|\n|:----:|:-----|\n|1.0.0|Initial public release, finally with support for datasets defined using dynamic record lookup|\n|1.0.1|Add `ProfileFromPath` and `BestRecordStructureFromPath`; ave\\_length bug fix|\n|1.0.2|Change attribute field in CorrelationsRec embedded dataset to STRING|\n|1.1.0|Add record count breakdown for low-cardinality field values; ProfileFromPath() returns correct record structure|\n|1.1.1|Examine UTF8 values for alternate best\\_attribute\\_type data types rather than just passing them through|\n|1.2.0|Add option to emit a suitable TRANSFORM function to BestRecordStructure and BestRecordStructureFromPath|\n|1.2.1|Just-sprayed CSV files now supported within BestRecordStructureFromPath|\n|1.2.2|Bug fix: Support datasets that contain reserved words as field names (e.g. loop)|\n|1.3.0|Support for embedded child records; bug fix for proper computing of upper quartile value|\n|1.3.1|Just-sprayed CSV files now supported within ProfileFromPath|\n|1.3.2|Allow most CSV attributes to acquire default values in ProfileFromPath and BestRecordStructureFromPath|\n|1.3.3|Add file kind gathering back to the code in ProfileFromPath and BestRecordStructureFromPath (regression from 1.3.2)|\n|1.3.4|When given explicit numeric attribute types, refrain from recommending a \"best\" attribute type|\n|1.3.5|Fix ordering of output in BestRecordStructure when TRANSFORM is emitted|\n|1.4.0|Automatically include improved visual results of Profile, including data distribution graphs (within workunit's Resources tab)|\n|1.4.1|Regression: Fix self-tests that were failing due to changes in v1.3.4|\n|1.4.2|String fields containing all numerics with leading zeros are now marked as string in best\\_attribute\\_type; string fields where the length varies by more than three orders of magnitude are now marked as string in best\\_attribute\\_type|\n|1.5.0|Add support for SET OF data types and child datasets|\n|1.5.1|Support for tabbed visual results of multiple profiles in a workunit's result; changes to avoid symbol collision in calling ECL code; visual report styling update|\n|1.5.2|Import the ECL Standard Library within the Profile() function macro so callers do not have to|\n|1.5.3|Fix leading-zero numeric test, ensuring that only all-numeric values are considered as string type candidates|\n|1.5.4|Fix tab issues that appeared when multiple profiling results were available|\n|1.5.5|Fix visualized report vertical scrolling problems; update dependency to resolve security issue; removed erroneous HTML fragment from reports|\n|1.5.7|Add NormalizeProfileResults() function macro (see below for details); fix ECL compiler problem accessing child datasets hosted within embedded child records; make sure empty child dataset information appears in the final output|\n|1.6.0|is\\_numeric result is now based upon best\\_attribute\\_type rather than given\\_attribute\\_type, and the numeric\\_xxxx results will appear for those attributes as well; renamed numeric\\_correlations result to simply correlations||\n|1.6.1|Fix problem where large datasets with implicit numeric conversions ran out of memory during the final phase of profiling|\n|1.6.2|Fix issue where a record definition END would appear in the wrong place within BestRecordStructure(); remove BestRecordStructureFromPath() and ProfileFromPath() -- they never worked in all circumstances|\n|1.6.3|Fix issue where fields in the NewLayout record definition emitted by BestRecordStructure were out of order|\n|1.6.4|Bump visualizer code, including dependencies, to latest versions; increase default lcbLimit value to 1000|\n|1.6.5|Significant (~75%) performance boost within the text pattern code  -- thanks to Manjunath Venkataswamy for finding the issue|\n|1.7.0|NormalizeProfileResults() now shows results for attributes within child datasets (text patterns, correlations, etc); addition of Benford() analysis function; add workaround to allow a child dataset to be cited in a fieldListStr argument in Profile()|\n|1.7.1|Fix digit selection code in Benford|\n|1.7.2|Benford: Recognize implied trailing zeros after a decimal point|\n|1.8.0|Addition of Validation module; minor optimization in text pattern generation|\n|1.8.1|Fix issue with correlation with a numeric field named 'row'|\n|1.8.2|Security: Bump Viz Versions|\n|1.9.0|New functionality:  Cardinality() function; improve handling of specific child dataset fields in fieldListStr parameter; security updates|\n|1.9.1|Fix IMPORT in (Profile) Tests module; support UTF-8 strings in Mode values and example text patterns|\n|1.9.2|Security updates|\n|1.9.3|Better identify upper- and lower-case Unicode characters in text patterns; scan Unicode and UTF-8 strings to see if they can be represented with a STRING data type instead|\n|1.9.4|README fixes and updates; improve UTF-8 detection and prevent buffer overruns during character scans; use short form of Unicode property names in regex|\n|1.9.5|Correct Unicode regex regression introduced in 1.9.4|\n|1.10.0|Security in visualization; expand \"record count\" fields from UNSIGNED4 to UNSIGNED6 -- thanks to Manjunath Venkataswamy for requesting this improvement; add UTF8-specific TRIM and regex calls to avoid casting if possible|\n|1.10.1|Security updates for visualization code|\n\u003c/details\u003e\n\n---\n\u003ca name=\"profile\"\u003e\u003c/a\u003e\n### Profile\n\nDocumentation as pulled from the beginning of [Profile.ecl](Profile.ecl):\n\n    Profile() is a function macro for profiling all or part of a dataset.\n    The output is a dataset containing the following information for each\n    profiled attribute:\n\n         attribute               The name of the attribute\n         given_attribute_type    The ECL type of the attribute as it was defined\n                                 in the input dataset\n         best_attribute_type     An ECL data type that both allows all values\n                                 in the input dataset and consumes the least\n                                 amount of memory\n         rec_count               The number of records analyzed in the dataset;\n                                 this may be fewer than the total number of\n                                 records, if the optional sampleSize argument\n                                 was provided with a value less than 100\n         fill_count              The number of rec_count records containing\n                                 non-nil values; a 'nil value' is an empty\n                                 string, a numeric zero, or an empty SET; note\n                                 that BOOLEAN attributes are always counted as\n                                 filled, regardless of their value; also,\n                                 fixed-length DATA attributes (e.g. DATA10) are\n                                 also counted as filled, given their typical\n                                 function of holding data blobs\n         fill_rate               The percentage of rec_count records containing\n                                 non-nil values; this is basically\n                                 fill_count / rec_count * 100\n         cardinality             The number of unique, non-nil values within\n                                 the attribute\n         cardinality_breakdown   For those attributes with a low number of\n                                 unique, non-nil values, show each value and the\n                                 number of records containing that value; the\n                                 lcbLimit parameter governs what \"low number\"\n                                 means\n         modes                   The most common values in the attribute, after\n                                 coercing all values to STRING, along with the\n                                 number of records in which the values were\n                                 found; if no value is repeated more than once\n                                 then no mode will be shown; up to five (5)\n                                 modes will be shown; note that string values\n                                 longer than the maxPatternLen argument will\n                                 be truncated\n         min_length              For SET datatypes, the fewest number of elements\n                                 found in the set; for other data types, the\n                                 shortest length of a value when expressed\n                                 as a string; null values are ignored\n         max_length              For SET datatypes, the largest number of elements\n                                 found in the set; for other data types, the\n                                 longest length of a value when expressed\n                                 as a string; null values are ignored\n         ave_length              For SET datatypes, the average number of elements\n                                 found in the set; for other data types, the\n                                 average length of a value when expressed\n         popular_patterns        The most common patterns of values; see below\n         rare_patterns           The least common patterns of values; see below\n         is_numeric              Boolean indicating if the original attribute\n                                 was a numeric scalar or if the best_attribute_type\n                                 value was a numeric scaler; if TRUE then the\n                                 numeric_xxxx output fields will be\n                                 populated with actual values; if this value\n                                 is FALSE then all numeric_xxxx output values\n                                 should be ignored\n         numeric_min             The smallest non-nil value found within the\n                                 attribute as a DECIMAL; this value is valid only\n                                 if is_numeric is TRUE; if is_numeric is FALSE\n                                 then zero will show here\n         numeric_max             The largest non-nil value found within the\n                                 attribute as a DECIMAL; this value is valid only\n                                 if is_numeric is TRUE; if is_numeric is FALSE\n                                 then zero will show here\n         numeric_mean            The mean (average) non-nil value found within\n                                 the attribute as a DECIMAL; this value is valid only\n                                 if is_numeric is TRUE; if is_numeric is FALSE\n                                 then zero will show here\n         numeric_std_dev         The standard deviation of the non-nil values\n                                 in the attribute as a DECIMAL; this value is valid only\n                                 if is_numeric is TRUE; if is_numeric is FALSE\n                                 then zero will show here\n         numeric_lower_quartile  The value separating the first (bottom) and\n                                 second quarters of non-nil values within\n                                 the attribute as a DECIMAL; this value is valid only\n                                 if is_numeric is TRUE; if is_numeric is FALSE\n                                 then zero will show here\n         numeric_median          The median non-nil value within the attribute\n                                 as a DECIMAL; this value is valid only\n                                 if is_numeric is TRUE; if is_numeric is FALSE\n                                 then zero will show here\n         numeric_upper_quartile  The value separating the third and fourth\n                                 (top) quarters of non-nil values within\n                                 the attribute as a DECIMAL; this value is valid only\n                                 if is_numeric is TRUE; if is_numeric is FALSE\n                                 then zero will show here\n         correlations            A child dataset containing correlation values\n                                 comparing the current numeric attribute with all\n                                 other numeric attributes, listed in descending\n                                 correlation value order; the attribute must be\n                                 a numeric ECL datatype; non-numeric attributes\n                                 will return an empty child dataset; note that\n                                 this can be a time-consuming operation,\n                                 depending on the number of numeric attributes\n                                 in your dataset and the number of rows (if you\n                                 have N numeric attributes, then\n                                 N * (N - 1) / 2 calculations are performed,\n                                 each scanning all data rows)\n\n    Most profile outputs can be disabled.  See the 'features' argument, below.\n\n    Data patterns can give you an idea of what your data looks like when it is\n    expressed as a (human-readable) string.  The function converts each\n    character of the string into a fixed character palette to produce a \"data\n    pattern\" and then counts the number of unique patterns for that attribute.\n    The most- and least-popular patterns from the data will be shown in the\n    output, along with the number of times that pattern appears and an example\n    (randomly chosen from the actual data).  The character palette used is:\n\n         A   Any uppercase letter\n         a   Any lowercase letter\n         9   Any numeric digit\n         B   A boolean value (true or false)\n\n    All other characters are left as-is in the pattern.\n\n    Function parameters:\n\n    @param   inFile          The dataset to process; this could be a child\n                             dataset (e.g. inFile.childDS); REQUIRED\n    @param   fieldListStr    A string containing a comma-delimited list of\n                             attribute names to process; use an empty string to\n                             process all attributes in inFile; OPTIONAL,\n                             defaults to an empty string\n    @param   maxPatterns     The maximum number of patterns (both popular and\n                             rare) to return for each attribute; OPTIONAL,\n                             defaults to 100\n    @param   maxPatternLen   The maximum length of a pattern; longer patterns\n                             are truncated in the output; this value is also\n                             used to set the maximum length of the data to\n                             consider when finding cardinality and mode values;\n                             must be 33 or larger; OPTIONAL, defaults to 100\n    @param   features        A comma-delimited string listing the profiling\n                             elements to be included in the output; OPTIONAL,\n                             defaults to a comma-delimited string containing all\n                             of the available keywords:\n                                 KEYWORD                 AFFECTED OUTPUT\n                                 fill_rate               fill_rate\n                                                         fill_count\n                                 cardinality             cardinality\n                                 cardinality_breakdown   cardinality_breakdown\n                                 best_ecl_types          best_attribute_type\n                                 modes                   modes\n                                 lengths                 min_length\n                                                         max_length\n                                                         ave_length\n                                 patterns                popular_patterns\n                                                         rare_patterns\n                                 min_max                 numeric_min\n                                                         numeric_max\n                                 mean                    numeric_mean\n                                 std_dev                 numeric_std_dev\n                                 quartiles               numeric_lower_quartile\n                                                         numeric_median\n                                                         numeric_upper_quartile\n                                 correlations            correlations\n                             To omit the output associated with a single keyword,\n                             set this argument to a comma-delimited string\n                             containing all other keywords; note that the\n                             is_numeric output will appear only if min_max,\n                             mean, std_dev, quartiles, or correlations features\n                             are active; also note that enabling the\n                             cardinality_breakdown feature will also enable\n                             the cardinality feature, even if it is not\n                             explicitly enabled\n    @param   sampleSize      A positive integer representing a percentage of\n                             inFile to examine, which is useful when analyzing a\n                             very large dataset and only an estimated data\n                             profile is sufficient; valid range for this\n                             argument is 1-100; values outside of this range\n                             will be clamped; OPTIONAL, defaults to 100 (which\n                             indicates that the entire dataset will be analyzed)\n    @param   lcbLimit        A positive integer (\u003c= 1000) indicating the maximum\n                             cardinality allowed for an attribute in order to\n                             emit a breakdown of the attribute's values; this\n                             parameter will be ignored if cardinality_breakdown\n                             is not included in the features argument; OPTIONAL,\n                             defaults to 64\n\nHere is a very simple example of executing the full data profiling code:\n\n```ECL\nIMPORT DataPatterns;\n\nfilePath := '~thor::my_sample_data';\nds := DATASET(filePath, RECORDOF(filePath, LOOKUP), FLAT);\nprofileResults := DataPatterns.Profile(ds);\nOUTPUT(profileResults, ALL, NAMED('profileResults'));\n```\n\n\u003ca name=\"summary_report_with_graphs\"\u003e\u003c/a\u003e\n### Profile(): Summary Report with Graphs\n\nA report is generated based on the output of `Profile()`. The report is\naccessible via a Workunit's *Resources* tab within ECL Watch. For example:\n\n![Screen capture displaying active Resources tab](https://user-images.githubusercontent.com/1891935/57020403-2ac29480-6bf7-11e9-9584-a6fd23a3b4c4.png)\n\nEvery `attribute` in the Profile result is represented by a row of information.\nEach row of information is organized into several columns. Here is a short description\nof each column:\n\n1. Type information, Cardinality Count \u0026 Filled Count\n2. Min, Avg, Max Length (for string attributes) or Mean, Std. Deviation, Quartiles (for numeric attributes)\n3. Quartile bell curve and candlestick\n    * only shown for attributes with `is_numeric` === `true`\n    * this column is omitted if the above condition fails for all attributes\n4. Cardinality Breakdown listed by count descending\n    * only shown for attributes with `cardinality_breakdown` content\n    * this column is omitted if the above condition fails for all attributes\n5. Popular Patterns\n    * only shown for attributes with `popular_patterns` content\n    * this column is omitted if the above condition fails for all attributes\n\nThis is a screen capture displaying a report row for a string attribute\n(\"Test\\_Name\") and a numeric attribute (\"Test\\_Score\"):\n\n![Screen capture of two report rows](https://user-images.githubusercontent.com/1891935/56989566-c228d880-6b60-11e9-87a8-c2aa1c76b3d8.png)\n\n---\n\u003ca name=\"normalizeprofileresults\"\u003e\u003c/a\u003e\n### NormalizeProfileResults\n\nThe result of a call to `Profile` is a rich dataset.\nThere are several fields (depending on the features requested) and some\nof them can include child datasets embedded for each field from the dataset\nbeing profiled.\n\nIn some circumstances, it would be advantageous to save the profile results\nin a more normalized format.  For instance, a normalized format would allow\nthe task of comparing one profile result to another to be much easier.\n\n`NormalizeProfileResults` accepts only one argument:  the dataset representing\nthe result of a call to either `Profile`.  The result\nis a dataset in the following format:\n\n    RECORD\n        STRING      attribute;  // Field from profiled dataset\n        STRING      key;        // Field from profile results\n        STRING      value;      // Value from profile results\n    END;\n\nSome profile results are represented with embedded child datasets (modes,\ncardinality breakdowns, text patterns, and correlations).  When normalizing,\nportions of these child datasets are converted to string values delimited\nby the '\u0026#124;' character.  If records within the child dataset contain\nadditional information, such as a record count, the additional information\nis delimited with a ':' character.\n\nSample code:\n\n```ECL\nIMPORT DataPatterns;\n\nfilePath := '~thor::my_sample_data';\nds := DATASET(filePath, RECORDOF(filePath, LOOKUP), FLAT);\nprofileResults := DataPatterns.Profile(ds);\nnormalizedResults := DataPatterns.NormalizeProfileResults(profileResults);\nOUTPUT(normalizedResults, ALL, NAMED('normalizedResults'));\n```\n\nprofileResults:\n\n|attribute|given\\_attribute\\_type|rec\\_count|fill\\_count|fill\\_rate|popular_patterns|\n|---|---|---|---|---|---|\n|field1|string|1000|1000|100|\u003ctable\u003e\u003ctr\u003e\u003cth\u003edata\\_patterns\u003c/th\u003e\u003cth\u003erec_count\u003c/th\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003eAAAAAA\u003c/td\u003e\u003ctd\u003e10\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003eAAA\u003c/td\u003e\u003ctd\u003e5\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\nnormalizedResults:\n\n|attribute|key|value|\n|---|---|---|\n|field1|given\\_attribute\\_type|string|\n|field1|rec\\_count|1000|\n|field1|fill\\_count|1000|\n|field1|fill\\_rate|100|\n|field1|popular_patterns|AAAAAA:10\u0026#124;AAA:5|\n\n---\n\u003ca name=\"bestrecordstructure\"\u003e\u003c/a\u003e\n### BestRecordStructure\n\nThis is a function macro that, given a dataset, returns a recordset containing\nthe \"best\" record definition for the given dataset.  By default, the entire\ndataset will be examined. You can override this behavior by providing a\npercentage of the dataset to examine (1-100) as the second argument.  This is\nuseful if you are checking a very large file and are confident that a sample\nwill provide correct results.\n\nSample call:\n\n```ECL\nIMPORT DataPatterns;\n\nfilePath := '~thor::my_sample_data';\nds := DATASET(filePath, RECORDOF(filePath, LOOKUP), FLAT);\nrecordDefinition := DataPatterns.BestRecordStructure(ds);\nOUTPUT(recordDefinition, NAMED('recordDefinition'), ALL);\n```\n\nThe result will be a recordset containing only a STRING field.  The first\nrecord will always contain 'RECORD' and the last record will always contain\n'END;'.  The records in between will contain declarations for the attributes\nfound within the given dataset.  The entire result can be copied and pasted\ninto an ECL code module.\n\nNote that, when outputing the result of `BestRecordStructure` to a workunit,\nit is a good idea to add an ALL flag to the OUTPUT function.  This ensures that\nall attributes will be displayed.  Otherwise, if you have more than 100\nattributes in the given dataset, the result will be truncated.\n\n---\n\u003ca name=\"cardinality\"\u003e\u003c/a\u003e\n### Cardinality\n\nA portion of `Profile()` deals with cardinality.  If there is a low-enough number of\nunique values within an attribute, `Profile()` will automatically show those values\nalong with the count of the number of records with each value.  But what if you're\n*really* interested in those values and want to see them all?  No matter how many\nthere are?  Enter the `Cardinality()` function macro.\n\n`Cardinality()` finds all the unique values in one or more fields and displays the\ncount of the number of records for each value, without limitation on the number of\nfields or the number of found values.  The result is a simple three-field dataset:\n\n    STRING      attribute;\n    UTF8        value;\n    UNSIGNED8   rec_count;\n\nThe only required parameter to `Cardinality()` is a dataset to process.  You can\noptionally provide a comma-delimited string naming specific fields, if you don't\nwant to process all of the fields.  You can also limit the analysis to only a portion\nof the dataset (though that is of probably limited usefulness).\n\nSample call:\n\n```ECL\nIMPORT DataPatterns;\n\nfilePath := '~thor::my_sample_data';\nds := DATASET(filePath, RECORDOF(filePath, LOOKUP), FLAT);\ncardinalityResults := DataPatterns.Cardinality(ds);\nOUTPUT(cardinalityResults, NAMED('cardinalityResults'));\n```\n\nSee the comments at the beginning of the [Cardinality.ecl](Cardinality.ecl) file\nfor more details.\n\n---\n\u003ca name=\"validation\"\u003e\u003c/a\u003e\n### Data Validation Submodule\n\nValidation exists as a submodule within DataPatterns.  It contains two function\nmacros:  `Validate()` and `Fix()`.\n\n`Validate()` provides an easy mechanism for testing expected field values at\nthe record level, then append those test results to each record in a\nstandardized layout.  Tests are named, and associated with each test is\na bit of ECL that defines what a valid field should look like.  Fields with\nvalues that do not pass that test are flagged.\n\n`Fix()` is the other half of the testing:  Once you have the output from\n`Validate()` you will need to handle the failing field values somehow.  The\n`Fix()` function macro processes records with failures and gives you the\nopportunity to correct the error or to omit the record entirely.\n\n\u003ca name=\"validation_validate\"\u003e\u003c/a\u003e\n#### Validation.Validate()\n\nDocumentation as pulled from [Validation.ecl](Validation.ecl):\n\nValidation checks are defined within a semicolon-delimited STRING.  Each check\nshould be in the following format:\n\n     \u003ctest_name\u003e:\u003ctest_ecl\u003e\n\n`test_name` should be a name somehow representing the check that is\nbeing performed.  The name will be included in the appended data if the\ncheck fails.  This name should clearly (but succinctly) describe what is\nbeing tested.  There is no requirement for a `test_name` to be unique\n(and there some use cases where you may not want it unique at all) but,\nin general, the name should be unique within a single `Validate()` call.\nNames should start with a letter and may contain letters, numbers, periods,\ndashes, and underscores.\n\n`test_ecl` is ECL code that performs the test.  If a string literal is\nincluded in the test then the apostrophes must be escaped because the test\nis being defined within a string.  If a `REGEXFIND()` or `REGEXREPLACE()`\nfunction is used and anything within the pattern needs to be escaped then\nthe backslash must be double-escaped.  ECL already requires a single escape\n(`\\\\.` or `\\\\d`) but including it in a test here means you have to\ndouble-escape the backslash: `\\\\\\\\.` or `\\\\\\\\d`.\n\nThe ECL code used during the test is executed within the scope of a single\ndataset record.  Syntax-wise, it is similar to creating an ECL filter clause.\nLike a filter, the ECL should evaluate to a `BOOLEAN` result and what you want\nto do is return `TRUE` if the data being tested is **valid**.  Invalid results,\nwhere the ECL returns `FALSE`, are what is appended to the dataset.\n\n`Validate()` imports the Std ECL library, so all standard library functions\nare available for use within a test.  Also, because `Validate()` is a function\nmacro, any function that is in scope when `Validate()` is called may also be\nused within a test.  This provides quite a bit of flexibility when it comes\nto writing tests.  The example code below references `StartsWithAA()` which\nis an example of one of these user-supplied tests.\n\n`Validate()` also includes a few internally-defined functions for use within\nyour tests as a convenience.  Some are coercion functions that alter a field's\nvalue, others are test functions.  These tests are not available for use in\nyour own custom, externally-defined tests.\n\nCoercion helpers:\n\n    OnlyDigits(s)       Convert a single argument to a string and remove\n                        everything but numeric digits; returns a STRING\n\n    OnlyChars(s)        Convert a single argument to a UTF-8 string and remove\n                        everything but alphabetic characters; returns a\n                        UTF8 string\n\n    WithoutPunct(s)     Convert a single argument to a UTF-8 string and remove\n                        all punctuation characters; returns a UTF8 string\n\n    Patternize(s)       Create a 'text pattern' from the single argument,\n                        mapping character classes to a fixed palette:\n                            lowercase character -\u003e a\n                            uppercase character -\u003e A\n                            numeric digit       -\u003e 9\n                            everything else     -\u003e unchanged\n                        The result is returned as a UTF8 string\n\nValue testing helpers:\n\n    StrLen(s)           Convert a single argument to a UTF-8 string and return\n                        its length as an unsigned integer\n\n    IsOnlyDigits(s)     Return TRUE if every character in the value is a digit\n\n    IsOnlyUppercase(s)  Return TRUE if every character in the value is an\n                        uppercase character\n\n    IsOnlyLowercase(s)  Return TRUE if every character in the value is a\n                        lowercase character\n\n    IsDecimalNumber(s)  Return TRUE if the value is a number, possibly prefixed\n                        by a negative sign, and possibly including a decimal\n                        portion\n\nRecord-level testing helpers:\n\n    AllFieldsFilled()   Tests every top-level field in the record by coercing\n                        the values to STRING and seeing if any of them are empty;\n                        returns TRUE if no field value is an empty string; note\n                        that this function accepts no argument\n\nExample test specifications:\n\n     MyValueIsPos:my_value \u003e 0 // my_value must be greater than zero\n     SomeNumInRange:some_num BETWEEN 50 AND 100 // some_num must be 50..100\n     FIPSLength:StrLen(fips) = 5 // length of FIPS code must be 5\n     DatesOrdered:dateBegin \u003c= dateEnd // make sure dates are not flipped\n\nHere is a complete example:\n\n```ECL\nIMPORT DataPatterns;\n\nfilePath := '~thor::stock_data.txt';\n\nDataRec := RECORD\n    STRING  trade_date;\n    STRING  exchange_code;\n    STRING  stock_symbol;\n    STRING  opening_price;\n    STRING  high_price;\n    STRING  low_price;\n    STRING  closing_price;\n    STRING  shares_traded;\n    STRING  share_value;\nEND;\n\nds := DATASET(filePath, DataRec, CSV(SEPARATOR('\\t'), HEADING(1)));\n\n// Custom, external field validation functions\nStartsWithAA(STRING s) := s[1..2] = 'AA';\nIsValidPrice(STRING price) := NOT(REGEXFIND('^\\\\d+?00$', price) AND (UNSIGNED)price \u003e= 10000);\n\nchecks := 'NonZeroLowPrice:(REAL)low_price \u003e 0'\n            + '; NonZeroHighPrice:(REAL)high_price \u003e 0'\n            + '; LowPriceLessOrEqualToHighPrice:(REAL)low_price \u003c= (REAL)high_price'\n            + '; OpeningPriceGreaterThanOne:(REAL)opening_price \u003e 1'\n            + '; OpeningPriceFormat:REGEXFIND(U8\\'9+(\\\\\\\\.9{1,2})?\\', Patternize(opening_price))'\n            + '; OpeningPriceValid:IsValidPrice(opening_price)'\n            + '; ClosingPriceValid:IsValidPrice(closing_price)'\n            + '; SymbolStartsWithAA:StartsWithAA(stock_symbol)'\n            + '; EveryFieldPresent:AllFieldsFilled()'\n            ;\n\nvalidationResult := DataPatterns.Validation.Validate(ds, specStr := checks);\nOUTPUT(validationResult, {validationResult}, '~thor::stock_data_validated', OVERWRITE, COMPRESSED);\n```\n\u003ca name=\"validation_fix\"\u003e\u003c/a\u003e\n#### Validation.Fix()\n\nFixes are defined within a semicolon-delimited STRING.  Each fix should\nbe in the following format:\n\n     \u003cmembership_test\u003e:\u003cfix_ecl\u003e\n\n`membership_test` is a logical clause testing whether one or more tests\nfrom the `Validate()` function is true for that record.  The entries here\ncorrespond to the `test_name` entries from the `Validate()` function and\nthey can optionally form a boolean expression using AND and OR operators.\nAt its simplest, a `membership_test` is just a single `test_name` entry and\nit will be interpreted as the following ECL:\n\n     ('test_name' IN vaidation_results.violations)\n\nMore complex boolean expressions will use that as the basis.  For instance,\ntesting for \"`test_name_1` OR `test_name_2`\" -- meaning, if either of the two\nvalidation checks failed, execute the `fix_ecl` code -- would be interpreted as the\nfollowing ECL:\n\n      (('test_name_1' IN vaidation_results.violations)\n       OR\n       ('test_name_2' IN vaidation_results.violations))\n\nThe NOT() operator is also available, so testing for the absence of a\nvalidation is supported.\n\n`fix_ecl` is ECL code that fixes the problem.  The most basic fix is\nredefining a field value (e.g. `my_field := new_value_expression`).\nIf a string literal is included in the fix then the apostrophes must be\nescaped because it is being defined within a string.  If a `REGEXFIND()`\nor `REGEXREPLACE()` function is used and anything within the pattern needs\nto be escaped then the backslash must be double-escaped.  ECL already\nrequires a single escape (`\\\\.` or `\\\\d`) but including it in a test here\nmeans you have to double-escape the backslash: `\\\\\\\\.` or `\\\\\\\\d`.\n\nThe ECL code used during the fix is executed within the scope of a single\ndataset record.  This means that the expression may reference any field\nin the record.  There is no need to include SELF or LEFT scoping prefixes\nwhen citing a dataset field name.\n\n`Fix()` imports the Std ECL library, so all standard library functions\nare available for use within a fix.  Also, because `Fix()` is a function\nmacro, any function that is in scope when `Fix()` is called may also be\nused within a fix.\n\n`Fix()` also includes a few internally-defined functions for use within\nyour fixes as a convenience:\n\n     OnlyDigits(s)       Convert a single argument to a UTF-8 string and remove\n                         everything but numeric digits\n\n     OnlyChars(s)        Convert a single argument to a UTF-8 string and remove\n                         everything but alphabetic characters\n\n     WithoutPunct(s)     Convert a single argument to a UTF-8 string and remove\n                         all punctuation characters\n\n     Swap(f1, f2)        Swap the contents of two named fields\n\n     SkipRecord()        Remove the current record from the dataset\n\nHere is a complete example:\n\n```ECL\nIMPORT DataPatterns;\n\nValRec := RECORD\n    UNSIGNED2       num_violations;\n    SET OF STRING   violations;\nEND;\n\nLAYOUT := RECORD\n    STRING  trade_date;\n    STRING  exchange_code;\n    STRING  stock_symbol;\n    STRING  opening_price;\n    STRING  high_price;\n    STRING  low_price;\n    STRING  closing_price;\n    STRING  shares_traded;\n    STRING  share_value;\n    ValRec  validation_results;\nEND;\n\nds := DATASET('~thor::stock_data_validated', LAYOUT, FLAT);\n\nrepairs := 'LowPriceLessThanOrEqualToHighPrice:Swap(high_price, low_price)'\n            + '; OpeningPriceValid AND ClosingPriceValid:SkipRecord()'\n            + '; OpeningPriceGreaterThanOne:opening_price := \\'2\\''\n            ;\n\nrepairResults := DataPatterns.Validation.Fix(ds, specStr := repairs);\nOUTPUT(repairResults, {repairResults}, '~thor::stock_data_fixed', OVERWRITE, COMPRESSED);\n```\n\n---\n\u003ca name=\"benford\"\u003e\u003c/a\u003e\n### Benford\n\nBenford's law, also called the Newcomb–Benford law, or the law of anomalous\nnumbers, is an observation about the frequency distribution of leading digits\nin many real-life sets of numerical data.\n\nBenford's law doesn't apply to every set of numbers, but it usually applies\nto large sets of naturally occurring numbers with some connection like:\n\n* Companies' stock market values\n* Data found in texts — like the Reader's Digest, or a copy of Newsweek\n* Demographic data, including state and city populations\n* Income tax data\n* Mathematical tables, like logarithms\n* River drainage rates\n* Scientific data\n\nThe law usually doesn't apply to data sets that have a stated minimum and\nmaximum, like interest rates or hourly wages. If numbers are assigned,\nrather than naturally occurring, they will also not follow the law. Examples\nof assigned numbers include: zip codes, telephone numbers and Social\nSecurity numbers.\n\nFor more information: https://en.wikipedia.org/wiki/Benford%27s_law\n\n**Note:**  This function is also available in the ECL Standard Library\nas `Std.DataPatterns.Benford()` as of HPCC version 7.12.0.\n\nDocumentation as pulled from the beginning of [Benford.ecl](Benford.ecl):\n\n    Note that when computing the distribution of the most significant digit,\n    the digit zero is ignored.  So for instance, the values 0100, 100, 1.0,\n    0.10, and 0.00001 all have a most-significant digit of '1'.  The digit\n    zero is considered for all other positions.\n\n    @param   inFile          The dataset to process; REQUIRED\n    @param   fieldListStr    A string containing a comma-delimited list of\n                             attribute names to process; note that attributes\n                             listed here must be top-level attributes (not child\n                             records or child datasets); use an empty string to\n                             process all top-level attributes in inFile;\n                             OPTIONAL, defaults to an empty string\n    @param   digit           The 1-based digit within the number to examine; the\n                             first significant digit is '1' and it only increases;\n                             OPTIONAL, defaults to 1, meaning the most-significant\n                             non-zero digit\n    @param   sampleSize      A positive integer representing a percentage of\n                             inFile to examine, which is useful when analyzing a\n                             very large dataset and only an estimated data\n                             analysis is sufficient; valid range for this\n                             argument is 1-100; values outside of this range\n                             will be clamped; OPTIONAL, defaults to 100 (which\n                             indicates that all rows in the dataset will be used)\n\n    @return  A new dataset with the following record structure:\n\n         RECORD\n             STRING      attribute;   // Name of data attribute examined\n             DECIMAL4_1  zero;        // Percentage of rows with digit of '0'\n             DECIMAL4_1  one;         // Percentage of rows with digit of '1'\n             DECIMAL4_1  two;         // Percentage of rows with digit of '2'\n             DECIMAL4_1  three;       // Percentage of rows with digit of '3'\n             DECIMAL4_1  four;        // Percentage of rows with digit of '4'\n             DECIMAL4_1  five;        // Percentage of rows with digit of '5'\n             DECIMAL4_1  six;         // Percentage of rows with digit of '6'\n             DECIMAL4_1  seven;       // Percentage of rows with digit of '7'\n             DECIMAL4_1  eight;       // Percentage of rows with digit of '8'\n             DECIMAL4_1  nine;        // Percentage of rows with digit of '9'\n             DECIMAL7_3  chi_squared; // Chi-squared \"fitness test\" result\n             UNSIGNED8   num_values;  // Number of rows with non-zero values for this attribute\n         END;\n\n    The named digit fields (e.g. \"zero\" and \"one\" and so on) represent the\n    digit found in the 'digit' position of the associated attribute.  The values\n    that appear there are percentages.  num_values shows the number of\n    non-zero values processed, and chi_squared shows the result of applying\n    that test using the observed vs expected distribution values.\n\n    The first row of the results will show the expected values for the named\n    digits, with \"-- EXPECTED DIGIT n --\" showing as the attribute name.'n' will\n    be replaced with the value of 'digit' which indicates which digit position\n    was examined.\n\nSample call:\n\n```ECL\nIMPORT DataPatterns;\n\nfilePath := '~thor::stock_data_';\n\nDataRec := RECORD\n\tUNSIGNED4   trade_date;\n\tSTRING1     exchange_code;\n\tSTRING9     stock_symbol;\n\tDECIMAL9_2  opening_price;\n\tDECIMAL9_2  high_price;\n\tDECIMAL9_2  low_price;\n\tDECIMAL9_2  closing_price;\n\tUNSIGNED4   shares_traded;\n\tUNSIGNED4   share_value;\nEND;\n\nds := DATASET(filePath, DataRec, FLAT);\n\n// Analyze only the opening_price, closing_price, and trade_date attributes\nbenfordResult := DataPatterns.Benford(ds, 'opening_price, closing_price, trade_date');\n\nOUTPUT(benfordResult, NAMED('benfordResult'), ALL);\n```\n\nThe result would look something like the following:\n\n|attribute|zero|one|two|three|four|five|six|seven|eight|nine|chi_squared|num_values|\n|---|---|---|---|---|---|---|---|---|---|---|---|---|\n|-- EXPECTED DIGIT 1 --|-1|30.1|17.6|12.5|9.7|7.9|6.7|5.8|5.1|4.6|20.09|20959177|\n|opening_price|-1|31.7|20|13.3|9.7|7.2|5.7|4.8|4.1|3.6|1.266|19082595|\n|closing_price|-1|31.7|20|13.3|9.7|7.2|5.7|4.8|4|3.6|1.307|19083933|\n|trade_date|-1|0|100|0|0|0|0|0|0|0|468.182|20959177|\n\nThe result contains the attribute name, expected and actual distributions of the digit\nas a percentage, the chi-squared computation indicating how well that attribute\nadheres to Benford's Law, and the number of records actually considered.\n\nBy definition, the most-significant digit will never be zero.  Therefore, when computing the\ndistribution of the most-significant digit, the 'zero' field will show -1 for all\nattributes in the result.\n\nThe chi\\_squared column represents the critical value for a chi-squared test.  If an\nattribute's chi\\_squared value is greater than the expected chi\\_squared value then that\nattribute does not follow Benford's Law.\n\nIn the above example, the trade\\_date attribute fails the chi-squared test, as 468.182 \u003e 20.09.\nThis makes sense, because the data in that attribute is a date in YYYYMMDD format represented\nas an unsigned integer, and the dataset contains stock data for only the past few years.\n\n---\n\u003ca name=\"testing\"\u003e\u003c/a\u003e\n### Profile() Testing\n\nThe data profiling code can be easily tested with the included Tests module.\nhthor or ROXIE should be used to execute the tests, simply because Thor takes a\nrelatively long time to execute them.  Here is how you invoke the tests:\n\n```ECL\nIMPORT DataPatterns;\nEVALUATE(DataPatterns.Tests);\n```\n\nIf the tests pass then the execution will succeed and there will be no output.\nThese tests may take some time to execute on Thor.  They run much faster on\neither hthor or ROXIE, due to the use of small inline datasets.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhpcc-systems%2Fdatapatterns","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhpcc-systems%2Fdatapatterns","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhpcc-systems%2Fdatapatterns/lists"}