{"id":22826241,"url":"https://github.com/adriens/schemacrawler-additional-command-lints-as-csv","last_synced_at":"2025-05-05T22:55:24.590Z","repository":{"id":48346397,"uuid":"149840057","full_name":"adriens/schemacrawler-additional-command-lints-as-csv","owner":"adriens","description":"An additional command for schemacrawler to dump lints as csv files and produce analytics","archived":false,"fork":false,"pushed_at":"2021-07-30T18:14:02.000Z","size":349,"stargazers_count":2,"open_issues_count":8,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-05-05T22:55:21.570Z","etag":null,"topics":["csv","dashoard","database","elasticsearch","kibana","lint","logstash","schemacrawler"],"latest_commit_sha":null,"homepage":"https://www.linkedin.com/pulse/continuous-database-linting-dashboards-bring-analytics-adrien-sales/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/adriens.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-09-22T02:57:14.000Z","updated_at":"2020-10-12T22:16:27.000Z","dependencies_parsed_at":"2022-08-29T17:11:19.200Z","dependency_job_id":null,"html_url":"https://github.com/adriens/schemacrawler-additional-command-lints-as-csv","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adriens%2Fschemacrawler-additional-command-lints-as-csv","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adriens%2Fschemacrawler-additional-command-lints-as-csv/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adriens%2Fschemacrawler-additional-command-lints-as-csv/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adriens%2Fschemacrawler-additional-command-lints-as-csv/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/adriens","download_url":"https://codeload.github.com/adriens/schemacrawler-additional-command-lints-as-csv/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252590550,"owners_count":21772936,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv","dashoard","database","elasticsearch","kibana","lint","logstash","schemacrawler"],"created_at":"2024-12-12T17:14:43.923Z","updated_at":"2025-05-05T22:55:24.571Z","avatar_url":"https://github.com/adriens.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/adriens/schemacrawler-additional-command-lints-as-csv.svg?branch=master)](https://travis-ci.org/adriens/schemacrawler-additional-command-lints-as-csv)\r\n[![](https://jitpack.io/v/adriens/schemacrawler-additional-command-lints-as-csv.svg)](https://jitpack.io/#adriens/schemacrawler-additional-command-lints-as-csv)\r\n\r\n# schemacrawler-additional-command-lints-as-csv\r\n\r\nAn additional [command](http://www.schemacrawler.com/plugins.html) for\r\n[schemacrawler](http://www.schemacrawler.com/) to dump lints as csv files,\r\nwith some additional fields.\r\n\r\nSee [first linkedIn article](https://www.linkedin.com/pulse/continuous-database-linting-dashboards-bring-analytics-adrien-sales/) for more details or check the [Pinterest album for more samples](https://www.pinterest.com/rastadidi/schemacrawler-lint-dashboards/)\r\n\r\nFor people interested by gdpr, take a look at this other [linkedIn article](https://www.linkedin.com/pulse/from-database-column-gdpr-analytics-elk-getting-map-your-adrien-sales/).\r\n\r\n\r\n# Usage\r\n\r\nThis additional schemacrawler command is dedicated to produce csv files that\r\ncan be used to produce advanced reporting for :\r\n\r\n- database lints\r\n- table sizes (number of rows and columns), so you can report and monitor them, etc...\r\n\r\nThanks to the fact that `csv` is a very common data format, it can be used to\r\nproduce intelligence and reporting in numerous technologies. I'll focus on\r\n[Elastic Search](https://www.elastic.co/) reporting, but it would also be very efficient to produce\r\nanalytics with any other reporting tools like [Jupyter Notebook](https://jupyter.org/)\r\nor [R](https://www.r-project.org/).\r\n\r\n\r\n# Install steps\r\n\r\nBuild the jar :\r\n\r\n```\r\nmvn clean package\r\n```\r\n\r\nthen copy the jar to `$SCHEMACRAWLER_HOME/lib` and you're done with install steps.\r\n\r\nTo get help, simply run :\r\n\r\n```\r\nschemacrawler -help -c=csv\r\n```\r\n\r\nThis `jar` adds the following command with the following options :\r\n\r\n```\r\n-c=csv -dbid=666 -dbenv=hell\r\n```\r\n\r\n- `-c=csv` : tells that we want to dump the lints as a csv file\r\n- `-dbid`  : optional paramater if you want to stick on a given database\r\n- `-dbenv` : optional paramater if you want to tag a database to an environment (typically prod, dev, test, ...)\r\n\r\nFor each run, you then get the following `csv` files in your working directory :\r\n\r\n- `schemacrawler-lints-\u003cUUID\u003e.csv` : this file contains lint outputs\r\n- `schemacrawler-tables-\u003cUUID\u003e.csv` : this file contains datas reporting number of rows and columns of tables, with schema, tableName, ...\r\n- `schemacrawler-columns-\u003cUUID\u003e.csv` : this file contains datas about table/columns\r\n\r\nTo load these files, you need the dedicated [logstash](https://www.elastic.co/products/logstash) configuration files.\r\nTherefore, you have two logstash configuration files samples :\r\n\r\n- for lints, check `logstash-lints.conf`\r\n- for table datas, check `logstash-tables.conf`\r\n- for column datas, check `logstash-columns.conf`\r\n\r\n**For each, you have to customize index names and `input.file.path` according to your needs.**\r\n\r\n# Contribute\r\n\r\nYou can contribute code, but also your own dashoard realizations. Therefore, just make a PR that :\r\n\r\n- add an image to the `img` directory\r\n- add the screenshot to the dedicated `SCREENSHOTS.md` file or a link to a video : any cool demo is welcome\r\n- also you can ask (fill and issue on Github for that) to contribute to the dedicated [Pinterest album](https://www.pinterest.com/rastadidi/schemacrawler-lint-dashboards/)\r\n\r\n# Contribute ideas\r\n\r\nIf you have ideas for dashboard but don't know how to create them, but still are\r\nconvinced that the are interesting, please fill an issue on the project,\r\nexplaining what you'd like to produce. A hand made drawing can also be a\r\nvery good beginning !\r\n\r\n# Details and samples\r\n\r\nSee [linkedIn article](https://www.linkedin.com/pulse/continuous-database-linting-dashboards-bring-analytics-adrien-sales/) for more details or visit the dedicated [Pinterest album](https://www.pinterest.com/rastadidi/schemacrawler-lint-dashboards/).\r\n\r\n![Simple Kibana dashboard screenshot](img/COVER.png \"Screenshot\")\r\n\r\n# Youtube demos\r\n\r\n- [running lints from command line](https://www.youtube.com/watch?v=sDM_el5Pk_A)\r\n- [Drilling into the biggest database](https://www.youtube.com/watch?v=GQ07UoC6IWg)\r\n- [Drilling into objects by their size](https://www.youtube.com/watch?v=9Ttszji3Zuw)\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadriens%2Fschemacrawler-additional-command-lints-as-csv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fadriens%2Fschemacrawler-additional-command-lints-as-csv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadriens%2Fschemacrawler-additional-command-lints-as-csv/lists"}