{"id":13625100,"url":"https://github.com/measuresforjustice/textricator","last_synced_at":"2025-04-16T06:31:44.239Z","repository":{"id":34195744,"uuid":"133551745","full_name":"measuresforjustice/textricator","owner":"measuresforjustice","description":"Textricator is a tool to extract text from documents and generate structured data.","archived":false,"fork":false,"pushed_at":"2023-12-08T13:25:06.000Z","size":827,"stargazers_count":344,"open_issues_count":10,"forks_count":38,"subscribers_count":28,"default_branch":"main","last_synced_at":"2024-08-01T22:03:57.745Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://textricator.mfj.io","language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/measuresforjustice.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-05-15T17:39:00.000Z","updated_at":"2024-07-17T23:47:11.000Z","dependencies_parsed_at":"2023-11-14T04:29:26.765Z","dependency_job_id":"7e29e197-ddfd-42bd-a528-24be9ce79369","html_url":"https://github.com/measuresforjustice/textricator","commit_stats":null,"previous_names":[],"tags_count":37,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/measuresforjustice%2Ftextricator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/measuresforjustice%2Ftextricator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/measuresforjustice%2Ftextricator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/measuresforjustice%2Ftextricator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/measuresforjustice","download_url":"https://codeload.github.com/measuresforjustice/textricator/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223700234,"owners_count":17188274,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T21:01:50.859Z","updated_at":"2024-11-08T14:30:27.722Z","avatar_url":"https://github.com/measuresforjustice.png","language":"Kotlin","funding_links":[],"categories":["Kotlin","人工智能"],"sub_categories":[],"readme":"# ![Textricator](./textricator-logo-text-paths.png)\n\n_Textricator_ is a tool to extract text from documents and generate structured data.\n\nIf you have a bunch of PDFs with the same format (or one big, consistently formatted PDF)\nand you want to extract the data to CSV, XML, or JSON, _Textricator_ can help!\nIt can even work on OCR'ed documents!\n\n_Textricator_ is released under the\n[GNU Affero General Public License Version 3](https://www.gnu.org/licenses/agpl-3.0.en.html).\n\n_Textricator_ is deployed to [Maven Central](https://repo1.maven.org/maven2/io/mfj/textricator/) with GAV `io.mfj:textricator`.\n\nThis application is actively used and developed by [Measures for Justice](https://measuresforjustice.org).\nWe welcome feedback, bug reports, and contributions. Create an issue, send a pull request,\nor email us at \u003ctextricator@mfj.io\u003e. If you use _Textricator_, please let us know.\nSend us your mailing address and we will mail you a sticker.\n\n`io.mfj.textricator.Textricator` is the main entry point for library usage.\n\n`io.mfj.textricator.cli.TextricatorCli` is the command-line interface.\n\nThe CLI has three subcommands, to use the three main features of Textricator:\n  * text - Extract text from the PDF and generate JSON.\n  * table - Parse the text that is in columns and rows. See [Table](#table) section.\n  * form - Parse the text with a configured finite state machine. See [Form](#form) section.\n\n## Quick Start\n\n- Install Java (version 11+)\n  * Windows \u0026 Macos: Download from [https://java.com](https://java.com) and install.\n  * Linux: Use your package manager.\n- Download the latest build of _Textricator_ from [https://repo1.maven.org/maven2/io/mfj/textricator/](https://repo1.maven.org/maven2/io/mfj/textricator/) - click on the directory for the latest version and download `textricator-VERSION-bin.tgz` (or `textricator-VERSION-bin.zip` for Windows).\n- Extract it.\n- Run a shell\n  * Windows: run _Windows Powershell_ (it should be in the start menu)\n    * The following examples start with `./textricator`. On Windows, use `.\\textricator.bat`.\n  * MacOS: Run _Terminal_ (type \"terminal\" in Spotlight)\n- Show help\n    - `./textricator --help`\n- Download the example files to the textricator directory:\n  * https://github.com/measuresforjustice/textricator/blob/main/src/test/resources/io/mfj/textricator/examples/school-employee-list.pdf\n  * https://github.com/measuresforjustice/textricator/blob/main/src/test/resources/io/mfj/textricator/examples/school-employee-list.yml\n- Extract raw text from a PDF to standard out\n    - `./textricator text --input-format=pdf.pdfbox school-employee-list.pdf`\n- Parse a PDF to CSV\n    - `./textricator form --config=school-employee-list.yml school-employee-list.pdf school-employee-list.csv`\n      * This uses the configuration file `school-employee-list.yml` to parse `school-employee-list.pdf`.\n        To parse your own PDF form, you will need to write your own configuration file.\n        See the [Form](#form) section for details.\n        If your PDF has a tabular layout, see the [Table](#table) section.\n\n## Logging\n\nUse the `--debug` flag to log everything.\nLogging is written to standard error.\n\nTextricator uses [SLF4J](https://slf4j.org) for logging, with the [Logback](https://logback.qos.ch) implementation.\nIf you are using Textricator as a library, you may want to exclude `ch.qos.logback:logback-classic`.\nTextricator does *not* include a `/logback.xml`, so it will not conflict with other logging configurations,\nso long as `TextricatorCli.main()` is not invoked.\n\n## Extracting text\n\nTo extract the text from a PDF, run\n`textrictor text --input-format=pdf.itext5 input.pdf input-text.csv`\nfor any `input.pdf` and then open `input-text.csv` in your favorite spreadsheet program.\nIt will show you every bit of text that Textricator sees with its position, size,\nand font information. This information is very useful for building configuration to parse\ntables or forms using Textricator (see the following two sections).\n\nTry `--input-format=pdf.itext7` and `--input-format=pdf.pdfbox` to see how Textricator\nextracts the texts using the different parser engines. Some work better for some documents\nthan others.\n\n## Table\n\nIn table mode, the data is grouped into columns based on the x-coordinate of the text.\n\n### Example\n\nThis is an example for `src/test/resources/io/mfj/textricator/examples/probes.pdf`.\n\n```yaml\n# All measurements are in points. 1 point = 1/72 of an inch.\n# x-coordinates are from the left edge of the page.\n# y-coordinates are from the top edge of the page.\n\n# Use the built-in pdfbox extractor\nextractor: \"pdf.pdfbox\"\n\n# Ignore everything above 88pt from the top\ntop: 88\n\n# Ignore everything below 170pt from the top\nbottom: 170\n\n# If multiple text segments are withing 2pt vertically, consider them in the same row.\nmaxRowDistance: 2\n\n# Define the columns, based on the x-coordinate where the column starts:\ncols:\n  \"name\": 0\n  \"launched\": 132\n  \"speed\": 235\n  \"cospar\": 249\n  \"power\": 355\n  \"mass\": 415\n\ntypes:\n  \"name\":\n    label: \"Name\"\n\n  \"launched\":\n    label: \"Launch Date\"\n\n  \"speed\":\n    label: \"Speed (km/s)\"\n    type: \"number\"\n\n  \"cospar\":\n    label: \"COSPAR ID\"\n\n  \"power\":\n    label: \"Power (watts)\"\n    type: \"number\"\n\n  \"mass\":\n    label: \"Mass (pounds)\"\n    # Add .0 to the end of mass\n    replacements:\n      -\n        pattern: \"^(.*)$\"\n        replacement: \"$1.0\"\n\n# Omit if Power is less than 200\nfilter: 'power \u003e= 200'\n```\n\n## Form\n\nIn form mode, the data is parsed by Textricator using a\n[finite-state machine](https://en.wikipedia.org/wiki/Finite-state_machine) (FSM),\nand the FSM and additional parsing and formatting parameters are configured with\n[YAML](https://en.wikipedia.org/wiki/YAML), indicated by command line option `--config`.\n\n### Conditions\n\nState transitions are selected by evaluating conditions. Conditions are expressions parsed by\n[Expr](https://github.com/measuresforjustice/expr).\n\n#### Available variables:\n\n  * `ulx` - x coordinate of the upper-left corner of the text box\n  * `uly` - y coordinate of the upper-left corner of the text box\n  * `lrx` - x coordinate of the lower-right corner of the text box\n  * `lry` - y coordinate of the lower-right corner of the text box\n  * `text` - the text\n  * `page` - page number\n  * `page_prev` - page number of the previous text\n  * `fontSize` - font size\n  * `font` - font name\n  * `color` - text color\n  * `bgcolor` - background color\n  * `width` - width of the text box\n  * `height` - height of the text box\n  * `ulx_rel` - difference in `ulx` between the previous and current texts\n  * `uly_rel` - difference in `uly` between the previous and current texts\n  * `lrx_rel` - difference in `lrx` between the previous and current texts\n  * `lry_rel` - difference in `lry` between the previous and current texts\n  * [added Variables](#variables)\n\n### Example\n\nThis is an example for `src/test/resources/io/mfj/textricator/examples/school-employee-list.pdf`.\n\n```yaml\n# Use the built-in pdfbox parser\nextractor: \"pdf.pdfbox\"\n\n# All measurements are in points. 1 point = 1/72 of an inch.\n# x-coordinates are from the left edge of the page.\n# y-coordinates are from the top edge of the page.\nheader:\n    # ignore anything less than this many points from the top, default and per-page\n  default: 130\nfooter:\n    # ignore anything more than this many points from the top, default and per-page\n  default: 700\n\n# Text segments are generally parsed in order, top to bottom, left to right.\n# If two text segments have y-coordinates within this many points, consider them on the same line,\n# and process the one further left first, even if it is 0.4pt lower on the page.\nmaxRowDistance: 2\n\n# Define the output data record.\n# Since the main record type we're collecting information on is our employees,\n# we'll have that be the root type for our harvested information.\nrootRecordType: employee\nrecordTypes:\n  employee:\n    label: \"employee\" # Labels are used when nested recordTypes come into play, like this document.\n    valueTypes:\n      # Not sure what to name a valueType? Just make something up!\n      - employee\n      - name\n      - hiredate\n      - occupation\n      - showinfo\n      - bool1\n      - bool2\n      - bool3\n      - salary\n    children:\n      # In this example, there are multiple children nested under an employee,\n      # so we'll treat it as a 'child' to the 'employee' recordType.\n      - child\n  child:\n    label: \"child\"\n    valueTypes:\n      - child\n      - grade\n\nvalueTypes:\n  employee:\n    # In the CSV, use \"Employee ID\" as the column header instead of \"employee\".\n    label: \"Employee ID\"\n  name:\n    label: \"Name\"\n  hiredate:\n    label: \"Hire Date\"\n  occupation:\n    label: \"Occupation\"\n  salary:\n    label: \"Salary\"\n  showinfo:\n    label: \"Important Info?\"\n  bool1:\n    label: \"Boolean 1\"\n  bool2:\n    label: \"Boolean 2\"\n  bool3:\n    label: \"Boolean 3\"\n  child:\n    label: \"Attending Child\"\n  grade:\n    label: \"Grade\"\n\n# Now we define the finite-state machine\n# Let's name the state that our machine starts off with:\ninitialState: \"INIT\"\n\n# When each text segment is encountered, each transition for the current state is checked.\nstates:\n  INIT:\n    transitions:\n      # The first bit of text we reach is 'ID-0001', so we'll try the only transition that should work here.\n      -\n        # If this condition matches (which it should)\n        condition: employee # Curious about the condition? Sxroll further down to the conditions section of this YAML.\n        # Then we'll switch to the 'employee' state!\n        nextState: employee\n\n  employee: # ID number with the format 'ID-####'\n    startRecord: true # When we enter this stage, we'll create a new \"case\" record.\n    transitions:\n      - # Now we move on to the name label. Once again, by varifying the condition and moving on after that.\n        condition: namelabel\n        nextState: namelabel\n\n  namelabel:\n    include: false # The label isn't important information in and of itself, so we can just not include it in the data.\n    transitions:\n      -\n        condition: name\n        nextState: name\n\n  name:\n    transitions:\n      -\n        # Sometimes a name will be in two segments, and we'll hit another 'name' text segment before anything else.\n        # In that case, a state can transition to itself, compounding the information picked up in it.\n        condition: name\n        nextState: name\n      -\n        # Does the first condition not match the text? We move onto the next one.\n        condition: hiredateLabel\n        nextState: hiredateLabel\n\n  hiredateLabel:\n    include: false\n    transitions:\n      -\n        condition: hiredateLabel\n        nextState: hiredateLabel\n      -\n        condition: hiredate\n        nextState: hiredate\n\n  hiredate:\n    transitions:\n      -\n        condition: occupationLabel\n        nextState: occupationLabel\n\n  occupationLabel:\n    include: false\n    transitions:\n      -\n        condition: occupation\n        nextState: occupation\n\n  occupation:\n    transitions:\n      -\n        condition: occupation\n        nextState: occupation\n      -\n        # This state and the next are an example of how you can choose, using conditions, what to include or not.\n        # They share the same area of a document, but have qualities to them that can be distinguishable.\n        # Does it meet 'showinfo' conditions? Then we go to the 'showinfo' state that includes it.\n        condition: showinfo\n        nextState: showinfo\n      -\n        # Doesn't meet 'showinfo'? Then check for 'notinfo' and exclude it.\n        condition: notinfo\n        nextState: notinfo\n  showinfo:\n    transitions:\n      -\n        condition: showinfo\n        nextState: showinfo\n      -\n        condition: bool1\n        nextState: bool1\n  notinfo:\n    include: false\n    transitions:\n      -\n        condition: notinfo\n        nextState: notinfo\n      -\n        condition: bool1\n        nextState: bool1\n\n  bool1:\n    transitions:\n      -\n        condition: bool2\n        nextState: bool2\n  bool2:\n    transitions:\n      -\n        condition: bool3\n        nextState: bool3\n  bool3:\n    transitions:\n      -\n        condition: salaryLabel\n        nextState: salaryLabel\n\n  salaryLabel:\n    include: false\n    transitions:\n      -\n        condition: salary\n        nextState: salary\n\n  salary:\n    transitions:\n      -\n        condition: childrenLabel\n        nextState: childrenLabel\n      -\n        condition: employee\n        nextState: employee\n      -\n        condition: end\n        nextState: end\n\n  childrenLabel:\n    include: false\n    transitions:\n      -\n        condition: childrenLabel\n        nextState: childrenLabel\n      -\n        condition: childLabel\n        nextState: childLabel\n\n  childLabel:\n    include: false\n    transitions:\n      -\n        condition: child\n        nextState: child\n\n  child:\n    # Here we reach a datatype nested within another datatype. We can start records using this child datatype.\n    # In the process, we'll be making multiple rows for the parent datatype, each one holding onto it's own child.\n    startRecord: true\n    transitions:\n      -\n        condition: child\n        nextState: child\n      -\n        condition: gradeLabel\n        nextState: gradeLabel\n      -\n        condition: childLabel\n        nextState: childLabel\n\n  gradeLabel:\n    include: false\n    transitions:\n      -\n        # Normally, there would always been an instance of a grade appearing right after the label.\n        # But in this document, we have one instance of that not happening under ID-0007's child.\n        condition: grade\n        nextState: grade\n      -\n        # So we just account for that possibility by adding a transition out of the label.\n        condition: employee\n        nextState: employee\n\n  grade:\n    transitions:\n      -\n        condition: employee\n        nextState: employee\n      -\n        condition: childLabel\n        nextState: childLabel\n      -\n        # Reach the end of the usable info in a document, but there's still text left to go?\n        # An easy fix is to just create a looping, not-included state to finish the document off.\n        condition: end\n        nextState: end\n\n  end:\n    # We reached a point in the document where all the useful information is gone, but we still have text to go.\n    include: false\n    transitions:\n      -\n        # By using an always-true condition such as 'any', we can loop this state until the document has been completely gone through.\n        condition: any\n        nextState: end\n\n# Here we define the conditions:\nconditions:\n\n  # An example of comparing text with regex.\n  # In this case, we're making sure that the text contains the characters 'ID-' followed by any amount of numbers.\n  employee: 'text =~ /ID-(\\\\d)*/'\n\n  # You can match based on the x- and y- coordinates of the upper left and lower right corners of the rectangle\n  # containing the text. ulx = Upper-Left X-coordinate. lry = Lower-Right Y-coordinate. Also uly and lrx.\n  # You can define the lower and upper limit for each, inclusive.\n  namelabel: '70 \u003c ulx \u003c 80 and font = \"BCDFEE+Calibri-Bold\"'\n\n  # You can also match based on the type of font used, including if it was bolded or italicized.\n  name: '112 \u003c ulx \u003c 200 and font = \"BCDEEE+Calibri\"'\n\n  hiredateLabel: '230 \u003c ulx \u003c 270 and font = \"BCDFEE+Calibri-Bold\"'\n\n  hiredate: '280 \u003c ulx \u003c 290 and font = \"BCDEEE+Calibri\"'\n\n  occupationLabel: '391 \u003c ulx \u003c 393 and font = \"BCDFEE+Calibri-Bold\"'\n\n  occupation: '394 \u003c ulx \u003c 700 and font = \"BCDEEE+Calibri\"'\n\n  showinfo: 'font = \"BCDJEE+Georgia\"'\n\n  notinfo:  'font = \"BCDEEE+Calibri\"'\n\n  bool1:  'font = \"BCDIEE+Cambria\"'\n\n  bool2:  'font = \"BCDIEE+Cambria\"'\n\n  bool3:  'font = \"BCDIEE+Cambria\"'\n\n  salaryLabel: '391 \u003c ulx \u003c 393 and font = \"BCDFEE+Calibri-Bold\"'\n\n  salary: '394 \u003c ulx \u003c 700 and font = \"BCDEEE+Calibri\"'\n\n  childrenLabel: '70 \u003c ulx \u003c 140 and font = \"BCDFEE+Calibri-Bold\" and text =~ /(Attending)|(Children:)/'\n\n  childLabel: '230 \u003c ulx \u003c 240 and font = \"BCDFEE+Calibri-Bold\"'\n\n  child: '230 \u003c ulx \u003c 380 and font = \"BCDEEE+Calibri\"'\n\n  gradeLabel: '391 \u003c ulx \u003c 393 and font = \"BCDFEE+Calibri-Bold\"'\n\n  grade: '394 \u003c ulx \u003c 700 and font = \"BCDEEE+Calibri\"'\n\n  # You can also match based on the size of the font and on specific text.\n  end: 'fontSize = 16.0 and text = \"TOTAL:\"'\n\n  # Need a condition that is always true? \"1=1\" does that for you.\n  any: \"1 = 1\"\n```\n\n### Advanced functionality\n\n#### Regex matching\n\nYou can use regular expression matching instead of exact string matching in conditions:\n\n```yaml\nconditions:\n  caseTypeLabel: 'text =~ /Case Type:?/' # maybe sometimes they forgot the \":\"\n```\n\n#### Font matching\n\nYou can match font and font size:\n\n```yaml\nconditions:\n  helvetica8: 'font =~ /.*Helvetica.*/ and fontSize = 8'\n```\n\n#### Page number matching\n\nYou can match the page number\n\n```yaml\nconditions:\n  # Match the textbox that starts at 100pt,120pt on page 9.\n  specificTextbox: 'page = 9 and ulx = 100 and uly = 120'\n```\n\n#### Requiring intermediate state\n\nThere may be cases where you want to transition from state \"B\" to state \"A\" and start a new record,\nbut ONLY if you were in state \"C\" since last starting a new record:\n\n```yaml\nstates:\n  A:\n    startRecord: true\n    startRecordRequiredState: C\n    # ...\n```\n\n#### Combining states\n\nYou may have 2 different states that you want to combine into one column in the output:\n\n```yaml\nrecordTypes:\n  record:\n    label: \"Record\"\n    valueTypes:\n      - A\n      # no A2\n      - B\n\nstates:\n  A:\n    # ...\n  A2:\n    valueTypes: [ A ] # combine this with state \"A\".\n    # ...\n```\n\n#### Joining text segments in a state\n\nWhen there are multiple text segments in the same state, by default they are concatenated with a space in between.\nAny string, including an empty string, can be used as the separator:\n\n```yaml\nvalueTypes:\n  recordId:\n    # concatenate without spaces between\n    label: \"Record ID\"\n    separator: \"\"\n  description:\n    # put newlines between text segments\n    label: \"Description\"\n    separator: \"\n\"\n```\n\n#### Exclude types from output\n\nA value type (`dataRecordMember`) can be excluded from the output.\nThis is useful if the type is repeated on the PDF and needed as a data type to set new records,\nbut should not be in the output.\n\n```yaml\nvalueTypes:\n  repeatedId:\n    include: false\n```\n#### Relative offsets\n\nConditions can evaluate coordinates relative to the coordinates of the previous text.\nThis is useful for matching only something on the next line:\n\n```yaml\nmaxRowDistance: 2\nconditions:\n  # These should generally be the same (absolute value) as maxRowDistance\n  descriptionSameLine: '-2 \u003c= uly_rel \u003c= 2'\n  descriptionOneLineDown: '12 \u003c= uly_rel \u003c= 16' # for 12pt, single-spaced font.\n```\n\n#### Control which page number in output\n\nIf a complex data record spans multiple pages, which page number is used for the output can be controlled.\nEach type (`dataRecordType`) has a page priority (default: 0).\nThe page number for the record comes from the type with the highest priority.\n\n```yaml\nrecordTypes:\n  agency:\n    label: \"Agency\"\n    valueTypes:\n      - agencyName\n    children:\n      - case\n    # Agency may span hundreds of pages\n    # default page priority of 0\n  case:\n    label: \"Case\"\n    valueTypes:\n      - caseNumber\n      - name\n    children:\n      - charge\n    # Case record may span multiple pages.\n    # Higher priority than agency but lower than charges,\n    so case is used if there are no charges.\n    pagePriority: 1\n  charge:\n    label: \"Charge\"\n    valueTypes:\n      - chargeNumber\n    # Highest priority. Use charge's page number.\n    pagePriority: 2\n```\n\n#### Exclude \n\nText segments can be excluded before processing by the finite-state machine\nby adding condition name to the `excludeConditions` list.\nIf any of the conditions match, the text segment is excluded.\n\nFor example, to exclude all text segments that consist solely of underscores:\n\n```yaml\nexcludeConditions:\n  - underline\n  \nconditions:\n  underline: 'text =~ /_+/'\n```\n\n#### Starting new records for each value\n\nTODO\n\n#### Regex replacement of values\n\nSometimes the value contains a label, or other text you want to remove.\nIn a `dataRecordMember`, add a `replacement`, which has a `pattern`,\nwhich is a regular expression with capturing groups,\nand `replacement`, which is a replacement string with group references.\nSee `java.util.regex.Pattern` and `java.util.regex.Matcher` for details.\n\n```yaml\n# Replace \"Bond Agency: Fred's Bonds\" with \"Fred's Bonds\"\nvalueTypes:\n  bondagency:\n    label: \"Bond Agency\"\n    replacements:\n      -\n        pattern: \"Bond Agency:\\ *(.*)\"\n        replacement: \"$1\"\n```\n\n#### Variables\u003ca name=\"variables\"\u003e\u003c/a\u003e\n\nOn entering a state, a variable can be set.\n\nIn `State`, add a `setVariable`, which has `name` of the variable and the `value` to set.\n\nIf the `value` starts with `{` and ends with `}`, the content can be any built-in variable or previously\nset `variable` - the same things usable as variables for conditions.\n\nFor example: a condition checks that has the same background color as the last caseNo:\n\n```yaml\nstates:\n  caseNo:\n    setVariables:\n      -\n        name: \"lastCaseBgColor\"\n        value: \"{bgcolor}\"\n\nconditions:\n  sameBgColor: \u003e\n    145 \u003c= ulx\n    and fontSize = 8\n    and bgcolor = lastCaseBgColor\n```\n\n#### Splitting one PDF field into multiple CSV columns\n\nOne field in the PDF can be broken up into multiple columns in the CSV file, based on\ndifferent regular expressions and replacements:\n\n```yaml\n# Take a field that contains lastName,firstName and split it into 2 fields - lastName and firstName\nrecordTypes:\n  inmateName:\n    label: \"inmateName\"\n    valueTypes:\n      - lastName\n      - firstName\n\nvalueTypes:\n  lastName:\n    label: \"Last Name\"\n    replacements:\n      -\n        # lastName column will contain what's before comma\n        pattern: \"(.*),.*\"\n        replacement: \"$1\"\n  firstName:\n    label: \"First Name\"\n    replacements:\n      -\n        # firstName column will contain what's after comma\n        pattern: \".*,(.*)\"\n        replacement: \"$1\"\n\nstates:\n  name:\n    # when FSM hits state \"name\", split data into lastName and firstName\n    valueTypes:\n      - lastName\n      - firstName\n```\n\n#### Hyperlinks\n\nIf the text is a hyperlink, the URL can be used intead of the text by\nsetting `valueTypes.attribute` to `url`.\n\nThis is supported only by the `itext5` and `itext7` parsers.\n\n```yaml\nrecordTypes:\n  company:\n    label: \"Company\"\n    valueTypes:\n      - name\n      - website\n\nvalueTypes:\n  name:\n    label: \"Company Name\"\n  website:\n    label: \"Website\"\n    attribute: url # use the link URL instead of the text.\n\nstates:\n  name:\n    # Put the value in both \"name\" and \"website\" to get both the\n    # text and the url into the output.\n    valueTypes:\n      - name\n      - website\n```\n\n#### Filtering records\n\nRecords can be filtered before output by setting a filter on the `dataRecordType`. Root records\nwith a non-matching filter will not be output. Child records with a non-matching filter will\nbe removed from their parent - the root will still be output.\n\nFilters are expressions parsed by\n[Expr](https://github.com/measuresforjustice/expr)\n\nThe variables are the fields in the record. The default type for variables is `STRING`.\nThe type can be set by setting `type` in the `dataRecordMember` to `string` or `number`.\n\n```yaml\nrecordTypes:\n  case:\n    label: \"Case\"\n    # Only include cases in 2009-2013\n    filter: \"2009 \u003c= year \u003c= 2013\"\n    valueTypes:\n      - year\n      - name\n    children:\n      - charge\n  charge:\n    label: \"Charge\"\n    # omit charges of type \"dummy\"\n    filter: \u003e\n      not( type = \"dummy\" )\n    valueTypes:\n      - code\n      - type\n\nvalueTypes:\n  year:\n    type: number\n```\n\nFiltering happens after replacements and before member include checking.\nThis allows splitting a field (e.g.: extracting a year from a date)\nto use in the filter, and not including the split field in the output.\n\n#### Processing multiple files together\n\nMultiple files using the same form config can be parsed and written to a single output\nwith the `forms` CLI subcommand or with `io.mfj.textricator.Textricator.parseForms`.\nThe output will include the source file name for each record.\n\n## Extractors\n\nExtractors extract text (instances of `io.mfj.textricator.text.Text`) from a source.\nThere are four included extractors:\n\n  * _pdf.pdfbox_ - Extract text from PDF files using [Apache PDFBox](https://pdfbox.apache.org).\n  * _pdf.itext5_ - Extract text from PDF files using [iText 5](https://itextpdf.com/itext-5-core).\n  * _pdf.itext7_ - Extract text from PDF files using [iText 7](https://itextpdf.com/itext-7-core).\n  * _json_ - Parse text from the JSON format generated by the \"text\" subcommand of the CLI and by `io.mfj.textricator.Textricator.parseText()`.\n  * _csv_ - Parse text from the CSV format generatetd by the \"text\" subcommand of the CLI and by `io.mfj.textricator.Textricator.parseText()`\n\nOther extractors modules may be loaded, which may support different source types,\ncapture different information or split the text up differently.\nAn extractor module is loaded by adding a properties file\n`io/mfj/textricator/extractor/textExtractor.properties` to the classpath with a single property -\nthe key is the extractor name and the value is the fully-qualified class name of an implementation of\n`io.mfj.textricator.extractor.TextExtractorFactory`.\nTypically an extractor module will be distributed as a JAR that includes `textExtractor.properties`.\n\nIndicate which extractor to use by setting `extractor` in the yaml configuration,\nor overriding it by passing the extractor name to the `inputFormat`/`--input-format` option.\nIf the extractor is not specified and the input is PDF, `pdf.itext5` is used.\n\n## Versioning\n\n_Textricator_'s version is of the format `major.minor.build`.\n\nThe major number is incremented for breaking changes or major new features.\n\nThe minor number is incremented for minor new features.\n\nThe build number is generated by [Measures for Justice](https://measuresforjustice.org)'s private CI tool.\nIt is incremented for each build, regardless of the major and minor numbers\n(it does not reset to zero when minor or major numbers are increased).\n\n## Acknowledgements\n\nMuch credit is due to some people who do not show up in the commit history:\n\n* Joe Hale, for the original idea and prototype\n* John Castaneda and Abbie Miehle who, as the first end users, provided excellent\nbug reports, improvements, documentation, and examples.\n\n![mascot](./textricator-mascot.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmeasuresforjustice%2Ftextricator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmeasuresforjustice%2Ftextricator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmeasuresforjustice%2Ftextricator/lists"}