https://github.com/opensavannah/liberate-the-pdf
Pro-hacks for scraping public data stuck in PDFs by Carl V. Lewis of Open Savannah
https://github.com/opensavannah/liberate-the-pdf
Last synced: 7 months ago
JSON representation
Pro-hacks for scraping public data stuck in PDFs by Carl V. Lewis of Open Savannah
- Host: GitHub
- URL: https://github.com/opensavannah/liberate-the-pdf
- Owner: opensavannah
- Created: 2017-03-16T23:40:33.000Z (over 9 years ago)
- Default Branch: master
- Last Pushed: 2017-03-19T01:01:02.000Z (over 9 years ago)
- Last Synced: 2025-02-01T20:11:25.995Z (over 1 year ago)
- Homepage:
- Size: 2.93 KB
- Stars: 0
- Watchers: 5
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# CIVIC-TECH 101: PRO-HACKS FOR PDF PROBLEMS
## :musical_note: I got 99 problems and a PDF is (more than) one :notes:
### What to do when government agencies can't, won't or don't know how to provide *public* data in machine readable formats
--
###### A tutorial by [Carl V. Lewis](http://carlvlew.is) for [Open Savannah](http://opensavannah.org), a [Code for America](http://codeforamerica.org) Brigade.
If you've ever sought government data, chances are likely you've come across more than one dataset that's only available as a PDF. While PDFs certainly have a valid role – they maintain the same style regardless of what operating system or device they're viewed on and have universal support– you can't exactly perform a *query* or build a *pivot table* on PDFs. While the best step to take in this situation is *always* to contact the government agency or, if that fails, [file a FOIA request](https://www.foia.gov/how-to.html) to see if you can get the original data that was used to create the PDF in machine-readable format. But life doesn't always work out perfectly, and sometimes you can't fit a square peg in a round hole.
But you **can** scrape it.
Don't worry. The term *scraping* isn't meant to conjure fear or panic. Often, you don't even need to know a single line of code to scrape data from PDFs. But let's first of all start out by determining what kind of data-silo beast we're dealing with.....
### MYTH: Not all PDFs are created equal
People assume that because PDFs are portable and universal that they are all the same. Quite the contrary, PDFs can take two different forms of evil.
Let's start out with the most benign species of PDF.
**1.Text-based PDFs** - This type of PDF––while still a PDF and thus a natural adversary of all Open Data people––is the friendliest type of PDF to work with, if that means much.
+ It is produced from a spreadsheet or other electronic report, and you can copy and paste text out of it with ease.
+ Many of this variety can be handled with open-source tools.
Now, let's meet the more evil PDF:
**2. Image-Based (Scanned) PDFs** - These are exponentially more difficult to scrape, as they weren't originally generated by a computer; or, if they were, the computer generated only print copies which were then scanned.
+ OCR (optimal character-recognition) is the way to go
+ Can usually be handled with open source tools, depending on quality.
### But how do I tell which type of PDF I'm working with?
Try doing an `Edit>Find` command to search for text in the PDF. If you see a word in the document and it's findable with Find/Replace, you've got yourself a text-based PDF. Congrats!
Also, see [this detailed StackOverflow discussion on further ways to determine the type of PDF you have](http://stackoverflow.com/questions/1489733/how-to-know-if-a-pdf-contains-only-images-or-has-been-ocr-scanned-for-searching)
<--WIP-->