{"id":13747384,"url":"https://github.com/michenriksen/birdwatcher","last_synced_at":"2025-04-04T07:04:57.989Z","repository":{"id":50345766,"uuid":"70418781","full_name":"michenriksen/birdwatcher","owner":"michenriksen","description":"Data analysis and OSINT framework for Twitter","archived":false,"fork":false,"pushed_at":"2020-10-18T21:46:58.000Z","size":710,"stargazers_count":413,"open_issues_count":9,"forks_count":62,"subscribers_count":25,"default_branch":"master","last_synced_at":"2025-04-02T05:53:55.407Z","etag":null,"topics":["framework","osint","ruby","security","twitter-api"],"latest_commit_sha":null,"homepage":"https://michenriksen.com/blog/birdwatcher-twitter-osint-framework/","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/michenriksen.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-10-09T17:22:21.000Z","updated_at":"2025-02-28T13:33:24.000Z","dependencies_parsed_at":"2022-09-14T04:01:05.359Z","dependency_job_id":null,"html_url":"https://github.com/michenriksen/birdwatcher","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michenriksen%2Fbirdwatcher","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michenriksen%2Fbirdwatcher/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michenriksen%2Fbirdwatcher/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michenriksen%2Fbirdwatcher/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/michenriksen","download_url":"https://codeload.github.com/michenriksen/birdwatcher/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247135141,"owners_count":20889420,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["framework","osint","ruby","security","twitter-api"],"created_at":"2024-08-03T06:01:27.034Z","updated_at":"2025-04-04T07:04:57.965Z","avatar_url":"https://github.com/michenriksen.png","language":"Ruby","readme":"# Birdwatcher\n\n[Birdwatcher](https://github.com/michenriksen/birdwatcher) is a data analysis and OSINT framework for Twitter. Birdwatcher supports creating multiple workspaces where arbitrary Twitter users can be added and their Tweets harvested through the Twitter API for offline storage and analysis. Birdwatcher comes with several modules which can be envoked to further enrich collected data or work with it, e.g. Retrieving user's Klout score, generating social graphs between users and weighted word clouds based on their Tweets.\n\n![Birdwatcher commands](img/birdwatcher_help.png)\n\n## Installation\n\n### 1. Ruby\n\nBirdwatcher is written in [Ruby](https://www.ruby-lang.org/) and requires at least version 1.9.3 or above. To check which version of Ruby you have installed, simply run `ruby --version` in a terminal.\n\nShould you have an older version installed, it is very easy to upgrade and manage different versions with the Ruby Version Manager ([RVM](https://rvm.io/)). Please see the [RVM website](https://rvm.io/) for installation instructions.\n\n### 2. RubyGems\n\nBirdwatcher is packaged as a Ruby gem to make it easy to install and update. To install Ruby gems you'll need the RubyGems tool installed. To check if you have it already, type `gem` in a Terminal. If you got it already, it is recommended to do a quick `gem update --system` to make sure you have the latest and greatest version. In case you don't have it installed, download it from [here](https://rubygems.org/pages/download) and follow the simple installation instructions.\n\n### 3. PostgreSQL\n\nBirdwatcher uses a PostgreSQL database to store all its data. If you are setting up Birdwatcher in the [Kali](https://www.kali.org/) linux distribution you already have it installed, you just need to make sure it's running by executing `service postgresql start` and perhaps install a dependency with `apt-get install libpq-dev` in a terminal. Here's an excellent [guide](https://www.digitalocean.com/community/tutorials/how-to-install-and-use-postgresql-9-4-on-debian-8) on how to install PostgreSQL on a Debian based Linux system. If you are setting up Birdwatcher on a Mac, the easiest way to install PostgreSQL is with [Homebrew](http://brew.sh/). Here's a [guide](http://exponential.io/blog/2015/02/21/install-postgresql-on-mac-os-x-via-brew/) on how to install PostgreSQL with Homebrew.\n\n#### 3.1 PostgreSQL user and database\n\nYou need to set up a user and a database in PostgreSQL for Birdwatcher. Execute the following commands in a terminal:\n\n    sudo su postgres # Not necessary on Mac OS X\n    createuser -s birdwatcher --pwprompt\n    createdb -O birdwatcher birdwatcher\n\nYou now have a new PostgreSQL user with the name `birdwatcher` and with the password you typed into the prompt. You also created a database with the name `birdwatcher` which is owned by the `birdwatcher` user.\n\n### 4. Graphviz\n\nSome Birdwatcher modules use [Graphviz](http://graphviz.org/) to generate visual graphs and other things. On a Mac you can install Graphviz with [homebrew](http://brew.sh/) by typing `brew update \u0026\u0026 brew install graphviz` in a terminal. On a Debian based Linux distro, Graphviz can be installed by typing `sudo apt-get update \u0026\u0026 sudo apt-get install graphviz` in a terminal.\n\n### 5. ImageMagick\n\nSome Birdwatcher modules use [ImageMagick](https://imagemagick.org/script/index.php) to generate images. On a Mac you can install Imagemagick with [homebrew](http://brew.sh/) by typing `brew update \u0026\u0026 brew install imagemagick` in a terminal. On a Debian based Linux distro, ImageMagick can be installed by typing `sudo apt-get update \u0026\u0026 sudo apt-get install libmagickwand-dev imagemagick` in a terminal.\n\n### 6. Birdwatcher\n\nFinally with all the dependencies in place, Birdwatcher can now be installed with a simple command in a terminal:\n\n    $ gem install birdwatcher\n\nThis will download and set up Birdwatcher and install all its code depencies.\n\n## Configuration\n\nBirdwatcher needs to know a bit about what database to connect to as well as API keys to use for API communication. For Twitter, you will need to register an application in order to get a consumer key and consumer secret. Head over to [apps.twitter.com](https://apps.twitter.com/) and set up your application. You don't need to have a valid callback URL as we won't be doing any OAuth authentication.\n\nTo make Birdwatcher even more useful, it is recommended to also obtain an API key for the [Klout API](https://klout.com/s/developers/v2) which is required for some modules to work. If you don't intend to use Klout modules, you can of course skip this step.\n\nOn the first run, Birdwatcher will automatically start a configuration wizard where it will ask for the configuration it needs. If you have PostgreSQL connection details, Twitter consumer key \u0026 secret and optional Klout API key, you can start Birdwatcher for the first time:\n\n    $ birdwatcher\n\nEnter the details into the configuration wizard:\n\n![Birdwatcher configuration wizard](img/birdwatcher_configuration.png)\n\nAs can be seen from the above screenshot, Birdwatcher supports multiple Twitter and Klout keys. If you configure Birdwatcher with several keys it will randomly shuffle between them when communicating with APIs to potentially avoid any rate limiting issues on extensive use.\n\nBirdwatcher will save its configuration to `~/.birdwatcherrc`. Be careful not to push this file up to any public code repositories!\n\n### System pager\n\nBirdwatcher pages long command output with the operating system's default pager command (usually `less`) however the command output can be colored and will show up strangely if the pager is not configured to render terminal colors. It is advised to add the following to your `~/.bash_profile` or similar file:\n\n    # Get color support for 'less'\n    export LESS=\"--raw-control-chars\"\n    export PAGER=\"less\"\n\nThis will set up `less` as your default pager command as well as configure `less` to support terminal colors. Execute the command `source ~/.bash_profile` to apply the configuration immediately instead of next time you open a terminal.\n\n## Usage\n\nBirdwatcher is built as a console and if you have any experience with other frameworks such as [Metasploit](https://www.metasploit.com/) or [Recon-ng](https://bitbucket.org/LaNMaSteR53/recon-ng), you should feel right at home as Birdwatcher has many of the same concepts and commands.\n\n### Workspaces\n\nBirdwatcher, like Metasploit and Recon-ng, works with the concept of Workspaces. Workspaces enable you to segment and manage users and data stored in the database. You can use workspaces to create logical separation between different users. For example, you may want to create a workspace for a company, a department or for a specific topic.\n\nBirdwatcher will always show the currently active workspace inside the square brackets on the command prompt:\n\n![Birdwatcher prompt](img/birdwatcher_prompt.png)\n\nThere will always be a default workspace with the name `default` which might be enough if you plan to use Birdwatcher for a small group of Twitter users. Let's create a new workspace called `top5` that we can use for experimentation:\n\n    birdwatcher[default]\u003e  workspace create top5\n    [+] Created workspace: top5\n    birdwatcher[top5]\u003e\n\nThe `workspace create` command created a new workspace and automatically made it the active workspace. Any user we add and any data we collect now will only be available in the `top5` workspace.\n\nIf we want to navigate between workspaces, we can do so by using the `workspace` command again:\n\n    birdwatcher[top5]\u003e workspace use default\n    [+] Now using workspace: default\n    birdwatcher[default]\u003e workspace use top5\n    [+] Now using workspace: top5\n    birdwatcher[top5]\u003e\n\nWith the above commands we switched over to the `default` workspace and then back again to the `top5` workspace.\n\n### Commands\n\nThe core of the Birdwatcher framework is of course its commands. You already got an introduction to the `workspace` command in the previous section, but there are many more. One of the most important commands is `help`:\n\n    birdwatcher[top5]\u003e help\n    [+] Available commands:\n\n        back               \t\tUnloads current module\n        exit               \t\tExit Birdwatcher\n        help [COMMAND]     \t\tShow help and detailed command usage\n        irb                \t\tStart an interactive Ruby shell\n        module ACTION      \t\tShow modules\n        query QUERY        \t\tExecute SQL query\n        query_csv QUERY    \t\tExecute SQL query and return result as CSV\n        resource FILE      \t\tExecute commands from a resource file\n        run                \t\tRun current module\n        schema [TABLE_NAME]\t\tShow schema for database table\n        set OPTION VALUE   \t\tSet module option\n        shell COMMAND      \t\tExecute shell command\n        show DETAILS       \t\tShow module details and options\n        status [ACTION]    \t\tManage statuses\n        unset OPTION       \t\tUnset module option\n        use MODULE_PATH    \t\tLoad specified module\n        user [ACTION]      \t\tManage users\n        workspace [ACTION] \t\tManage workspaces\n\n    birdwatcher[top5]\u003e\n\nThe help command simply lists the available commands with a short description of what they do. If we want to get more information on a specific command, e.g. the `workspace` command, we can execute the following:\n\n    birdwatcher[top5]\u003e help workspace\n\n    Workspaces enable you to segment and manage users and data stored in the database.\n    You can use workspaces to create logical separation between different users.\n    For example, you may want to create a workspace for a company, a department or\n    for a specific topic.\n\n    There will always be a default workspace with the name default which might be enough\n    if you plan to use Birdwatcher for a small group of Twitter users.\n\n    USAGE:\n\n    List available workspaces:\n      workspace list\n\n    Create a new workspace:\n      workspace create NAME [DESCRIPTION]\n\n    Switch to a workspace:\n      workspace use NAME\n\n    Delete a workspace:\n      workspace delete NAME\n\n    Rename a workspace\n      workspace rename NAME NEW_NAME\n\n    birdwatcher[top5]\u003e\n\nAnother core command in Birdwatcher is the `user` command which can be used to add users to the workspace as well as updating or removing them at a later point.\n\nAs an example, let's add the top 5 most followed Twitter users from [this list](http://twittercounter.com/pages/100). At the time of writing it is:\n\n * Katy Perry ([@katyperry](https://twitter.com/katyperry))\n * Justin Bieber ([@justinbieber](https://twitter.com/justinbieber))\n * Taylor Swift ([@taylorswift13](https://twitter.com/taylorswift13))\n * Barack Obama ([@BarackObama](https://twitter.com/BarackObama))\n * Rihanna ([@rihanna](https://twitter.com/rihanna))\n\nExecute the following command to add them to the workspace:\n\n     birdwatcher[top5]\u003e user add katyperry justinbieber taylorswift13 BarackObama rihanna\n     [+] Added katyperry to workspace\n     [+] Added justinbieber to workspace\n     [+] Added taylorswift13 to workspace\n     [+] Added BarackObama to workspace\n     [+] Added rihanna to workspace\n     birdwatcher[top5]\u003e\n\nBirdwatcher fetched basic information on the users through the Twitter API. A summary of users in the workspace can be seen with the `user list` command. For more information on what the `user` command can do, simply enter `help user`.\n\nNow that we have a couple of users in the workspace we can look at the `status` command which is another core command of the Birdwatcher framework:\n\n    birdwatcher[top5]\u003e status fetch\n    [+] Fetching statuses for BarackObama... done\n    [+] Processing 1000 statuses... done\n    [+] Fetching statuses for justinbieber... done\n    [+] Processing 997 statuses... done\n    [+] Fetching statuses for katyperry... done\n    [+] Processing 998 statuses... done\n    [+] Fetching statuses for rihanna... done\n    [+] Processing 999 statuses... done\n    [+] Fetching statuses for taylorswift13... done\n    [+] Processing 996 statuses... done\n    birdwatcher[top5]\u003e\n\n The `status fetch` command fetches up to 1.000 statuses from the users in the workspace and saves them to the underlying database. The statuses are also processed to extract URLs, mentions and hashtags to separate database tables. Running `status fetch` at a later time will fetch any new statuses that the users might have posted since last fetch.\n\nWe can page through the last 1.000 statuses across all the users with the `status list` command:\n\n    . . . .\n\n    Justin Bieber (@justinbieber) * Oct  5, 23:50\n    Proud of my buddies https://t.co/4vc0qfmSyA\n    Favorites: 51505 | Retweets: 22434\n\n    ================================================================================\n\n    KATY PERRY (@katyperry) * Oct  5, 23:36\n    You hear it as \"excellent\" I hear it as \"egg salad with lint\" 🤔\n    Favorites: 14204 | Retweets: 4340\n\n    ================================================================================\n\n    Justin Bieber (@justinbieber) * Oct  5, 23:29\n    Great show https://t.co/jgtRbU4RHC\n    Favorites: 50799 | Retweets: 22617\n\n    ================================================================================\n\n    Barack Obama (@BarackObama) * Oct  5, 21:30\n    This historic step in the fight to #ActOnClimate came faster than anyone predicted. https://t.co/W2rtcNXkI7\n    Favorites: 5053 | Retweets: 1254\n\n    ================================================================================\n\n    Barack Obama (@BarackObama) * Oct  5, 20:44\n    \"Today is a historic day in the fight to protect our planet for future generations.\" —President Obama #ActOnClimate https://t.co/x3dJSCYUcj\n    Favorites: 5500 | Retweets: 1652\n\n    . . . .\n\nIt is also possible to page through statuses which contain a certain word or phrase with the `status search \u003cword\u003e` command. See `help status` for more usage on the `status` command.\n\n### Modules\n\nBeing able to fetch Twitter users and statuses from the API is cool and all, but if that was all Birdwatcher would just be an offline Twitter client. Modules are where the fun begins. Modules either enrich collected data with more data (e.g. Klout score) or do some sort of work on the collected data.\n\nHere are some of the things the modules can do:\n\n  * Generate weighted word clouds based on statuses\n  * Listing the most shared URLs in a certain time frame\n  * Generate visual social graphs between users\n  * Crawl shared URLs to fetch status codes, content types and page titles\n  * Generate [KML](https://developers.google.com/kml/) files with geo-enabled statuses for viewing in [Google Earth](https://www.google.com/earth/)\n  * Generate Punchard-style plots of when users are most engaged with Twitter\n\nTo see all available modules, use the `module list` command:\n\n    birdwatcher[top5]\u003e module list\n    [+] Available Modules:\n\n           Name: KML Document\n    Description: Creates a KML document of statuses with Geo locations\n           Path: statuses/kml\n\n    ================================================================================\n\n           Name: Status Sentiment Analysis\n    Description: Enrich statuses with sentiment score\n           Path: statuses/sentiment\n\n    ================================================================================\n\n    . . . .\n\n    ================================================================================\n\n           Name: User Klout Topics\n    Description: Enrich users with their Klout topics\n           Path: users/klout_topics\n\n    ================================================================================\n\n           Name: Social Graph\n    Description: Graphs the social relations between users\n           Path: users/social_graph\n\n    ================================================================================\n\n    birdwatcher[top5]\u003e\n\nThe name, description and _path_ is listed for each available module. The module path is how the modules are divided up into folders on the file system. The folders can be seen as _namespaces_ and gives a clue about what sort of objects they work on.\n\nThe path is used to select modules with the `use` command:\n\n    birdwatcher[top5]\u003e use statuses/word_cloud\n    birdwatcher[top5][statuses/word_cloud]\u003e\n\nThe `use` command simply loads a module by providing the modules path as an argument. This also changes the command prompt to display the currently loaded module in square brackets next to the currently active workspace.\n\nNow that we are inside the `statuses/word_cloud` module we may want to get a bit more information about it:\n\n    birdwatcher[top5][statuses/word_cloud]\u003e show info\n\n           Name: Word Cloud\n    Description: Generates a word cloud from statuses\n         Author: Michael Henriksen \u003cmichenriksen@neomailbox.ch\u003e\n           Path: statuses/word_cloud\n\n    ================================================================================\n\n    The Word Cloud module can generate a classic weighted word cloud from words used\n    in statuses across all or specific users and between different times.\n\n    The module is heavily configurable; have a look at the options with show options\n\n    Please note that configuring the module with a long timespan might result in a\n    very long execution time when the word cloud image is generated.\n\n    The generated image will be in PNG format.\n\n    birdwatcher[top5][statuses/word_cloud]\u003e\n\nThe `show info` command shows additional information on the module if available. It can also be used to see any configuration options the module might have:\n\n    birdwatcher[top5][statuses/word_cloud]\u003e show options\n\n    ------------------------------------------------------------------------------------------------------------------------------------------\n     Name                 Current Setting              Required  Description\n    ------------------------------------------------------------------------------------------------------------------------------------------\n     DEST                                              yes       Destination file\n     USERS                                             no        Space-separated list of screen names (all users if empty)\n     SINCE                                             no        Process statuses posted since specified time (last 7 days if empty)\n     BEFORE                                            no        Process statuses posted before specified time (from now if empty)\n     MIN_WORD_COUNT       3                            no        Exclude words mentioned fewer times than specified\n     MIN_WORD_LENGTH      3                            no        Exclude words smaller than specified\n     EXCLUDE_STOPWORDS    true                         no        Exclude english stopwords\n     EXCLUDE_COMMON       true                         no        Exclude common english words\n     EXCLUDE_WORDS                                     no        Space-separated list of words to exclude\n     EXCLUDE_HASHTAGS     false                        no        Exclude Hashtags\n     EXCLUDE_MENTIONS     true                         no        Exclude @username mentions\n     INCLUDE_PAGE_TITLES  false                        no        Include web page titles from shared URLs (requires crawling with urls/crawl)\n     WORD_CAP             200                          no        Cap list of words to specified amount\n     PALETTE              #8F99AB #A3ADC2 #272A2F ...  yes       Space-separated list of hex color codes to use for word cloud\n     IMAGE_WIDTH          1024                         yes       Image width in pixels\n     IMAGE_HEIGHT         1024                         yes       Image height in pixels\n    ------------------------------------------------------------------------------------------------------------------------------------------\n\n    birdwatcher[top5][statuses/word_cloud]\u003e\n\nThe `show options` command shows all of the configuration options for the loaded module. The `statuses/word_cloud` module happens to be one of the more configurable modules, but looking at the table we can see that only required option we need to set is the `DEST` option which tells the module where to write the final word cloud image. Because the workspace doesn't contain a whole lot of users we will also set the `SINCE` option to `6 months ago` in order to make a word cloud of what the Top 5 Twitter users have been talking about through the last 6 months. On a side note, all module options that have something to do with dates and times are processed with [Chronic](https://github.com/mojombo/chronic) gem for natural language processing; see the [examples](https://github.com/mojombo/chronic#examples) for what is supported.\n\n    birdwatcher[top5][statuses/word_cloud]\u003e set DEST /tmp/wordcloud.png\n    birdwatcher[top5][statuses/word_cloud]\u003e set SINCE 6 months ago\n    birdwatcher[top5][statuses/word_cloud]\u003e run\n    [+] Processing 1542 statuses... done\n    [+] Generating word cloud, patience please... done\n    [+] Word cloud written to /tmp/wordcloud.png\n    birdwatcher[top5][statuses/word_cloud]\u003e\n\nAfter a couple of seconds the module wrote the word cloud image to `/tmo/wordcloud.png`. The result is:\n\n![Word cloud of Top-5 Twitter users](img/wordcloud.png)\n\n### Power User Features\n\nSince everything is stored in a database, it is possible to perform arbitrary SQL queries against the data if you know the SQL language:\n\n#### Raw SQL\n\n    birdwatcher[top5]\u003e query select name, screen_name, followers_count from users where workspace_id = 2 order by followers_count DESC\n\n    +---------------+---------------+-----------------+\n    | name          | screen_name   | followers_count |\n    +---------------+---------------+-----------------+\n    | KATY PERRY    | katyperry     | 93379238        |\n    | Justin Bieber | justinbieber  | 88715066        |\n    | Taylor Swift  | taylorswift13 | 81158756        |\n    | Barack Obama  | BarackObama   | 77850900        |\n    | Rihanna       | rihanna       | 66491106        |\n    +---------------+---------------+-----------------+\n\nThe `query` command will execute any SQL query it receives as an argument and output the result in a table. As can be seen in the command example, the `query` command does not know how to scope the data to the currently active workspace, so you will have to take care of that in your queries. Usually rows can be scoped by filtering on `workspace_id = ?` in tables. The current workspace ID can be retrieved by issuing the `workspace` command.\n\n#### Outputting in CSV Format\n\nIf you want to extract data with raw SQL and want it to be easily parsable by other applications or code, you can use the `query_csv` command. It works similarly to `query` but outputs the result in CSV format:\n\n    birdwatcher[top5]\u003e query_csv select name, screen_name, followers_count from users where workspace_id = 2 order by followers_count DESC\n    name,screen_name,followers_count\n    KATY PERRY,katyperry,93379238\n    Justin Bieber,justinbieber,88715066\n    Taylor Swift,taylorswift13,81158756\n    Barack Obama,BarackObama,77850900\n    Rihanna,rihanna,66491106\n\n#### Getting Schema Information\n\nPerforming raw SQL queries against Birdwatcher's database can be hard if you don't know how the tables are layed out. To get a list of available tables to inspect, you can use the `schema` command:\n\n    birdwatcher[default]\u003e schema\n    [+] Available tables:\n\n     * hashtags\n     * hashtags_statuses\n     * influencees\n     * influencees_users\n     * influencers\n     * influencers_users\n     * klout_topics\n     * klout_topics_users\n     * mentions\n     * mentions_statuses\n     * schema_info\n     * statuses\n     * statuses_urls\n     * urls\n     * users\n     * workspaces\n\n    birdwatcher[default]\u003e\n\nGiving the `schema` command a table name as an argument will show detailed schema information, indexes and foreign keys for that table:\n\n    birdwatcher[default]\u003e schema urls\n    [+] Schema information for table urls:\n\n    +--------------+-----------------------------+----------------------------------+------------+-------------+\n    | Column Name  | Type                        | Default                          | Allow NULL | Primary Key |\n    +--------------+-----------------------------+----------------------------------+------------+-------------+\n    | id           | integer                     | nextval('urls_id_seq'::regclass) | No         | Yes         |\n    | workspace_id | integer                     | NULL                             | Yes        | No          |\n    | url          | text                        | NULL                             | Yes        | No          |\n    | final_url    | text                        | NULL                             | Yes        | No          |\n    | http_status  | integer                     | NULL                             | Yes        | No          |\n    | content_type | text                        | NULL                             | Yes        | No          |\n    | title        | text                        | NULL                             | Yes        | No          |\n    | crawled_at   | timestamp without time zone | NULL                             | Yes        | No          |\n    | updated_at   | timestamp without time zone | NULL                             | Yes        | No          |\n    | created_at   | timestamp without time zone | NULL                             | Yes        | No          |\n    +--------------+-----------------------------+----------------------------------+------------+-------------+\n\n    [+] Indexes on table urls:\n\n    +-------------------------+--------------+--------+\n    | Index Name              | Column(s)    | Unique |\n    +-------------------------+--------------+--------+\n    | urls_content_type_index | content_type | No     |\n    | urls_crawled_at_index   | crawled_at   | No     |\n    | urls_created_at_index   | created_at   | No     |\n    | urls_final_url_index    | final_url    | No     |\n    | urls_http_status_index  | http_status  | No     |\n    | urls_title_index        | title        | No     |\n    | urls_updated_at_index   | updated_at   | No     |\n    | urls_url_index          | url          | No     |\n    | urls_workspace_id_index | workspace_id | No     |\n    +-------------------------+--------------+--------+\n\n    [+] Foreign keys on table urls:\n\n    +--------------+------------------+----------------------+\n    | Column(s)    | Referenced Table | Referenced Column(s) |\n    +--------------+------------------+----------------------+\n    | workspace_id | workspaces       | id                   |\n    +--------------+------------------+----------------------+\n\n    birdwatcher[default]\u003e\n\nIf you want a graphical ER diagram, you can [find one here](img/birdwatcher_schema.png)\n\n#### Interactive Ruby Shell\n\nIf you really know what you are doing and know the ins and outs of Birdwatcher's code, you can use the `irb` command to get an interactive Ruby shell ([IRB](http://ruby-doc.org/stdlib-2.0.0/libdoc/irb/rdoc/IRB.html)) where all of Birdwatcher's classes and models are available. You can use this command for debugging or for more complex, one-off data manipulation where building a module isn't suitable.\n\n## Development\n\n### Modules\n\nFor information on how to create a Birdwatcher module, please see this [Wiki page](https://github.com/michenriksen/birdwatcher/wiki/Creating-a-Birdwatcher-Module).\n\n## License\n\nBirdwatcher is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).\n\n","funding_links":[],"categories":["Tools by Category","Ruby","Ruby (88)"],"sub_categories":["📱 Social Media Intelligence"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichenriksen%2Fbirdwatcher","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmichenriksen%2Fbirdwatcher","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichenriksen%2Fbirdwatcher/lists"}