{"id":16986904,"url":"https://github.com/amamenko/nypd-arrest-map","last_synced_at":"2025-04-12T03:41:19.245Z","repository":{"id":49777695,"uuid":"300127070","full_name":"amamenko/nypd-arrest-map","owner":"amamenko","description":"Data visualization application for year-to-date NYPD arrests ","archived":false,"fork":false,"pushed_at":"2023-02-20T16:55:53.000Z","size":25737,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-25T23:14:18.759Z","etag":null,"topics":["age","arrest","borough","crime","data","deck-gl","graph","new","new-york","nyc","nypd","police","race","trends","visualization","websockets","york"],"latest_commit_sha":null,"homepage":"https://nypd-arrest-map.onrender.com","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amamenko.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-10-01T03:20:25.000Z","updated_at":"2023-02-20T17:27:23.000Z","dependencies_parsed_at":"2022-09-11T17:54:10.762Z","dependency_job_id":null,"html_url":"https://github.com/amamenko/nypd-arrest-map","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amamenko%2Fnypd-arrest-map","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amamenko%2Fnypd-arrest-map/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amamenko%2Fnypd-arrest-map/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amamenko%2Fnypd-arrest-map/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amamenko","download_url":"https://codeload.github.com/amamenko/nypd-arrest-map/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248514217,"owners_count":21116899,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["age","arrest","borough","crime","data","deck-gl","graph","new","new-york","nyc","nypd","police","race","trends","visualization","websockets","york"],"created_at":"2024-10-14T02:47:17.709Z","updated_at":"2025-04-12T03:41:19.224Z","avatar_url":"https://github.com/amamenko.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![MIT License][license-shield]][license-url]\n[![LinkedIn][linkedin-shield]][linkedin-url]\n\n\u003c!-- PROJECT LOGO --\u003e\n\u003cbr /\u003e\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/amamenko/nypd-arrest-map\"\u003e\n    \u003cimg src=\"Client/public/android-chrome-192x192.png\" alt=\"Logo\" width=\"80\" height=\"80\" /\u003e\n  \u003c/a\u003e\n\n  \u003ch3 align=\"center\"\u003eNYPD Arrest Map\u003c/h3\u003e\n\n  \u003cp align=\"center\"\u003e\n    Data visualization and analysis tool for year-to-date NYPD arrests \n    \u003cbr /\u003e\n    \u003cbr /\u003e\n    \u003ca href=\"https://nypd-arrest-map.onrender.com\"\u003eView Demo\u003c/a\u003e\n    ·\n    \u003ca href=\"https://www.youtube.com/watch?v=eO54xtfrfPk\"\u003eWatch Video Demo\u003c/a\u003e\n    ·\n    \u003ca href=\"https://github.com/amamenko/nypd-arrest-map/issues\"\u003eReport Issue\u003c/a\u003e \n  \u003c/p\u003e\n\u003c/p\u003e\n\n## Introduction\n\nThe New York Police Department (NYPD) publishes data that reflect every arrest that takes place in New York City (NYC) during the current year. This dataset is manually extracted\nand updated on a quarterly basis by the Office of Management Analysis and Planning on [NYC Open Data's NYPD Arrest Data (Year to Date)](https://data.cityofnewyork.us/Public-Safety/NYPD-Arrest-Data-Year-to-Date-/uip8-fykc) website.\nOn this website, the NYPD states that \"this data can be used by the public to explore the nature of police enforcement activity.\"\n\nEvery arrest that is published includes an approximate location and date, as well as demographical information about the arrestee. Other details about the nature of the arrest,\nsuch as whether a felony, misdemeanor, or violation has taken place, are also included.\n\nPlease note that occasionally, arrests in the NYPD datasets will be erroneously attributed to one borough, while having the longitude and latitude of another. These have been corrected in the NYPD Arrest Map in\nfavor of the arrest's geocoordinates.\n\n## About the Map\n\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://nypd-arrest-map.onrender.com\"\u003e\n    \u003cimg  src=\"Client/images/ResponsiveDesignDemo.png\" alt=\"NYPD Arrest Map Responsiveness Demo Screenshots\" /\u003e\n\u003c/a\u003e\n\u003c/span\u003e\n\u003cbr /\u003e\n\u003cbr /\u003e\n\nThe NYPD Arrest Map is an application built with the React library (using Redux for state management) as well as the Express framework for Node.js. The majority of the functionality of the\nwebsite can be broken down as follows:\n\n\u003cstrong\u003eServer-side\u003c/strong\u003e\n\n- Scrapes the [NYC Open Data's NYPD Arrest Data (Year to Date)](https://data.cityofnewyork.us/Public-Safety/NYPD-Arrest-Data-Year-to-Date-/uip8-fykc) page\n  daily using [Puppeteer](https://github.com/puppeteer/puppeteer) and [node-cron](https://www.npmjs.com/package/node-cron).\n- Compares the last updated date/data numbers to those stored in [Contentful](https://www.contentful.com).\n- Parses and formats the dataset if any updates are detected.\n- Modifies the Contentful data stores (if updates detected) via Contentful's [Content Management API](https://www.contentful.com/developers/docs/references/content-management-api/).\n- Gzip-compresses and uploads the JSON dataset to a [Google Cloud Storage](https://cloud.google.com/storage) (GCS) bucket.\n- Upon request, uses [Oboe.js](http://oboejs.com) to stream the latest JSON data from GCS to the client in chunks by means of the [WebSocket API](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API).\n\n\u003cstrong\u003eClient-side\u003c/strong\u003e\n\n- Compiles received dataset chunks and performs analyses/formatting in background threads using [Web Workers](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API).\n- Serves map assets and geomarkers using [deck.gl](https://deck.gl) and [Mapbox](https://www.mapbox.com).\n- Mounts [carousel](https://www.npmjs.com/package/react-alice-carousel) (only upon request, due to memory allocation issues) showing [Google Charts](https://www.npmjs.com/package/react-google-charts) of various natures (pie charts, bar charts, line charts) based on current filtered data.\n- When prompted, filters currently shown arrest geomarkers and graph data in background thread.\n\n## Deployment\n\nServer-side and client-side initially deployed to [Heroku](https://www.heroku.com/), now deployed on [Render.com](https://render.com).\n\n## Local Development\n\nTo set up this project locally, you can follow the steps below.\n\n### Prerequisites\n\nYou will need to have the following software installed:\n\n- npm\n- Git\n- Node.js\n\n### Installation\n\n1. Get a free Mapbox API token at [https://www.mapbox.com](https://www.mapbox.com).\n2. Create a new [Google Cloud Storage](https://cloud.google.com/storage) project in the Cloud Console.\n3. Create a new service account key JSON file.\n4. Clone the Github repository.\n   ```sh\n   git clone https://github.com/amamenko/nypd-arrest-map.git\n   ```\n5. Install all server-side NPM packages.\n   ```sh\n   npm install\n   ```\n6. Install all client-side NPM packages.\n   ```sh\n   cd Client\n   npm install\n   ```\n7. Webpack will throw a `Conflict: Multiple assets emit to the same filename` warning due to web workers creating multiple background threads and not emitting to dynamic\n   filenames. Edit the `webpack.config.js` file, specifically the `output` key - concerning the build folder - of the returned object with the following values:\n\n```JS\n    filename: isEnvProduction\n          ? 'static/js/[name].[contenthash:8].js'\n          : isEnvDevelopment \u0026\u0026 '[name].bundle.js'\n```\n\n```JS\n    chunkFilename: isEnvProduction\n          ? 'static/js/[name].[contenthash:8].chunk.js'\n          : isEnvDevelopment \u0026\u0026 'static/js/[name].chunk.js'\n```\n\n8. Enter your Mapbox API token as a client-side environment variable.\n   ```JS\n   REACT_APP_MAPBOX_TOKEN=YOUR MAPBOX TOKEN\n   REACT_APP_CONTENTFUL_SPACE_ID=YOUR CONTENTFUL SPACE ID\n   REACT_APP_CONTENTFUL_ACCESS_TOKEN=YOUR CONTENTFUL ACCESS TOKEN\n   ```\n9. Enter your Google Cloud Storage JSON and Contentful information as server-side environment variables.\n   ```JS\n   PROJECT_ID=YOUR PROJECT ID\n   PRIVATE_KEY_ID=YOUR PRIVATE KEY ID\n   PRIVATE_KEY=YOUR PRIVATE KEY\n   CLIENT_EMAIL=YOUR CLIENT EMAIL\n   CLIENT_ID=YOUR CLIENT ID\n   AUTH_URI=YOUR AUTH URI\n   TOKEN_URI=YOUR TOKEN URI\n   PROVIDER_CERT_URL=YOUR PROVIDER CERT URL\n   CLIENT_CERT_URL=YOUR CLIENT CERT URL\n   CONTENTFUL_SPACE_ID=YOUR CONTENTFUL SPACE ID\n   CONTENTFUL_ENTRY_ID=YOUR CONTENTFUL ENTRY ID\n   CONTENTFUL_ACCESS_TOKEN=YOUR CONTENTFUL ACCESS TOKEN\n   CONTENTFUL_MANAGEMENT_TOKEN=YOUR CONTENTFUL MANAGEMENT TOKEN\n   ```\n10. Build for production.\n    ```JS\n    npm run build\n    ```\n\n## Known Issue\n\niPhone 11 phones running iOS 14.1 have been found to experience \"zoom breathing\" when geomarkers are selected. The scatterplot layer will scale and zoom in for a moment before resetting itself.\n[I have opened an issue with deck.gl regarding this](https://github.com/visgl/deck.gl/issues/5140) and, although the bug is still open, the issue is otherwise fixed in iOS 14.2.\n\n\u003c!-- CONTRIBUTING --\u003e\n\n## Contributing\n\nContributions are welcome!\n\n1. Fork the Project\n2. Create your Feature Branch (`git checkout -b feature/MyFeature`)\n3. Commit your Changes (`git commit -m 'Add my feature'`)\n4. Push to the Branch (`git push origin feature/MyFeature`)\n5. Open a Pull Request\n\n\u003c!-- LICENSE --\u003e\n\n## License\n\nDistributed under the MIT License. See `LICENSE.txt` for more information.\n\n\u003c!-- CONTACT --\u003e\n\n## Contact\n\nAvraham (Avi) Mamenko - avimamenko@gmail.com\n\nProject Link: [https://github.com/amamenko/nypd-arrest-map](https://github.com/amamenko/nypd-arrest-map)\n\n\u003c!-- ACKNOWLEDGEMENTS --\u003e\n\n## Acknowledgements\n\n- [NYPD / NYC Open Data](https://opendata.cityofnewyork.us)\n- [deck.gl](https://deck.gl)\n- [Mapbox](https://www.mapbox.com)\n- [Render.com](https://render.com)\n- [React Icons](https://react-icons.github.io/react-icons)\n- [React Alice Carousel](https://www.npmjs.com/package/react-alice-carousel)\n- [React Google Charts](https://react-google-charts.com)\n- [Tippy.js for React](https://www.npmjs.com/package/@tippyjs/react)\n- [React Radio Buttons](https://www.npmjs.com/package/react-radio-buttons)\n- [React Ladda](https://www.npmjs.com/package/react-ladda)\n- [React Burger Menu](https://github.com/negomi/react-burger-menu)\n- [ReactJS Popup](https://www.npmjs.com/package/reactjs-popup)\n- [React Circular Progressbar](https://www.npmjs.com/package/react-circular-progressbar)\n- [React Spinners](https://www.npmjs.com/package/react-spinners)\n- [react-div-100vh](https://www.npmjs.com/package/react-div-100vh)\n- [React Modal](https://www.npmjs.com/package/react-modal)\n- [worker-loader](https://www.npmjs.com/package/worker-loader)\n- [Oboe.js](http://oboejs.com)\n- [Puppeteer](https://www.npmjs.com/package/puppeteer)\n- [node-cron](https://www.npmjs.com/package/node-cron)\n- [csvtojson](https://www.npmjs.com/package/csvtojson)\n- [Google Cloud Storage](https://cloud.google.com/storage)\n- [Day.js](https://github.com/iamkun/dayjs)\n- [express-sslify](https://www.npmjs.com/package/express-sslify)\n- [Best-README-Template](https://github.com/othneildrew/Best-README-Template)\n\n\u003c!-- MARKDOWN LINKS \u0026 IMAGES --\u003e\n\u003c!-- https://www.markdownguide.org/basic-syntax/#reference-style-links --\u003e\n\n[license-shield]: https://img.shields.io/github/license/othneildrew/Best-README-Template.svg?style=for-the-badge\n[license-url]: https://github.com/amamenko/nypd-arrest-map/blob/master/LICENSE.txt\n[linkedin-shield]: https://img.shields.io/badge/-LinkedIn-black.svg?style=for-the-badge\u0026logo=linkedin\u0026colorB=555\n[linkedin-url]: https://www.linkedin.com/in/avrahammamenko\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famamenko%2Fnypd-arrest-map","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famamenko%2Fnypd-arrest-map","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famamenko%2Fnypd-arrest-map/lists"}