{"id":21130923,"url":"https://github.com/paulpierre/pp_canary","last_synced_at":"2026-04-07T07:47:00.089Z","repository":{"id":122996453,"uuid":"156763170","full_name":"paulpierre/pp_canary","owner":"paulpierre","description":"Canary is a platform that helps anticipate fraud and fulfillment delays in Shopify","archived":false,"fork":false,"pushed_at":"2018-11-08T20:52:34.000Z","size":16403,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-10-07T07:54:58.610Z","etag":null,"topics":["anti-fraud","app","canary","delay","ecommerce","fraud","fulfillment","merchant","php","platform","shopify"],"latest_commit_sha":null,"homepage":"http://paulpierre.com/#canary","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/paulpierre.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-11-08T20:13:49.000Z","updated_at":"2025-03-21T04:38:42.000Z","dependencies_parsed_at":null,"dependency_job_id":"ffecfba4-a189-4182-adc7-4df59ae2d044","html_url":"https://github.com/paulpierre/pp_canary","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/paulpierre/pp_canary","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulpierre%2Fpp_canary","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulpierre%2Fpp_canary/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulpierre%2Fpp_canary/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulpierre%2Fpp_canary/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/paulpierre","download_url":"https://codeload.github.com/paulpierre/pp_canary/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulpierre%2Fpp_canary/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31504897,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-07T03:10:19.677Z","status":"ssl_error","status_checked_at":"2026-04-07T03:10:13.982Z","response_time":105,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anti-fraud","app","canary","delay","ecommerce","fraud","fulfillment","merchant","php","platform","shopify"],"created_at":"2024-11-20T05:41:35.131Z","updated_at":"2026-04-07T07:47:00.068Z","avatar_url":"https://github.com/paulpierre.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Canary\n- - -\n\n### Canary is a platform that helps anticipate fraud and fulfillment delays in Shopify\n\n\n![Canary](http://paulpierre.com/img/canary.jpg)\n\n\nbuilt by [http://paulpierre.com](http://paulpierre.com) in 2017-2018\n\nCanary is a tool I built for my e-commerce team to help address fulfillment problems. Shipments from China would be held up in customs, or customers would never receive an item and eventually after a volume of data we were able to develop heuristics around this and build a tool that was not available to the public at the time. \n\n![Canary](http://paulpierre.com/img/canary_1.jpg) \n\n![Built with](https://i.imgur.com/IRsBvDx.png) \n\n\nI built the platform end to end using the basic Linux LAMP stack in conjunction with a tracking API from Chinese tracking company 17track. Every few hours I would poll our Shopify API and 17 Track to check the fulfillment status and based on the codified heuristics we were able to determine the likelihood status of orders so our fulfillment team can react in time and salvage a sale/fulfillment process from an upset customer and ultimately save our company hundreds and thousands of dollars in the process. \n\nLive app and servers no longer maintained, but demo available upon request.\n\n---\n\n### Documentation\n\n*****\nIndex\n*****\n1. Purpose of application\n2. Customer Lifecycle\n3. App Lifecycle\n4. Directory Structure\n5. Setup\n6. Apache Setup\n\n\n### Purpose of application\n---\n\n    This application is supposed to track customer orders. It grabs information from the Shopify API:\n\n    https://help.shopify.com/api/reference\n\n    and stores the relevant objects like Order and Fulfillment into our database. It also interfaces\n    with Shopify's webhooks:\n\n    https://help.shopify.com/api/reference/webhook\n\n    To grab orders as they are created and determine the status of an order during it's entire lifetime.\n\n    Shopify does NOT track the actual shipping of an order. It is only aware when an order is created,\n    when it is fulfilled (meaning we take the order and we ship it to the customer). It is not aware\n    of WHERE the package is. This application leverages the 17track.net API to determine the state of\n    shipment.\n\n    The key objective of this application is to keep awareness of the state of an order during the\n    shipping process. So to summarize, this app stores Shopify order information into the database\n    and checks shipping status for orders hourly.\n\n    In the front end, we must display high risk orders so we can bring it to the attention of the team.\n\n\n\n\n### Customer Lifecycle\n---\n\n    Let me explain how the high level lifeycle of our Shopify works.\n\n    1) Customer makes an order. We store customer order information like:\n        • Customer shipping details\n        • Product details like SKU, price, etc.\n    2) Order is stored in Shopify database, we also store this.\n    3) When we are ready to fulfill the product (e.g. package and ship to customer) a fulfillment is created.\n        Fulfillment information includes:\n        • Tracking number and carrier\n        • Status of shipment\n    4) A customer may make MULTIPLE ORDERS, so each may be shipped at different times, and thus fulfillment\n        events will be different for different items in an order\n    5) An order is considered complete when all products ordered have been fulfilled.\n\n\n\n\n### App Lifecycle\n---\n    Now that I've explained the customer lifecycle, it's important to understand how this translates into\n    code. The core life cycle of the app is\n\n    1) Grab ORDER data including customer information, product ordered. This will come from:\n        • Hourly cron job from: api/scripts/canary_orders which grabs latest orders from Shopify from the last hour\n        • Shopify webhooks. Shopify will PUSH via HTTP REST to our servers Order and Fulfillment Object data to our\n          API endpoints. Currently there are 4:\n\n            ------------------\n            Fulfillment Create\n            ------------------\n                URL: http://api.thecanary.io/webhook/fulfillment/create/\n                This is when we decide to fulfill part of an order. If our team ships an item for\n                a customer, they will send JSON to the above URL real-time.\n\n            ------------------\n            Fulfillment Update\n            ------------------\n                URL: https://api.thecanary.io/webhook/fulfillment/update/\n                If anything changes within a fulfillment, like a tracking number has been added\n                Shopify will fire the current Fulfillment object to this end point\n\n            ---------------\n            Order fulfilled\n            ---------------\n                URL: https://api.thecanary.io/webhook/order/fulfilled/\n                When an order has been fulfilled by our shipping partner\n\n            ------------\n            Order Update\n            ------------\n                URL: http://api.thecanary.io/webhook/order/update/\n                If Order information has been changed, for example if a customer requests a refund, their\n                Order object will be changed and we will get this information\n\n\n            **NOTE**: We do not request ORDER CREATE webhook, because we are already grabbing it hourly from the\n            API as stated in the first point above. We only need webhooks for when any data about the ORDER\n            or FULFILLMENT is MODIFIED\n\n    2) External data sources like CRONJOBS and WEBHOOKS send JSON data to our controller end points. These\n        controllers parse the data and update the models which stores it into our database.\n\n    3) Above is the flow for creating and updating our data structure from Shopify. But Shopify does not understand\n        the state of an Order's SHIPPING STATUS. We get this information from a shipment tracking API called 17track.net\n        Every hour the BASH SCRIPT api/scripts/canary_crawler.sh is ran via CRON JOB. The lifecycle for this is:\n\n        • Run canary_crawler.sh every hour, this will update the shipment tracking status for all fulfillment records in the DB\n\n        • canary_craw.sh:\n            - Grab fulfillment records that have not been checked in 1 day\n            - Grab order_id and fulfillment_id and tracking_number and convert list to JSON\n            - Pass JSON to controller \"crawler.controller.php\"\n\n        • crawler.controller.php\n            - Connect to 17track.net API and grab status of the ORDER\n            - Pass results to tracking.controller.php\n\n        • tracking.controller.php\n            - Determine status and whether we need to write alert status, this will be explained later\n            - If call is successful to 17track.net, store this in api_status so we monitor API usage since we are charged per call\n            - Store status into Fulfillment object and to the database\n\n\n    3) That's it! Basically the app needs to store ALL information about orders. The status of the shipment of the orders.\n        This is the back-end. Basically the front-end will display order status information that is\n        important for the team to know so they can react quickly with shipments that have problems.\n\n        Details on the dashboard are here:\n\n        https://docs.google.com/document/d/1huP_YYSE1jTC16-N2f11vTdeSDikYQWgYtj4ysoArmk/edit\n\n\n\n### Directory Structure\n---\n\n    The app is a customer framework I created. .htaccess grabs the URL and routes it MVC style.\n    This is to help redundancy. Controllers match the name of models and are all loaded automatically.\n\n    An API object like \"order\" (url: http://api.thecanary.io/order/) has a matching controller and\n    model.\n\n\n    ---------\n    index.php\n    ---------\n\n    The application uses APACHE rewrites via .htaccess. Please enable this in your apache configuration file.\n\n    Basically when the browser requests a URL, .htaccess will pass the query string to index.php instead\n    of accessing a file or directory directly. It will skip index.php if the file DOES exist.\n\n    So, if the browser requests http://api.thecanary.io/controllerObject/controllerFunction/controllerID/controllerData\n\n    index.php will grab $_SERVER['REQUEST_URI'] (or argv for BASH SHELL SCRIPTS) and explode the URL\n    into an array. Each \"/\" will denote different parts of the controller.\n\n     • controllerObject\n            This is the name of the controller, the first part of the URL. index.php will look for a file\n            in the api/controller directory and execute it. It will also pass the arguments from the URL\n            via: global $controllerObject,$controllerFunction,$controllerID,$controllerData\n\n     • controllerFunction\n            This is what your want to do with the object\n\n     • controllerID\n            This is the object's row ID in the database when you want to reference a particular object\n\n     • controllerData\n            These are additional parameters you want to pass to the object's controller\n\n     At the end, when the API requests an object, it needs to provide a response to the client.\n     This is done via the api_response() function in index.php\n\n     The structure for the response is in JSON and is structured like this:\n\n     [{\n        code: \u003cresponse_code\u003e,\n        msg: \"\u003cmessage_to_client\u003e\",\n        data: {\n            \u003csupporting_data_for_query\u003e\n        }\n     }]\n\n\n    ----------\n    api/backup\n    ----------\n    IGNORE THIS DIRECTORY. Contains backup files form previous versions of the app that are deprecated\n    but kept in the repository for future purposes\n\n\n    ---------\n    api/class\n    ---------\n\n    Contains classes for objects that have complex functionality\n\n        • crawler.class.php - IGNORE THIS FILE. IT IS NO LONGER USED\n\n\n    --------------\n    api/controller\n    --------------\n\n    Contains controllers for API object end-points. routed from index.php\n\n        • crawler.controller.php\n            The Crawler controller is called by the system cron job which crawls fulfillment records flagged in the\n            database for shipment tracking. Everyday it will crawl http://17track.net to parse and store / update the\n            shipping status of all outstanding fulfillment orders.\n\n        • cron.controller.php\n            This controller gets called by the system CRON JOB to grab hourly data from Shopify API\n            and insert new Orders and related Fulfillment data\n\n            It also contains spreadsheet parsing data for inserting new tracking #s\n            This will be a separate project and explained later\n\n        • manual.controller.php\n            You can IGNORE THIS file, this is used for debugging and for me to manually insert\n            data from the Shopify API when I have issues or am missing data. This file can be\n            explained later, but will not be used by you\n\n        • order.controller.php\n            This file will process Order information whether it is Order creation which will create\n            Item objects as well or Order UPDATE which will grab an order object, query if it exists\n            in the database, if it does, only update NEW information in the relevant columns. It will\n            also parse through Order data and create or update related Fulfillment objects\n\n        • report.controller.php\n             This is the main API endpoint for the dashboard. This will request a particular SQL\n             report query from the file sql_queries.php and return the appropriate JSON to the\n             user-facing client. This is where you will be adding a lot of code\n\n        • spytool.controller.php\n            Ignore this file, I will delete it later.\n\n        • test.controller.php\n            Ignore this file, I use this for debugging and testing when adding new controllers or features\n\n        • tracking.controller.php\n            This controller is used internally. crawler.controller.php will pass this file the\n            JSON results from the tracking API 17track.net and will PARSE this data and update\n            the relevant objects and the database.\n\n            This will also keep track of the tracking API usage\n\n        • webhook.controller.php\n            This controller will process data sent from the Shopify API webhooks, please refer above\n            where I explain all the webhooks\n\n\n\n    --------\n    api/data\n    --------\n\n    For now ignore this folder. This directory is used for another related feature in this project which\n    you will work on after the dashboard. Basically when a user uploads EXCEL files, this is where they\n    are stored and archived.\n\n\n    --------\n    api/logs\n    --------\n\n    This is where log files we generate for debugging are stored. Please add this to the .gitignore file\n    on your local and stage machine.\n\n\n    ---------\n    api/model\n    ---------\n\n    This contains all the models for all the objects in the API. Basically models represent the data\n    structure for our objects and are in charge for storing the data into the database.\n\n        • crawler.model.php\n            IGNORE THIS, currently not used\n\n        • fulfillment.model.php\n            Holds the data structure and database functions for Fulfillment objects\n            This is the most important part of the application because the user of our\n            dashboard needs to understand specific statuses of the order process that\n            this model must indicate, specifically \"delivery_status\" and \"alert_status\"\n            I will explain this more later.\n\n            A Fulfillment object is the child of an Order object.\n\n        • item.model.php\n            This contains details about an Item in Shopify which is a product a customer\n            ordered. An item object is the child of an Order object.\n\n        • order.model.php\n            This contains details of an order. This is the parent object. This model\n            contains functions to add, create, pull a list of Orders from the database\n\n    -----------\n    api/scripts\n    -----------\n\n    This directory contains BASH SHELL SCRIPTS that are executed in specific intervals determined\n    by CRONJOBs. Only two scripts are used:\n\n\n        • canary_crawler.sh\n            This is ran EVERY HOUR to pull eligible Fulfillment objects and check the 17track API\n            for the current status of an order's shipment, and updates the database with this\n            data.\n\n        • canary_orders.sh\n            This is ran EVERY HOUR to pull Order objects from the Shopify API and store new\n            Orders and relevant Fulfillments and Items in the database.\n\n\n\n    --------------\n    api/shared/lib\n    --------------\n\n    Contains shared libraries used by the application\n\n        • php-excel-reader/\n            You will use this for the next project. IGNORE THIS FOR NOW\n\n        • phpFastCache/\n            Currently not used, but will be used later for caching MySQL queries for better app performance\n\n        • database.class.php\n            This should really be in the class folder, but since I re-use this alot, I put it in the\n            shared folder. This class is basically the interface to the database. It contains\n            functions that allow to create, delete, add, and update rows in the database via\n            a simple Array hash methodology.\n\n            EVERY MODEL INHERITS THIS CLASS to make it easy to store and modify Object data to the\n            database. DO NOT modify this class as it has been optimized already. If you wish to modify\n            it or optimize it, just let me know\n\n        • utility.php\n            Contains useful functions I use universally across all applications. Feel free to add to this.\n\n        • other files\n            Ignore these, they will be used for the next project for this application\n\n\n    -------\n    api/tmp\n    -------\n\n    Temporary directory, mostly used by phpFastCache library. Ignore for now.\n\n\n    ----------\n    config.php\n    ----------\n\n    This contains app configuration information.\n\n    • ENABLE_CACHE\n        Make sure this stays false for now\n\n    • MODE\n        Within this switch statement, I've added your local machine information so that the codebase\n        works on both our machines as well as our stage and production machines. Please modify and edit\n        the CASE statements that are relevant only to YOUR server\n\n    • ENABLE_DEBUG\n        This will set the app in debug mode and display data in certain areas VERBOSELY. You will not\n        really need this, I use this flag for testing to make sure data is accurate, etc.\n\n    • ENABLE_LOGS\n        When set to TRUE this will write to the HTTP error log in Apache and display useful data as\n        the app runs through its lifecycle. This is extremely helpful to me during debugging an app\n        when adding a new feature.\n\n        If you want to write to the error log, PLEASE use the function: log_error($output)\n        It will write to the error log if ENABLE_LOGS == TRUE. If FALSE, it will ignore log_error\n        and not write to the HTTP error log\n\n    -------------\n    constants.php\n    -------------\n\n    Contains constants. You and I will add to this as the project grows. Contains API data as well.\n\n\n    ---------------\n    sql_queries.php\n    ---------------\n\n    Since the dashboard reporting will contain complex JOINs in MySQL, I have put all the reporting\n    queries inside this file. Basically the reporting queries are set as CONSTANTS in a pre-defined\n    format so it is easy to edit. When report.controller.php grabs the query string, it will replace\n    the relevant variables in the query string like {ID} or {DATE_RANGE} to execute the query for\n    the dashboard.\n\n    You will be heavily editing this file.\n\n\n\n### Model \u0026 Controller structure\n---\n\n    The way I designed the Model files is so that you only need to declare the class/instance variables\n    once inside the class file.\n\n    It will dynamically also generate the setter and getter functions, although I don't really use them.\n\n    Each model has a serialize_object function which will grab all the instance variables\n    and convert them to an Array which will be used by database.class.php to store into the DB\n\n    You can initialize any model and if you do not pass a parameter in the contructor, it will create\n    a blank Object.\n\n    If you pass the constructor an Array, it will create an Object with these values in the respective\n    instance variables\n\n    If you pass the contructor an ID it will look up this row ID in the database and load the Object\n    into the instance you've declared.\n\n    So it's quite dynamic.\n\n\n\n### API end points\n---\n\n    Internally there are a few endpoints, for the client-facing side like the dashboard\n    it will only use the /report  end point\n\n    • canary_crawl.sh - to update tracking status information in Fulfillment Objects\n    • canary_orders.sh - to grab new Orders from Shopify API and store to DB\n\n\n### Cron Jobs\n---\n\n    Currently there are only 2 cron jobs that run every hour.\n\n\n\n### Dashboard\n---\n\n    The majority of your focus will be to build the front-end dashboard for the team.\n\n    Details are here:\n\n            https://docs.google.com/document/d/1huP_YYSE1jTC16-N2f11vTdeSDikYQWgYtj4ysoArmk/edit\n\n    The template files are inside api/dashboard/\n\n    report.js contains the logic for calling the API and arranging the data appropriately in the dashboard UI\n\n\n\n\n\n\n### MIT License\n- - -\n\nCopyright (c) 2019 Paul Pierre\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in allcopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaulpierre%2Fpp_canary","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpaulpierre%2Fpp_canary","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaulpierre%2Fpp_canary/lists"}