{"id":19591317,"url":"https://github.com/humanloop/test","last_synced_at":"2026-02-19T06:02:34.077Z","repository":{"id":166046683,"uuid":"336440886","full_name":"humanloop/test","owner":"humanloop","description":"Take home test for full-stack roles","archived":false,"fork":false,"pushed_at":"2021-07-05T13:31:41.000Z","size":108,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-10-23T15:36:03.408Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/humanloop.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-02-06T02:48:12.000Z","updated_at":"2021-07-05T13:31:44.000Z","dependencies_parsed_at":null,"dependency_job_id":"ed3448ab-c551-473a-bb8f-8d1f7ccdabbc","html_url":"https://github.com/humanloop/test","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/humanloop/test","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/humanloop%2Ftest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/humanloop%2Ftest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/humanloop%2Ftest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/humanloop%2Ftest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/humanloop","download_url":"https://codeload.github.com/humanloop/test/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/humanloop%2Ftest/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29604552,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-19T05:11:50.834Z","status":"ssl_error","status_checked_at":"2026-02-19T05:11:38.921Z","response_time":117,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T08:28:29.628Z","updated_at":"2026-02-19T06:02:34.026Z","avatar_url":"https://github.com/humanloop.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Humanloop _Data Debugger_ Challenge \n\nThis is an engineering exercise that will give you some insight into the\ntypes of things we are building at Humanloop. We use the same test for ML engineers\nand full-stack engineers but the guidelines and evaluation differ slightly.\n \nAt Humanloop we want to make programming computers as natural as teaching a colleague\n so that anyone can collaborate with AI to achieve their goals.\n \nAn initial step towards this goal is creating tools for\nprogramming a system by simply providing carefully curated data. And an essential\nfeature of any reliable programming environment is the debugger...\n   \nOne of the most common challenges in the lifecycle of a Machine Learning\nproject is trying to improve the performance of an existing trained model. An\n important route to tackling this is to try and understand\nwhere in the data the model is under-performing. \n      \nThe ability to diagnose which data points the model is struggling with can provide a\n host of benefits. For example it can expose issues with the original training data (e.g\n  labelling mistakes). It can also inform strategies for what additional types of\n   data might be most useful to spend time collecting. This knowledge, coupled with\n    techniques like [active learning](https://humanloop.com/blog/why-you-should-be-using-active-learning/),\ncan help to rapidly enhance the performance of a model through additional targeted\n training.\n \n**For this exercise we would like you to build a small application that\n helps a user debug the data associated to a classifier model trained using the\n  [Humanloop Platform](https://humanloop.com/signup).**\n  \n## Background\n\nA company have used Humanloop to create a simple chat-bot\nintent classifier. Their team of in-house client servicing experts trained the\n model by annotating a proportion of the available data on the Humanloop Platform.\n \nTheir data team now wish to debug the data to understand the\nquality of the annotation process and interrogate the model's\nstrengths and weaknesses. This will be used to help inform what the next steps for improving\nthe data might be.\n\n**You have access to:**\n\n1. An API to get batch predictions and associated confidence scores\n from the classifier.\n2. An export of the full dataset used – broken down into the training, validation and test sets.\n \n## Requirements\n\nThe basics of your app should:\n\n- Store the dataset in a suitable database of your choice.\n- Provide a debugger service that operates on the data.\n- Serve a simple user interface for users to get value from the\n debugger. \n \nPlease create and share with us a github repository with your solution and provide clear\n instructions for how to run your application locally.\n\n  \n### Dataset \nThe dataset is included as a json file (_chatbot_intent_classifier_data.json.zip_) in this directory.\n\nThe `complete` field indicates whether an annotation has been\nprovided and the `usage` field indicates whether the annotated data point was\nused for training, testing or validation purposes. \n  \nWe also provide an active learning `score` field as a bonus. This is the [entropy](https://en.wikipedia.org/wiki/Entropy_(information_theory)) of the softmax classification output of the model –– which can be used as an estimate for how confused the model is with that datapoint.\n\n### Prediction API \n\nYou can leverage the [prediction api](https://humanloop.com/docs#operation/create_prediction_projects__id__predict_post) \nfrom the intent classifier project: \n\n    X-API-KEY: sk_63f406a2690ccd01a3234492e2e99996\n\nSnippet:\n    \n    curl -H \"X-API-Key: sk_63f406a2690ccd01a3234492e2e99996\" \\\n    -H \"Content-Type: application/json\" \\\n    -d '{\"data\":[{\"text\": \"What is my new credit limit?\"}], \"n_best\":1}' \\\n    -X POST https://api.humanloop.com/projects/716/predict\n\nYou should limit batch sizes to 20 data points at a time and can use this to enrich\n the data.\n \n## Guidelines\n- Ideally we'd like you to spend around 4 hours working on this, so keep things simple.\n Please let us know how much time you do end up spending so we can calibrate\n  expectations. \n \n- You are free to use any languages and libraries you want. \n \n- Given the indicated time-frame, we don't expect you to spend much time in this\n  exercise on 'non-functional' aspects such as authentication, testing, performance, deployment or documentation. We are most interested in what useful debugging functionality you are able to ship here and expect a prototypical solution - although we do care a lot about code quality and style!\n\n### For Full-Stack Engineers \n\n- Whilst we are hoping to see creativity and ML knowledge for the debugging\n approach, it is far more important that you deliver some small piece of functionality that\n works. ML knowledge and creativity are a bonus. An example feature could be: providing a view of the data to the user where the model most\n  confidently disagrees with the annotations provided - what might this uncover? \n\n- Although the UI is not the focus of this exercise, we do care a lot\n about UX across all aspects of product at Humanloop and are very interested in ways\n  to communicate potentially complex data in intuitive user friendly ways.\n\n### For ML Engineers\n    \n- Unlike the full-stack guidelines, we're far more interested in your creativity\nand ML knowledge. It's less important that you are able to code a fully functional app, and more important\nthat you can leverage what you know of ML to help the end user improve their model or data. \nFor example, could you make use of pre-trained models or embeddings? What visualisation or clustering techniques\nmight help? There's no right answer here, so feel free to be creative.\n  \n  \nAny questions, please let us know! careers@humanloop.com\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhumanloop%2Ftest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhumanloop%2Ftest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhumanloop%2Ftest/lists"}