{"id":23016378,"url":"https://github.com/davidmaceachern/playground-data-collection-rust","last_synced_at":"2026-04-30T01:35:46.815Z","repository":{"id":143991090,"uuid":"288240877","full_name":"davidmaceachern/playground-data-collection-rust","owner":"davidmaceachern","description":"🐈 🦀 Code to accompany a 'How to get started' with Rust blog post, originally intended for introducing a team to the ecosystem.","archived":false,"fork":false,"pushed_at":"2023-01-04T08:05:06.000Z","size":11,"stargazers_count":0,"open_issues_count":2,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-02T18:13:40.598Z","etag":null,"topics":["cats","filesystem","json","rust"],"latest_commit_sha":null,"homepage":"https://davidmaceachern.com/posts/collecting-data-from-an-api","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/davidmaceachern.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-08-17T17:16:18.000Z","updated_at":"2020-11-02T10:31:04.000Z","dependencies_parsed_at":null,"dependency_job_id":"2702052d-a2ef-4040-9e88-c05bf0bae049","html_url":"https://github.com/davidmaceachern/playground-data-collection-rust","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/davidmaceachern/playground-data-collection-rust","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidmaceachern%2Fplayground-data-collection-rust","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidmaceachern%2Fplayground-data-collection-rust/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidmaceachern%2Fplayground-data-collection-rust/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidmaceachern%2Fplayground-data-collection-rust/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/davidmaceachern","download_url":"https://codeload.github.com/davidmaceachern/playground-data-collection-rust/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidmaceachern%2Fplayground-data-collection-rust/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32451474,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-29T22:27:22.272Z","status":"ssl_error","status_checked_at":"2026-04-29T22:10:49.234Z","response_time":110,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cats","filesystem","json","rust"],"created_at":"2024-12-15T11:15:05.566Z","updated_at":"2026-04-30T01:35:46.797Z","avatar_url":"https://github.com/davidmaceachern.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Collecting Data from an API\n\nData is essential when building applications. \n\nLet's collect some data that we can use to write an application.\n\nContinue reading if you want to:\n- Query an API.\n- Handle some Http errors.\n- Collect some data and save it as JSON.\n- Write an application in Rust.\n\nIf you are looking for the complete code you can find it [here](github.com/davidmaceachern/playground-data-collection-rust).\n\n## Requirements\n\nBefore going further let's define how we can address the requirements of the application to avoid doing more than necessary.\n\n- **Finding data** we need to have some open data we can collect from an API somewhere. It needs to be open because like Music and Films we need to have permission to use it.\n- **Collecting data** we can do this using a HTTP client, it's worth noting that a data structure that is transmitted via HTTP is serialized as a string.\n- **Storing data** can be done using a filesystem. This can be done by converting the strings we collect to a suitable data structure and outputting it to a file. The format we choose will depend on what we want to do with the data later on.\n\n## Finding Data \n\nFirstly we need an API we can query for some data if you do not already have one chosen, a good place to start is this [repository](https://github.com/public-apis/public-apis).\n\nFor this exercise, we are going to get some [cat facts](https://alexwohlbruck.github.io/cat-facts/docs/). Taking a look through the repository we can see a [repository licence](https://github.com/alexwohlbruck/cat-facts/blob/master/LICENSE), which looks permissive enough to use this data.\n\nWe can use a web browser to call a `GET` endpoint, pasting this endpoint `https://cat-fact.herokuapp.com/facts/random` into the address bar returns the following response.\n\n``` Json\n{\n  \"used\": false,\n  \"source\": \"api\",\n  \"type\": \"cat\",\n  \"deleted\": false,\n  \"_id\": \"591f98703b90f7150a19c151\",\n  \"__v\": 0,\n  \"text\": \"Cat families usually play best in even numbers. Cats and kittens should be aquired in pairs whenever possible.\",\n  \"updatedAt\": \"2020-06-30T20:20:33.478Z\",\n  \"createdAt\": \"2018-01-04T01:10:54.673Z\",\n  \"status\": {\n    \"verified\": true,\n    \"sentCount\": 1\n  },\n  \"user\": \"5a9ac18c7478810ea6c06381\"\n}\n```\n\n## Writing an Application in Rust\n\n### Setting up the project\n\nWe are going to write our application in Rust, if you haven't already you can install Rust using the instructions [here](https://www.rust-lang.org/learn/get-started).\n\nNext, we want to check that the installation provided the `Cargo` package manager, we can do this by running: \n\n``` Bash\n$ cargo --version\n```\nIf that returned the version we have, we can then initialize a new project that uses the current folder as the name of the project by running:\n\n``` Bash\n$ cargo init --bin\n```\n\nOne thing I like about Rust is the ecosystem, as functionalities that other languages have built-in can be provided through `Crates` until the Rust Language team adopt them.\n\nI would recommend installing `cargo-edit` for adding packages the same way you might do in Javascript when running `$ npm install --save`.\n\n``` Rust\n$ cargo install cargo-edit\n```\n\nTo address the problem our application will solve, we can use the following crates together:\n- The HTTP client we can use to send the request for our data is very helpfully called [reqwest](https://crates.io/crates/reqwest)\n- Filesystem interactions will be provided by a JSON file store called [jfs](https://crates.io/crates/jfs)\n- To convert our strings to data structures and data structures to strings we can use [serde](https://crates.io/crates/serde)\n  - For dealing with JSON data structures we can use [serde_json](https://crates.io/crates/serde_json)\n- To avoid worrying about how we implement errors for now we can use [anyhow](https://crates.io/crates/anyhow)\n\n\n``` Bash\n$ cargo add reqwest serde serde_json jfs anyhow \n```\n\nThe output will look like:\n\n``` Bash\n      Adding reqwest v0.10.7 to dependencies\n      Adding serde v1.0.115 to dependencies\n      Adding serde_json v1.0.57 to dependencies\n      Adding jfs v0.6.2 to dependencies\n      Adding anyhow v1.0.32 to dependencies\n```\n\nWithout specifying a version number for these libraries, we will want to check the versions we are telling Cargo to use because that will determine which version of the documentation we need to look at.\n\nInside the `Cargo.toml` we can see:\n\n``` Toml\n[dependencies]\nreqwest =  \"0.10\",\nserde = \"1.0.115\",\nserde_json = \"1.0.57\"\njfs = \"0.6.2\"\nanyhow = \"1.0.32\"\n```\n\nSo next we can build the application to install the dependencies, we can do this by running:\n\n``` Rust\n$ cargo build\n```\nWhat I am going to do from here onwards however:\n\n``` Rust\n$ cargo run\n```\nWhich will build and execute the code for us so we can have some feedback from theError: Server responded with: 404 Not Found output:\n``` Bash \n    Finished dev [unoptimized + debuginfo] target(s) in 0.06s\n     Running `target/debug/data-collection-rust`\nHello, world!!\n```\nNormally if we had tests written then we could have them watch for new  file changes but that is out of scope for the article today.\n\n### Calling the API\n\nWe will start by replicating calling the API in our application.\n\nIn a similar way to how the Web Browser was our client before, we must have a client that will interact with the API.\n\n``` diff\n fn main() {\n-    println!(\"Hello, world!\");\n+    let client = reqwest::blocking::Client::new();\n }\n```\nRunning this code, we encounter an error.\n``` Bash\nerror[E0433]: failed to resolve: could not find `blocking` in `reqwest`\n --\u003e src/main.rs:2:27\n  |\n2 |     let client = reqwest::blocking::Client::new();\n  |                           ^^^^^^^^ could not find `blocking` in `reqwest`\n```\n\nWhen using some crates we must specify the features that our application will use in the `Cargo.toml`.\n\n``` diff \n [dependencies]\n-reqwest = \"0.10\"\n+reqwest = { version = \"0.10\", features = [\"blocking\"] }\n```\nOk so let's add another line to call out endpoint\n``` diff\n     let client = reqwest::blocking::Client::new();\n+    let uri = \"https://cat-fact.herokuapp.com/fact/random\";\n+    let response = client.get(uri).send()?;\n```\n\nUpon running our application again we see another error:\n\n``` Bash\nerror[E0277]: the `?` operator can only be used in a function that returns `Result` or `Option` (or another type that implements...\n```\n\nWe're using the `?` operator here to handle calling a function that could throw an error. Let's do as the compiler suggests:\n\n``` diff\n-fn main() {\n+fn main() -\u003e Result {\n```\nSeems that isn't exactly what the compiler wants:\n``` Bash\nerror[E0107]: wrong number of type arguments: expected 2, found 0\n --\u003e src/main.rs:1:14\n  |\n1 | fn main() -\u003e Result {\n  |              ^^^^^^ expected 2 type arguments\n```\nThe Result will only be the return type if our code is successful, if not then this function will return an error. This is where we can use `anyhow`:\n``` diff\n-fn main() {\n+fn main() -\u003e Result\u003c(),  anyhow::Error\u003e {\n```\nOk, so we have another compiler error...\n``` Bash\nerror[E0308]: mismatched types\n --\u003e src/main.rs:1:14\n  |\n1 | fn main() -\u003e Result\u003c(), anyhow::Error\u003e {\n  |    ----      ^^^^^^^^^^^^^^^^^^^^^^^^^ expected enum `std::result::Result`, found `()`\n  |    |\n  |    implicitly returns `()` as its body has no tail or `return` expression\n```\nWe need to add the following it seems, for any successful outcome that doesn't throw an error.\n``` diff\n     let response = client.get(uri).send()?;\n+    Ok(())\n })\n```\n\nSo we probably want to know what the response looks like. We can take a quick look using the following macro:\n\n``` diff\n+    dbg!(response);\n     Ok(())\n```\nAnd it appears that I might have mistyped the url, as we receive a `not found` error.\n\n``` Bash\n[src/main.rs:5] response = Response {\n    url: \"https://cat-fact.herokuapp.com/fact/random\",\n    status: 404,\n    headers: {\n        \"server\": \"Cowboy\",\n        \"connection\": \"keep-alive\",\n        \"x-powered-by\": \"Express\",\n        \"access-control-allow-origin\": \"*\",\n        \"content-security-policy\": \"default-src 'none'\",\n        \"x-content-type-options\": \"nosniff\",\n        \"content-type\": \"text/html; charset=utf-8\",\n        \"content-length\": \"150\",\n        \"set-cookie\": \"connect.sid=s%3A5IS9zYZqbamwJECS6C5JrdcDfIBJ8epX.Lbh4Zl5C21jdFOyih1RgS1%2FiZr2c8jxbEc1l1XiwTvo; Path=/; HttpOnly\",\n        \"date\": \"Tue, 25 Aug 2020 17:46:27 GMT\",\n        \"via\": \"1.1 vegur\",\n    },\n}\n```\n\nI checked the API documentation and indeed I had mistyped the url. \n\n``` diff\n-    let uri = \"https://cat-fact.herokuapp.com/fact/random\";\n+    let uri = \"https://cat-fact.herokuapp.com/facts/random\";\n```\nAfter correction, we get the correct status code.\n\n``` diff\n    url: \"https://cat-fact.herokuapp.com/facts/random\",\n    status: 200,\n    headers: {\n```\n\nIt would probably be a good idea to handle errors when we don't get a `200` response. Let's check the response value so we can add a condition.\n\n``` diff\n-    dbg!(response);\n+    dbg!(response.status());\n```\nOk.\n``` Bash\n[src/main.rs:5] response.status() = 200\n```\nNow let's add a conditional.\n\n``` diff\n-    dbg!(response.status());\n+    if(response.status() == 200) {\n+        println!(\"{}\", response.status());\n+    }\n```\nCool.\n\n``` Bash\n200 OK\n```\nHowever we want it to throw an error right, seems `reqwest` might let us do this, let's force it to fail again by adding the typo back in.\n\n``` diff\n-    let uri = \"https://cat-fact.herokuapp.com/facts/random\";\n+    let uri = \"https://cat-fact.herokuapp.com/fact/random\";\n     let response = client.get(uri).send()?;\n-    if(response.status() == 200) {\n+    if response.status().is_client_error() || response.status().is_server_error() {\n         println!(\"{}\", response.status());\n     } \n```\nAnd let's have the application return the error it encounters to avoid running any other code. We can do this by using a macro bundled with anyhow.\n\n``` diff\n+use anyhow::anyhow;\n+\n fn main() -\u003e Result\u003c(), anyhow::Error\u003e {\n     let client = reqwest::blocking::Client::new();\n     let uri = \"https://cat-fact.herokuapp.com/facts/random\";\n     let response = client.get(uri).send()?;\n     if(response.status().is_client_error() || response.status().is_server_error()) {\n-        println!(\"{}\", response.status());\n+        return Err(anyhow!(\"Server responded with: {}\", response.status()));\n     } \n     Ok(())\n```\nSeems the compiler is warning us about something:\n``` Bash\nwarning: unnecessary parentheses around `if` condition\n```\nRemove the parentheses and now upon receiving a status code that isn't `200`\n``` Bash\nError: Server responded with: 404 Not Found\n```\nGreat. Next, let's look at getting our facts out of the response.\n\n### Deserializing Data\n\nLet's start by trying to store the body's text as a string.\n\n``` diff\n+    let string = serde_json::from_str(response.text());\n     Ok(())\n```\n\nThe compiler complains...\n\n``` Bash\nerror[E0308]: mismatched types\n  --\u003e src/main.rs:10:39\n   |\n10 |     let string = serde_json::from_str(response.text());\n   |                                       ^^^^^^^^^^^^^^^ expected `\u0026str`, found enum `std::result::Result`\n```\nWe can try to get the result by telling Rust that we might expect an error on both function calls, allowing us to access the result which hopefully is of type `\u0026str`.\n\n``` diff\n-    let string = serde_json::from_str(response.text());\n+    let string = serde_json::from_str(response.text()?)?;\n```\n\n``` Bash\nerror[E0308]: try expression alternatives have incompatible types\n  --\u003e src/main.rs:10:39\n   |\n10 |     let string = serde_json::from_str(response.text()?);\n   |                                       ^^^^^^^^^^^^^^^^\n   |                                       |\n   |                                       expected `\u0026str`, found struct `std::string::String`\n   |                                       help: consider borrowing here: `\u0026response.text()?`\n```\n\nWow, the compiler tells us what we might be able to do to fix the problem. Ok so if we reference the response instead...\n\n``` Bash\nwarning: unused variable: `string`\n  --\u003e src/main.rs:10:9\n   |\n10 |     let string = serde_json::from_str(\u0026response.text()?)?;\n   |         ^^^^^^ help: if this is intentional, prefix it with an underscore: `_string`\n   |\n   = note: `#[warn(unused_variables)]` on by default\n\nwarning: 1 warning emitted\n\n    Finished dev [unoptimized + debuginfo] target(s) in 3.20s\n     Running `target/debug/data-collection-rust`\nError: invalid type: map, expected unit at line 1 column 0\n```\n\nOk, nice so we only have one warning which we can ignore because we plan to use the variable. The error however might take some figuring out. So it turns out we can coerce Rust to try to parse using a specific type provided by serde_json. This will only work if the item is valid JSON.\n\n``` diff\n-    let string = serde_json::from_str(\u0026response.text()?)?;\n+    let string: Value = serde_json::from_str(\u0026response.text()?)?;\n```\nSo the code compiles, we can print out what string now holds to check what the response body looks like.\n\n``` diff\n+    dbg!(string);\n```\n\n``` bash\n[src/main.rs:12] string = Object({\n    \"__v\": Number(\n        0,\n    ),\n    \"_id\": String(\n        \"591f97d48dec2e14e3c20aff\",\n    ),\n    \"createdAt\": String(\n        \"2018-01-04T01:10:54.673Z\",\n    ),\n    \"deleted\": Bool(\n        false,\n    ),\n    \"source\": String(\n        \"api\",\n    ),\n    \"status\": Object({\n        \"sentCount\": Number(\n            1,\n        ),\n        \"verified\": Bool(\n            true,\n        ),\n    }),\n    \"text\": String(\n        \"Cats have the largest eyes of any mammal.\",\n    ),\n    \"type\": String(\n        \"cat\",\n    ),\n    \"updatedAt\": String(\n        \"2020-08-23T20:20:01.611Z\",\n    ),\n    \"used\": Bool(\n        false,\n    ),\n    \"user\": String(\n        \"5a9ac18c7478810ea6c06381\",\n    ),\n})\n```\n\nAwesome, so this looks more like the kind of data we might want to use later. Rust even provides us with what types it thinks the fields are.\n\n### Determining the types\n\nRust is a strongly typed language so this means we define the types that our application needs to know about, and should do so when we can because the compiler is not always able to infer. It's useful to define the structure of our data for future reference so that when it comes to expanding our application we might need to understand the shape of our data.\n\nHaving checked the serde_json documentation we will need to make the following changes:\n1. At the top of the `main.rs`\n    ``` diff\n    -use serde_json::Value;\n    +use serde::{Deserialize, Serialize};\n    ```\n2. Further down the `main.rs`\n    ``` diff\n    -    let string: Value = serde_json::from_str(\u0026response.text()?)?;\n    +    let string: CatFact = serde_json::from_str(\u0026response.text()?)?;\n        Ok(())\n    }\n    +\n    +#[derive(Debug, Serialize, Deserialize)]\n    +struct CatFact {\n    +    used: bool,\n    +    source: String,\n    +    r#type: String,\n    +    deleted: bool,\n    +    _id: String,\n    +    __v: i32,\n    +    text: String,\n    +    updatedAt: String,\n    +    createdAt: String,\n    +    status: Status,\n    +    user: String\n    +}\n    +\n    +#[derive(Debug, Serialize, Deserialize)]\n    +struct Status {\n    +    verified: bool,\n    +    sentCount: i32\n    +}\n    ```\n3. In the `Cargo.toml`:\n    ``` diff\n    -serde = \"1.0.115\"\n    +serde = { version = \"1.0.115\", features = [\"derive\"] }\n    ```\n\nNotice that when attempting to define the types for a JSON record, if the field name (also known as a key) happens to be a reserved keyword then the compiler handily points this out. \n\n``` Rust\nerror: expected identifier, found keyword `type`\n  --\u003e src/main.rs:34:5\n   |\n34 |     type: String,\n   |     ^^^^ expected identifier, found keyword\n   |\nhelp: you can escape reserved keywords to use them as identifiers\n   |\n34 |     r#type: String,\n   |     ^^^^^^\n\nerror: aborting due to previous error\n\nerror: could not compile `playground-data-collection-rust`.\n```\n\nSo when we run the application and `dbg!(string)` we see we have a cat fact.\n\n``` Bash\n[src/main.rs:12] string = CatFact {\n    used: false,\n    source: \"api\",\n    type: \"cat\",\n    deleted: false,\n    etc...\n```\nThen if we want to access a specific field we use dot notation.\n\n``` diff\n+    dbg!(string.text);\n     Ok(())\n```\n\nNow we have the data we can store it somewhere.\n\n### Persisting the data locally\n\nIf we want the data to be used after our application has finished running, we need to consider using a persistence layer. This is useful if the application crashes, or we have another application that will use the data elsewhere. For this exercise let's consider the scope of how we can store data.\n\n- We can write a new file for each dataset that we collect. \n- We restrict the number of calls we are writing a single record at a time, so our program is synchronous.\n- We can use a loop to run our program for the prescribed number of times, and we pause the thread's execution so that we don't flood the API with requests. This can be based on whether the API we are calling has a throttling limit.\n\nIn the future we can increase the collection frequency then we might want to consider a different storage layer that considers scalability.\n\nWe have chosen to use `jfs` which will use our filesystem to store the data. If we wanted to analyze our data straight away we could have considered `sqlite`.\n\nOk so let's add the ability to save our data structure.\n\nStart by importing:\n\n``` diff\n use anyhow::anyhow;\n+use jfs::Store;\n```\nAnd we can print out the key that it uses as the file name.\n``` diff\n+    let d: Store = Store::new(\"data\")?;\n+    let key = db.save(\u0026string)?;\n+    dbg!(key);\n     Ok(())\n```\nLet's add some logging capability so that we can let our application log the result of what it's doing.\n``` Bash\n$ cargo add env_log\n```\nAnd adding the following in the `main.rs`.\n\n``` diff\n use jfs::Store;\n+#[macro_use]\n+extern crate log;\n...\n+    env_logger::init();\n+    info!(\"Starting up\");\n....\n-    dbg!(key);\n+    info!(\"Written one file with key: {}\", key);\n```\nWhen we run our application with an environment variable.\n``` Bash\nRUST_LOG=info cargo run\n```\nIt will now print out some details for us.\n``` Log\n[2020-08-27T17:54:41Z INFO  data_collection_rust] Starting up\n[2020-08-27T17:54:42Z INFO  data_collection_rust] Written one file with key: 032bfc7b-f1c8-4cdd-bb9a-f29b3f1fa9c4\n```\n\nGreat, but what if we want to keep collecting items, can we make the application do this?\n\n### Collecting more than one item\n\nSo now to collect more than one item.\n\nWe can add a loop that will run infinitely. You will notice that I have moved items out of the loop as it should be more efficient to only run them once at start-up.\n\n``` diff\nfn main() -\u003e Result\u003c(), anyhow::Error\u003e {\n     info!(\"Starting up\");\n     let client = reqwest::blocking::Client::new();\n     let uri = \"https://cat-fact.herokuapp.com/facts/random\";\n-    let response = client.get(uri).send()?;\n-    if response.status().is_client_error() || response.status().is_server_error() {\n-        return Err(anyhow!(\"Server responded with: {}\", response.status()));\n-    }\n-    let string: CatFact = serde_json::from_str(\u0026response.text()?)?;\n     let db: Store = Store::new(\"data\")?;\n-    let key = db.save(\u0026string)?;\n-    info!(\"Written one file with key: {}\", key);\n+    loop {\n+        let response = client.get(uri).send()?;\n+        if response.status().is_client_error() || response.status().is_server_error() {\n+            return Err(anyhow!(\"Server responded with: {}\", response.status()));\n+        }\n+        let string: CatFact = serde_json::from_str(\u0026response.text()?)?;\n+        let key = db.save(\u0026string)?;\n+        info!(\"Written one file with key: {}\", key);\n+    }\n     Ok(())\n }\n```\n\nGreat but we can probably slow down our requests so we don't DDOS or throttle the service we're using.\n\n``` log\n[2020-08-28T13:17:57Z INFO  data_collection_rust] Starting up\n[2020-08-28T13:18:04Z INFO  data_collection_rust] Written one file with key: 6ec26b1e-51e9-46d7-92fc-5b5d848f3b85\n[2020-08-28T13:18:04Z INFO  data_collection_rust] Written one file with key: e731bec5-ef6e-4f64-93c9-abd2df4d837b\n[2020-08-28T13:18:05Z INFO  data_collection_rust] Written one file with key: fecd2a8b-2f3c-4f3b-a15e-f87f577a1116\n[2020-08-28T13:18:05Z INFO  data_collection_rust] Written one file with key: b2220b56-ed18-4a60-a79a-b096f308cfae\n[2020-08-28T13:18:05Z INFO  data_collection_rust] Written one file with key: c38acf34-edee-4e24-938f-2bf3f6b08e7d\n[2020-08-28T13:18:05Z INFO  data_collection_rust] Written one file with key: 1e0f9ca5-d2fe-44a0-801b-869bee21233c\n```\n\nWe can do this be making the thread this process run in sleep for some time. So let's import this functionality.\n\n``` diff\n use serde::{Deserialize, Serialize};\n+use std::thread;\n+use std::time::Duration;\n```\nAnd add this to call the method.\n\n``` diff\n         info!(\"Written one file with key: {}\", key);\n+        thread::sleep(Duration::from_millis(5000));\n```\nThen when we run our application.\n\n``` bash\n[2020-08-28T13:24:08Z INFO  data_collection_rust] Starting up\n[2020-08-28T13:24:08Z INFO  data_collection_rust] Written one file with key: 30183e77-c25c-453c-922b-e027ba9a0e54\n[2020-08-28T13:24:14Z INFO  data_collection_rust] Written one file with key: f1b63d41-8452-442f-8e3d-c4fea7bf80d0\n```\nGreat, you can see the time difference between the two requests is greater now, we could set this duration as an environment variable if we wanted later. \n\n``` bash\nwarning: unreachable expression\n  --\u003e src/main.rs:26:5\n   |\n16 | /     loop {\n17 | |         let response = client.get(uri).send()?;\n18 | |         if response.status().is_client_error() || response.status().is_server_error() {\n19 | |             return Err(anyhow!(\"Server responded with: {}\", response.status()));\n...  |\n24 | |         thread::sleep(Duration::from_millis(5000));\n25 | |     }\n   | |_____- any code following this expression is unreachable\n26 |       Ok(())\n   |       ^^^^^^ unreachable expression\n   |\n   = note: `#[warn(unreachable_code)]` on by default\n```\n\nThe compiler is warning us that our function is never going to return `OK` because it never leaves the loop. We can add a counter and break condition to resolve this.\n\nFor our use case, we can use an unsigned integer since we know we will never have a negative number when incrementing our counter. We also make it mutable because we want to change it.\n\n``` diff\n     info!(\"Starting up\");\n+    let mut count = 0u32;\n```\n``` diff\nloop {\n+    count += 1;\n```\nAnd add the break condition.\n\n``` diff\n         thread::sleep(Duration::from_millis(5000));\n+        if count == 5 {\n+            break;\n+        } else {\n+            continue;\n+        }\n```\n\n\n## Future work\n\nSome things that we could do next:\n- Add a test that will mock calling the API.\n- Create a data factory for generating random test data.\n- Use a storage layer such as Amazon S3 to enable scaling.\n\nThanks for reading, I hope this helped you, please reach out to me on Twitter if you have any questions and/or suggestions!\n\n## Misc\n\n### Response\n\n``` Rust\nResponse {\n    url: \"https://cat-fact.herokuapp.com/facts/random\",\n    status: 200,\n    header: {\n        \"server\": \"Cowboy\",\n        \"connection\": \"keep-alive\",\n        \"x-powered-by\": \"Express\",\n        \"access-control-allow-origin\": \"*\",\n        \"content-type\": \"application/json; charset=utf-8\",\n        \"content-length\": \"305\",\n        \"etag\": \"W/\\\"131-4MvaJDAXqtlkSUWatxfF1BCPTek\\\"\",\n        \"set-cookie\": \"connect.sid=s%3Ap0LnvYyEQUN9plmg--3mnx7DNd1dyhI4.Ms1oCBB5sWMAMwCtgfye572bD%2FBfxlCRkRRoq8cLzEY; Path=/; HttpOnly\",\n        \"date\": \"Tue, 18 Aug 2020 20:04:01 GMT\",\n        \"via\": \"1.1 vegur\",\n    },\n}\n```\n\n### Compiler Warnings\n\nThis type of warning doesn't stop the code from compiling, it's a bit like Javascripts ESLint telling us that our code isn't idiomatic. \n``` Bash\nwarning: structure field `sentCount` should have a snake case name\n  --\u003e src/main.rs:49:5\n   |\n49 |     sentCount: i32\n   |     ^^^^^^^^^ help: convert the identifier to snake case: `sent_count`\n\nwarning: 3 warnings emitted\n```\n\nThis can be disabled whilst developing if the noise gets in the way with the following at the top of the file.\n\n`#![allow(non_snake_case)]`","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidmaceachern%2Fplayground-data-collection-rust","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdavidmaceachern%2Fplayground-data-collection-rust","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidmaceachern%2Fplayground-data-collection-rust/lists"}