{"id":39655192,"url":"https://github.com/pdidev/tutorial","last_synced_at":"2026-01-18T09:16:29.948Z","repository":{"id":54203759,"uuid":"254289301","full_name":"pdidev/tutorial","owner":"pdidev","description":"PDI hands-on exercises","archived":false,"fork":false,"pushed_at":"2026-01-14T14:29:46.000Z","size":5444,"stargazers_count":1,"open_issues_count":9,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-01-14T15:44:20.239Z","etag":null,"topics":["hdf5","pdi","yaml"],"latest_commit_sha":null,"homepage":"https://pdi.dev/master/Hands_on.html","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pdidev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-04-09T06:26:47.000Z","updated_at":"2025-06-23T09:26:00.000Z","dependencies_parsed_at":"2024-06-06T15:29:54.584Z","dependency_job_id":"9f623bee-debe-4360-bc7a-75e5224305bf","html_url":"https://github.com/pdidev/tutorial","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/pdidev/tutorial","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pdidev%2Ftutorial","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pdidev%2Ftutorial/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pdidev%2Ftutorial/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pdidev%2Ftutorial/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pdidev","download_url":"https://codeload.github.com/pdidev/tutorial/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pdidev%2Ftutorial/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28534156,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-18T00:39:45.795Z","status":"online","status_checked_at":"2026-01-18T02:00:07.578Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hdf5","pdi","yaml"],"created_at":"2026-01-18T09:16:29.634Z","updated_at":"2026-01-18T09:16:29.922Z","avatar_url":"https://github.com/pdidev.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Hands-on tutorial {#Hands_on}\n\nIn this tutorial, you will build a PDI-enabled application step-by-step from a\nPDI-free base.\nYou will end-up building the C version of the\n\\ref PDI_example \"example provided with PDI\" for the\n\\ref trace_plugin \"Trace\", \\ref Decl_HDF5_plugin \"Decl'HDF5\",\n\\ref pycall_plugin \"Pycall\",\n\\ref user_code_plugin \"user_code\" and\n\\ref set_value_plugin \"set_value\" plugins.\nAdditional [examples are available for the other plugins](https://gitlab.maisondelasimulation.fr/pdidev/pdi/-/tree/master/example).\n\n\n## Setup\n\n\\attention\nTo run this hands-on tutorial, you first need to \\ref Installation \"install PDI\"\nand setup your environment.\n\n### PDI installation\n\n\\ref Installation \"PDI installation\" is documented in a\n\\ref Installation \"dedicated page\".\n\n\n### Hands-on tutorial setup\n\nOnce %PDI is installed, you can proceed with getting the sources for the\nhands-on tutorial from [github](https://github.com/pdidev/tutorial):\n```bash\ngit clone https://github.com/pdidev/tutorial.git\ncd tutorial\n```\n\n\n### Compilation\n\nBefore compilation, configure the tutorial by detecting all dependencies:\n```bash\npdirun cmake .\n```\n\n\\attention\nIf you installed PDI in a standard path, the `pdirun` prefix is never required.\n\nOnce you have correctly modified each exercise according to instructions, you\ncan compile it by running:\n```bash\npdirun make ex?\n```\nWhere `?` is the number of the exercise.\n\n\n### Execution\n\nYou can run each exercise with the following command:\n```bash\npdirun mpirun -n 4 ./ex?\n```\nWhere `?` is the number of the exercise and 4 represents the number of MPI\nprocesses to use.\n\n#### Execution with storage of the log {#execution-with-storage-of-the-log}\n\nTo store the logs for later comparison, you can use the following command (for\nexample for ex2.):\n```bash\npdirun mpirun -n 1 ./ex2 | tee ex2.result.log\n```\n\n### How to compare a h5 file with a h5dump file {#compare_h5_h5dump}\n\nUsing h5dump, you can create a `h5` file with a readable format using.\n```bash\n  h5dump ex3.h5 \u003e ex3.result.h5dump\n```\nTo compare with the h5dump file of the tutorial (for example ex3).\n```bash\n  diff ex3.h5dump ex3.result.h5dump\n```\nMoreover, you can see with your editor the two `h5dump` files.\n\n\nThe comparison can be done in one line without creating the `h5dump` file.\n```bash\n  diff ex3.h5dump \u003c(h5dump ex3*.h5)\n```\n\nNow you're ready to work, **good luck**!\n\n\n## PDI-free code\n\n### Ex1. Getting started\n\nEx1. implements a simple heat equation solver using an explicit forward finite\ndifference scheme parallelized with MPI.\nThe code uses a block domain decomposition where each process holds a 2D block\nof data.\n\n![Data domain decomposition in the example](heat_global_matrix.jpg)\n\nLocally, each process holds its local block of data with one additional element\non each side for ghost zones.\n\n![Data domain decomposition in the example](heat_local_matrix.jpg)\n\nIn the following exercises however, %PDI will only be used to decouple I/O\noperations.\nThere is no need to fully dive in the core of the solver described in the\n\\ref heat_algorithm \"PDI example algorithm\" and implemented in the `iter` and\n`exchange` functions.\n\nThe specification tree in the `.yml` files and the `main` function are the\nlocations where all the I/O-related aspects will be handled and the only ones\nyou will actually need to fully understand or modify.\n\n* Examine the source code, compile it and run it.\n  There is no input/output operations in the code yet, so you can not see any\n  result.\n\nThis example gets its configuration from a file in the \\ref YAML \"YAML format\":\n`ex1.yml` file.\nIf you're not familiar with YAML, please have a look at our quick\n\\ref YAML \"YAML format documentation\" to understand it.\nThe example uses the [paraconf library](https://github.com/pdidev/paraconf) to\nread this file.\n\n* Play with and understand the code parameters in `ex1.yml`.\n\n* Set values in `ex1.yml` to be able to run the code with 4 MPI processes.\n\n```bash\nmpirun -np 4 ./ex1\n```\n\n## PDI core \u0026 trace plugin\n\n### Ex2. Now with some PDI\n\nEx2. C code is similar to ex1. C code with %PDI calls added in `main` function.\nThis exercise will be run sequentially.\n\n* Examine the source code, compile it and run it.\n\nIn our YAML file (`ex2.yml`), the `pdi` key is added. The sub-tree, defined\nafter this key, is the %PDI specification tree passed to %PDI at initialization.\n\n* To observe %PDI calls on the standard output, add \\ref trace_plugin\n \"Trace plugin\" (`trace`) plugin of %PDI in our YAML file (`ex2.yml`).\n\n* In the C file (`ex2.c`), add `::PDI_share` and `::PDI_reclaim` call to share\n  some data with %PDI.\n  The shared data are defined in the line that starts with \"//***\" in `ex2.c`.\n\nHere, the objective is to match the output of `ex2.log` file.\nIn this file, only the line corresponding to `[Trace-plugin]` have been kept.\nMoreover, the time are given for each %PDI calls.\nTo compare, we need to remove this information from the log.\nIt is done by adding this line in the sub-tree of the trace plugin.\n\n```yaml\n  logging: { pattern: '[PDI][%n-plugin] *** %l: %v' }\n```\nAdditionally, we run sequentially to facilitate the comparison between logs\n(in parallel each rank send a `trace` message and the order of writing can be\ndifferent).\n\n* Add the previous line in the sub-tree of \\ref trace_plugin \"Trace plugin\"\n(don't forget to indent this line correctly).\nUsing the previous section [Execution with storage of the log](#execution-with-storage-of-the-log),\nrun this exercise and save the output log in the file `ex2.result.log`.\nAfter that you can easily check if the files are the same by running the command:\n\n```bash\n  diff ex2.log \u003c(grep Trace-plugin ex2.result.log)\n```\n\n\\attention\nIn this exercise, the shared variable and the reclaimed variable are not defined\nin the YAML file (see ex3. and further for this).\n\n#### Note on the order of share/reclaim calls {#ShareReclaimCallsOrder}\n\nNotice that some share/reclaim pairs come one after the other while others are\ninterlaced.\nIs one better than the other?\nIf you do not know the answer to this question, just wait until ex5. :)\n\n## Decl'HDF5 plugin\n\nFrom exercise 3 to exercise 9 included, we present the \\ref Decl_HDF5_plugin \n\"Decl'HDF5 plugin\" (`decl_hdf5`). We will introduce some keywords (`when`, \n`datasets`, ...) in the sub-tree of `decl_hdf5` in configuration YAML file.\n\nAll keywords are defined in the last section **full configuration example** of \n\\ref Decl_HDF5_plugin \"Decl'HDF5 plugin\".\n\n### Ex3. HDF5 through PDI\n\nIn this exercise, the C code is the same as in ex2. You only need to modify the \nYAML file (`ex3.yml`).\n\n* Examine the YAML file, compile the code and run it.\n\nThe \\ref Decl_HDF5_plugin \"Decl'HDF5 plugin\" (`decl_hdf5`) is added in the\nspecification tree. \nIn its configuration, the `dsize` variable is defined as a metadata for %PDI.\n\n* Write the `psize` and `pcoord` variables in addition to `dsize` in a file `ex3.h5`\nwith one MPI process.\nTo achieve this result, you will need to fill 2 sections in the YAML file:\n\n  1. The `data` section to indicate to %PDI the\n  \\ref datatype_node \"data_type\" of the fields that are exposed.\n\n  2. The `decl_hdf5` section for the configuration of the\n  \\ref Decl_HDF5_plugin \"Decl'HDF5 plugin\".\n\n* Use the `h5dump` command to see the content of your HDF5 output file in the \nsame format as the h5dump file `ex3.h5dump`. You can easily check if the files \nare the same by running:\n```bash\n  diff ex3.h5dump \u003c(h5dump ex3*.h5)\n```\nTo see your `h5` file in readable file format, you can check the section\n[Comparison with the `h5dump` command](#compare_h5_h5dump).\n\n\\warning\nIf you relaunch the executable, remember to delete your old `ex3.h5` file before,\notherwise the data will not be changed.\n\n\\warning\nWhen more than one MPI rank is used, we write to the same location in the file\nindependently of the MPI rank. For this reason, this exercise will fail. The next\nexercise solves this issue.\n\n### Ex4. Writing some real data\n\nIn this exercise each MPI process will write its local 2D array block contained\nin the `main_field` variable to a separate HDF5 file.\nOnce again, this can be done by modifying the YAML file only, no need to touch\nthe C file.\n\n* Examine the YAML file, compile the code and run it.\n\n\\note Notice that in the YAML file `ex4.yml`, a list was used in the `decl_hdf5`\nsection with multiple write blocks instead of a single one as before in order\nto write to multiple files.\n\n\\note Notice that we have moved fields (`dsize`, `psize` and `pcoord`) in the \n`metadata` section.\n```yaml\npdi:\n  metadata: # small values for which PDI keeps a copy\n    #*** add ii as metadata\n    #...\n    dsize: { type: array, subtype: int, size: 2 }\n```\nBy definition, a metadata is a variable that can be used to describe other data\n(for example, the size of a vector).\nYou can reference them from dynamic \"$-expressions\" in the configuration file.\n\n\\remark Also notice that this example now runs in parallel with 4 processes.\nTo ensure we do not write to the same file, we need  to specify the file name\nusing \"$-expressions\" for the different process rank. \n\nUnlike the other fields manipulated until now, the type of `main_field` is not\nfully known, its size is dynamic. Therefore, we need to define the size in YAML\nfile for %PDI using \"$-expressions\".\n\n* Describe the temperature data on the current iteration by using a $-expression\nto specify the size of `main_field` in `data` section.\n\nUnlike the other fields manipulated until now, `main_field` is exposed multiple \ntimes along execution.\nIn order not to overwrite it every time it is exposed, you propose to write one\nfile per rank only at the first iteration (`ii=1`) with the directive `when`.\n\n* Add the iteration loop `ii` as a metadata.\n* Write the curent temperature field in one file per process at first iteration.\n\nYou should be able to match the expected output described in `ex4.h5dump`.\nYou can easily check if the files are the same by running:\n```bash\n  diff ex4.h5dump \u003c(h5dump ex4-data-*.h5)\n```\nTo see your `h5` file in readable file format, you can check the section\n[Comparison with the `h5dump` command](#compare_h5_h5dump).\n\n\\note A definition of `metadata` and `data` can be:\n\n- `metadata`: small values for which PDI keeps a copy. These value can be\nreferenced by using \"$-expressions\" in the configuration YAML file.\n\n- `data`    : values for which PDI does not keep a copy.\n\n### Ex5. Introducing events and group of dataset\n\nThis exercise is done sequentially to facilitate the comparison between logs.\n\n#### Ex 5.1 PDI event and on_event\n\nIn ex4, two variables (`ii` and `main_field`) were written to `ex4-data-*.h5`,\nbut the files were opened and closed for each and every write.\n\nSince \\ref Decl_HDF5_plugin \"Decl'HDF5 plugin\" only sees the data appear\none after the other, it does not keep the file open.\nSince `ii` and `main_field` are shared in an interlaced way,\nthey are both available to %PDI at the same time after the second\n`::PDI_share`.\n```C\n  // share the loop counter \u0026 main field at each iteration\n  PDI_share(\"ii\",         \u0026ii, PDI_OUT);\n  PDI_share(\"main_field\", cur, PDI_OUT);\n  PDI_reclaim(\"main_field\");\n  PDI_reclaim(\"ii\");\n```\nTherefore at that moment, they could be written without opening the file twice.\nYou have to use events for that.\n\n\\attention\nThe answer to section [Note on the order of share/reclaim calls](#ShareReclaimCallsOrder)\nis given here.\nInterlacing share/reclaim pairs is better when we want to give access to\nmultiple data to %PDI or %PDI plugins at the same time.\n\nIn this section of ex5, you will modify both the C and YAML file.\n\n* Examine the YAML file and source code.\n* In the C file, add a %PDI event named `loop` when both `ii` and\n  `main_field` are shared.\n\n  With the \\ref trace_plugin \"Trace plugin\", check that the event is indeed \n  triggered at the expected time as described in `ex5.log` (only the lines\n  matching `[Trace-plugin]` have been kept).\n  Using the previous section [Execution with storage of the log](#execution-with-storage-of-the-log),\n  run  this exercise in saving the output log in the `ex5.result.log`.\n  After that you can easily check if the files are the same by running:\n```bash\n  diff ex5.log \u003c(grep Trace-plugin ex5.result.log)\n```\n* In the YAML file, use the `on_event` mechanism to trigger the write of `ii` and\n `main_field` for event `loop` only. This mechanism can be combined with a `when`\n directive, in that case the write is only executed when both mechanisms agree.\n* Add the `when` directive to write only at iteration 1 and 2. Use the symbol `\u0026` \n  which corresponds to the logical operation `and`.\n\n#### Ex 5.2 group of dataset\nIn ex4, the name of the datasets of `.h5` file are `ii` and `main_field`(see ex4.h5dump).\nUsing the keyword `dataset`, it is possible to have a different name from the \n%PDI variable name.\n\nThe name of the dataset is given after the definition of the data \n```yaml\n       write:\n          ii: # name of the PDI data to write\n            dataset: 'new_name' \n```\nUsing this mechanism, it is possible to define a group object of hdf5 see\nhttps://support.hdfgroup.org/documentation/hdf5/latest/_h5_g__u_g.html.\nIf you want to add a dataset `my_data` in the sub-group `groupA` of the group \n`my_group`, the name of the dataset will be:\n```yaml\ndataset: 'my_group/groupA/my_data'.\n```\nwhere the symbol \"/\" is used to separate groups in path.\n\n* Change the YAML file to write `main_field` and `ii` at iterations 1 and 2,\nin two distinct groups `iter1` and `iter2`.\n\n  To match the expected output described in `ex5.h5dump`. You can easily check\n  if the files are the same by running:\n```bash\n  diff ex5.h5dump \u003c(h5dump ex5-data-*.h5)\n```\n\n  To see your `h5` file in readable file format, you can check the section\n  [Comparison with the `h5dump` command](#compare_h5_h5dump).\n\n### Ex6. Simplifying the code\n\nAs you can notice, the %PDI code is quite redundant.\nIn this exercise, you will use `::PDI_expose` and `::PDI_multi_expose` to\nsimplify the code while keeping the exact same behaviour.\nFor once, there is no need to modify the YAML file here, you only need to modify\nthe C file. Moreover, this exercise will be launched sequentially to facilitate\nthe comparison between logs.\n\n* Examine the source code, compile it and run it.\n\n\\remark At the end of the iteration loop, a new event `finalization` is added.\nTo gain a better understanding of the information provided by the trace plugin,\nthe call order of different `::PDI_share` is changed in the ex6 after the main\nloop.\n\nThere are lots of matched `::PDI_share`/`::PDI_reclaim` in the code.\n\n* Replace these by `::PDI_expose` that is the exact equivalent of a\n  `::PDI_share` followed by a matching `::PDI_reclaim`.\n\nThis replacement is not possible for interlaced `::PDI_share`/`::PDI_reclaim`\nwith events in the middle.\nThis case is however handled by `::PDI_multi_expose` call that exposes all data,\nthen triggers an event and finally does all the reclaim in reverse order.\n\n* Replace the remaining `::PDI_share`/`::PDI_reclaim` by `::PDI_expose`s and\n  `::PDI_multi_expose`s and ensure that your code keeps the exact same behaviour\n  as in previous exercise by comparing its trace to `ex6.log`\n  (only the lines matching `[Trace-plugin]` have been kept).\n  Using the previous section\n  [Execution with storage of the log](#execution-with-storage-of-the-log),\n  run  this exercise in saving the output log in the `ex6.result.log`.\n  After that you can easily check if the files are the same by running:\n```bash\n  diff ex6.log \u003c(grep Trace-plugin ex6.result.log)\n```\n\nIn summary:\n\n1. `::PDI_expose` is equivalent to `::PDI_share` + `::PDI_reclaim`.\n\n2. `::PDI_multi_expose` for one data is equivalent to\n`::PDI_share` + `::PDI_event` + `::PDI_reclaim`.\n\n3. `::PDI_multi_expose` for data `A` and data `B` is equivalent to\n`::PDI_share` `A` + `::PDI_share` `B`+ `::PDI_event` + `::PDI_reclaim` `B` + `::PDI_reclaim` `A`.\n\n\\attention\nThe `::PDI_multi_expose` is implemented with interlaced share/reclaim pairs.\n\n\\attention\nWhen we used `::PDI_multi_expose` with multiple data, the order of appearance\nof the arguments of the function corresponds to the order of the `::PDI_share`.\n\n\\attention\nIn a `::PDI_multi_expose` if you have a data1 that depends on the data2,\nyou need to pass the arguments corresponding to data2 before the arguments\ncorresponding to data1 in this function. With `::PDI_share` and\n`::PDI_reclaim` functions, you need to share data2 before data1.\nFor example, a vector `V` that depends on its size `N`:\n```C\n  PDI_multi_expose(\"save_vector_V\",\n          \"size_of_vector\", \u0026N, PDI_OUT,\n          \"vector_V\", V, PDI_OUT,\n          NULL);\n```\n```C\n  PDI_share(\"size_of_vector\", \u0026N, PDI_OUT)\n  PDI_share(\"vector_V\", V, PDI_OUT)\n  PDI_reclaim(\"size_of_vector\")\n  PDI_reclaim(\"vector_V\")\n```\n### Ex7. Writing a selection\n\nIn this exercise, you will only write a selection of the 2D array in memory\nexcluding ghosts to the HDF5 file.\nOnce again, you only need to modify the YAML file in this exercise, no need to\ntouch the C file.\n\n\\remark This exercise will run with 4 MPI processes.\n\n* Examine the YAML file and compile the code.\n\nAs you can notice, now the dataset is independently described in the file.\n\n* Restrict the selection to the non-ghost part of the array in memory (excluding\n  one element on each side).\n\nYou can achieve this by using the `memory_selection` directive in ex7.yml that \nspecifies the selection of data from memory to write.\n\nYou should be able to match the expected output described in `ex7.h5dump`. \nYou can easily check if the files are the same by running:\n```bash\n  diff ex7.h5dump \u003c(h5dump ex7*.h5)\n```\nTo see your `h5` file in readable file format, \nyou can check the section [Comparison with the `h5dump` command](#compare_h5_h5dump).\n\n![graphical representation](PDI_hdf5_selection.jpg)\n\n### Ex8. Selecting on the dataset size\n\nIn this exercise, you will once again change the YAML file to handle a selection\nin the dataset in addition to the selection in memory from the previous\nexercise.\nIn this exercise, you don't want to get one output file per iteration.\nYou will write the 2D array from the previous exercise as a slice of a 3D dataset\nincluding a time dimension for iterations 1 to 3 (inclusive).\n\nOnce again, you only need to modify the YAML file in this exercise, no need to\ntouch the C file.\n\n* Examine the YAML file and compile the code.\n\nNotice how the dataset is extended with an additional dimension \n```yaml\n      datasets:\n        #*** add one dimention to main_field datasets to represent the time step\n        main_field: { type: array, subtype: double, size: [..., '$dsize[0]-2', '$dsize[1]-2' ] }\n```\n\n* replace `...` in previous line by the number of iteration time, we want to save\n  in this exercise.\n\n* Change the `when` directive to write the `main_field` at iterations 1 to 3 inclusive.\n\n* Write the 2D selection from `main_field` at iterations 1 to 3 inclusive into\n  slices at coordinate 0 to 2 of the first dimension of the 3D dataset.\n\nYou can achieve this by using the `dataset_selection` directive that specifies\nthe selection where to write in the file dataset.\n\nYou should be able to match the expected output described in `ex8.h5dump`.\nYou can easily check if the files are the same by running:\n```bash\n  diff ex8.h5dump \u003c(h5dump ex8*.h5)\n```\nTo see your `h5` file in readable file format, you can check the section\n[Comparison with the `h5dump` command](#compare_h5_h5dump).\n\n![graphical representation](PDI_hdf5_selection_advanced.jpg)\n\n## parallel Decl'HDF5\n\n### Ex9. Going parallel\n\nRunning the code from the previous exercises in parallel should already work and\nyield one file per process containing the local data block.\nIn this exercise you will write one single file `ex9.h5`(see `ex9.yml`) with \nparallel HDF5 whose content should be independent from the number of \nprocesses used.\n\n\\attention You need to do this exercise with a parallel version of HDF5 and the \n\\ref Decl_HDF5_plugin \"Decl'HDF5 plugin\" compiled in parallel.\n\nOnce again, you only need to modify the YAML file in this exercise,\nno need to touch the C file.\n\n* Examine the YAML file and compile the code.\n\n* Load the `mpi` plugin to make sharing MPI communicators possible.\n\n* Define the `communicator` directive of the \\ref Decl_HDF5_plugin \"Decl'HDF5 plugin\"\n  to switch to parallel I/O for HDF5. Set the value of the communicator to MPI_COMM_WORLD.\n\n\\note We have added the directive `collision_policy: write_into` of the \n\\ref Decl_HDF5_plugin \"Decl'HDF5 plugin\" (see section COLLISION_POLICY). \nThis parameter is used to define what to do when writing to a file or dataset\n that already exists.\n\n* Set the size of the dataset to take the global (parallel) array size into account. \n  You will need to multiply the local size by the number of processes in each\n  dimension (use `psize`).\n\n* Ensure the dataset selection of each process does not overlap with the others.\n  You will need to make a selection in the dataset that depends on the global\n  coordinate of the local data block (use `pcoord`).\n\nYou should be able to match the expected output described in `ex9.h5dump`. \nYou can easily check if the files are the same by running:\n```bash\n  diff ex9.h5dump \u003c(h5dump ex9*.h5)\n```\nTo see your `h5` file in readable file format,\nyou can check the section [Comparison with the `h5dump` command](#compare_h5_h5dump).\n\n\\warning\nIf you relaunch the executable, remember to delete your old `ex9.h5` file before,\notherwise the data will not be changed correctly.\n\n![graphical representation of the parallel I/O](PDI_hdf5_parallel.jpg)\n\n\n## Pycall\n\n### Ex10. Post-processing the data in python\n\n\\attention\nYou need to have a version %PDI with the \\ref pycall_plugin \"Pycall plugin\"\nto do this exercise.\n\nIn this exercise, you will once again modify the YAML file only and use python\nto post-process the data in situ before writing it to HDF5.\nHere, you will write the square root of the raw data to HDF5 instead of the\ndata itself.\n\n* Examine the YAML file and compile the code.\n\n* Load the \\ref pycall_plugin \"Pycall plugin\" and enable this plugin\n  when the `loop` event is triggered.\n\nSome variables of the python script inside `ex10.yml` are not defined.\nThe `with` directive of this plugin allows to specify input variables (parameters)\nto pass to Python as a set of \"$-expressions\".\nThese parameters can be given as multiple blocks.\n\n* Add a `with` block with the missing parameter to let the Python code process\n  the data exposed in `main_field` for event `loop`.\n\n* Use the keyword `exec` of \\ref pycall_plugin \"Pycall plugin\".\n  After the colon (\":\"), add a space and a vertical bar (\" |\").\n  Uncomment the python script.\n\nNotice that the Decl'HDF5 configuration was simplified, no memory selection is\napplied, the `when` condition disappeared because it is done in the python script:\n```python\n  if 0 \u003c iter_id \u003c 4:\n    transformed_field = np.sqrt(source_field[1:-1,1:-1])\n    pdi.expose('transformed_field', transformed_field, pdi.OUT)\n```\nThe last line of the python script allows to expose the transformed field to %PDI.\nMoreover, this data is known to %PDI in this call.\n\n* Modify the Decl'HDF5 configuration to write the new data `transformed_field`\nexposed from Python.\n\n\\attention\nThe dataset name is however explicitly specified now because it does not match\nthe %PDI variable name anymore, you will instead write a new variable exposed\nfrom python.\n\nYou should be able to match the expected output described in `ex10.h5dump`.\nYou can easily check if the files are the same by running:\n```bash\n  diff ex10.h5dump \u003c(h5dump ex10*.h5)\n```\nTo see your `h5` file in readable file format,\nyou can check the section [Comparison with the `h5dump` command](#compare_h5_h5dump).\n\n\\warning\nIf you relaunch the executable, remember to delete your old `ex10.h5` file before,\notherwise the data will not be changed.\n\n\\attention\nIn a more realistic setup, one would typically not write much code in the YAML\nfile directly, but would instead call functions specified in a `.py` file on the side.\n\n##  Call a user C function\n\n### Ex11. user_code plugin\n\nIn this exercise, you will learn how to call a user C function in %PDI with the\n\\ref user_code_plugin \"user_code plugin\".\nMoreover, to reduce the standard output log,\nwe run sequentially and we reduce the number of iterations.\n\n\\attention\nYou need to read the section \\ref important_notes_node \"Important notes\"\nbefore using \\ref user_code_plugin \"user_code plugin\".\nTo compile the program `ex11` with Wl,`--export-dynamic` or `-rdynamic`,\nwe have added the following line:\n```cmake\nset_target_properties(ex11 PROPERTIES ENABLE_EXPORTS TRUE)\n```\nin the CMakeList.txt provided.\n\nThe objective is to write the integral of `main_field` on the file\n`integral.dat`.\nThis integral is computed in the C function `compute_integral`\ndefined in `ex11.c`.\n\n* Examine the C file, the YAML file and compile the code.\n\n\\remark In the `compute_integral` function, the %PDI function `::PDI_access`\nand `::PDI_release` are called (see \\ref annotation \"Code annotation\").\nFor example,\n```c\n  int *iter;\n  PDI_access(\"ii\", (void **)\u0026iter, PDI_IN);\n  PDI_release(\"ii\");\n```\n`::PDI_access` sets our pointer (`iter`) to the data location of `ii`.\nWe need to pass `PDI_IN` because data flows from PDI to our application.\nWe also want to use `::PDI_release`, because `::PDI_reclaim` would end\nthe sharing status of this descriptor and we reclaim this data later in\nthe main function.\n\nThe keyword `on_event` allows to call a C function inside `::PDI_event`.\nYou can call a user C function inside `::PDI_share`\nusing the keyword `on_data` in the \\ref user_code_plugin \"user_code plugin\".\n\nIn this exercise, we only modify `ex11.yml`.\n\n* Open the file `integral.dat` at event “initialization” for recording\nthe integral of the `main_field` in calling `open_file` function.\n\n* Close the file `integral.dat` at event “finalization” in calling `close_file`\nfunction.\n\n* Compute the integral of `main_field` and write this to the file `integral.dat`\n(using `compute_integral` function) when `main_field` is shared to %PDI.\n\nYou should be able to match the expected output described\nin `integral_solution.dat`.\nYou can easily check if the files are the same by running:\n```bash\n  diff integral_solution.dat integral.dat\n```\n\n* Check that the call of C functions defined by `on_event` and `on_data` are\nindeed done in the expected order in the log.\n\n\\remark The keywords `on_event` and `on_data` are also used in other plugins\nto execute instructions in `::PDI_event` and `::PDI_share` respectively.\n\n\n## set value of data and metadata\n\n### Ex12. set_value plugin\n\nThe \\ref set_value_plugin \"set_value\" plugin allows setting values to data and\nmetadata descriptors from the YAML file.\nIn the  \\ref set_value_plugin \"set_value\", the user can trigger action upon:\n`on_init`, `on_finalize`, `on_data`, `on_event`.\nIn this plugin, we have five different types of action:\n- Share data (`share`) - plugin will share new allocated data with given values.\n- Release data  (`release`) - plugin will release shared data.\n- Expose data (`expose`) - plugin will expose new allocated data\nwith given values.\n- Set data (`set`) - plugin will set given values to the already shared data.\n- Calling an event (`event`) - plugin will call an event.\n\n\\note examples with keywords `share`, `release`, `expose`, `set` and `event`\nare given at the end of section \"set_value\" plugin in the website.\n\nIn this exercise, we expose the integer `switch` to %PDI at each iteration.\nThis interger is used to enable or to disable the writing of `main_field`.\nWe want to start writing once this integer passes 50 and stop when it's below 25.\nFor this purpose, we introduce a new logical variable `should_output`\nin `ex12.yml`.\nThe value of `should_output` is defined by:\n```c\n  if(switch \u003e 50) should_output = true\n  if(switch \u003c 25) should_output = false\n  //otherwise the value of should_output doesn't change.\n```\n\n* At initialization of %PDI, define the `should_output` to 0 (false).\n* At finalization, release the variable `should_output`.\n* When `switch` is shared with %PDI, set the value of `should_output`\naccording to its definition.\n\\attention\n%PDI doesn't known directive `if` in YAML file. Therefore, you need to redefine\nthe value of `should_output` according to the previous value of `should_output`\nand the value of `switch`.\n\nYou should be able to match the expected output described in\n`should_output_solution.dat`.\nYou can easily check if the files are the same by running:\n```bash\n  diff should_output_solution.dat should_output.dat\n```\n\n* Enable the writing of `main_field` according to the value of `should_output`.\n\nYou should be able to match the expected output described in `ex12.h5dump`.\nYou can easily check if the files are the same by running:\n```bash\n  diff ex12.h5dump \u003c(h5dump ex12*.h5)\n```\nTo see your `h5` file in readable file format,\nyou can check the section [Comparison with the `h5dump` command](#compare_h5_h5dump).\n\n## What next ?\n\nIn this tutorial, you used the C API of %PDI and from YAML, you used the \n\\ref trace_plugin \"Trace\", \\ref Decl_HDF5_plugin \"Decl'HDF5\",\n\\ref pycall_plugin \"Pycall\", \\ref user_code_plugin \"user_code\" and\n\\ref set_value_plugin \"set_value\" plugins.\n\nIf you want to try PDI from another language (Fortran, python, ...) or if you\nwant to experiment with other \\ref Plugins \"PDI plugins\", have a look at the\n[examples provided with PDI](https://gitlab.maisondelasimulation.fr/pdidev/pdi/-/tree/master/example).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpdidev%2Ftutorial","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpdidev%2Ftutorial","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpdidev%2Ftutorial/lists"}