{"id":13665530,"url":"https://github.com/leeper/conjoint-example","last_synced_at":"2025-08-28T08:04:35.018Z","repository":{"id":41583659,"uuid":"76790470","full_name":"leeper/conjoint-example","owner":"leeper","description":"An Example Conjoint Experimental Design in Qualtrics","archived":false,"fork":false,"pushed_at":"2018-03-12T17:42:44.000Z","size":90,"stargazers_count":65,"open_issues_count":1,"forks_count":24,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-06-20T06:38:19.624Z","etag":null,"topics":["conjoint","factorial-experiment","javascript","qualtrics","r","randomization","survey-experiment"],"latest_commit_sha":null,"homepage":null,"language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/leeper.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-12-18T15:21:13.000Z","updated_at":"2025-06-02T13:07:42.000Z","dependencies_parsed_at":"2022-09-16T16:43:55.543Z","dependency_job_id":null,"html_url":"https://github.com/leeper/conjoint-example","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/leeper/conjoint-example","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leeper%2Fconjoint-example","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leeper%2Fconjoint-example/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leeper%2Fconjoint-example/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leeper%2Fconjoint-example/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/leeper","download_url":"https://codeload.github.com/leeper/conjoint-example/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leeper%2Fconjoint-example/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272466637,"owners_count":24939445,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-28T02:00:10.768Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["conjoint","factorial-experiment","javascript","qualtrics","r","randomization","survey-experiment"],"created_at":"2024-08-02T06:00:41.249Z","updated_at":"2025-08-28T08:04:34.992Z","avatar_url":"https://github.com/leeper.png","language":"HTML","funding_links":[],"categories":["HTML"],"sub_categories":[],"readme":"# Example Conjoint Experimental Design in Qualtrics\n\nThis is an example of conducting a conjoint experiment in Qualtrics. It benefitted heavily from Kyle Dropp's guide, [\"Implementing a Conjoint Analysis in Qualtrics\"](http://www.weebly.com/uploads/1/2/0/9/12094568/conjoint.pdf).\n\nA conjoint is a fully randomized, factorial experiment involving a potentially large number of factors and factor levels. Typically, participants are exposed to a vignette wherein features of the vignette (e.g., a person description) are randomized. To gain statistical power and to avoid severe sparsity problems in such a high-dimensional design, within-subjects comparisons are typically leveraged. That is, a respondent may be exposed to 5-10 vignettes in order to multiple the statistical power of the design (under some plausible assumptions). \n\nFor example, in this example, participants are shown descriptions of two hypothetical political candidates with numerous demographic and attitudinal features:\n\n![example](example.png)\n\nParticipants then indicate which of the two alternatives they prefer and the data are analyzed using regression with respondent fixed effects (and/or standard errors clustered by respondent) in order to estimate the marginal contribution of each feature (trait factor) averaging across all of the other combinations of feature levels.\n\nThis design creates approximately 2.5 million experimental cells. So it is impossible to physically create every possible combination of candidate profiles. Thus to actually conduct this experiment requires some dynamic programming that generates the profiles randomly. Building on Dropp's techniques, I show how we (Joshua Robison, Aarhus University) and I implemented this in Qualtrics.\n\nThe basic workflow is this:\n\n 1. Create a \"template\" HTML document that lays out the candidate profiles, without any of the feature information populated. Use this template as the body of a Qualtrics multiple choice question.\n \n 2. Write a small js script that randomly samples from all possible traits and save this as javascript in the Qualtrics question. The js script should also save the profile data to Qualtrics [\"embedded data\"](https://www.qualtrics.com/support/survey-platform/survey-module/survey-flow/standard-elements/embedded-data/) fields so that it the output data file contains a record of which profiles were shown to which respondents.\n \n 3. Repeat this for however many profiles are needed. We used 5 in this study, so that each respondent was shown five pairs of candidates. In essence this multiplies our sample size from `n=600` to `n=3000` minus whatever uncertainty is introduced by using non-independent observations.\n \n 4. Run the study, download the data, and construct profile-describing variables from stored profile information.\n \n## 1. HTML Template\n\nA complete template HTML file is provided in [template.html](template.html). It is simply an HTML file:\n\n```html\n\u003ctable style=\"width:90%; align:center; margin-left:auto; margin-right:auto; text-align:center; table-layout:fixed;\" cellspacing=\"4\" \u003e\n    \u003ctr\u003e\n        \u003ctd style=\"font-weight:bold; font-size:1.25em; font-align:center;\"\u003eCandidate A\u003c/td\u003e\n        \u003ctd\u003e\u003c/td\u003e\n        \u003ctd style=\"font-weight:bold; font-size:1.25em; font-align:center;\"\u003eCandidate B\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd id=\"a1\"\u003e\u003c/td\u003e\n        \u003ctd style=\"font-weight:bold;\"\u003eAge\u003c/td\u003e\n        \u003ctd id=\"b1\"\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd id=\"a2\"\u003e\u003c/td\u003e\n        \u003ctd style=\"font-weight:bold;\"\u003eSex\u003c/td\u003e\n        \u003ctd id=\"b2\"\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd id=\"a3\"\u003e\u003c/td\u003e\n        \u003ctd style=\"font-weight:bold;\"\u003eRace\u003c/td\u003e\n        \u003ctd id=\"b3\"\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd id=\"a4\"\u003e\u003c/td\u003e\n        \u003ctd style=\"font-weight:bold;\"\u003eReligion\u003c/td\u003e\n        \u003ctd id=\"b4\"\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd id=\"a5\"\u003e\u003c/td\u003e\n        \u003ctd style=\"font-weight:bold;\"\u003eOccupation\u003c/td\u003e\n        \u003ctd id=\"b5\"\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd id=\"a6\"\u003e\u003c/td\u003e\n        \u003ctd style=\"font-weight:bold;\"\u003eParty\u003c/td\u003e\n        \u003ctd id=\"b6\"\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd id=\"a7\"\u003e\u003c/td\u003e\n        \u003ctd style=\"font-weight:bold;\"\u003eMilitary Service\u003c/td\u003e\n        \u003ctd id=\"b7\"\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd id=\"a8\"\u003e\u003c/td\u003e\n        \u003ctd style=\"font-weight:bold;\"\u003eEducation\u003c/td\u003e\n        \u003ctd id=\"b8\"\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003c/td\u003e\n        \u003ctd style=\"font-style:italic; text-align:left;\"\u003eThe candidates have reported opinions on the following issues:\u003c/td\u003e\n        \u003ctd\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd id=\"a9\" style=\"vertical-align:top;\"\u003e\u003c/td\u003e\n        \u003ctd style=\"font-weight:bold; vertical-align:top;\"\u003eRatification of the Trans-Pacific Partnership (TPP)\u003c/td\u003e\n        \u003ctd id=\"b9\" style=\"vertical-align:top;\"\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd id=\"a10\" style=\"vertical-align:top;\"\u003e\u003c/td\u003e\n        \u003ctd style=\"font-weight:bold; vertical-align:top;\"\u003eDeploying ground troops to combat ISIS\u003c/td\u003e\n        \u003ctd id=\"b10\" style=\"vertical-align:top;\"\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd id=\"a11\" style=\"vertical-align:top;\"\u003e\u003c/td\u003e\n        \u003ctd style=\"font-weight:bold; vertical-align:top;\"\u003eA carbon tax (\"cap and trade\") system for greenhouse gases\u003c/td\u003e\n        \u003ctd id=\"b11\" style=\"vertical-align:top;\"\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd id=\"a12\" style=\"vertical-align:top;\"\u003e\u003c/td\u003e\n        \u003ctd style=\"font-weight:bold; vertical-align:top;\"\u003eIncreased taxes on those making over $250,000\u003c/td\u003e\n        \u003ctd id=\"b12\" style=\"vertical-align:top;\"\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd id=\"a13\" style=\"vertical-align:top;\"\u003e\u003c/td\u003e\n        \u003ctd style=\"font-weight:bold; vertical-align:top;\"\u003ePath to citizenship for illegal immigrants brought to US as children\u003c/td\u003e\n        \u003ctd id=\"b13\" style=\"vertical-align:top;\"\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n\u003c/table\u003e\n\u003cbr /\u003e\n\u003cspan\u003e\n    \u003cp style=\"font-weight:bold;\"\u003eWhich candidate do you prefer?\u003c/p\u003e\n\u003c/span\u003e\n```\n\nAs you will see, it is an HTML table with placeholders for each feature value. For example, the second row of the table shows:\n\n```html\n\u003ctr\u003e\n    \u003ctd id=\"a1\"\u003e\u003c/td\u003e\n    \u003ctd style=\"font-weight:bold;\"\u003eAge\u003c/td\u003e\n    \u003ctd id=\"b1\"\u003e\u003c/td\u003e\n\u003c/tr\u003e\n```\n\nThis is a single row of the table, with the first column cell given the html id `\"a1\"`, the center column serving as the label for the experimental factor (\"Age\"), and the right column cell given the html id `\"b1\"`. These `id` values are critical as they will be used to populate the candidate profiles using javascript. The physical layout and styling here are totally irrelevant. All that matters is that there is an html element (`\u003ctd\u003e\u003c/td\u003e`, `\u003cspan\u003e\u003c/span\u003e`, `\u003cp\u003e\u003c/p\u003e`, etc.) for each profile feature and that each one of these has a systematically named and unique `id` value. We used `a*` to indicate features of \"Candidate A\" and `b*` to indicate features of \"Candidate B\" and numbered the attributes vertically top to bottom but this is totally arbitrary.\n\nNote: [Dropp's guide](http://www.weebly.com/uploads/1/2/0/9/12094568/conjoint.pdf) discusses how to also randomize the order of the attributes within the profiles. We chose not to do that here because we thought it would be confusing to respondents.\n\nThis html should then be used as the question content in Qualtrics.\n\n## 2. Javascript\n\nThe javascript code needs to do four things: (a) define all of the possible feature values, (b) randomly sample from those values, (c) display them to the respondent, and (d) save the sampled values for later use in analysis. The [conjoint.js](conjoint.js) contains our complete javascript code, which I will explain a bit here.\n\nThe first two steps involve defining feature values and sampling from them. For discrete features, this is pretty simple. You just create a javascript array containing all the values and draw a random value from the array. We also added three additional complexities:\n\n 1. One of our features, age, was continuous so we simply randomly sample numbers in a given range (35-75).\n \n 2. We constrained another feature, party, to always display one Democrat and one Republican, so the code for this is a bit different. \n \n 3. We constrained two other features (race and religion) to use unequal sampling probabilities. This was to prevent huge numbers of \"rare\" profiles from appearing.\n\n \nThe code for the \"simple\" features is straightforward because we simply define arrays with all possible feature values:\n\n```js\n// Create Variables for Traits associated with each dimension.\nvar vsex = [\"Male\", \"Female\"];\nvar voccupation = [\"State Governor\", \"U.S. Senator\", \"Member of Congress\", \"CEO\"];\nvar vmilitary = [\"Served\", \"Did not serve\"];\nvar veduc = [\"Community college\", \"State university\", \"Small college\", \"Ivy League university\"];\nvar vopinion = [\"Strongly oppose\", \"Moderately oppose\", \"Slightly oppose\", \"Neither Support Nor Oppose\", \"Slightly support\", \"Moderately support\", \"Strongly support\"];\n```\n\nAnd the code for party is simple, too. Here we just randomly sort a length-2 array:\n\n```js\nif (Math.random() \u003e= 0.5) {\n    var vparty = [\"Republican\", \"Democrat\"];\n} else {\n    var vparty = [\"Democrat\", \"Republican\"];\n}\n```\n\nThe unequal sampling probabilities for race and religion involved creating functions that drew a random value between 0 and 100 and then used `if-else` statements to draw one of the possible values out of an array. The unequal sampling problems are expressed in the cut points across the 0-100 range. Note also that javascript is 0-indexed so that the first value in the array is extracted using `array[0]` (unlike say R).:\n\n```js\n// Functions for setting race and religion approximately proportionately\nfunction getRace(){\n  // 60% non-hispanic white; 15% black; 15% hispanic; 10% asian\n  var n = Math.floor(Math.random()*100);\n  if (n\u003c10) {\n    var out = 3;\n  } else if (n \u003c25) {\n    var out = 2;\n  } else if (n\u003c40) {\n    var out = 1;\n  } else {\n    var out = 0;\n  }\n  var vrace = [\"White\", \"African American\", \"Hispanic\", \"Asian American\"];\n  return vrace[out];\n}\nfunction getReligion(){\n  // 20% evangelical; 20% mainline; 20% catholic; 10% jewish; 10% muslim; 20% none\n  var n = Math.floor(Math.random()*100);\n  if (n\u003c20) {\n    var out = 5;\n  } else if (n\u003c30) {\n    var out = 4;\n  } else if (n\u003c40) {\n    var out = 3;\n  } else if (n\u003c60) {\n    var out = 2;\n  } else if (n\u003c80) {\n    var out = 1;\n  } else {\n    var out = 0;\n  }\n  var vreligion = [\"Evangelical protestant\", \"Mainline protestant\", \"Catholic\", \"Jewish\", \"Muslim\", \"None\"];\n  return vreligion[out];\n}\n```\n\nIn the below code you will also see how we generated a random continuous age value using:\n\n```js\n(Math.floor(Math.random() * (75 - 35 + 1)) + 35).toString()\n```\n\nWhich draws a random number between 0 and 1, converts that to the range 35-75, and converts the value to a string.\n\nPiecing all of that together, we produce an array of candidate traits for each candidate. We repeatedly sample from the `vopinion` array because we display candidates' opinions on five different issues, each o fwhich has the same set of possible values.\n\n\n```js\n// Use math.random to randomly select traits for each dimension for candidate A\ntraits_a = [(Math.floor(Math.random() * (75 - 35 + 1)) + 35).toString(),\n            vsex[Math.floor(Math.random()*vsex.length)],\n            getRace(),\n            getReligion(),\n            voccupation[Math.floor(Math.random()*voccupation.length)],\n            vparty[0],\n            vmilitary[Math.floor(Math.random()*vmilitary.length)],\n            veduc[Math.floor(Math.random()*veduc.length)],\n            vopinion[Math.floor(Math.random()*vopinion.length)],\n            vopinion[Math.floor(Math.random()*vopinion.length)],\n            vopinion[Math.floor(Math.random()*vopinion.length)],\n            vopinion[Math.floor(Math.random()*vopinion.length)],\n            vopinion[Math.floor(Math.random()*vopinion.length)] ];\n\n// Use math.random to randomly select traits for each dimension for candidate B\ntraits_b = [(Math.floor(Math.random() * (75 - 35 + 1)) + 35).toString(),\n            vsex[Math.floor(Math.random()*vsex.length)],\n            getRace(),\n            getReligion(),\n            voccupation[Math.floor(Math.random()*voccupation.length)],\n            vparty[1],\n            vmilitary[Math.floor(Math.random()*vmilitary.length)],\n            veduc[Math.floor(Math.random()*veduc.length)],\n            vopinion[Math.floor(Math.random()*vopinion.length)],\n            vopinion[Math.floor(Math.random()*vopinion.length)],\n            vopinion[Math.floor(Math.random()*vopinion.length)],\n            vopinion[Math.floor(Math.random()*vopinion.length)],\n            vopinion[Math.floor(Math.random()*vopinion.length)] ];\n```\n\nIn this code, the order of the feature values is critical because the order determines how we enter the values into the displayed table and how we store them in Qualtrics embedded data. Other approaches are possible, such as creating a variable for each candidate-trait combination, but we do not do that here.\n\n\n### Controlling randomization\n\nOne concern with using javscript to randomly sample is that javascript is [a client-side technology](https://en.wikipedia.org/wiki/Client-side), so the randomizations are executed by the respondent's browser rather than the Qualtrics server. This introduces a potentially rare but important risk in the design. If a respondent refreshes their browser, all of this code is re-executed such that a different set of profile features are shown to the respondent. This poses real analytic and user-experience risks, so we need to find a way to ensure that the displayed profiles are unaffected by any refreshes or browser mishaps.\n\nIn many languages that would be easy to achieve with a [seeded pseudo-random number generator](https://en.wikipedia.org/wiki/Random_seed), but javascript's PRNG is not seeded. Luckily, David Bau has written [seedrandom.js](https://github.com/davidbau/seedrandom/) to achieve this. Serious props to him!\n\nTo use it, we need to create some PRNG seeds - one for every profile - that are held constant for the user. To do that, we rely on Qualtrics random number [web service](https://www.qualtrics.com/support/survey-platform/survey-module/survey-flow/advanced-elements/web-service/) to create embedded data fields that store seeds for each respondent. Follow the directions on that link to get this setup in the Qualtrics Survey Flow (button at the top of the survey page). Create a \"Web Service\" entry with the following configuration:\n\n - URL: `http://reporting.qualtrics.com/projects/randomNumGen.php`\n - Setup `min` argument: `0`\n - Setup `max` argument: `99999999999` (or whatever seems reasonable for your project)\n - Setup \"Set Embedded Data\" field name: `seed1`\n - Setup \"Set Embedded Data\" service value: `random`\n\nYou will need to setup one of these \"Web Service\" entries for each conjoint profile, saving the seed as a unique embedded data field. So, if you have five conjoint profiles to display, you will need to setup five Web Service entries and five corresponding embedded data seed fields (`seed1`, `seed2`, `seed3`, `seed4`, `seed5`) that will be populated by the web service calls and retrieved via the javascript. The randomseed.js code will then be added to each conjoint profile question's javascript field, grabbing the right embedded data seed using the Qualtrics `${e://Field/EMBEDDEDDATAFIELD}` notation to set the seed for that profile page:\n\n```js\n// import seeded random number generator code\n!function(a,b){function c(c,j,k){var n=[];j=1==j?{entropy:!0}:j||{};var s=g(f(j.entropy?[c,i(a)]:null==c?h():c,3),n),t=new d(n),u=function(){for(var a=t.g(m),b=p,c=0;q\u003ea;)a=(a+c)*l,b*=l,c=t.g(1);for(;a\u003e=r;)a/=2,b/=2,c\u003e\u003e\u003e=1;return(a+c)/b};return u.int32=function(){return 0|t.g(4)},u.quick=function(){return t.g(4)/4294967296},u[\"double\"]=u,g(i(t.S),a),(j.pass||k||function(a,c,d,f){return f\u0026\u0026(f.S\u0026\u0026e(f,t),a.state=function(){return e(t,{})}),d?(b[o]=a,c):a})(u,s,\"global\"in j?j.global:this==b,j.state)}function d(a){var b,c=a.length,d=this,e=0,f=d.i=d.j=0,g=d.S=[];for(c||(a=[c++]);l\u003ee;)g[e]=e++;for(e=0;l\u003ee;e++)g[e]=g[f=s\u0026f+a[e%c]+(b=g[e])],g[f]=b;(d.g=function(a){for(var b,c=0,e=d.i,f=d.j,g=d.S;a--;)b=g[e=s\u0026e+1],c=c*l+g[s\u0026(g[e]=g[f=s\u0026f+b])+(g[f]=b)];return d.i=e,d.j=f,c})(l)}function e(a,b){return b.i=a.i,b.j=a.j,b.S=a.S.slice(),b}function f(a,b){var c,d=[],e=typeof a;if(b\u0026\u0026\"object\"==e)for(c in a)try{d.push(f(a[c],b-1))}catch(g){}return d.length?d:\"string\"==e?a:a+\"\\0\"}function g(a,b){for(var c,d=a+\"\",e=0;e\u003cd.length;)b[s\u0026e]=s\u0026(c^=19*b[s\u0026e])+d.charCodeAt(e++);return i(b)}function h(){try{if(j)return i(j.randomBytes(l));var b=new Uint8Array(l);return(k.crypto||k.msCrypto).getRandomValues(b),i(b)}catch(c){var d=k.navigator,e=d\u0026\u0026d.plugins;return[+new Date,k,e,k.screen,i(a)]}}function i(a){return String.fromCharCode.apply(0,a)}var j,k=this,l=256,m=6,n=52,o=\"random\",p=b.pow(l,m),q=b.pow(2,n),r=2*q,s=l-1;if(b[\"seed\"+o]=c,g(b.random(),a),\"object\"==typeof module\u0026\u0026module.exports){module.exports=c;try{j=require(\"crypto\")}catch(t){}}else\"function\"==typeof define\u0026\u0026define.amd\u0026\u0026define(function(){return c})}([],Math);\n\n// seed random number generator from embedded data fields\n// conjoint profile 1\nMath.seedrandom('${e://Field/seed1}');\n// conjoint profile 2\n//Math.seedrandom('${e://Field/seed2}');\n// conjoint profile 3\n//Math.seedrandom('${e://Field/seed3}');\n// conjoint profile 4\n//Math.seedrandom('${e://Field/seed4}');\n// conjoint profile 5\n//Math.seedrandom('${e://Field/seed5}');\n```\n\nI show all of the seed setting code above, with only the first line uncommented. For subsequent profiles, uncomment only the relevant seed setting line. All of this code has to come before everything above (again, see [conjoint.js](conjoint.js) for a complete working example).\n\nNow, with the seeds configured and used to seed the PRNGs, and the feature values randomly sampled, we can now use the Qualtrics [Javascript API](https://www.qualtrics.com/support/survey-platform/survey-module/question-options/add-javascript/) to [Question API](https://s.qualtrics.com/WRAPI/QuestionAPI/classes/Qualtrics%20JavaScript%20Question%20API.html) to modify the displayed question text and store the displayed values as a further embedded data field.\n\nOur javascript code includes two arrays that store the html `id`'s for each attribute value:\n\n```js\n// Create list of variables to use when setting attributes\na_list = [\"a1\",\"a2\",\"a3\",\"a4\",\"a5\",\"a6\",\"a7\",\"a8\",\"a9\",\"a10\",\"a11\",\"a12\",\"a13\"]; \nb_list = [\"b1\",\"b2\",\"b3\",\"b4\",\"b5\",\"b6\",\"b7\",\"b8\",\"b9\",\"b10\",\"b11\",\"b12\",\"b13\"]; \n```\n\nWe then loop over these (a loop from 0 to 12, given javascript's base 0 indexing), and set the \"innerHTML\" of html element `a_list[i]` (\"a1\", \"a2\", etc.) and `b_list[i]` (\"b1\", \"b2\", etc.) to the value of the traits in `traits_a[i]` and `traits_b[i]`, respectively.\n\n```js\n// set html values in conjoint table\nfor(i=0;i\u003c13;i++){\n    document.getElementById(a_list[i]).innerHTML = traits_a[i];\n    document.getElementById(b_list[i]).innerHTML = traits_b[i];\n}\n```\n\nThen, we need to save the displayed traits in embedded data fields so that we can access them later. This requires first defining the embedded data fields in the Qualtrics editor, leaving the values blank. We created one embedded data field for each candidate in each profile pair. The code `traits_a.join(\"|\")` binds the values of the trait arrays into a single pipe-separated character string and the `Qualtrics.SurveyEngine.setEmbeddedData()` function stores that string in the named embedded data field. Here's the code for every profile showing only the first profile pair uncommented:\n\n```js\n// store values as embedded data fields\nQualtrics.SurveyEngine.setEmbeddedData('traits1a', traits_a.join(\"|\"));\nQualtrics.SurveyEngine.setEmbeddedData('traits1b', traits_b.join(\"|\"));\n\n//Qualtrics.SurveyEngine.setEmbeddedData('traits2a', traits_a.join(\"|\"));\n//Qualtrics.SurveyEngine.setEmbeddedData('traits2b', traits_b.join(\"|\"));\n\n//Qualtrics.SurveyEngine.setEmbeddedData('traits3a', traits_a.join(\"|\"));\n//Qualtrics.SurveyEngine.setEmbeddedData('traits3b', traits_b.join(\"|\"));\n\n//Qualtrics.SurveyEngine.setEmbeddedData('traits4a', traits_a.join(\"|\"));\n//Qualtrics.SurveyEngine.setEmbeddedData('traits4b', traits_b.join(\"|\"));\n\n//Qualtrics.SurveyEngine.setEmbeddedData('traits5a', traits_a.join(\"|\"));\n//Qualtrics.SurveyEngine.setEmbeddedData('traits5b', traits_b.join(\"|\"));\n```\n\nThat's it on javascript!\n\n\n## 3. Setup and run the study\n\nOnce question is setup with the relevant html and javascript, it can simply be copied into a new question for each profile you want to show. The HTML doesn't need to be changed, but the javascript should be commented/uncommented to draw the relevant seeds and store the feature values in the relevant embedded data field.\n\nThere is one very important final issue that **must** be addressed for this to work correctly. In the \"Look \u0026 Feel\" settings of the survey, you have to set \"Page Transitions\" to \"none\"\n\n![look-and-feel](look-and-feel.png)\n\nThis prevents \"preloading\" of the javascript (and thus failure of the the whole thing).\n\n(Note: There is some argumentation online about whether setting an additional query string argument in the survey URL (specifically, `Q_JFE=0`) will solve this. In my experience, it does not; and it also has the consequence of making surveys behave weirdly on mobile devices. The page transitions is a pretty small price to pay for getting this to otherwise work.)\n\nWith that done, the survey should be good to go!\n\n## 4. Analyze\n\nThe resulting feature values come out in the output data as a character variable named for the embedded data fields. These look like:\n\n```\n60|Female|Asian American|Jewish|U.S. Senator|Republican|Served|Community college|Slightly support|Slightly support|Slightly oppose|Strongly support|Slightly oppose\n```\n\nYou can then use simple string manipulation to convert these to separate variables. In Stata it would be something like:\n\n```stata\nsplit trait1a, parse(|)\n```\n\nIn R, there are multiple strategies but something like:\n\n```R\ndo.call(rbind.data.frame, strsplit(d[[\"trait1a\"]], \"\\\\|\"))\n```\n\nwould give you a data frame that could be `cbind()`-ed on to the original data frame.\n\nWith that setup, you can simply analyze the data using regular regression analysis where the new variables are used as regressors.\n\n---\n\nThanks to Kyle Dropp for guidance on how to do this! And thanks to Isaac from Qualtrics support for help getting the page transition addressed in June 2017.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleeper%2Fconjoint-example","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fleeper%2Fconjoint-example","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleeper%2Fconjoint-example/lists"}