Questionnaires are excellent tools for data capture. They allow tight control over what data is gathered and ensure information is gathered consistently across multiple users. However, data gathered using different questionnaires - or even different versions of the same questionnaire - is often not comparible. It is also not very searchable or easily integrated with discrete data sources. Because of this, the general recommendation in FHIR is to use questionnaires for raw data capture but then to convert the resulting QuestionnaireResponse instances into other FHIR resources - Observations, MedicationStatements, FamilyMemberHistories, etc. This allows the data gathered to then be easily combined with other data into FHIR documents and messages and exposed over FHIR REST interfaces.

Such conversion can be done with custom code written on a Questionnaire by Questionnaire basis. However, it makes the process much easier if it's possible to write generic software that can convert any arbitrary QuestionnaireResponse into appropriate FHIR resources leveraging metadata embedded in the Questionnaire. This portion of the SDC guide defines mechanisms for doing so.

9.1 Caveats and considerations with form population

• Extraction is a step that only makes sense to occur once a QuestionnaireResponse is completed. Prior to that, the information to create valid resource instances may not be available and conversion logic is likely to fail. The types of data produced will vary by Questionnaire.
• Some questionnaires might result in a single resource. Others will produce a Bundle of resources. In some cases, the result might be a transaction intended to create some resources and update others.
• If the Questionnaire is being designed with conversion to a resource in mind, conversion will be a pretty straight-forward process because the questions will align well with how FHIR stores data - and the questions can even be tuned to align with particular profiles around the use of terminology, which elements are mandatory, etc. However, if converting data from questionnaires designed without FHIR in mind, mapping may be more challenging. Required elements may need to be inferred, codes may need to be transformed and other transformations may be necessary to ensure that converted data meets expectations for use, such as aligning with country-specific implementation guides. In some cases, alignment may not be possible.
• Once a QuestionnaireResponse has been converted, it might not be necessary to retain the QuestionnaireResponse any longer as all data access and subsequent maintenance will occur through the 'traditional' resources. However, in many environments the QuestionnaireResponse will be retained anyhow to keep a record of the original source-of-truth and for traceability reasons.
• When resources have been generated from a QuestionnaireResponse, the Provenance instance associated with the creation of the resource instance(s) can and should include an entity reference of type 'source' that points back to the original QuestionnaireResponse. If Observations are generated, they can also have an explicit derivedFrom link pointing back to the QuestionnaireRsponse
• In theory, it's possible to use Questionnaire as a user-facing interface to allow maintenance of one or more resources. The source data can be used to populate the QuestionnaireResponse and once the response is 'submitted', the data can then be extracted and used to update the existing resources. For this to work, the id of the resource must be retained through the round-trip process to allow for the update. This can be supported by storing the id as a hidden readOnly question not shown to the user.
• Sometimes the author interested in making a Questionnare "extractable" doesn't have authority to make changes to the "official" Questionnaire. In other cases, there might be one official Questionnaire, but a need to create extracted resources that comply with different sets of profiles - and thus a need for different metadata in the Questionnaire to support the extraction process. In this case, rather than basing the extraction on the original Questionnaire, it can be based on a derived Questionnaire - one that has a Questionnaire.derivedFrom relationship to the same canonical URL the QuestionnaireResponse refers to. The derived Questionnaire would contain the same content as the base Questionnaire, but would have additional extensions inserted to support data extraction.

9.2 Extraction service

Like Questionnaire population, extracting data from a QuestionnaireResponse is a complex process involving querying existing FHIR data and using more advanced technologies such as FHIRPath and StructureMap. It's therefore a function that systems may also wish to offload to a separate system. The Questionnaire extract has been created for this purpose. It takes in a completed QuestionnaireResponse and returns either an individual FHIR resource or a Bundle of resources, depending on the type of Questionnaire. The operation doesn't actually post the created resources to a server. It's up to the client system to determine what action(s) to take with the created content.

NOTE: It's the responsibility of the client system to ensure that any generated resources are valid against necessary profiles, etc. before using content produced by this operation.

9.1 Designing Questionnaires to support data extraction

This specification defines three different mechanisms to embed information in Questionnaires to support subsequent resource extraction:

• Observation-based
• definition-based
• StructureMap-based

Systems are free to experiment with other extraction mechanisms but cannot expect support for those from other SDC-conformant systems.

The sdc-questionnaire-extract profile includes the data elements and extensions relevant for supporting all of these mechanisms. None of the additional resource elements or extensions are marked as "mustSupport" because there is no expectation that systems will support all of (or any of) the different extraction mechanisms. Instead, each system should choose which approach(es) it wishes to use and support the elements described in that section of the implementation guide.

9.1.1 Observation-based extraction

This is the simplest of the extraction mechanisms. It leverages the same data elements as are used for the Observation-based population mechanism. It takes advantage of the fact that the vast majority of questions in the healthcare space typically correspond to the value element of an Observation. It also takes advantage of the Questionnaire.item.code element that identifies what a concept each question or group corresponds to.

To use this method:

1. Include the item.code element on each question to be populated. Typically this will be a LOINC code, but in some jurisdictions/environments, SNOMED CT or other codes may be relevant
2. Groups can also have a item.code present - this might represent the code of the a panel or the Observation.code of an Observation with no value but with multiple Observation.component elements. Child question items can then assert the item.code of the "member-of" Observations or the Observation.component.code values
3. To signal that the item.code is actually intended for use in extraction (as opposed to just providing metadata about the Questionnaire item, the questionnaire-observationLinkPeriod extension must also be included. This extension indicates the period of time over which to search for matching observations. If there are no observations within that window, no population will occur and any data entered will be treated as a create. For observations where how recent they are doesn't matter (e.g. blood type), simply set the duration to a long period of time - e.g. 200 years.
4. Multiple item.code elements might be present. If so, each are considered to be one of the Observation.code codings in the resulting extracted Observation.

For example:

<item> <extension url="http://hl7.org/fhir/StructureDefinition/questionnaire-observationLinkPeriod"> <valueDuration> <value value="3"/> <system value="http://unitsofmeasure.org"/> <code value="mo"/> </valueDuration> </extension> <linkId value="code-pop-demo"/> <code> <system value="http://loinc.org"/> <code value="29463-7"/> <display value="Body weight"/> </code> <code> <system value="http://loinc.org"/> <code value="3141-9"/> <display value="Body weight Measured"/> </code> <code> <system value="http://loinc.org"/> <code value="8341-0"/> <display value="Dry body weight Measured"/> </code> <text value="What is your current weight?"/> <type value="quantity"/> <answerOption> <valueCoding> <system value="http://unitsofmeasure.org"/> <code value="kg"/> </valueCoding> </answerOption> <answerOption> <valueCoding> <system value="http://unitsofmeasure.org"/> <code value="[lb_av]"/> </valueCoding> </answerOption> </item>

When performing the extraction process, the system will create a batch that will contain creates or updates of Observation instances. It will go through the QuestionnaireResponse and identify all answers for which the corresponding Questionnaire item has a questionnaire-observationLinkPeriod extension. For each of those it will then determine whether to create a new observation, update an existing observation or do nothing. Guidelines for making this decision are as follows:

Update if:	the answer was populated from an existing Observation; the system rendering rendering the QuestionnaireResponse is able to retain context and knows the 'id' of Observation to update; the author of the original Observation is the same as the current author of the QuestionnaireResponse the answer has changed from the populated value; and the context of the questionnaire and the use of it is one where updates are appropriate - as opposed to asserting a new Observation with a new performer and date
Take no action if:	the answer was populated from an existing Observation; the system rendering rendering the QuestionnaireResponse is able to retain context and knows the 'id' of Observation to update; the author of the original Observation is the same as the current author of the QuestionnaireResponse the answer has notchanged from the populated value; and the context of the questionnaire and the use of it is one where updates are appropriate - as opposed to asserting a new Observation with a new performer and date
Create a new Observation if:	the conditions in the preceding two rows are not met

If updating, simply adjust the original Observation to have the new value and change the status to "amended", then PUT it to the source system. If creating, data elements should be populated as follows:

• Observation.basedOn and Observation.partOf - copy from QuestionnaireResponse elements of the same name
• Observation.status - set to 'final'
• Observation.category - if this can be inferred from any of the Questionnaire.item.code values or from known context of the Questionnaire itself, then fill it in, otherwise ommit.
• Observation.code - add all of the Questionnaire.item.code values as Observation.code.coding instances
• Observation.subject - set to QuestionnaireResponse.subject
• Observation.encounter - set to QuestionnaireResponse.context (if an Encounter)
• Observation.effectiveDateTime - set to QuestionnaireResponse.authored.
  
  Note, this is an inference. It is important that the question text implies that the value is 'current' not 'historical' for this to be safe - otherwise don't include the 'observationLinkPeriod' extension that marks the question as appropriate for population and extraction.
• Observation.issued - set to QuestionnaireResponse.authored
• Observation.performer - set to QuestionnaireResponse.author
• Observation.value[x] - set to QuestionnaireResponse.item.answer.value[x]
• Observation.derivedFrom - set to a reference to the QuestionnaireResponse
• Observation.interpretation and Observation.referenceRange - if these can be inferred from the QuestionnaireResponse.item.code (and for interpretation the answer value too), they can be populated, otherwise omit

If the Questionnaire.item that is linked to an Observation contains child items that are also linked to Observations, then things get more complex as a determination will need to be made on whether to link the parent to child as Observation.component or as Observation.hasMember. In the ideal situation, the system will recognize the Observation.item.code and know which approach is correct for that type of Observation. If not, then the system could query for other records of the child type and see if they appear as components anywhere. If unsure, systems should use "hasMember".

Considerations when using this approach:

• If a questionnaire item has the questionnaire-unit extension, the Observation.value should be a valueQuantity rather than integer or decimal and the units should be taken from the extension value.
• If a question is skipped (no answer) or cleared, no Observation should be created. Existing Observations SHALL NOT be deleted
• If a question has multiple answers, each answer will be a separate Observation instance
• There is no mechanism to support items that are mapped to Observation codes which then have nested items without codes - e.g. to capture the text description for an "other - please specify" code - one of the other extraction mechanisms will need to be used
• Implementers are free to try combining this mechanism with the Definition-based approach. If they do, they should take care that a given item (and its children) are only handled by one approach or the other - not both.
• This approach does not allow for observations where Observation.focus is relevant or for capturing Observation.dataAbsentReason
• Where an Observation is known to directly correlate to another resource element value (e.g. LOINC 21112-8 corresponds to Patient.birthDate), systems MAY take advantage of this knowledge to update the value of resources other than Observations, however such use is discouraged - using one of the other extraction techniques is likely better and safer.
• Obviously, this mechanism only works for questionnaire items that correspond to Observation values.

9.1.2 Definition-based extraction

This approach to population is more generic. It supports extracting data into any type of FHIR resource rather than being limited to only Observation. It also supports more Observation data elements than can be gathered using the Observation element - for example, explicit effective time ranges, interpretations, comments, etc.

To use this method:

1. Include the questionnaire-itemContext extension either on the Questionnaire root or on 'group' items within the Questionnaire to identify the resource that will serve as the context for any population.
2. On descendant items of that element, fill in the Questionnaire.definition to point to the resource or profile element that Questionnaire item corresponds to. (Profiles may be relevant for data that is sliced or has fixed values for some properties.). The definition SHALL have the full canonical URL of the resource (or profile) followed by '#' followed by the snapshot.path of the element the Questionnaire item corresponds to.
3. If necessary, define questionnaire-hidden items that have Questionnaire.initialValue[x] or that use the questionnaire-initialExpression extension to define their content to use to populate resource elements that the user will not be filling in. (The initialExpressions might in turn depend on variable and questionnaire-context extensions, used as described in the Path-based population section.

To perform the population process, first determine whether the context resource should be updated or created:

• If the context resource existed and was used in the population of the resource and data has changed, the resource will typically be updated. (Note that this means the system must capture the resource ids as hidden items in the Questionnaire so they're available for update.)
• In other cases, the record will be created

If updating, the answer values from the questionnaire (including hidden, calculated answers) will be propagated to the context resource. If creating, then for each occurrence of each item asserting a questionnaire-context, a resource of the context type will be defined. It will be populated with answers to questions (hidden or not) that have a definition that matches the resource type of the context. As well, if the definitions refer to a profile, any child elements of the profile that have fixed values or patterns declared will also be included in the instance with values matching the fixed or pattern values.

​ An example of a questionnaire using this approach can be found here.

Considerations when using this approach:

• FHIR queries found in the questionnaire-context variables may contain embedded FHIRPath expressions (surrounded by double curly-braces). Systems SHALL evaluate and substitute the results of such queries before executing them
• If the result of evaluating the FHIRPath expressions is an invalid query, that is an error. Systems SHOULD log it and continue on with population as if the query had returned no data.
• When crafting queries, be sure to filter all relevent elements. For example, ensuring status excludes entered-in-error elements, practitioners are active, etc.
• In some cases, the context of an element will switch. For example, an item might have a definition of "http://hl7.org/fhir/MedicationStatement#MedicationStatement.medicationReference" and a questionnaire-context extension that shifts the context to "Medication". Subsequent children would then uave definitions based on the "http://hl7.org/fhir/Medication" resource.
• Note that only one context can be in play at the same time. When a new context is declared, it takes the place of the old context.
• The items in the Questionnaire might not have the same order as those in the resource. When serializing into XML, the official order must still be respected.

9.1.3 StructureMap-based extraction

The StructureMap approach is the most sophisticated approach of the three - and the most powerful. It allows significant transformation of data, including code translations when generating output resources. It also allows the conversion process between data and Questionnaire to be maintained independently and to draw on shared sources across Questionnaires. This can be an advantage in certain environments where the content of the questionnaire may need tight control but the data environment can be more dynamic. This comes at the cost of requiring expertise in the FHIR mapping language, which is not (yet?) a common skill.

To use this method:

1. Include the questionnaire-targetStructureMap extension. This SHALL define a transform between the QuestionnaireResponse and either a single resource or a transaction Bundle containing the set of resources extracted from the QuestionnaireResponse.

​ To extract data from the completed QuestionnaireResponse, simply invoke execute the StructureMap on it. A sample Questionnaire and associated StructureMap that shows this approach can be found here.

Considerations when using this approach:

• This mode has the drawback that if the StructureMap execution fails, there will generally not be any data extracted from the Questionnaire. With the other approaches, if one Observation or context fails, the others might still work. As a result, the StructureMap must be designed to be very robust in the face of missing or potentially 'bad' data.
• The ability of StructureMaps to reference other StructureMaps allows for the possibility of re-use if certain sections of multiple questionnaires are consistent