Publish-box (todo)

3.2.6.1 Asynchronous Bulk Data Request Pattern

FHIR Infrastructure Work Group

Maturity Level: 4

Standards Status: Draft

3.2.6.1.1 Use Case

All of the interactions defined in the RESTful API are described as synchronous operations - that is, the client makes a query and waits for the server to respond with the outcome in the HTTP response. This pattern is not always suitable when substantial amounts of data must be returned.

A good example of this is the $export operation , where simple searches may result in large amounts of data.

The asynchronous request pattern, based on rfc 7240 , caters to this use case and is applicable for for use in Operations and Defined Interactions interactions. Servers may choose which interactions the pattern should be supported on (if at all).

3.2.6.1.1.1 Related Pattern: Asynchronous Interaction Request

For use cases that return small amounts of data but may take a lot of time to process, see Asynchronous Interaction Request.

3.2.6.1.2 Kick-off Request

The request will support the HTTP methods, URLs, headers and other parameters that normally apply, but servers SHALL also support the Prefer header described below, and SHOULD support the Accept header and _outputFormat parameter described below.

3.2.6.1.2.0.1 Headers

• Accept (string)

  Specifies the format of the optional OperationOutcome Resource response to the kick-off request. A client SHOULD provide this header. A server may support any subset of the Serialization Format Representations. If omitted, the server MAY return an error or MAY process the request and return a format selected by the server.

• Prefer (string, required)

  Specifies whether the response is immediate or asynchronous. Setting this to respond-async triggers this async pattern.

3.2.6.1.2.0.2 Parameters

• _outputFormat (string, optional, defaults to application/fhir+ndjson)

  The format for the generated bulk data files. Currently, ndjson must be supported, though servers may choose to also support other output formats. Servers SHALL support the full content type of application/fhir+ndjson as well as abbreviated representations including application/ndjson and ndjson.

  For requests types where the server supports either the Asynchronous Bulk Data Request pattern or the Asynchronous Interaction Request pattern, requests that include the _outputFormat parameter SHALL trigger the Asynchronous Bulk Data Request pattern.

3.2.6.1.2.0.3 Response - Success

• HTTP Status Code of 202 Accepted
• Content-Location header with the absolute URL of an endpoint for subsequent status requests (polling location)
• Optionally, a OperationOutcome Resource in the body

3.2.6.1.2.0.4 Response - Error (e.g. unsupported search parameter)

• HTTP Status Code of 4XX or 5XX
• The body SHALL be a OperationOutcome Resource

3.2.6.1.3 Delete Request

After a Bulk Data request has been started, a client MAY send a http DELETE request to the URL provided in the Content-Location header to cancel the request. If the request has been completed, a server MAY use the request as a signal that a client is done retrieving files and that it is safe for the sever to remove those from storage. Following the delete request, when subsequent requests are made to the polling location, the server SHALL return a 404 Not Found error and an associated OperationOutcome Resource.

3.2.6.1.3.0.1 Response - Success

• HTTP Status Code of 202 Accepted
• Optionally a FHIR OperationOutcome in the body

3.2.6.1.3.0.2 Response - Error

• HTTP status code of 4XX or 5XX
• Optionally a OperationOutcome Resource in the body

3.2.6.1.4 Status Request

After a Bulk Data request has been started, the client MAY poll the status URL provided in the Content-Location header by issuing HTTP GET requests to the location.

A client SHOULD follow an exponential backoff approach when polling for status. A server SHOULD supply a Retry-After header with a with a delay time in seconds (e.g., 120 to represent two minutes) or a http-date (e.g., Fri, 31 Dec 1999 23:59:59 GMT). When provided, a client SHOULD use this information to inform the timing of future polling requests. The server SHOULD keep an accounting of status queries received from a given client, and if a client is polling too frequently, the server SHOULD respond with a 429 Too Many Requests status code in addition to a Retry-After header, and optionally a OperationOutcome Resource with further explanation. If excessively frequent status queries persist, the server MAY return a 429 Too Many Requests status code without returning a status answer. It may either return a Retry-after header with the 429 indicating the amount of time that needs to pass before it will again respond to a status check, or it may choose to abandon the asynchronous operation entirely and force a retry. Other standard HTTP 4XX as well as 5XX status codes may be used to identify errors as mentioned.

When requesting status, the client SHOULD use an Accept header indicating a content type of application/json. In the case that errors prevent the asynchronous operation from completing, the server SHOULD respond with a OperationOutcome Resource in JSON format.

3.2.6.1.4.0.1 Response - In-Progress

• HTTP Status Code of 202 Accepted
• Optionally an X-Progress header with a text description of the status of the request that is less than 100 characters. The format of this description is at the server's discretion and MAY be a percentage complete value, or MAY be a more general status such as "in progress". The client MAY parse the description, display it to the user, or log it.

3.2.6.1.4.0.2 Response - Error

• HTTP status code of 4XX or 5XX
• Content-Type header of application/fhir+json when body is a OperationOutcome Resource
• The body of the response SHOULD be a OperationOutcome Resource in JSON format. If this is not possible (for example, the infrastructure layer returning the error is not FHIR aware), the server MAY return an error message in another format and include a corresponding value for the Content-Type header.

In the case of a polling failure that does not indicate failure of the original request, a server SHOULD use a transient code from the IssueType valueset when populating the OperationOutcome Resource issue.code element to indicate to the client that it should retry the request at a later time.

Note: Even if some of the requested resources cannot successfully be returned, the overall request MAY still succeed. In this case, the Response.error array of the completion response body SHALL be populated with one or more files in ndjson format containing OperationOutcome Resources to indicate what went wrong (see below). In the case of a partial success, the server SHALL use a 200 status code instead of 4XX or 5XX. The choice of when to determine that a job has failed in its entirety (error status) vs. returning a partial success (complete status) is left up to the server implementer.

3.2.6.1.4.0.3 Response - Complete

• HTTP status of 200 OK
• Content-Type header of application/json
• The server SHOULD return an Expires header indicating when the files listed will no longer be available for access.
• A body containing a JSON object providing metadata, and links to the generated Bulk Data files. The files SHALL be accessible to the client at the URLs advertised. These URLs MAY be served by file servers other than a FHIR-specific server.

Fields:

{
 "resourceType": "Bundle",
 "id": "bundle-transaction",
 "meta": {"lastUpdated: "2020-04-27T02:56:00Z},
 "type": "transaction",
 "entry":[{
  "request": {"method": "DELETE", "url": "Patient/123"}
  ...
 }]
}

Example response body:

{
  "transactionTime": "2021-01-01T00:00:00Z",
  "request" : "https://example.com/fhir/Patient/$export?_type=Patient,Observation",
  "requiresAccessToken" : true,
  "output" : [{
    "type" : "Patient",
    "url" : "https://example.com/output/patient_file_1.ndjson"
  },{
    "type" : "Patient",
    "url" : "https://example.com/output/patient_file_2.ndjson"
  },{
    "type" : "Observation",
    "url" : "https://example.com/output/observation_file_1.ndjson"
  }],
  "deleted": [{
    "type" : "Bundle",
    "url" : "https://example.com/output/del_file_1.ndjson"      
  }],
  "error" : [{
    "type" : "OperationOutcome",
    "url" : "https://example.com/output/err_file_1.ndjson"
  }],
  "extension":{"https://example.com/extra-property": true}
}

3.2.6.1.5 Output File Request

Using the URLs supplied by the FHIR server in the Complete Status response body, a client MAY download the generated Bulk Data files (one or more per resource type) within the time period specified in the Expires header (if present). If the requiresAccessToken field in the Complete Status body is set to true, the request SHALL include a valid access token.

As long as servers are following relevant security guidance, they MAY choose to generate output manifests where requiresAccessToken is true or false; this applies even for servers available on the public internet. When requiresAccessToken is set to false and no additional authorization-related extensions are present on the completion manifest's output entry, then the output URLs SHALL be dereferenceable directly, and SHALL be short lived following the expiration timing requirement that have been documented for bearer tokens in the SMART Backend Services Authorization Guide . In this case, a client MAY re-fetch the output manifest if output links have expired and MAY use the Expires header on the output response, when present, as a hint to know when capability URLs will expire.

The output data SHALL include only the most recent version of any FHIR resources unless the client explicitly requests different behavior in a fashion supported by the server (e.g., via a new query parameter yet to be defined). Inclusion of the Resource.meta information in the resources is at the discretion of the server (as it is for all FHIR interactions).

References in the resources returned MAY be relative URLs with the format resourceType/id, or MAY be absolute URLs with the same structure rooted in the base URL for the server from which the asynchronous operation was performed.

Example NDJSON output file:

{"resourceType":"Patient","id":"5c41cecf-cf81-434f-9da7-e24e5a99dbc2","name":[{"given":["Brenda"],"family":["Jackson"]}],"gender":"female","birthDate":"1956-10-14T00:00:00.000Z"}
{"resourceType":"Patient","id":"3fabcb98-0995-447d-a03f-314d202b32f4","name":[{"given":["Bram"],"family":["Sandeep"]}],"gender":"male","birthDate":"1994-11-01T00:00:00.000Z"}
{"resourceType":"Patient","id":"945e5c7f-504b-43bd-9562-a2ef82c244b2","name":[{"given":["Sandy"],"family":["Hamlin"]}],"gender":"female","birthDate":"1988-01-24T00:00:00.000Z"}

3.2.6.1.5.0.1 Headers

• Accept (optional, defaults to application/fhir+ndjson)

Specifies the format of the file being requested.

3.2.6.1.5.0.2 Response - Success

• HTTP status of 200 OK
• Content-Type header that matches the file format being delivered. For files in ndjson format, SHALL be application/fhir+ndjson
• Body of FHIR resources in newline delimited json - ndjson or other requested format

3.2.6.1.5.0.3 Response - Error

• HTTP Status Code of 4XX or 5XX

3.2.6.1.5.0.4 Attachments

If resources in an output file contain elements of the type Attachment, the server SHOULD populate the Attachment.contentType code as well as either the data element or the url element. When populated, the url element SHALL be an absolute url that can be de-referenced to the attachment’s content.

When the url element is populated with an absolute URL and the requiresAccessToken field in the Complete Status body is set to true, the url location must be accessible by a client with a valid access token, and SHALL NOT require the use of additional authentication credentials. When the url element is populated and the requiresAccessToken field in the Complete Status body is set to false and no additional authorization-related extensions are present on a the Complete Status body's output entry, the url location SHALL be dereferenceable directly, and SHALL be short lived following the expiration timing requirement that have been documented for bearer tokens in the SMART Backend Services Authorization Guide .

Note that if a server copies files to the Bulk Data output endpoint or proxies requests to facilitate access from this endpoint, it may need to modify the Attachment.url element when generating the Bulk Data output files.