Analyzing FHIR Data in a Tabular Format With Python
Learning objectives
Understand the high-level approaches for converting FHIR-formatted data into tabular for analysis in Python.
Learn how the FHIR-PYrate library facilitates requesting data from a FHIR server, and creating tidy tabular data tables.
Relevant roles:
Informaticist
Data analysis approaches in Python often use Pandas DataFrames to store tabular data. There are two primary approaches to loading FHIR-formatted data into Pandas DataFrames:
Writing Python code to manually convert FHIR instances in JSON format into DataFrames.
This does not require any special skills beyond data manipulation in Python, but in practice can be laborious (especially with large number of data elements) and prone to bugs.
Using a purpose-built library like FHIR-PYrate to automatically convert FHIR instances into DataFrames.
It is recommended to try this approach first, and only fall back to (1) if needed.
To use FHIR-PYrate, you will need a Python 3 runtime with FHIR-PYrate and Pandas installed.
However, any FHIR server loaded with testing data can be used. See Standing up a FHIR Testing Server for instructions to set up your own test server.
The code blocks in the following section show sample output immediately after. This is similar to the code cells and results in a Jupyter notebook.
2 Retrieving FHIR data
Once your environment is set up, you can run the following Python code to retrieve instances of the Patient resource from a test server:
# Load dependenciesfrom fhir_pyrate import Pirateimport pandas as pd# Instantiate a Pirate object using the FHIR-PYrate library to query a test FHIR serversearch = Pirate( auth=None, base_url="https://api.logicahealth.org/FHIRResearchSynthea/open/", print_request_url=True,)# Use the whimsically named `steal_bundles()` method to instantiate a search interaction## For more information, see https://github.com/UMEssen/FHIR-PYrate/#piratebundles = search.steal_bundles( resource_type='Patient', request_params={"_count": 10# Get 10 instances per page }, num_pages=1# Get 1 page (so a total of 10 instances))# Execute the search and convert to a Pandas DataFramedf = search.bundles_to_dataframe(bundles)df.head(5)
It is easier to see the contents of this DataFrame by printing out its first row vertically:
# Print the first row of the DataFrame vertically for easier reading.pd.set_option('display.max_rows', 100) # Show all rowsdf.head(1).T
0
resourceType
Patient
id
1
meta_versionId
1
meta_lastUpdated
2023-04-06T20:52:11.000+00:00
meta_source
#wQwWCylvgEiNKNbB
text_status
generated
text_div
<div xmlns="http://www.w3.org/1999/xhtml">Gene...
extension_0_url
http://hl7.org/fhir/us/core/StructureDefinitio...
extension_0_extension_0_url
ombCategory
extension_0_extension_0_valueCoding_system
urn:oid:2.16.840.1.113883.6.238
extension_0_extension_0_valueCoding_code
2054-5
extension_0_extension_0_valueCoding_display
Black or African American
extension_0_extension_1_url
text
extension_0_extension_1_valueString
Black or African American
extension_1_url
http://hl7.org/fhir/us/core/StructureDefinitio...
extension_1_extension_0_url
ombCategory
extension_1_extension_0_valueCoding_system
urn:oid:2.16.840.1.113883.6.238
extension_1_extension_0_valueCoding_code
2186-5
extension_1_extension_0_valueCoding_display
Not Hispanic or Latino
extension_1_extension_1_url
text
extension_1_extension_1_valueString
Not Hispanic or Latino
extension_2_url
http://hl7.org/fhir/StructureDefinition/patien...
extension_2_valueString
Delois358 Hintz995
extension_3_url
http://hl7.org/fhir/us/core/StructureDefinitio...
extension_3_valueCode
M
extension_4_url
http://hl7.org/fhir/StructureDefinition/patien...
extension_4_valueAddress_city
Lawrence
extension_4_valueAddress_state
Massachusetts
extension_4_valueAddress_country
US
extension_5_url
http://synthetichealth.github.io/synthea/disab...
extension_5_valueDecimal
0.098274
extension_6_url
http://synthetichealth.github.io/synthea/quali...
extension_6_valueDecimal
62.901726
identifier_0_system
https://github.com/synthetichealth/synthea
identifier_0_value
439b24b4-6f25-4093-b101-47a39bd061ca
identifier_1_type_coding_0_system
http://terminology.hl7.org/CodeSystem/v2-0203
identifier_1_type_coding_0_code
MR
identifier_1_type_coding_0_display
Medical Record Number
identifier_1_type_text
Medical Record Number
identifier_1_system
http://hospital.smarthealthit.org
identifier_1_value
439b24b4-6f25-4093-b101-47a39bd061ca
identifier_2_type_coding_0_system
http://terminology.hl7.org/CodeSystem/v2-0203
identifier_2_type_coding_0_code
SS
identifier_2_type_coding_0_display
Social Security Number
identifier_2_type_text
Social Security Number
identifier_2_system
http://hl7.org/fhir/sid/us-ssn
identifier_2_value
999-57-3355
identifier_3_type_coding_0_system
http://terminology.hl7.org/CodeSystem/v2-0203
identifier_3_type_coding_0_code
DL
identifier_3_type_coding_0_display
Driver's License
identifier_3_type_text
Driver's License
identifier_3_system
urn:oid:2.16.840.1.113883.4.3.25
identifier_3_value
S99925942
identifier_4_type_coding_0_system
http://terminology.hl7.org/CodeSystem/v2-0203
identifier_4_type_coding_0_code
PPN
identifier_4_type_coding_0_display
Passport Number
identifier_4_type_text
Passport Number
identifier_4_system
http://standardhealthrecord.org/fhir/Structure...
identifier_4_value
X42032818X
name_0_use
official
name_0_family
Moen819
name_0_given_0
Willian804
name_0_prefix_0
Mr.
telecom_0_system
phone
telecom_0_value
555-135-7303
telecom_0_use
home
gender
male
birthDate
1955-10-09
address_0_extension_0_url
http://hl7.org/fhir/StructureDefinition/geoloc...
address_0_extension_0_extension_0_url
latitude
address_0_extension_0_extension_0_valueDecimal
42.319305
address_0_extension_0_extension_1_url
longitude
address_0_extension_0_extension_1_valueDecimal
-71.173653
address_0_line_0
545 Tromp Port Unit 55
address_0_city
Needham
address_0_state
Massachusetts
address_0_postalCode
02492
address_0_country
US
maritalStatus_coding_0_system
http://terminology.hl7.org/CodeSystem/v3-Marit...
maritalStatus_coding_0_code
M
maritalStatus_coding_0_display
M
maritalStatus_text
M
multipleBirthBoolean
False
communication_0_language_coding_0_system
urn:ietf:bcp:47
communication_0_language_coding_0_code
en-US
communication_0_language_coding_0_display
English
communication_0_language_text
English
If you look at the output above, you can see FHIR-PYrate collapsed the hierarchical FHIR data structure into DataFrame columns. FHIR-PYrate does this by taking an element from the FHIR-formatted data like Patient.identifier[0].value and converting to an underscore-delimited column name like identifier_0_value. (Note that Patient.identifier has multiple values in the FHIR data, so there are multiple identifier_N_... columns in the DataFrame.)
3 Selecting specific columns
Usually not every single value from a FHIR instance is needed for analysis. There are two ways to get a more concise DataFrame:
Use the approach above to load all elements into a DataFrame, remove the unneeded columns, and rename the remaining columns as needed. The process_function capability in FHIR-PYrate allows you to integrate this approach into the bundles_to_dataframe() method call.
Use FHIRPath to select specific elements and map them onto column names.
The second approach is typically more concise. For example, to generate a DataFrame like this…
id
gender
date_of_birth
marital_status
…
…
…
…
…you could use the following code:
# Instantiate and perform the FHIR search interaction in a single function calldf = search.steal_bundles_to_dataframe( resource_type='Patient', request_params={"_count": 10# Get 10 instances per page }, num_pages=1, # Get 1 page (so a total of 10 instances) fhir_paths=[ ("id", "identifier[0].value"), ("gender", "gender"), ("date_of_birth", "birthDate"), ("marital_status", "maritalStatus.coding[0].code") ])df
While FHIRPath can be quite complex, its use in FHIR-PYrate is often straight forward. Nested elements are separated with ., and elements with multiple sub-values are identified by [N] where N is an integer starting at 0. The element paths can typically be constructed by loading all elements into a DataFrame and then manually deriving the FHIRPaths from the column names, or by looking at the hierarchy resource pages in the FHIR specification (see Key FHIR Resources for more information on reading the FHIR specification).
4 Elements with multiple sub-values
There are multiple identifier[N].value values for each instance of Patient in this dataset.
# Instantiate and perform the FHIR search interaction in a single function calldf = search.steal_bundles_to_dataframe( resource_type='Patient', request_params={"_count": 10# Get 10 instances per page }, num_pages=1, # Get 1 page (so a total of 10 instances) fhir_paths=[ ("id", "identifier[0].value"), ("identifiers", "identifier.value") ])df
This will give you separate identifier_0, identifier_1, … columns for each Patient.identifier[N] value.
5 Retrieving related data
To retrieve instances of related resources, additional request_params can be added. See Using the FHIR API to Access Data for more information on constructing the parameters for FHIR search interactions.
In the example below, instances of Patient and instances of related Observation resources are requested:
# Instantiate and perform the FHIR search interaction in a single function calldfs = search.steal_bundles_to_dataframe( resource_type='Patient', request_params={# Get instances of Observation where `Observation.patient` refers to a fetched Patient instance"_revinclude": "Observation:patient","_count": 10# Get 10 instances per page }, num_pages=1, # Get 1 page (so a total of 10 instances))# `dfs` is a dictionary where the key is the FHIR resource type, and the value is the DataFrame## Split these into separate variables for easy access:df_patients = dfs['Patient']df_observations = dfs['Observation']# Look at the first row of the Observations DataFramedf_observations.head(1).T
To work around this, you can also iterate over all the rows in a DataFrame and request related resources using trade_rows_for_bundles():
df_observations2 = search.trade_rows_for_dataframe( df_patients, resource_type="Observation",# Limit to 10 instances of Observation per patient request_params={"_count": "10" }, num_pages=1,# Load Observations where `Observation.subject` references the instance of Patient# identified by `id` in the `df_patients` DataFrame df_constraints={"subject": "id"}, fhir_paths=[ ("observation_id", "id"), ("patient", "subject.reference"), ("status", "status"), ("code", "code.coding[0].code"), ("code_display", "code.coding[0].display"), ("value", "valueQuantity.value"), ("value_units", "valueQuantity.unit"), ("datetime", "effectiveDateTime") ])# Look at the first row of the Observations DataFramedf_observations2.head(15)
Note that this will only display value for instances of Observation that record a value in Observation.valueQuantity. Typically, you would filter by Observation.code and then choose the appropriate data type for Observation.value[x] to import. For example, http://loinc.org|72166-2 is the LOINC for smoking status. To get smoking status records for all patients in df_patients:
df_observations2 = search.trade_rows_for_dataframe( df_patients, resource_type="Observation",# Limit to 10 instances of Observation per patient request_params={"_count": "10","code": "http://loinc.org|72166-2" }, num_pages=1,# Load Observations where `Observation.subject` references the instance of Patient# identified by `id` in the `df_patients` DataFrame df_constraints={"subject": "id"}, fhir_paths=[ ("observation_id", "id"), ("patient", "subject.reference"), ("status", "status"), ("code", "code.coding[0].code"), ("code_display", "code.coding[0].display"), ("value", "valueCodeableConcept.coding[0].code"), ("value_display", "valueCodeableConcept.coding[0].display"), ("datetime", "effectiveDateTime") ])# Look at the first row of the Observations DataFramedf_observations2.head(15)
NIH’s Office of Data Science Strategy has online exercises for converting FHIR-formatted data into tabular format for further analysis. These exercises include implementations in both Python and R.
Note that the Python version of these exercises was created before the FHIR-PYrate library existed, so they use the “writing Python code to manually convert FHIR instances in JSON Format into DataFrames” approach. This approach may still be useful for use cases that are not well-supported by FHIR-PYrate or for integrating FHIR data with data from other web APIs.