Analyzing FHIR Data in a Tabular Format With Python

min. reading time 

Not yet updated for 2026

Roles: Informaticist

Learning objectives

1. Understand the high-level approaches for converting FHIR-formatted data into tabular for analysis in Python.
2. Learn how the FHIR-PYrate library facilitates requesting data from a FHIR server, and creating tidy tabular data tables.

Data analysis approaches in Python often use Pandas DataFrames to store tabular data. There are two primary approaches to loading FHIR-formatted data into Pandas DataFrames:

1. Writing Python code to manually convert FHIR instances in JSON format into DataFrames.

   This does not require any special skills beyond data manipulation in Python, but in practice can be laborious (especially with large number of data elements) and prone to bugs.

2. Using a purpose-built library like FHIR-PYrate to automatically convert FHIR instances into DataFrames.

   It is recommended to try this approach first, and only fall back to (1) if needed.

To use FHIR-PYrate, you will need a Python 3 runtime with FHIR-PYrate and Pandas installed.

1 FHIR testing server

The examples in this module use a FHIR testing server populated with Synthea data in FHIR R4 format via public HAPI Test Server operated by HAPI FHIR.

The endpoint for this testing server is:

https://hapi.fhir.org/baseR4

However, any FHIR server loaded with testing data can be used. See Standing up a FHIR Testing Server for instructions to set up your own test server.

The code blocks in the following section show sample output immediately after. This is similar to the code cells and results in a Jupyter notebook.

2 Retrieving FHIR data

Once your environment is set up, you can run the following Python code to retrieve instances of the Patient resource from a test server:

# Load dependencies
from fhir_pyrate import Pirate
import pandas as pd

# Instantiate a Pirate object using the FHIR-PYrate library to query a test FHIR server
search = Pirate(
    auth=None,
    base_url="https://hapi.fhir.org/baseR4",
    print_request_url=True,
)

# Use the whimsically named `steal_bundles()` method to instantiate a search interaction
#
# For more information, see https://github.com/UMEssen/FHIR-PYrate/#pirate
bundles = search.steal_bundles(
    resource_type="Patient",
    request_params={
        "_count": 10,  # Get 10 instances per page
        "identifier": "https://github.com/synthetichealth/synthea|",
    },
    num_pages=1,  # Get 1 page (so a total of 10 instances)
)

# Execute the search and convert to a Pandas DataFrame
df = search.bundles_to_dataframe(bundles)

df.head(5)

/home/runner/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhirpathpy/engine/nodes.py:7: SyntaxWarning: "\." is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\\."? A raw string is also an option.
  timeRE = '([01][0-9]|2[0-3]):([0-5][0-9]):([0-5][0-9]|60)(\.[0-9]+)?'
/home/runner/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhirpathpy/engine/nodes.py:8: SyntaxWarning: "\+" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\\+"? A raw string is also an option.
  dateTimeRE = '%s(T%s(Z|(\+|-)((0[0-9]|1[0-3]):[0-5][0-9]|14:00)))?)?)?' % (dateFormat, timeRE)

https://hapi.fhir.org/baseR4/Patient?_count=10&identifier=https://github.com/synthetichealth/synthea|

Query (Patient):   0%|          | 0/1 [00:00<?, ?it/s]Query (Patient): 100%|██████████| 1/1 [00:00<00:00, 11066.77it/s]

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[1], line 27
     24 # Execute the search and convert to a Pandas DataFrame
     25 df = search.bundles_to_dataframe(bundles)
---> 27 df.head(5)

AttributeError: 'dict' object has no attribute 'head'

It is easier to see the contents of this DataFrame by printing out its first row vertically:

# Print the first row of the DataFrame vertically for easier reading.
pd.set_option("display.max_rows", 100)  # Show all rows
df.head(1).T

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[2], line 3
      1 # Print the first row of the DataFrame vertically for easier reading.
      2 pd.set_option("display.max_rows", 100)  # Show all rows
----> 3 df.head(1).T

AttributeError: 'dict' object has no attribute 'head'

If you look at the output above, you can see FHIR-PYrate collapsed the hierarchical FHIR data structure into DataFrame columns. FHIR-PYrate does this by taking an element from the FHIR-formatted data like Patient.identifier[0].value and converting to an underscore-delimited column name like identifier_0_value. (Note that Patient.identifier has multiple values in the FHIR data, so there are multiple identifier_N_... columns in the DataFrame.)

3 Selecting specific columns

Usually not every single value from a FHIR instance is needed for analysis. There are two ways to get a more concise DataFrame:

1. Use the approach above to load all elements into a DataFrame, remove the unneeded columns, and rename the remaining columns as needed. The process_function capability in FHIR-PYrate allows you to integrate this approach into the bundles_to_dataframe() method call.
2. Use FHIRPath to select specific elements and map them onto column names.

The second approach is typically more concise. For example, to generate a DataFrame like this…

id	gender	date_of_birth	marital_status
…	…	…	…

…you could use the following code:

# Instantiate and perform the FHIR search interaction in a single function call
df = search.steal_bundles_to_dataframe(
    resource_type="Patient",
    request_params={
        "_count": 10,  # Get 10 instances per page
        "identifier": "https://github.com/synthetichealth/synthea|",
    },
    num_pages=1,  # Get 1 page (so a total of 10 instances)
    fhir_paths=[
        ("id", "identifier[0].value"),
        ("gender", "gender"),
        ("date_of_birth", "birthDate"),
        ("marital_status", "maritalStatus.coding[0].code"),
    ],
)
df

---------------------------------------------------------------------------
OptionalImportError                       Traceback (most recent call last)
Cell In[3], line 2
      1 # Instantiate and perform the FHIR search interaction in a single function call
----> 2 df = search.steal_bundles_to_dataframe(
      3     resource_type="Patient",
      4     request_params={
      5         "_count": 10,  # Get 10 instances per page
      6         "identifier": "https://github.com/synthetichealth/synthea|",
      7     },
      8     num_pages=1,  # Get 1 page (so a total of 10 instances)
      9     fhir_paths=[
     10         ("id", "identifier[0].value"),
     11         ("gender", "gender"),
     12         ("date_of_birth", "birthDate"),
     13         ("marital_status", "maritalStatus.coding[0].code"),
     14     ],
     15 )
     16 df

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/pirate.py:282, in Pirate.steal_bundles_to_dataframe(self, resource_type, request_params, num_pages, process_function, fhir_paths, build_df_after_query)
    251 def steal_bundles_to_dataframe(
    252     self,
    253     resource_type: str,
   (...)    258     build_df_after_query: bool = False,
    259 ) -> Union[pd.DataFrame, Dict[str, pd.DataFrame]]:
    260     """
    261     Execute a request, iterates through the result pages, and builds a DataFrame with their
    262     information. The DataFrames are either built after each
   (...)    280     returned.
    281     """
--> 282     return self._query_to_dataframe(self._get_bundles)(
    283         resource_type=resource_type,
    284         request_params=request_params,
    285         num_pages=num_pages,
    286         silence_tqdm=False,
    287         process_function=process_function,
    288         fhir_paths=fhir_paths,
    289         build_df_after_query=build_df_after_query,
    290         disable_multiprocessing_build=True,
    291         always_return_dict=False,
    292     )

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/pirate.py:1441, in Pirate._query_to_dataframe.<locals>.wrap(process_function, fhir_paths, build_df_after_query, disable_multiprocessing_build, always_return_dict, *args, **kwargs)
   1436 if fhir_paths is not None:
   1437     logger.info(
   1438         f"The selected process_function {process_function.__name__} will be "
   1439         f"overwritten."
   1440     )
-> 1441     process_function = self._set_up_fhirpath_function(fhir_paths)
   1442 return self._bundles_to_dataframe(
   1443     bundles=bundles_function(
   1444         *args, **kwargs, tqdm_df_build=not build_df_after_query
   (...)   1449     always_return_dict=always_return_dict,
   1450 )

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/pirate.py:1349, in Pirate._set_up_fhirpath_function(self, fhir_paths)
   1331             if (
   1332                 re.search(pattern=rf"{token}[\.\[]|[\.\]]{token}$", string=path)
   1333                 is not None
   1334             ):
   1335                 warnings.warn(
   1336                     f"You are using the term {token} in of your FHIR path {path}. "
   1337                     f"Please keep in mind that this token can be used a function according "
   (...)   1346                     stacklevel=2,
   1347                 )
   1348 compiled_paths = [
-> 1349     (name, fhirpathpy.compile(path=path)) for name, path in fhir_paths_with_name
   1350 ]
   1351 return partial(parse_fhir_path, compiled_fhir_paths=compiled_paths)

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/util/imports.py:106, in optional_import.<locals>._LazyRaise.__getattr__(self, name)
    101 def __getattr__(self, name: str) -> str:
    102     """
    103     Raise:
    104         OptionalImportError: When you call this method.
    105     """
--> 106     raise self._exception

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/util/imports.py:59, in optional_import(module, name, descriptor, allow_namespace_pkg)
     57     actual_cmd = f"import {module}"
     58 try:
---> 59     the_module = import_module(module)
     60     if not allow_namespace_pkg:
     61         is_namespace = getattr(the_module, "__file__", None) is None and hasattr(
     62             the_module, "__path__"
     63         )

File /opt/hostedtoolcache/Python/3.14.2/x64/lib/python3.14/importlib/__init__.py:88, in import_module(name, package)
     86             break
     87         level += 1
---> 88 return _bootstrap._gcd_import(name[level:], package, level)

File <frozen importlib._bootstrap>:1398, in _gcd_import(name, package, level)
   1396 if level > 0:
   1397     name = _resolve_name(name, package, level)
-> 1398 return _find_and_load(name, _gcd_import)

File <frozen importlib._bootstrap>:1371, in _find_and_load(name, import_)
   1369     module = sys.modules.get(name, _NEEDS_LOADING)
   1370     if module is _NEEDS_LOADING:
-> 1371         return _find_and_load_unlocked(name, import_)
   1373 # Optimization: only call _bootstrap._lock_unlock_module() if
   1374 # module.__spec__._initializing is True.
   1375 # NOTE: because of this, initializing must be set *before*
   1376 # putting the new module in sys.modules.
   1377 _lock_unlock_module(name)

File <frozen importlib._bootstrap>:1342, in _find_and_load_unlocked(name, import_)
   1340     parent_spec._uninitialized_submodules.append(child)
   1341 try:
-> 1342     module = _load_unlocked(spec)
   1343 finally:
   1344     if parent_spec:

File <frozen importlib._bootstrap>:938, in _load_unlocked(spec)
    935             raise ImportError('missing loader', name=spec.name)
    936         # A namespace package so do nothing.
    937     else:
--> 938         spec.loader.exec_module(module)
    939 except:
    940     try:

File <frozen importlib._bootstrap_external>:759, in _LoaderBasics.exec_module(self, module)
    756 if code is None:
    757     raise ImportError(f'cannot load module {module.__name__!r} when '
    758                       'get_code() returns None')
--> 759 _bootstrap._call_with_frames_removed(exec, code, module.__dict__)

File <frozen importlib._bootstrap>:491, in _call_with_frames_removed(f, *args, **kwds)
    483 def _call_with_frames_removed(f, *args, **kwds):
    484     """remove_importlib_frames in import.c will always remove sequences
    485     of importlib frames that end with a call to this function
    486 
   (...)    489     module code)
    490     """
--> 491     return f(*args, **kwds)

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhirpathpy/__init__.py:2
      1 from fhirpathpy.engine.invocations.constants import constants
----> 2 from fhirpathpy.parser import parse
      3 from fhirpathpy.engine import do_eval
      4 from fhirpathpy.engine.util import arraify, get_data, set_paths

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhirpathpy/parser/__init__.py:2
      1 import sys
----> 2 from antlr4 import *
      3 from antlr4.tree.Tree import ParseTreeWalker
      4 from antlr4.error.ErrorListener import ErrorListener

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/antlr4/__init__.py:6
      4 from antlr4.StdinStream import StdinStream
      5 from antlr4.BufferedTokenStream import TokenStream
----> 6 from antlr4.CommonTokenStream import CommonTokenStream
      7 from antlr4.Lexer import Lexer
      8 from antlr4.Parser import Parser

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/antlr4/CommonTokenStream.py:33
      1 #
      2 # Copyright (c) 2012-2017 The ANTLR Project. All rights reserved.
      3 # Use of this file is governed by the BSD 3-clause license that
   (...)     29 # channel.</p>
     30 #/
     32 from antlr4.BufferedTokenStream import BufferedTokenStream
---> 33 from antlr4.Lexer import Lexer
     34 from antlr4.Token import Token
     37 class CommonTokenStream(BufferedTokenStream):

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/antlr4/Lexer.py:12
      1 # Copyright (c) 2012-2017 The ANTLR Project. All rights reserved.
      2 # Use of this file is governed by the BSD 3-clause license that
      3 # can be found in the LICENSE.txt file in the project root.
   (...)      9 #  of speed.
     10 #/
     11 from io import StringIO
---> 12 from typing.io import TextIO
     13 import sys
     14 from antlr4.CommonTokenFactory import CommonTokenFactory

OptionalImportError: import fhirpathpy (No module named 'typing.io'; 'typing' is not a package).

While FHIRPath can be quite complex, its use in FHIR-PYrate is often straight forward. Nested elements are separated with ., and elements with multiple sub-values are identified by [N] where N is an integer starting at 0. The element paths can typically be constructed by loading all elements into a DataFrame and then manually deriving the FHIRPaths from the column names, or by looking at the hierarchy resource pages in the FHIR specification (see Key FHIR Resources for more information on reading the FHIR specification).

4 Elements with multiple sub-values

There are multiple identifier[N].value values for each instance of Patient in this dataset.

# Instantiate and perform the FHIR search interaction in a single function call
df = search.steal_bundles_to_dataframe(
    resource_type="Patient",
    request_params={
        "_count": 10,  # Get 10 instances per page
        "identifier": "https://github.com/synthetichealth/synthea|",
    },
    num_pages=1,  # Get 1 page (so a total of 10 instances)
    fhir_paths=[("id", "identifier[0].value"), ("identifiers", "identifier.value")],
)
df

---------------------------------------------------------------------------
OptionalImportError                       Traceback (most recent call last)
Cell In[4], line 2
      1 # Instantiate and perform the FHIR search interaction in a single function call
----> 2 df = search.steal_bundles_to_dataframe(
      3     resource_type="Patient",
      4     request_params={
      5         "_count": 10,  # Get 10 instances per page
      6         "identifier": "https://github.com/synthetichealth/synthea|",
      7     },
      8     num_pages=1,  # Get 1 page (so a total of 10 instances)
      9     fhir_paths=[("id", "identifier[0].value"), ("identifiers", "identifier.value")],
     10 )
     11 df

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/pirate.py:282, in Pirate.steal_bundles_to_dataframe(self, resource_type, request_params, num_pages, process_function, fhir_paths, build_df_after_query)
    251 def steal_bundles_to_dataframe(
    252     self,
    253     resource_type: str,
   (...)    258     build_df_after_query: bool = False,
    259 ) -> Union[pd.DataFrame, Dict[str, pd.DataFrame]]:
    260     """
    261     Execute a request, iterates through the result pages, and builds a DataFrame with their
    262     information. The DataFrames are either built after each
   (...)    280     returned.
    281     """
--> 282     return self._query_to_dataframe(self._get_bundles)(
    283         resource_type=resource_type,
    284         request_params=request_params,
    285         num_pages=num_pages,
    286         silence_tqdm=False,
    287         process_function=process_function,
    288         fhir_paths=fhir_paths,
    289         build_df_after_query=build_df_after_query,
    290         disable_multiprocessing_build=True,
    291         always_return_dict=False,
    292     )

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/pirate.py:1441, in Pirate._query_to_dataframe.<locals>.wrap(process_function, fhir_paths, build_df_after_query, disable_multiprocessing_build, always_return_dict, *args, **kwargs)
   1436 if fhir_paths is not None:
   1437     logger.info(
   1438         f"The selected process_function {process_function.__name__} will be "
   1439         f"overwritten."
   1440     )
-> 1441     process_function = self._set_up_fhirpath_function(fhir_paths)
   1442 return self._bundles_to_dataframe(
   1443     bundles=bundles_function(
   1444         *args, **kwargs, tqdm_df_build=not build_df_after_query
   (...)   1449     always_return_dict=always_return_dict,
   1450 )

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/pirate.py:1349, in Pirate._set_up_fhirpath_function(self, fhir_paths)
   1331             if (
   1332                 re.search(pattern=rf"{token}[\.\[]|[\.\]]{token}$", string=path)
   1333                 is not None
   1334             ):
   1335                 warnings.warn(
   1336                     f"You are using the term {token} in of your FHIR path {path}. "
   1337                     f"Please keep in mind that this token can be used a function according "
   (...)   1346                     stacklevel=2,
   1347                 )
   1348 compiled_paths = [
-> 1349     (name, fhirpathpy.compile(path=path)) for name, path in fhir_paths_with_name
   1350 ]
   1351 return partial(parse_fhir_path, compiled_fhir_paths=compiled_paths)

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/util/imports.py:106, in optional_import.<locals>._LazyRaise.__getattr__(self, name)
    101 def __getattr__(self, name: str) -> str:
    102     """
    103     Raise:
    104         OptionalImportError: When you call this method.
    105     """
--> 106     raise self._exception

    [... skipping hidden 1 frame]

Cell In[3], line 2
      1 # Instantiate and perform the FHIR search interaction in a single function call
----> 2 df = search.steal_bundles_to_dataframe(
      3     resource_type="Patient",
      4     request_params={
      5         "_count": 10,  # Get 10 instances per page
      6         "identifier": "https://github.com/synthetichealth/synthea|",
      7     },
      8     num_pages=1,  # Get 1 page (so a total of 10 instances)
      9     fhir_paths=[
     10         ("id", "identifier[0].value"),
     11         ("gender", "gender"),
     12         ("date_of_birth", "birthDate"),
     13         ("marital_status", "maritalStatus.coding[0].code"),
     14     ],
     15 )
     16 df

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/pirate.py:282, in Pirate.steal_bundles_to_dataframe(self, resource_type, request_params, num_pages, process_function, fhir_paths, build_df_after_query)
    251 def steal_bundles_to_dataframe(
    252     self,
    253     resource_type: str,
   (...)    258     build_df_after_query: bool = False,
    259 ) -> Union[pd.DataFrame, Dict[str, pd.DataFrame]]:
    260     """
    261     Execute a request, iterates through the result pages, and builds a DataFrame with their
    262     information. The DataFrames are either built after each
   (...)    280     returned.
    281     """
--> 282     return self._query_to_dataframe(self._get_bundles)(
    283         resource_type=resource_type,
    284         request_params=request_params,
    285         num_pages=num_pages,
    286         silence_tqdm=False,
    287         process_function=process_function,
    288         fhir_paths=fhir_paths,
    289         build_df_after_query=build_df_after_query,
    290         disable_multiprocessing_build=True,
    291         always_return_dict=False,
    292     )

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/pirate.py:1441, in Pirate._query_to_dataframe.<locals>.wrap(process_function, fhir_paths, build_df_after_query, disable_multiprocessing_build, always_return_dict, *args, **kwargs)
   1436 if fhir_paths is not None:
   1437     logger.info(
   1438         f"The selected process_function {process_function.__name__} will be "
   1439         f"overwritten."
   1440     )
-> 1441     process_function = self._set_up_fhirpath_function(fhir_paths)
   1442 return self._bundles_to_dataframe(
   1443     bundles=bundles_function(
   1444         *args, **kwargs, tqdm_df_build=not build_df_after_query
   (...)   1449     always_return_dict=always_return_dict,
   1450 )

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/pirate.py:1349, in Pirate._set_up_fhirpath_function(self, fhir_paths)
   1331             if (
   1332                 re.search(pattern=rf"{token}[\.\[]|[\.\]]{token}$", string=path)
   1333                 is not None
   1334             ):
   1335                 warnings.warn(
   1336                     f"You are using the term {token} in of your FHIR path {path}. "
   1337                     f"Please keep in mind that this token can be used a function according "
   (...)   1346                     stacklevel=2,
   1347                 )
   1348 compiled_paths = [
-> 1349     (name, fhirpathpy.compile(path=path)) for name, path in fhir_paths_with_name
   1350 ]
   1351 return partial(parse_fhir_path, compiled_fhir_paths=compiled_paths)

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/util/imports.py:106, in optional_import.<locals>._LazyRaise.__getattr__(self, name)
    101 def __getattr__(self, name: str) -> str:
    102     """
    103     Raise:
    104         OptionalImportError: When you call this method.
    105     """
--> 106     raise self._exception

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/util/imports.py:59, in optional_import(module, name, descriptor, allow_namespace_pkg)
     57     actual_cmd = f"import {module}"
     58 try:
---> 59     the_module = import_module(module)
     60     if not allow_namespace_pkg:
     61         is_namespace = getattr(the_module, "__file__", None) is None and hasattr(
     62             the_module, "__path__"
     63         )

File /opt/hostedtoolcache/Python/3.14.2/x64/lib/python3.14/importlib/__init__.py:88, in import_module(name, package)
     86             break
     87         level += 1
---> 88 return _bootstrap._gcd_import(name[level:], package, level)

File <frozen importlib._bootstrap>:1398, in _gcd_import(name, package, level)
   1396 if level > 0:
   1397     name = _resolve_name(name, package, level)
-> 1398 return _find_and_load(name, _gcd_import)

File <frozen importlib._bootstrap>:1371, in _find_and_load(name, import_)
   1369     module = sys.modules.get(name, _NEEDS_LOADING)
   1370     if module is _NEEDS_LOADING:
-> 1371         return _find_and_load_unlocked(name, import_)
   1373 # Optimization: only call _bootstrap._lock_unlock_module() if
   1374 # module.__spec__._initializing is True.
   1375 # NOTE: because of this, initializing must be set *before*
   1376 # putting the new module in sys.modules.
   1377 _lock_unlock_module(name)

File <frozen importlib._bootstrap>:1342, in _find_and_load_unlocked(name, import_)
   1340     parent_spec._uninitialized_submodules.append(child)
   1341 try:
-> 1342     module = _load_unlocked(spec)
   1343 finally:
   1344     if parent_spec:

File <frozen importlib._bootstrap>:938, in _load_unlocked(spec)
    935             raise ImportError('missing loader', name=spec.name)
    936         # A namespace package so do nothing.
    937     else:
--> 938         spec.loader.exec_module(module)
    939 except:
    940     try:

File <frozen importlib._bootstrap_external>:759, in _LoaderBasics.exec_module(self, module)
    756 if code is None:
    757     raise ImportError(f'cannot load module {module.__name__!r} when '
    758                       'get_code() returns None')
--> 759 _bootstrap._call_with_frames_removed(exec, code, module.__dict__)

File <frozen importlib._bootstrap>:491, in _call_with_frames_removed(f, *args, **kwds)
    483 def _call_with_frames_removed(f, *args, **kwds):
    484     """remove_importlib_frames in import.c will always remove sequences
    485     of importlib frames that end with a call to this function
    486 
   (...)    489     module code)
    490     """
--> 491     return f(*args, **kwds)

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhirpathpy/__init__.py:2
      1 from fhirpathpy.engine.invocations.constants import constants
----> 2 from fhirpathpy.parser import parse
      3 from fhirpathpy.engine import do_eval
      4 from fhirpathpy.engine.util import arraify, get_data, set_paths

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhirpathpy/parser/__init__.py:2
      1 import sys
----> 2 from antlr4 import *
      3 from antlr4.tree.Tree import ParseTreeWalker
      4 from antlr4.error.ErrorListener import ErrorListener

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/antlr4/__init__.py:6
      4 from antlr4.StdinStream import StdinStream
      5 from antlr4.BufferedTokenStream import TokenStream
----> 6 from antlr4.CommonTokenStream import CommonTokenStream
      7 from antlr4.Lexer import Lexer
      8 from antlr4.Parser import Parser

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/antlr4/CommonTokenStream.py:33
      1 #
      2 # Copyright (c) 2012-2017 The ANTLR Project. All rights reserved.
      3 # Use of this file is governed by the BSD 3-clause license that
   (...)     29 # channel.</p>
     30 #/
     32 from antlr4.BufferedTokenStream import BufferedTokenStream
---> 33 from antlr4.Lexer import Lexer
     34 from antlr4.Token import Token
     37 class CommonTokenStream(BufferedTokenStream):

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/antlr4/Lexer.py:12
      1 # Copyright (c) 2012-2017 The ANTLR Project. All rights reserved.
      2 # Use of this file is governed by the BSD 3-clause license that
      3 # can be found in the LICENSE.txt file in the project root.
   (...)      9 #  of speed.
     10 #/
     11 from io import StringIO
---> 12 from typing.io import TextIO
     13 import sys
     14 from antlr4.CommonTokenFactory import CommonTokenFactory

OptionalImportError: import fhirpathpy (No module named 'typing.io'; 'typing' is not a package).

To convert to separate columns, you can do the following:

df.join(pd.DataFrame(df.pop("identifiers").values.tolist()).add_prefix("identifier_"))

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[5], line 1
----> 1 df.join(pd.DataFrame(df.pop("identifiers").values.tolist()).add_prefix("identifier_"))

AttributeError: 'dict' object has no attribute 'join'

This will give you separate identifier_0, identifier_1, … columns for each Patient.identifier[N] value.

5 Retrieving related data

To retrieve instances of related resources, additional request_params can be added. See Using the FHIR API to Access Data for more information on constructing the parameters for FHIR search interactions.

In the example below, instances of Patient and instances of related Observation resources are requested:

# Instantiate and perform the FHIR search interaction in a single function call
dfs = search.steal_bundles_to_dataframe(
    resource_type="Patient",
    request_params={
        # Get instances of Observation where `Observation.patient` refers to a fetched Patient instance
        "_revinclude": "Observation:patient",
        "identifier": "https://github.com/synthetichealth/synthea|",
        "_count": 10,  # Get 10 instances per page
    },
    num_pages=1,  # Get 1 page (so a total of 10 instances)
)

# `dfs` is a dictionary where the key is the FHIR resource type, and the value is the DataFrame
#
# Split these into separate variables for easy access:
df_patients = dfs["Patient"]
df_observations = dfs["Observation"]

# Look at the first row of the Observations DataFrame
df_observations.head(1).T

https://hapi.fhir.org/baseR4/Patient?_count=10&_revinclude=Observation:patient&identifier=https://github.com/synthetichealth/synthea|

Query & Build DF (Patient):   0%|          | 0/1 [00:00<?, ?it/s]Query & Build DF (Patient): 100%|██████████| 1/1 [00:00<00:00, 12905.55it/s]

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[6], line 16
      2 dfs = search.steal_bundles_to_dataframe(
      3     resource_type="Patient",
      4     request_params={
   (...)     10     num_pages=1,  # Get 1 page (so a total of 10 instances)
     11 )
     13 # `dfs` is a dictionary where the key is the FHIR resource type, and the value is the DataFrame
     14 #
     15 # Split these into separate variables for easy access:
---> 16 df_patients = dfs["Patient"]
     17 df_observations = dfs["Observation"]
     19 # Look at the first row of the Observations DataFrame

KeyError: 'Patient'

As of April 2023, FHIR-PYrate does not have a good approach to fhir_paths for searches that return instances of multiple FHIR resource types.

To work around this, you can also iterate over all the rows in a DataFrame and request related resources using trade_rows_for_bundles():

df_observations2 = search.trade_rows_for_dataframe(
    df_patients,
    resource_type="Observation",
    request_params={
        "_count": "10",  # Get 10 instances per page
    },
    num_pages=1,
    # Load Observations where `Observation.subject` references the instance of Patient
    # identified by `id` in the `df_patients` DataFrame
    df_constraints={"subject": "id"},
    fhir_paths=[
        ("observation_id", "id"),
        ("patient", "subject.reference"),
        ("status", "status"),
        ("code", "code.coding[0].code"),
        ("code_display", "code.coding[0].display"),
        ("value", "valueQuantity.value"),
        ("value_units", "valueQuantity.unit"),
        ("datetime", "effectiveDateTime"),
    ],
)

# Look at the first row of the Observations DataFrame
df_observations2.head(15)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[7], line 2
      1 df_observations2 = search.trade_rows_for_dataframe(
----> 2     df_patients,
      3     resource_type="Observation",
      4     request_params={
      5         "_count": "10",  # Get 10 instances per page
      6     },
      7     num_pages=1,
      8     # Load Observations where `Observation.subject` references the instance of Patient
      9     # identified by `id` in the `df_patients` DataFrame
     10     df_constraints={"subject": "id"},
     11     fhir_paths=[
     12         ("observation_id", "id"),
     13         ("patient", "subject.reference"),
     14         ("status", "status"),
     15         ("code", "code.coding[0].code"),
     16         ("code_display", "code.coding[0].display"),
     17         ("value", "valueQuantity.value"),
     18         ("value_units", "valueQuantity.unit"),
     19         ("datetime", "effectiveDateTime"),
     20     ],
     21 )
     23 # Look at the first row of the Observations DataFrame
     24 df_observations2.head(15)

NameError: name 'df_patients' is not defined

Note that this will only display value for instances of Observation that record a value in Observation.valueQuantity. Typically, you would filter by Observation.code and then choose the appropriate data type for Observation.value[x] to import. For example, http://loinc.org|72166-2 is the LOINC for smoking status. To get smoking status records for all patients in df_patients:

# Directly search for smoking status observations

df_observations2 = search.steal_bundles_to_dataframe(
    resource_type="Observation",
    request_params={
        "code": "http://loinc.org|72166-2",  # LOINC code for smoking status
        "_count": 20,  # Get more observations since we're not limiting by patient
    },
    num_pages=1,
    fhir_paths=[
        ("observation_id", "id"),
        ("patient", "subject.reference"),
        ("status", "status"),
        ("code", "code.coding[0].code"),
        ("code_display", "code.coding[0].display"),
        ("value", "valueCodeableConcept.coding[0].code"),
        ("value_display", "valueCodeableConcept.coding[0].display"),
        ("datetime", "effectiveDateTime"),
    ],
)

# Look at the first row of the Observations DataFrame
df_observations2.head(15)

---------------------------------------------------------------------------
OptionalImportError                       Traceback (most recent call last)
Cell In[8], line 3
      1 # Directly search for smoking status observations
----> 3 df_observations2 = search.steal_bundles_to_dataframe(
      4     resource_type="Observation",
      5     request_params={
      6         "code": "http://loinc.org|72166-2",  # LOINC code for smoking status
      7         "_count": 20,  # Get more observations since we're not limiting by patient
      8     },
      9     num_pages=1,
     10     fhir_paths=[
     11         ("observation_id", "id"),
     12         ("patient", "subject.reference"),
     13         ("status", "status"),
     14         ("code", "code.coding[0].code"),
     15         ("code_display", "code.coding[0].display"),
     16         ("value", "valueCodeableConcept.coding[0].code"),
     17         ("value_display", "valueCodeableConcept.coding[0].display"),
     18         ("datetime", "effectiveDateTime"),
     19     ],
     20 )
     22 # Look at the first row of the Observations DataFrame
     23 df_observations2.head(15)

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/pirate.py:282, in Pirate.steal_bundles_to_dataframe(self, resource_type, request_params, num_pages, process_function, fhir_paths, build_df_after_query)
    251 def steal_bundles_to_dataframe(
    252     self,
    253     resource_type: str,
   (...)    258     build_df_after_query: bool = False,
    259 ) -> Union[pd.DataFrame, Dict[str, pd.DataFrame]]:
    260     """
    261     Execute a request, iterates through the result pages, and builds a DataFrame with their
    262     information. The DataFrames are either built after each
   (...)    280     returned.
    281     """
--> 282     return self._query_to_dataframe(self._get_bundles)(
    283         resource_type=resource_type,
    284         request_params=request_params,
    285         num_pages=num_pages,
    286         silence_tqdm=False,
    287         process_function=process_function,
    288         fhir_paths=fhir_paths,
    289         build_df_after_query=build_df_after_query,
    290         disable_multiprocessing_build=True,
    291         always_return_dict=False,
    292     )

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/pirate.py:1441, in Pirate._query_to_dataframe.<locals>.wrap(process_function, fhir_paths, build_df_after_query, disable_multiprocessing_build, always_return_dict, *args, **kwargs)
   1436 if fhir_paths is not None:
   1437     logger.info(
   1438         f"The selected process_function {process_function.__name__} will be "
   1439         f"overwritten."
   1440     )
-> 1441     process_function = self._set_up_fhirpath_function(fhir_paths)
   1442 return self._bundles_to_dataframe(
   1443     bundles=bundles_function(
   1444         *args, **kwargs, tqdm_df_build=not build_df_after_query
   (...)   1449     always_return_dict=always_return_dict,
   1450 )

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/pirate.py:1349, in Pirate._set_up_fhirpath_function(self, fhir_paths)
   1331             if (
   1332                 re.search(pattern=rf"{token}[\.\[]|[\.\]]{token}$", string=path)
   1333                 is not None
   1334             ):
   1335                 warnings.warn(
   1336                     f"You are using the term {token} in of your FHIR path {path}. "
   1337                     f"Please keep in mind that this token can be used a function according "
   (...)   1346                     stacklevel=2,
   1347                 )
   1348 compiled_paths = [
-> 1349     (name, fhirpathpy.compile(path=path)) for name, path in fhir_paths_with_name
   1350 ]
   1351 return partial(parse_fhir_path, compiled_fhir_paths=compiled_paths)

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/util/imports.py:106, in optional_import.<locals>._LazyRaise.__getattr__(self, name)
    101 def __getattr__(self, name: str) -> str:
    102     """
    103     Raise:
    104         OptionalImportError: When you call this method.
    105     """
--> 106     raise self._exception

    [... skipping hidden 1 frame]

Cell In[4], line 2
      1 # Instantiate and perform the FHIR search interaction in a single function call
----> 2 df = search.steal_bundles_to_dataframe(
      3     resource_type="Patient",
      4     request_params={
      5         "_count": 10,  # Get 10 instances per page
      6         "identifier": "https://github.com/synthetichealth/synthea|",
      7     },
      8     num_pages=1,  # Get 1 page (so a total of 10 instances)
      9     fhir_paths=[("id", "identifier[0].value"), ("identifiers", "identifier.value")],
     10 )
     11 df

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/pirate.py:282, in Pirate.steal_bundles_to_dataframe(self, resource_type, request_params, num_pages, process_function, fhir_paths, build_df_after_query)
    251 def steal_bundles_to_dataframe(
    252     self,
    253     resource_type: str,
   (...)    258     build_df_after_query: bool = False,
    259 ) -> Union[pd.DataFrame, Dict[str, pd.DataFrame]]:
    260     """
    261     Execute a request, iterates through the result pages, and builds a DataFrame with their
    262     information. The DataFrames are either built after each
   (...)    280     returned.
    281     """
--> 282     return self._query_to_dataframe(self._get_bundles)(
    283         resource_type=resource_type,
    284         request_params=request_params,
    285         num_pages=num_pages,
    286         silence_tqdm=False,
    287         process_function=process_function,
    288         fhir_paths=fhir_paths,
    289         build_df_after_query=build_df_after_query,
    290         disable_multiprocessing_build=True,
    291         always_return_dict=False,
    292     )

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/pirate.py:1441, in Pirate._query_to_dataframe.<locals>.wrap(process_function, fhir_paths, build_df_after_query, disable_multiprocessing_build, always_return_dict, *args, **kwargs)
   1436 if fhir_paths is not None:
   1437     logger.info(
   1438         f"The selected process_function {process_function.__name__} will be "
   1439         f"overwritten."
   1440     )
-> 1441     process_function = self._set_up_fhirpath_function(fhir_paths)
   1442 return self._bundles_to_dataframe(
   1443     bundles=bundles_function(
   1444         *args, **kwargs, tqdm_df_build=not build_df_after_query
   (...)   1449     always_return_dict=always_return_dict,
   1450 )

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/pirate.py:1349, in Pirate._set_up_fhirpath_function(self, fhir_paths)
   1331             if (
   1332                 re.search(pattern=rf"{token}[\.\[]|[\.\]]{token}$", string=path)
   1333                 is not None
   1334             ):
   1335                 warnings.warn(
   1336                     f"You are using the term {token} in of your FHIR path {path}. "
   1337                     f"Please keep in mind that this token can be used a function according "
   (...)   1346                     stacklevel=2,
   1347                 )
   1348 compiled_paths = [
-> 1349     (name, fhirpathpy.compile(path=path)) for name, path in fhir_paths_with_name
   1350 ]
   1351 return partial(parse_fhir_path, compiled_fhir_paths=compiled_paths)

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/util/imports.py:106, in optional_import.<locals>._LazyRaise.__getattr__(self, name)
    101 def __getattr__(self, name: str) -> str:
    102     """
    103     Raise:
    104         OptionalImportError: When you call this method.
    105     """
--> 106     raise self._exception

    [... skipping hidden 1 frame]

Cell In[3], line 2
      1 # Instantiate and perform the FHIR search interaction in a single function call
----> 2 df = search.steal_bundles_to_dataframe(
      3     resource_type="Patient",
      4     request_params={
      5         "_count": 10,  # Get 10 instances per page
      6         "identifier": "https://github.com/synthetichealth/synthea|",
      7     },
      8     num_pages=1,  # Get 1 page (so a total of 10 instances)
      9     fhir_paths=[
     10         ("id", "identifier[0].value"),
     11         ("gender", "gender"),
     12         ("date_of_birth", "birthDate"),
     13         ("marital_status", "maritalStatus.coding[0].code"),
     14     ],
     15 )
     16 df

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/pirate.py:282, in Pirate.steal_bundles_to_dataframe(self, resource_type, request_params, num_pages, process_function, fhir_paths, build_df_after_query)
    251 def steal_bundles_to_dataframe(
    252     self,
    253     resource_type: str,
   (...)    258     build_df_after_query: bool = False,
    259 ) -> Union[pd.DataFrame, Dict[str, pd.DataFrame]]:
    260     """
    261     Execute a request, iterates through the result pages, and builds a DataFrame with their
    262     information. The DataFrames are either built after each
   (...)    280     returned.
    281     """
--> 282     return self._query_to_dataframe(self._get_bundles)(
    283         resource_type=resource_type,
    284         request_params=request_params,
    285         num_pages=num_pages,
    286         silence_tqdm=False,
    287         process_function=process_function,
    288         fhir_paths=fhir_paths,
    289         build_df_after_query=build_df_after_query,
    290         disable_multiprocessing_build=True,
    291         always_return_dict=False,
    292     )

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/pirate.py:1441, in Pirate._query_to_dataframe.<locals>.wrap(process_function, fhir_paths, build_df_after_query, disable_multiprocessing_build, always_return_dict, *args, **kwargs)
   1436 if fhir_paths is not None:
   1437     logger.info(
   1438         f"The selected process_function {process_function.__name__} will be "
   1439         f"overwritten."
   1440     )
-> 1441     process_function = self._set_up_fhirpath_function(fhir_paths)
   1442 return self._bundles_to_dataframe(
   1443     bundles=bundles_function(
   1444         *args, **kwargs, tqdm_df_build=not build_df_after_query
   (...)   1449     always_return_dict=always_return_dict,
   1450 )

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/pirate.py:1349, in Pirate._set_up_fhirpath_function(self, fhir_paths)
   1331             if (
   1332                 re.search(pattern=rf"{token}[\.\[]|[\.\]]{token}$", string=path)
   1333                 is not None
   1334             ):
   1335                 warnings.warn(
   1336                     f"You are using the term {token} in of your FHIR path {path}. "
   1337                     f"Please keep in mind that this token can be used a function according "
   (...)   1346                     stacklevel=2,
   1347                 )
   1348 compiled_paths = [
-> 1349     (name, fhirpathpy.compile(path=path)) for name, path in fhir_paths_with_name
   1350 ]
   1351 return partial(parse_fhir_path, compiled_fhir_paths=compiled_paths)

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/util/imports.py:106, in optional_import.<locals>._LazyRaise.__getattr__(self, name)
    101 def __getattr__(self, name: str) -> str:
    102     """
    103     Raise:
    104         OptionalImportError: When you call this method.
    105     """
--> 106     raise self._exception

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhir_pyrate/util/imports.py:59, in optional_import(module, name, descriptor, allow_namespace_pkg)
     57     actual_cmd = f"import {module}"
     58 try:
---> 59     the_module = import_module(module)
     60     if not allow_namespace_pkg:
     61         is_namespace = getattr(the_module, "__file__", None) is None and hasattr(
     62             the_module, "__path__"
     63         )

File /opt/hostedtoolcache/Python/3.14.2/x64/lib/python3.14/importlib/__init__.py:88, in import_module(name, package)
     86             break
     87         level += 1
---> 88 return _bootstrap._gcd_import(name[level:], package, level)

File <frozen importlib._bootstrap>:1398, in _gcd_import(name, package, level)
   1396 if level > 0:
   1397     name = _resolve_name(name, package, level)
-> 1398 return _find_and_load(name, _gcd_import)

File <frozen importlib._bootstrap>:1371, in _find_and_load(name, import_)
   1369     module = sys.modules.get(name, _NEEDS_LOADING)
   1370     if module is _NEEDS_LOADING:
-> 1371         return _find_and_load_unlocked(name, import_)
   1373 # Optimization: only call _bootstrap._lock_unlock_module() if
   1374 # module.__spec__._initializing is True.
   1375 # NOTE: because of this, initializing must be set *before*
   1376 # putting the new module in sys.modules.
   1377 _lock_unlock_module(name)

File <frozen importlib._bootstrap>:1342, in _find_and_load_unlocked(name, import_)
   1340     parent_spec._uninitialized_submodules.append(child)
   1341 try:
-> 1342     module = _load_unlocked(spec)
   1343 finally:
   1344     if parent_spec:

File <frozen importlib._bootstrap>:938, in _load_unlocked(spec)
    935             raise ImportError('missing loader', name=spec.name)
    936         # A namespace package so do nothing.
    937     else:
--> 938         spec.loader.exec_module(module)
    939 except:
    940     try:

File <frozen importlib._bootstrap_external>:759, in _LoaderBasics.exec_module(self, module)
    756 if code is None:
    757     raise ImportError(f'cannot load module {module.__name__!r} when '
    758                       'get_code() returns None')
--> 759 _bootstrap._call_with_frames_removed(exec, code, module.__dict__)

File <frozen importlib._bootstrap>:491, in _call_with_frames_removed(f, *args, **kwds)
    483 def _call_with_frames_removed(f, *args, **kwds):
    484     """remove_importlib_frames in import.c will always remove sequences
    485     of importlib frames that end with a call to this function
    486 
   (...)    489     module code)
    490     """
--> 491     return f(*args, **kwds)

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhirpathpy/__init__.py:2
      1 from fhirpathpy.engine.invocations.constants import constants
----> 2 from fhirpathpy.parser import parse
      3 from fhirpathpy.engine import do_eval
      4 from fhirpathpy.engine.util import arraify, get_data, set_paths

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/fhirpathpy/parser/__init__.py:2
      1 import sys
----> 2 from antlr4 import *
      3 from antlr4.tree.Tree import ParseTreeWalker
      4 from antlr4.error.ErrorListener import ErrorListener

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/antlr4/__init__.py:6
      4 from antlr4.StdinStream import StdinStream
      5 from antlr4.BufferedTokenStream import TokenStream
----> 6 from antlr4.CommonTokenStream import CommonTokenStream
      7 from antlr4.Lexer import Lexer
      8 from antlr4.Parser import Parser

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/antlr4/CommonTokenStream.py:33
      1 #
      2 # Copyright (c) 2012-2017 The ANTLR Project. All rights reserved.
      3 # Use of this file is governed by the BSD 3-clause license that
   (...)     29 # channel.</p>
     30 #/
     32 from antlr4.BufferedTokenStream import BufferedTokenStream
---> 33 from antlr4.Lexer import Lexer
     34 from antlr4.Token import Token
     37 class CommonTokenStream(BufferedTokenStream):

File ~/work/fhir-for-research/fhir-for-research/.venv/lib/python3.14/site-packages/antlr4/Lexer.py:12
      1 # Copyright (c) 2012-2017 The ANTLR Project. All rights reserved.
      2 # Use of this file is governed by the BSD 3-clause license that
      3 # can be found in the LICENSE.txt file in the project root.
   (...)      9 #  of speed.
     10 #/
     11 from io import StringIO
---> 12 from typing.io import TextIO
     13 import sys
     14 from antlr4.CommonTokenFactory import CommonTokenFactory

OptionalImportError: import fhirpathpy (No module named 'typing.io'; 'typing' is not a package).

More information about the search interaction used above to filter Observations is here.