Loading SDF Files

Loading SDF Files#

There are two main ways to load EPOCH SDF files into xarray objects; using the dedicated sdf_xarray functions or using the standard xarray interface with our custom engine.

The sdf_xarray functions are wrappers designed specifically for SDF data, providing the most straightforward experience:

Single files: Use sdf_xarray.open_dataset or sdf_xarray.open_datatree
Multiple files: Use sdf_xarray.open_mfdataset or sdf_xarray.open_mfdatatree
Raw files: use sdf_xarray.sdf_interface.SDFFile

If you prefer using the native xarray functions, you can use the xarray.open_dataset, xarray.open_datatree and xarray.open_mfdataset. Strangely there is no function in xarray for xarray.open_mfdatatree.

These functions should all work out of the box as long as sdf_xarray is installed on your system. If you are having issues reading files, you might need to pass the parameter engine=sdf_engine when calling any of the above xarray functions.

Important

When loading SDF files, variables related to boundaries, cpu and output file are excluded as they are problematic. If you wish to load these variables in see Loading raw files.

Note

All code examples shown here are visualised using Jupyter notebooks so that you can interactively explore the datasets. To do this on your machine make sure that you have the necessary dependencies installed:

uv pip install "sdf-xarray[jupyter]"

Additionally, if you wish to follow along using the same datasets featured in this documentation, you can download them from Zenodo (10.5281/zenodo.17618509) using sdf_xarray.download.fetch_dataset.

Loading raw files#

If you wish to load data directly from the SDF.C library and ignore the xarray interface layer.

import sdf_xarray as sdfxr

raw_ds = sdfxr.SDFFile("tutorial_dataset_1d/0010.sdf")
raw_ds.variables.keys()

dict_keys(['Wall-time', 'Electric Field/Ex', 'Electric Field/Ey', 'Magnetic Field/Bz', 'Total Particle Energy/Electron', 'Total Particle Energy/Ion', 'Total Particle Energy in Simulation', 'Total Field Energy in Simulation', 'Derived/Number_Density/Electron', 'Derived/Number_Density/Ion', 'Derived/Temperature/Electron', 'Derived/Temperature/Ion', 'dist_fn/px_py/Electron', 'Electron_Probe/Px', 'Electron_Probe/Py', 'Electron_Probe/Pz', 'Electron_Probe/weight', 'Absorption/Total Laser Energy Injected', 'Absorption/Fraction of Laser Energy Absorbed', 'CPUs/Original rank', 'CPUs/Current rank'])

Loading multiple files#

Multiple files can be loaded using one of two methods. The first of which is by using the sdf_xarray.open_mfdataset.

Tip

If your simulation includes multiple output blocks that specify different variables for output at various time steps, variables not present at a specific step will default to a nan value. To remove these nan values we suggest using the xarray.DataArray.dropna function or following our implmentation in Loading sparse data.

import sdf_xarray as sdfxr

sdfxr.open_mfdataset("tutorial_dataset_1d/*.sdf")

<xarray.Dataset> Size: 14MB
Dimensions:                                       (time: 41, X_Grid_mid: 200,
                                                   Px_px_py_Electron: 200,
                                                   Py_px_py_Electron: 200,
                                                   X_Grid: 201,
                                                   Px_px_py_Electron_mid: 199,
                                                   Py_px_py_Electron_mid: 199)
Coordinates:
  * time                                          (time) float64 328B 1.334e-...
  * X_Grid_mid                                    (X_Grid_mid) float64 2kB -4...
  * Px_px_py_Electron                             (Px_px_py_Electron) float64 2kB ...
  * Py_px_py_Electron                             (Py_px_py_Electron) float64 2kB ...
  * X_Grid                                        (X_Grid) float64 2kB -5e-06...
  * Px_px_py_Electron_mid                         (Px_px_py_Electron_mid) float64 2kB ...
  * Py_px_py_Electron_mid                         (Py_px_py_Electron_mid) float64 2kB ...
Data variables: (12/15)
    Wall_time                                     (time) float64 328B 0.00419...
    Electric_Field_Ex                             (time, X_Grid_mid) float64 66kB dask.array<chunksize=(1, 200), meta=np.ndarray>
    Electric_Field_Ey                             (time, X_Grid_mid) float64 66kB dask.array<chunksize=(1, 200), meta=np.ndarray>
    Magnetic_Field_Bz                             (time, X_Grid_mid) float64 66kB dask.array<chunksize=(1, 200), meta=np.ndarray>
    Total_Particle_Energy_Electron                (time) float64 328B 3.639e+...
    Total_Particle_Energy_Ion                     (time) float64 328B 3.57e+0...
    ...                                            ...
    Derived_Number_Density_Ion                    (time, X_Grid_mid) float64 66kB dask.array<chunksize=(1, 200), meta=np.ndarray>
    Derived_Temperature_Electron                  (time, X_Grid_mid) float64 66kB dask.array<chunksize=(1, 200), meta=np.ndarray>
    Derived_Temperature_Ion                       (time, X_Grid_mid) float64 66kB dask.array<chunksize=(1, 200), meta=np.ndarray>
    dist_fn_px_py_Electron                        (time, Px_px_py_Electron, Py_px_py_Electron) float64 13MB dask.array<chunksize=(1, 200, 200), meta=np.ndarray>
    Absorption_Total_Laser_Energy_Injected        (time) float64 328B 1.416e+...
    Absorption_Fraction_of_Laser_Energy_Absorbed  (time) float64 328B 0.0 ......
Attributes: (12/22)
    filename:         /home/docs/checkouts/readthedocs.org/user_builds/sdf-xa...
    file_version:     1
    file_revision:    4
    code_name:        Epoch1d
    step:             0
    time:             1.3342563807926084e-16
    ...               ...
    compile_flags:    unknown
    defines:          50364612
    compile_date:     Wed May 14 13:13:41 2025
    run_date:         Wed Nov 12 12:44:42 2025
    io_date:          Wed Nov 12 12:44:42 2025
    deck:             {'constant': {'I_peak': 1e+24, 'lambda_L': 1e-06, 't_fw...

Alternatively, files can be loaded using xarray.open_mfdataset. However, when loading in all the files we have do some processing of the data so that we can correctly align it along the time dimension; This is done via the preprocess parameter utilising the sdf_xarray.SDFPreprocess function.

import xarray as xr

xr.open_mfdataset(
   "tutorial_dataset_1d/*.sdf",
   join="outer",
   compat="no_conflicts",
   preprocess=sdfxr.SDFPreprocess()
)

<xarray.Dataset> Size: 14MB
Dimensions:                                       (time: 41, X_Grid_mid: 200,
                                                   Px_px_py_Electron: 200,
                                                   Py_px_py_Electron: 200,
                                                   X_Grid: 201,
                                                   Px_px_py_Electron_mid: 199,
                                                   Py_px_py_Electron_mid: 199)
Coordinates:
  * time                                          (time) float64 328B 1.334e-...
  * X_Grid_mid                                    (X_Grid_mid) float64 2kB -4...
  * Px_px_py_Electron                             (Px_px_py_Electron) float64 2kB ...
  * Py_px_py_Electron                             (Py_px_py_Electron) float64 2kB ...
  * X_Grid                                        (X_Grid) float64 2kB -5e-06...
  * Px_px_py_Electron_mid                         (Px_px_py_Electron_mid) float64 2kB ...
  * Py_px_py_Electron_mid                         (Py_px_py_Electron_mid) float64 2kB ...
Data variables: (12/15)
    Wall_time                                     (time) float64 328B 0.00419...
    Electric_Field_Ex                             (time, X_Grid_mid) float64 66kB dask.array<chunksize=(1, 200), meta=np.ndarray>
    Electric_Field_Ey                             (time, X_Grid_mid) float64 66kB dask.array<chunksize=(1, 200), meta=np.ndarray>
    Magnetic_Field_Bz                             (time, X_Grid_mid) float64 66kB dask.array<chunksize=(1, 200), meta=np.ndarray>
    Total_Particle_Energy_Electron                (time) float64 328B 3.639e+...
    Total_Particle_Energy_Ion                     (time) float64 328B 3.57e+0...
    ...                                            ...
    Derived_Number_Density_Ion                    (time, X_Grid_mid) float64 66kB dask.array<chunksize=(1, 200), meta=np.ndarray>
    Derived_Temperature_Electron                  (time, X_Grid_mid) float64 66kB dask.array<chunksize=(1, 200), meta=np.ndarray>
    Derived_Temperature_Ion                       (time, X_Grid_mid) float64 66kB dask.array<chunksize=(1, 200), meta=np.ndarray>
    dist_fn_px_py_Electron                        (time, Px_px_py_Electron, Py_px_py_Electron) float64 13MB dask.array<chunksize=(1, 200, 200), meta=np.ndarray>
    Absorption_Total_Laser_Energy_Injected        (time) float64 328B 1.416e+...
    Absorption_Fraction_of_Laser_Energy_Absorbed  (time) float64 328B 0.0 ......
Attributes: (12/22)
    filename:         /home/docs/checkouts/readthedocs.org/user_builds/sdf-xa...
    file_version:     1
    file_revision:    4
    code_name:        Epoch1d
    step:             0
    time:             1.3342563807926084e-16
    ...               ...
    compile_flags:    unknown
    defines:          50364612
    compile_date:     Wed May 14 13:13:41 2025
    run_date:         Wed Nov 12 12:44:42 2025
    io_date:          Wed Nov 12 12:44:42 2025
    deck:             {'constant': {'I_peak': 1e+24, 'lambda_L': 1e-06, 't_fw...

You can also load the data in as a xarray.DataTree, which organises the data hierarchically into groups (for example grouping related quantities such as the individual components of the electric and magnetic fields) while keeping each item as a xarray.Dataset.

import sdf_xarray as sdfxr

sdfxr.open_mfdatatree("tutorial_dataset_1d/*.sdf")

<xarray.DataTree>
Group: /
│   Dimensions:                              (time: 41)
│   Coordinates:
│     * time                                 (time) float64 328B 1.334e-16 ... 2....
│   Data variables:
│       Wall-time                            (time) float64 328B 0.004197 ... 1.028
│       Total_Particle_Energy_in_Simulation  (time) float64 328B 7.209e+06 ... 1....
│       Total_Field_Energy_in_Simulation     (time) float64 328B 0.0 ... 1.25e+09
│   Attributes: (12/22)
│       filename:         /home/docs/checkouts/readthedocs.org/user_builds/sdf-xa...
│       file_version:     1
│       file_revision:    4
│       code_name:        Epoch1d
│       step:             0
│       time:             1.3342563807926084e-16
│       ...               ...
│       compile_flags:    unknown
│       defines:          50364612
│       compile_date:     Wed May 14 13:13:41 2025
│       run_date:         Wed Nov 12 12:44:42 2025
│       io_date:          Wed Nov 12 12:44:42 2025
│       deck:             {'constant': {'I_peak': 1e+24, 'lambda_L': 1e-06, 't_fw...
├── Group: /Electric_Field
│       Dimensions:     (time: 41, X_Grid_mid: 200)
│       Coordinates:
│         * X_Grid_mid  (X_Grid_mid) float64 2kB -4.95e-06 -4.85e-06 ... 1.495e-05
│       Data variables:
│           Ex          (time, X_Grid_mid) float64 66kB dask.array<chunksize=(1, 200), meta=np.ndarray>
│           Ey          (time, X_Grid_mid) float64 66kB dask.array<chunksize=(1, 200), meta=np.ndarray>
├── Group: /Magnetic_Field
│       Dimensions:     (time: 41, X_Grid_mid: 200)
│       Coordinates:
│         * X_Grid_mid  (X_Grid_mid) float64 2kB -4.95e-06 -4.85e-06 ... 1.495e-05
│       Data variables:
│           Bz          (time, X_Grid_mid) float64 66kB dask.array<chunksize=(1, 200), meta=np.ndarray>
├── Group: /Total_Particle_Energy
│       Dimensions:   (time: 41)
│       Data variables:
│           Electron  (time) float64 328B 3.639e+06 3.601e+06 ... 1.083e+10 1.07e+10
│           Ion       (time) float64 328B 3.57e+06 3.57e+06 ... 1.301e+09 1.313e+09
├── Group: /Absorption
│       Dimensions:                            (time: 41)
│       Data variables:
│           Total_Laser_Energy_Injected        (time) float64 328B 1.416e+06 ... 1.70...
│           Fraction_of_Laser_Energy_Absorbed  (time) float64 328B 0.0 0.9661 ... 0.5871
├── Group: /Derived
│   ├── Group: /Derived/Number_Density
│   │       Dimensions:     (time: 41, X_Grid_mid: 200)
│   │       Coordinates:
│   │         * X_Grid_mid  (X_Grid_mid) float64 2kB -4.95e-06 -4.85e-06 ... 1.495e-05
│   │       Data variables:
│   │           Electron    (time, X_Grid_mid) float64 66kB dask.array<chunksize=(1, 200), meta=np.ndarray>
│   │           Ion         (time, X_Grid_mid) float64 66kB dask.array<chunksize=(1, 200), meta=np.ndarray>
│   └── Group: /Derived/Temperature
│           Dimensions:     (time: 41, X_Grid_mid: 200)
│           Coordinates:
│             * X_Grid_mid  (X_Grid_mid) float64 2kB -4.95e-06 -4.85e-06 ... 1.495e-05
│           Data variables:
│               Electron    (time, X_Grid_mid) float64 66kB dask.array<chunksize=(1, 200), meta=np.ndarray>
│               Ion         (time, X_Grid_mid) float64 66kB dask.array<chunksize=(1, 200), meta=np.ndarray>
└── Group: /dist_fn
    └── Group: /dist_fn/px_py
            Dimensions:            (time: 41, Px_px_py_Electron: 200, Py_px_py_Electron: 200)
            Coordinates:
              * Px_px_py_Electron  (Px_px_py_Electron) float64 2kB -1.492e-21 ... 1.492e-21
              * Py_px_py_Electron  (Py_px_py_Electron) float64 2kB -1.492e-21 ... 1.492e-21
            Data variables:
                Electron           (time, Px_px_py_Electron, Py_px_py_Electron) float64 13MB dask.array<chunksize=(1, 200, 200), meta=np.ndarray>

Loading sparse data#

When dealing with sparse data (where different variables are saved at different, non-overlapping time steps) you can optimize memory usage by loading the data with sdf_xarray.open_mfdataset using the parameter separate_times=True. This approach creates a distinct time dimension for each output block, avoiding the need for a single, large time dimension that would be filled with nan values. This significantly reduces memory consumption, though it requires more deliberate handling if you need to compare variables that exist on these different time coordinates.

import sdf_xarray as sdfxr

sdfxr.open_mfdataset("tutorial_dataset_1d/*.sdf", separate_times=True)

<xarray.Dataset> Size: 14MB
Dimensions:                                       (time0: 41, X_Grid_mid: 200,
                                                   Px_px_py_Electron: 200,
                                                   Py_px_py_Electron: 200,
                                                   X_Grid: 201,
                                                   Px_px_py_Electron_mid: 199,
                                                   Py_px_py_Electron_mid: 199)
Coordinates:
  * time0                                         (time0) float64 328B 1.334e...
  * X_Grid_mid                                    (X_Grid_mid) float64 2kB -4...
  * Px_px_py_Electron                             (Px_px_py_Electron) float64 2kB ...
  * Py_px_py_Electron                             (Py_px_py_Electron) float64 2kB ...
  * X_Grid                                        (X_Grid) float64 2kB -5e-06...
  * Px_px_py_Electron_mid                         (Px_px_py_Electron_mid) float64 2kB ...
  * Py_px_py_Electron_mid                         (Py_px_py_Electron_mid) float64 2kB ...
Data variables: (12/15)
    Wall_time                                     (time0) float64 328B 0.0041...
    Electric_Field_Ex                             (time0, X_Grid_mid) float64 66kB dask.array<chunksize=(1, 200), meta=np.ndarray>
    Electric_Field_Ey                             (time0, X_Grid_mid) float64 66kB dask.array<chunksize=(1, 200), meta=np.ndarray>
    Magnetic_Field_Bz                             (time0, X_Grid_mid) float64 66kB dask.array<chunksize=(1, 200), meta=np.ndarray>
    Total_Particle_Energy_Electron                (time0) float64 328B 3.639e...
    Total_Particle_Energy_Ion                     (time0) float64 328B 3.57e+...
    ...                                            ...
    Derived_Number_Density_Ion                    (time0, X_Grid_mid) float64 66kB dask.array<chunksize=(1, 200), meta=np.ndarray>
    Derived_Temperature_Electron                  (time0, X_Grid_mid) float64 66kB dask.array<chunksize=(1, 200), meta=np.ndarray>
    Derived_Temperature_Ion                       (time0, X_Grid_mid) float64 66kB dask.array<chunksize=(1, 200), meta=np.ndarray>
    dist_fn_px_py_Electron                        (time0, Px_px_py_Electron, Py_px_py_Electron) float64 13MB dask.array<chunksize=(1, 200, 200), meta=np.ndarray>
    Absorption_Total_Laser_Energy_Injected        (time0) float64 328B 1.416e...
    Absorption_Fraction_of_Laser_Energy_Absorbed  (time0) float64 328B 0.0 .....
Attributes: (12/18)
    file_version:     1
    file_revision:    4
    code_name:        Epoch1d
    jobid1:           1762951482
    jobid2:           578
    code_io_version:  1
    ...               ...
    compile_machine:  uoy24x520
    compile_flags:    unknown
    defines:          50364612
    compile_date:     Wed May 14 13:13:41 2025
    run_date:         Wed Nov 12 12:44:42 2025
    deck:             {'constant': {'I_peak': 1e+24, 'lambda_L': 1e-06, 't_fw...

Loading particle data#

Warning

It is not recommended to use xarray.open_mfdataset or sdf_xarray.open_mfdataset to load particle data from multiple SDF outputs. The number of particles often varies between outputs, which can lead to inconsistent array shapes that these functions cannot handle. Instead, consider loading each file individually and then concatenating them manually.

Note

When loading multiple probes from a single SDF file, you must use the probe_names parameter to assign a unique name to each. For example, use probe_names=["Front_Electron_Probe", "Back_Electron_Probe"]. Failing to do so will result in dimension name conflicts.

By default, particle data isn’t kept as it takes up a lot of space. Pass keep_particles=True as a keyword argument to xarray.open_dataset (for single files) or xarray.open_mfdataset (for multiple files).

Loading specific variables#

When loading datasets containing several (>10) coordinates/dimensions using sdf_xarray.open_mfdataset, xarray may struggle to locate the necessary RAM to concatenate all of the data (as seen in Issue #57). In this instance, you can optimize memory usage by loading only the data you need using the keyword argument data_vars and passing one or more variables. This creates a dataset consisting only of the given variable(s) and the relevant coordinates/dimensions, significantly reducing memory consumption.

import sdf_xarray as sdfxr

sdfxr.open_mfdataset("tutorial_dataset_1d/*.sdf", data_vars=["Electric_Field_Ex"])

<xarray.Dataset> Size: 68kB
Dimensions:            (time: 41, X_Grid_mid: 200)
Coordinates:
  * time               (time) float64 328B 1.334e-16 5.07e-15 ... 2.001e-13
  * X_Grid_mid         (X_Grid_mid) float64 2kB -4.95e-06 ... 1.495e-05
Data variables:
    Electric_Field_Ex  (time, X_Grid_mid) float64 66kB dask.array<chunksize=(1, 200), meta=np.ndarray>
Attributes: (12/22)
    filename:         /home/docs/checkouts/readthedocs.org/user_builds/sdf-xa...
    file_version:     1
    file_revision:    4
    code_name:        Epoch1d
    step:             0
    time:             1.3342563807926084e-16
    ...               ...
    compile_flags:    unknown
    defines:          50364612
    compile_date:     Wed May 14 13:13:41 2025
    run_date:         Wed Nov 12 12:44:42 2025
    io_date:          Wed Nov 12 12:44:42 2025
    deck:             {'constant': {'I_peak': 1e+24, 'lambda_L': 1e-06, 't_fw...

Loading the input.deck#

When loading SDF files, sdf_xarray will attempt to automatically load the input.deck file used to initialise the simulation from the same directory as the SDF file. If the file is not found, it will silently fail and continue loading the SDF file as normal. This file contains the initial simulation setup information which is not present in SDF outputs. By loading this file, you can access these parameters as part of your dataset’s metadata. To do this, use the deck_path parameter when loading an SDF file with sdf_xarray.open_dataset, xarray.open_dataset, sdf_xarray.open_datatree, xarray.open_datatree, sdf_xarray.open_mfdataset or sdf_xarray.open_mfdatatree.

There are a few ways you can load an input deck:

Default behaviour: The input deck is loaded from the same directory as the SDF file if it exists. If it does not exist, it will silently fail.
Relative path: (e.g. "template.deck") Searches for that specific filename within the same directory as the SDF file.
Absolute path: (e.g. "/path/to/input.deck") Uses the full, specified path to locate the file.

An example of loading a deck can be seen below

import json
from IPython.display import Code
import sdf_xarray as sdfxr

ds = sdfxr.open_dataset("tutorial_dataset_1d/0010.sdf")
# The results are accessible by calling
deck = ds.attrs["deck"]

# Some prettification to make it looks nice in jupyter notebooks
json_str = json.dumps(deck, indent=4)
Code(json_str, language='json')

Show code cell output

Hide code cell output

{
    "constant": {
        "I_peak": 1e+24,
        "lambda_L": 1e-06,
        "t_fwhm_L": 1.6e-14,
        "n_elec": 1e+27,
        "density_profile": "if(x lt 0, 0, 1)",
        "mppc": 50,
        "min_den_fact": 0.0001,
        "t_w": "t_fwhm_L / (sqrt(2*loge(2)))",
        "t_hw01m": "t_w * sqrt(loge(10))"
    },
    "control": {
        "t_end": 2e-13,
        "nx": 200,
        "x_min": -5e-06,
        "x_max": 1.5e-05,
        "nparticles": "nint( x_max / (x_max - x_min) * nx * mppc)",
        "dt_multiplier": 0.8,
        "smooth_currents": true,
        "dlb_threshold": 0.8,
        "stdout_frequency": 100
    },
    "laser": {
        "boundary": "x_min",
        "intensity": "I_peak",
        "lambda": "lambda_L",
        "t_profile": "gauss(time, t_hw01m, t_w)",
        "phase": 0
    },
    "boundaries": {
        "bc_x_min_field": "simple_laser",
        "bc_x_max_field": "simple_outflow",
        "bc_x_min_particle": "reflect",
        "bc_x_max_particle": "reflect"
    },
    "species": {
        "Electron": {
            "name": "Electron",
            "frac": 0.5,
            "temp_ev": 1000.0,
            "number_density_min": "n_elec * min_den_fact",
            "number_density": "n_elec * density_profile",
            "identify": "electron"
        },
        "Ion": {
            "name": "Ion",
            "frac": 0.5,
            "temp": "temp_x(Electron)",
            "number_density_min": "n_elec * min_den_fact",
            "number_density": "n_elec * density_profile",
            "identify": "proton"
        }
    },
    "dist_fn": {
        "px_py": {
            "name": "px_py",
            "ndims": 2,
            "dumpmask": "always",
            "direction1": "dir_px",
            "direction2": "dir_py",
            "range1": [
                -1.5e-21,
                1.5e-21
            ],
            "range2": [
                -1.5e-21,
                1.5e-21
            ],
            "resolution1": 200,
            "resolution2": 200,
            "include_species": "Electron"
        }
    },
    "probe": {
        "Electron_Probe": {
            "name": "Electron_Probe",
            "point": 0,
            "normal": -1,
            "ek_min": 0,
            "ek_max": -1,
            "include_species": "Electron",
            "dumpmask": "always"
        }
    },
    "output_global": {
        "force_final_to_be_restartable": false
    },
    "output": {
        "normal": {
            "name": "normal",
            "dt_snapshot": 5e-15,
            "particles": "never",
            "px": "never",
            "py": "never",
            "pz": "never",
            "particle_weight": "never",
            "particle_probes": "always",
            "grid": "always",
            "ex": "always",
            "ey": "always",
            "bz": "always",
            "number_density": "always + species + no_sum",
            "temperature": "always + species + no_sum",
            "total_energy_sum": "always + species",
            "distribution_functions": "always",
            "absorption": "always"
        }
    }
}

Loading SDF Files

Contents