Application Programming Interface

This library contains components for PyVALION software.

PyVALION.library.apply_median_filter(data, half_width, min_valid, mean_width=0)

Apply a median filter with optional averaging to a 1D array.

This function applies a median filter to the input data, preserving NaN values. An optional local averaging around the median can be applied by specifying mean_width. Filtering is only applied if at least min_valid non-NaN points exist in the window.

Parameters:

datanp.ndarray

1D array of data to be filtered.

half_widthint

Half-width of the window for the median filter.

min_validint

Minimum number of valid (non-NaN) points required to compute the median in each window.

mean_widthint, optional

Half-width of averaging around the median. Default is 0 (no averaging).

Returns:

resultnp.ndarray

1D array of filtered data with the same shape as data. NaNs in the original array remain at the same positions.

PyVALION.library.bilinear_weights(alat, alon, ob_lat, ob_lon)

Compute bilinear interpolation weights for a lat/lon grid.

Parameters:

alatarray-like

1D array of model latitudes (monotonic).

alonarray-like

1D array of model longitudes (monotonic).

ob_latfloat

Observation latitude.

ob_lonfloat

Observation longitude.

Returns:

weightsndarray

Array of four bilinear interpolation weights.

i_latndarray

Array of four latitude indices.

i_lonndarray

Array of four longitude indices.

PyVALION.library.compute_jason_tec(iono_ku_delay)

Compute Jason TEC.

See OSTM/Jason-2 Products Handbook Section 4.2.5 for details.

Parameters:

iono_ku_delayfloat

Jason-2 or Jason-3 Ku-band ionospheric delay.

Returns:

tecfloat

Total electron content in TECU.

PyVALION.library.compute_lanczos_kernel(N, cutoff)

Build a symmetric Lanczos kernel with half-width N and cutoff.

This function creates a normalized symmetric Lanczos kernel using a sinc filter multiplied by a Lanczos window.

Parameters:

Nint

Half-width of the kernel.

cutofffloat

Cutoff frequency.

Returns:

kernelnp.ndarray

Normalized symmetric Lanczos kernel.

PyVALION.library.concat_data_dicts(a_dict, b_dict)

Concatenate two dictionaries with the same field names.

This function merges the contents of dictionaries A and B, assuming both have identical field names. The data from A appears before B.

Parameters:

a_dictdict

Dictionary with identical field names to B.

b_dictdict

Dictionary with identical field names to A.

Returns:

c_dictdict

Dictionary with all the data from A and B combined (A preceding B).

PyVALION.library.count_cycle_subcatalogs(url)

Count the number of cycle sub-catalogs in a THREDDS catalog.

Parameters:

urlstr

Base catalog url from www.ncei.noaa.gov THREDDS containing “cycle” sub-catalogs.

Returns:

cycle_numint

Total number of cycle sub-catalogs in base url.

PyVALION.library.create_manifest(output_file)

Create a new manifest of all Jason-2 and Jason-3 .nc files.

Scans all THREDDS catalog URLs, collects netCDF file URLs, deduplicates by filename, and saves the sorted list to the specified manifest file.

Parameters:

output_filestr

Path to the manifest file to create.

PyVALION.library.create_or_update_manifest(output_file)

Create a new manifest or update an existing one.

Checks if the manifest file exists. If not, creates a new manifest; otherwise, updates it with new files from recent cycles.

Parameters:

output_filestr

Path to the manifest file to create or update.

PyVALION.library.download_GIRO_parameters(time_start, time_finish, ion_name, data_save_dir, save_dir, name_run, clean_directory=True, filter_CS=90)

Retrieve GIRO ionospheric parameters from fromlgdc.uml.edu.

Parameters:

time_startdatetime.datetime

Start date and time for the validation period

time_finishdatetime.datetime

End date and time for the validation period

ion_namearray-like

String arrays of GIRO ionosondes names

data_save_dirstr

Directory where to save the downloaded files

save_dirstr

Directory where to save the downloaded files

name_runstr

String to add to the name of the files for saved results

clean_directorybool

If set to True the downloaded .txt ionosonde file will be deleted (default=True)

filter_CSfloat

Minimum accepted Autoscaling Confidence Score. Expects values from 0 to 100, with flag values of 999 for manual scaling and -1 if unknown. (default=90)

Returns:

data_alldict

Dictionary with all the data combined.

PyVALION.library.download_Jason_TEC(time_start, time_finish, save_dir, name_run='', save_data_option=False, include_neg=True, sat_names=array(['JA2', 'JA3'], dtype='<U3'), jason_manifest_filename='jason_manifest.txt')

Retrieve Jason ionospheric TEC from from www.ncei.noaa.gov THREDDS.

Parameters:

time_startdatetime.datetime

Start date and time for the validation period.

time_finishdatetime.datetime

End date and time for the validation period.

save_dirstr

Directory where to save the downloaded data and where Jason file manifest is stored.

name_runstr

String to add to the name of the files for saved results. Defaults to empty string.

save_data_optionbool

Option to save data as a pickle (.p) file into save_dir. Defaults to False.

include_negbool

Option to include negative TEC values in the output. Defaults to True.

sat_namesarray-like

String arrays of Jason satellite names (“JA2” and/or “JA3”).

jason_manifest_filenamestr

String to designate an alternative Jaosn file manifest filename.

Returns:

data_alldict

Dictionary with all the data combined.

PyVALION.library.downsample_Jason_TEC(data_all, ddeg, save_dir='', name_run='', save_data_option=False)

Downsample all fields in a Jason data dictionary by a specified degree.

Parameters:

data_alldict

Dictionary of raw Jason data.

ddegfloat

Model resolution in degrees. Data will be resampled to approximately half this resolution in order to maintain adequate sampling.

save_dirstr, optional

Directory where the resampled data will be saved. Defaults to ‘’.

name_runstr, optional

String appended to the output filename. Defaults to ‘’.

save_data_optionbool, optional

If True, save data as a pickle (.p) file in save_dir. Defaults to False.

Returns:

data_resampdict

Dictionary of Jason data with downsampled fields.

PyVALION.library.downsample_dict(data_raw, N)

Downsample all fields in a dictionary by keeping every Nth point.

Parameters:

data_rawdict of array-like (np.ndarray or pd.Series)

Dictionary of data fields to downsample.

Nint

Step size indicating how frequently to sample. Keeps every Nth point.

Returns:

dict_resampdict

Dictionary with downsampled fields.

PyVALION.library.extract_cycle_number(url)

Extract the cycle number from a URL string.

Extract the cycle number from a URL string (e.g., ‘cycle345’ → 345). Used to update Jason file manifest from last recorded cycle. Returns -1 if not found.

Parameters:

urlstr

URL containing Jason filename from www.ncei.noaa.gov THREDDS catalog.

Returns:

cycle_numint

Cycle number from Jason filename URL.

PyVALION.library.extract_first_date(url)

Extract first date from URL.

Extract the first occurrence of a datetime stamp in the format YYYYMMDD_HHMMSS from the filename. Returns a datetime object if found, otherwise a fallback datetime far in the future.

Parameters:

urlstr

URL containing Jason filename from www.ncei.noaa.gov THREDDS catalog.

Returns:

first_datedatetime.datetime

Start date of the Jason file.

PyVALION.library.fill_small_gaps(data, max_gap_size)

Linearly interpolate gaps smaller than max_gap_size.

Parameters:

datanp.ndarray

1D array of data with NaN values representing gaps.

max_gap_sizeint

Maximum consecutive NaN values to interpolate. Gaps longer than this will remain as NaN.

Returns:

interp_allnp.ndarray

1D array with small gaps linearly interpolated, preserving larger gaps as NaN.

PyVALION.library.filter_GIRO_parameters(time_start, time_finish, ion_name, save_dir, filter_CS=90)

Download data from all GIRO ionosondes for the whole validation period.

Parameters:

time_startdatetime.datetime

Start date and time for the validation period

time_finishdatetime.datetime

End date and time for the validation period

ion_namearray-like

String arrays of GIRO ionosondes names

save_dirstr

Directory where the downloaded files are

filter_CSfloat

Minimum accepted Autoscaling Confidence Score. Expects values from 0 to 100, with flag values of 999 for manual scaling and -1 if unknown. (default=90)

Returns:

data_alldict

Dictionary with all the data combined.

PyVALION.library.filter_urls_by_timerange(manifest_filepath, user_start, user_finish, sat_names)

Filter list of URLs by user timespan.

Reads a .txt file of URLs and returns only those where the file’s time range overlaps with the user-specified datetime window.

Parameters:

manifest_filepathstr

Path to Jason file manifest (.txt file containing URLs).

user_startdatetime.datetime

Start date and time for the validation period.

user_finishdatetime.datetime

End date and time for the validation period.

sat_namesnp.ndarray

Array of satellite name substrings to include (e.g., “JA2”, “JA3”).

Returns:

matching_urlslist of tuples

Each tuple contains (url, start_datetime, end_datetime) for files that overlap with the specified time window.

PyVALION.library.find_G_and_y(adtime, alon, alat, data, save_dir, name_run, interp_method='bilinear', pickle_outputs=True)

Create geometry matrix G and observation vector.

Parameters:

adtimedatetime.datetime

Array of time

alonarray-like

Model array of longitude in degrees (from -180 to 180)

alatarray-like

Model array of latitudes in degrees (from -90 to 90)

datadict

Dictionary of GIRO data produced by filter_GIRO_parameters.

save_dirstr

Directory where to save the downloaded files

name_runstr

String to add to the name of the files for saved results

interp_methodstr, optional

Method for interpolating model data to observation points. Options are ‘bilinear’ (default) and ‘nearest’. ‘bilinear’ performs bilinear interpolation using the four nearest grid points, while ‘nearest’ assigns the value of the nearest model grid point to each observation.

pickle_outputsbool

It true the outputs are also pickled in the root directory, default is True.

Returns:

ydict

Dictionary that contains observation vectors: ay_nmf2 : array-like

Array of GIRO NmF2 parameter in m-3.

ay_hmf2array-like

Array of GIRO hmF2 parameter in km.

ay_B0array-like

Array of GIRO B0 parameter in km.

ay_B1array-like

Array of GIRO B1 parameter, unitless.

ay_lonarray-like

Array of observation longitude in degrees.

ay_latarray-like

Array of observation latitudes in degrees.

ay_namearray-like

Array of ionosonde names.

ay_timearray-like

Array of observation time.

unitsdict

Dictionary with strings of units for the keys in dict y

Garray-like

Geometry matrix (N_obs, N_time, N_lat, N_lon)

ion_infodict

Dictionary with ionosonde information lon : array-like

Array of longitudes for the stations in degrees.

ay_latarray-like

Array of latitudes latitudes for the stations in degrees.

namearray-like

Array of names of the stations.

PyVALION.library.find_Jason_G_and_y(adtime, alon, alat, data, interp_method='bilinear')

Create geometry matrix G and observation vector y.

Parameters:

adtimearray-like of datetime.datetime

Array of time stamps for the model.

alonarray-like

Array of model longitudes in degrees (-180 to 180).

alatarray-like

Array of model latitudes in degrees (-90 to 90).

datadict

Dictionary of Jason data produced by download_Jason_TEC or downsample_Jason_TEC.

interp_methodstr, optional

Method for interpolating model data to observation points. Options are ‘bilinear’ (default) and ‘nearest’. ‘bilinear’ performs bilinear interpolation using the four nearest grid points, while ‘nearest’ assigns the value of the nearest model grid point to each observation.

Returns:

ydict

Dictionary containing the following keys and values: TEC : array-like

Jason total electron content in TECU.

lonarray-like

Observation longitudes in degrees.

latarray-like

Observation latitudes in degrees.

namearray-like

Jason satellite names.

timearray-like

Observation times.

unitsdict

Dictionary mapping keys in y to their units as strings.

Gndarray

Geometry matrix with shape (N_obs, N_time, N_lat, N_lon).

PyVALION.library.find_Jason_residuals(model, G, obs_data, units)

Find residuals from model data for the given field and forward operator.

Parameters:

modeldict

Dictionary with model parameters in shape [N_time, N_lat, N_lon].

Garray-like

Geometry matrix [N_obs, N_time, N_lat, N_lon].

obs_datadict

Dictionary containing the following keys and values: TEC : array-like

Array of Jason total electron content in TECU.

lonarray-like

Array of observation longitude in degrees.

latarray-like

Array of observation latitudes in degrees.

namearray-like

Array of Jason satellite names.

timearray-like

Array of observation time.

unitsdict

Dictionary with strings of units for the keys in dict y.

Returns:

model_datadict

Array of model data, the expected data according to the model.

residualsdict

Dictionary containing residuals (obs_data - model_data).

model_unitsdict

Dictionary with strings of units for the keys in dict model_data.

PyVALION.library.find_model_data(field, G)

Find model data for the given field and forward operator.

Parameters:

fieldarray-like

2-D filed of a model parameter in shape [N_time, N_lat, N_lon].

Garray-like

Forward operator matrix for the ionosonde data in shape [N_obs, N_time, N_lat, N_lon].

Returns:

model_dataarray-like

Array of model data, the expected data according to the model.

PyVALION.library.find_residuals(model, G, obs_data, obs_info, units)

Find model data for the given field and forward operator.

Parameters:

modeldict

Dictionary with model parameters in shape [N_time, N_lat, N_lon].

Garray-like

Forward operator matrix for the ionosonde data in shape [N_obs, N_time, N_lat, N_lon].

obs_datadict

Dictionary that contains observation vectors: ay_nmf2 : array-like

Array of GIRO NmF2 parameter in m-3.

ay_hmf2array-like

Array of GIRO hmF2 parameter in km.

ay_B0array-like

Array of GIRO B0 parameter in km.

ay_B1array-like

Array of GIRO B1 parameter, unitless.

ay_lonarray-like

Array of observation longitude in degrees.

ay_latarray-like

Array of observation latitudes in degrees.

ay_namearray-like

Array of ionosonde names.

ay_timearray-like

Array of observation time.

obs_infodict

Dictionary with ionosonde information lon : array-like

Array of longitudes for the stations in degrees.

ay_latarray-like

Array of latitudes latitudes for the stations in degrees.

namearray-like

Array of names of the stations.

unitsdict

Dictionary with strings of units for the keys in dict y

Returns:

model_datadict

Array of model data, the expected data according to the model.

residualsdict

Array of residuals.

model_unitsdict

Dictionary with strings of units for the keys in dict model_data

res_iondict

Array of mean residuals for each ionosonde station.

PyVALION.library.freq2den(freq)

Convert ionospheric frequency to plasma density.

Parameters:

freqarray-like

ionospheric freqeuncy in MHz.

Returns:

densarray-like

plasma density in m-3.

Notes

This function converts ionospheric frequency to plasma density.

PyVALION.library.lanczos_filter(signal, N, cutoff, min_valid=3)

Apply a Lanczos filter to a 1D signal, ignoring NaNs.

This function convolves the input signal with a symmetric Lanczos kernel of half-width N and the specified cutoff frequency. Elements with NaN values are ignored, and filtering is applied only if at least min_valid non-NaN points are available in the kernel window.

Parameters:

signalnp.ndarray

1D array of data to be filtered.

Nint

Half-width of the Lanczos kernel.

cutofffloat

Cutoff frequency for the filter.

min_validint, optional

Minimum number of valid (non-NaN) points required to apply the filter at each position. Default is 3.

Returns:

filterednp.ndarray

1D array of the filtered signal, with the same shape as signal. Positions with insufficient valid points remain NaN.

PyVALION.library.list_nc_files(url, root_base=None, progress=None)

Recursively list netCDF files in a THREDDS catalog.

This function traverses a THREDDS catalog at the given URL and returns all .nc file URLs served via the HTTPServer interface. If a root_base is provided, it is used as the base URL for constructing full paths. An optional progress object can be updated for each subcatalog processed.

Parameters:

urlstr

URL of the THREDDS catalog (ending with catalog.xml).

root_basestr, optional

Base URL to use when constructing full dataset URLs. If None, the base is inferred from url.

progressobject, optional

Progress indicator with an update(n) method (e.g., a tqdm instance) to track progress while iterating through subcatalogs.

Returns:

nc_urlslist of str

List of .nc file URLs accessible via HTTPServer (converted to DODS/C URLs for subsequent downloads).

PyVALION.library.make_empty_dict_data()

Make empty dictionary to collect GIRO data.

Returns:

datadict

Dictionary with empty elements.

PyVALION.library.make_empty_dict_data_jason()

Make empty dictionary to collect Jason-2 and -3 data.

Returns:

datadict

Dictionary with empty elements.

PyVALION.library.mask_dict(data_raw, mask)

Mask data within all data dictionary fields.

This function removes elements from each field in the data dictionary according to the mask provided: 1 for keep, 0 for discard.

PyVALION.library.nearest_element(array, value)

Locate nearest element in array.

Parameters:

arrayarray-like

Given array.

valeuflt

Given value.

Returns:

smallest_difference_indexint

Index of the element.

PyVALION.library.read_GIRO_line(line_arr)

Read 14-element line from GIRO ionosonde file.

Parameters:

line_arrstr

String from the GIRO file.

Returns:

stampdatetime.datetime

Datetime stamp from the line.

scorefloat

Confidence score CS.

fof2float

Critical frequency of F2 layer in MHz.

hmf2float

Height of the F2 layer in km.

B1float

Bottom-side thickness of F2 layer in km.

B0float

Bottom-side shape parameter unitless.

PyVALION.library.read_jason2_file(j2url, include_neg=True)

Read in relevant data from Jason-2 OPeNDAP URL.

See OSTM/Jason-2 Products Handbook Section 4.2.5 for details.

Parameters:

j2urlstr

Jason-2 OPeNDAP url.

include_negbool

Option to include negative TEC values in the output. Defaults to True.

Returns:

data_alldict

Dictionary with all the data from a single file combined.

PyVALION.library.read_jason3_file(j3url, include_neg=True)

Read in relevant data from Jason-3 OPeNDAP URL.

See OSTM/Jason-3 Products Handbook for details.

Parameters:

j3urlstr

Jason-3 OPeNDAP url.

include_negbool

Option to include negative TEC values in the output. Defaults to True.

Returns:

data_alldict

Dictionary with all the data from a single file combined.

PyVALION.library.robust_iterative_filter(data_raw, INIT_SIGMA=3, DATA_GAP_MAX=0, MEDIAN_NB_PTS=30, MEDIAN_NB_PTS_MIN=30, MEDIAN_NB_PTS_MEAN=0, LANCZOS_NB_PTS=50, LANCZOS_NB_PTS_CUTOFF=50, LANCZOS_NB_PTS_MIN=10, CONVERGENCE_THRESHOLD=0)

Robust iterative filtering to remove outliers from a 1D signal.

This function performs a multi-step iterative filtering procedure to remove outliers from the input data. Steps include: 1. Initial global sigma-based outlier detection. 2. Optional linear interpolation of small gaps. 3. Median filtering. 4. Lanczos filtering. 5. Residual-based outlier flagging. The iteration continues until convergence criteria are met.

Parameters:

data_rawnp.ndarray

1D array of raw input data.

Returns:

data_lanczosnp.ndarray

Filtered 1D array after robust iterative filtering.

outlier_masknp.ndarray of bool

Boolean mask indicating which points were flagged as outliers.

PyVALION.library.round_and_stringify(lat, lon, rounding_interval)

Round latitude and longitude values and generate string identifiers.

Rounds latitude and longitude values to the nearest multiple of a specified model resolution and returns a string array used to determine the resampling rate of Jason data.

Parameters:

latndarray of float

Latitude values.

lonndarray of float

Longitude values.

rounding_intervalfloat

Model resolution in degrees.

Returns:

coor_strndarray of str

Array of strings containing rounded coordinates. Values are scaled to the nearest integer by a scaling factor. For example, if rounding_interval=0.25, values will be scaled by 4.

lat_roundedndarray of float

Array of rounded latitude values.

lon_roundedndarray of float

Array of rounded longitude values.

PyVALION.library.safe_float(line_arr, index, label)

Check if element is float.

Parameters:

line_arrstr

String from the GIRO file.

indexind

Index of the array to check.

labelstr

Label of the array to print out.

Returns:

resflt

If given element is can be converted to float return it, if not, returns nan.

PyVALION.library.sza_data_space(dtime, alon, alat)

Compute solar zenith angles for a sequence of times and locations.

Parameters:

dtimenp.ndarray

Array of datetime objects.

alonnp.ndarray

Array of longitudes (degrees East).

alatnp.ndarray

Array of latitudes (degrees North).

Returns:

solzennp.ndarray

Array of solar zenith angles in degrees.

Raises:

ValueError

If input arrays have different shapes

PyVALION.library.update_manifest(output_file)

Update a manifest file by appending new .nc files from newer cycles.

Scans ‘cycleXXX’ folders with cycle numbers equal to or higher than the last entry and appends only files with timestamps newer than the most recent entry in the manifest.

Parameters:

output_filestr

Path to the manifest file to update.

This library contains components visualization routines for PyIRI.

PyVALION.plotting.plot_TEC_residuals_histogram(residuals, model_units, dtime, save_option=False, save_dir='', plot_name='TEC_Residuals')

Plot residuals for TEC as a histogram.

Parameters:

residualsdict

Dictionary with residuals.

model_units: dict

Dictionary with units for each key in residuals.

dtimedatetime.datetime

Datetime of the validation day(s).

plot_dirstr

Direction where to save the figure.

plot_namestr

Output name, without directory, for the saved figure (default=’TEC_Residuals.pdf’).

Returns:

figmatplotlib.figure.Figure

Figure containing the residual histogram.

PyVALION.plotting.plot_TEC_residuals_map(alat, alon, residuals, dtime, save_option=False, save_dir='', plot_name='TEC_Residuals_Map')

Plot residual map for TEC.

Parameters:

alonarray-like (float)

Flattened array of longitudes in degrees.

alatarray-like (float)

Flattened array of latitudes in degrees.

residualsdict

Dictionary with TEC residuals.

model_units: dict

Dictionary with units for each key in residuals.

dtimedatetime.datetime

Datetime of the validation day.

plot_dirstr

Direction where to save the figure.

plot_namestr

Output name, without directory, for the saved figure (default=’TEC_Residuals_Map.pdf’).

Returns:

figmatplotlib.figure.Figure

Figure containing the residual map.

PyVALION.plotting.plot_histogram(residuals, model_units, dtime, plot_dir, plot_name='Residuals.pdf')

Plot residuals for each model parameter.

Parameters:

residualsdict

Dictionary with residuals.

model_units: dict

Dictionary with units for each key in residuals.

dtimeclass:datetime.datetime

Datetime of the validation day.

plot_dirstr

Direction where to save the figure.

plot_namestr

Output name, without directory, for the saved figure (default=’Residuals.pdf’)

PyVALION.plotting.plot_individual_mean_residuals(res_ion, ion_info, model_units, dtime, plot_dir, plot_name='Ion_Residuals.pdf')

Plot residuals for each ionosonde.

Parameters:

res_iondict

Dictionary with mean residuals for individual ionosondes.

ion_infodict

Dictionary with ionosonde information lon : array-like

Array of longitudes for the stations in degrees.

ay_latarray-like

Array of latitudes latitudes for the stations in degrees.

namearray-like

Array of names of the stations.

model_units: dict

Dictionary with units for each key in res_ion.

dtimeclass:datetime.datetime

Datetime of the validation day.

plot_dirstr

Direction where to save the figure.

plot_namestr

Output name, without directory, for the saved figure (default=’Ion_Residuals.pdf’)

Notes

Each residuals parameter will be saved in a different plot, with that parameter appended to the start of the default plot name (e.g., ‘ay_nmf2_Ion_Residuals.pdf’)

PyVALION.plotting.plot_ionosondes(y_instr_info, dtime, plot_dir, plot_name='Ionosondes_Map.pdf')

Plot map with GIRO ionosondes that were used for validation.

Parameters:

y_instr_infodict

Dictionary with ionosonde information lon : array-like

Array of longitudes for the stations in degrees.

ay_latarray-like

Array of latitudes latitudes for the stations in degrees.

namearray-like

Array of names of the stations.

dtimeclass:datetime.datetime

Datetime of the validation day.

plot_dirstr

Direction where to save the figure.

plot_namestr

Output name, without directory, for the saved figure (default=’Ionosondes_Map.pdf’)