CalSciPy
CalSciPy is a utility toolbox for calcium imaging experiments. It contains a variety of useful features, from interactive visualization of data to computer-generated holography for “read/write” experiments, and everything in-between. Essentially, it’s a collection of code written for my imaging experiments that might be considered useful to others.
Motivation
While working on my Ph.D. I noticed I was constantly re-writing boilerplate, “one-liners” and copying/pasting more-extensive code between environments. Ever try to use a minimally-documented function written during an exploratory analysis on a different operating system two years later? How much time have you spent debugging hardcoded variables after adapting functions from one study to the next? During some lunchtime reflection, I realized that a lot of my colleagues were simultaneously writing (and re-writing) very similar code to mine. How much time have scientists spent reinventing the wheel? Why did each of us write our own GUI for interacting looking at traces? And why do they all rely unstandardized, idiosyncratic organizations of datasets. I decided to spend a little extra time thoroughly documenting and testing to put together this package so at least my friends and I could easily analyze data across various environments, interpreters and operating systems. While having all your code so well documented and tested takes more time–I found it does wonders for your stress levels (documenting is a relaxing monotonous activity) and people like you more. Plus, all the abstraction and documentation makes your analysis easy to read and follow. As my programming skills and scientific expertise have grown, I’ve started to roadmap the inclusion of some more complex functionality–refactoring my existing code of interest, writing code for my projects with future distribution in mind, and implementing new features the science community could benefit from–because making scientific software open-source and proliferating expertise is simply the right thing to do.
Highlights
Useful, well-documented functions often used in boilerplate code alongside registration and roi extraction software packages such as Caiman, SIMA, and Suite2P.
Flexible & extensible system for neatly organizing data, timestamping analysis, & single-step analyses.
Quality-of-life utilities for Bruker’s PrairieView software
Interactive data visualization
Tools for optics & optogenetics
Getting Started
Installation
Eventually I will break the distributions up so you don’t have to necessarily install some of the more tangential modules. For right now, everything is in one package.
Enter pip install CalSciPy in your terminal.
If you want to use gpu-parallelization, you will need to install CuPy by following these instructions.
Limitations
The current implementation is partially finished. Please be patient, this is a pet-project, I have a newborn, and I need to finish my thesis. Until things are more finalized, I’ll explicitly note which modules are stable, have >90% test coverage, and are ready-to-use. Anything not explicitly listed as stable should be considered experimental and should be used at your own risk. If you really need something labeled experimental I can probably provide offline code that will probably prove more useful short-term. While the majority of features are already written, they need to be refined for maintainability, readability, and to remove any bugs.
Stable Modules
The following modules are stable as of the indicated version. New features added will be demarcated with a version added tag in the documentation and there will be no further breaking changes.
color_scheme0.7.5events0.7.5images0.7.5io_tools0.7.5traces0.7.5
Unstable Modules
These modules are considered unstable. They may be experimental, undocumented, untested, or unfinished.
interactiveopticsorganization
Loading and Saving Data
Simple importing and exporting of imaging data is a significant source of boilerplate code in most processing pipelines.
The io_tools module provides a few functions for loading, converting, and saving imaging
data with very simple syntax and good performance. Currently, *.tif, *.gif, *.mp4, and *.bin (binary) file
types are supported. Every function either returns or expects the images in the form of a
numpy arrays with shape frames, y-pixels, x-pixels. They also all use a similar syntax:
load_{file_type}(path) for loading and save_{file_type}(path, images) for saving.
Note
Color videos and gifs additionally accept numpy arrays with
shape frames, y-pixels, x-pixels, color.
Loading .tif’s
CalScipy offers a single, simple function for loading images with the *.tif file format and other closely associated
formats like *.ome.tiff: load_images.
Loading a 2D-image
images = load_images("single_image.tif")
Loading a 3D-stack
images = load_images("imaging_stack.tif")
Loading entire folders of imaging stacks
images = load_images("imaging_folder")
Easy, huh?
Saving .tif’s
CalScipy also offers a single, simple function for saving images with the *.tif file format. If the image size is larger the size_cap limit, the stack will be automatically split into chunks of size size_cap. By default, the size_cap is set to limit .tif stacks to less than 4GB each for compatibility with the majority of *.tif readers.
Saving images to file
save_images("single_image.tif", images)
Saving images to a folder
save_images("desired_folder", images)
Saving images to a folder with specified name
save_images("desired_folder", images, name="example_images")
Loading .bin’s
Binary data in CalSciPy can be loaded using the load_binary function with a
similar syntax. However, additional arguments are available to load the images without reading the entire file into
memory (i.e., memory-mapping).
Loading binary data directly from file
images = load_binary("binary.bin")
Loading binary data directly from a folder
images = load_binary("desired_folder")
Loading memory mapped binary data
images = load_binary("desired_folder", mapped=True, mode="r")
Loading binary data with missing metadata
missing_metadata = {"frames": 100, "y": 100, "dtype": int}
images = load_binary("desired_folder", missing_metadata=missing_metadata)
Saving .bin’s
Binary data can be saved to file using the save_binary <CalSciPy.io_tools.save_binary() function.
Saving binary to file
save_binary("binary_file.bin", images)
Saving binary to folder
save_binary("desired_folder", images)
Saving binary to folder with specified name
save_binary("desired_folder", images, name="example_binary")
Tip
This language-agnostic format is ideal for optimal read/write speeds, larger-than-memory data, and is highly-robust to corruption. However, it does have downsides. First, the images and their metadata are split into two separate files: “.bin” and “.json” respectively. If you happen to lose the metadata file, fear not! As long as you have the datatype and 2 of the 3 dimensions you can still load the data. A second disadvantage is a lack of compression. Using binary is excellent in cases where storage space is “cheaper” than I/O time: for example, when data is still being regularly accessed and not simply sitting in “cold storage”.
Loading .mp4’s
Loading *.mp4's uses the load_video function, returning the video as
a numpy array with shape frames, y-pixels, x-pixels, colors.
Loading video from file
images = load_video("video_file.mp4")
Loading video from folder
images = load_video("desired_folder")
Saving .mp4’s
Saving *.mp4's uses the save_video function. The frame rate of the video can be
set with the frame_rate argument.
Saving video to file
save_video("video_file.mp4", images)
Saving video to folder
save_video("desired_folder", images)
Saving video to folder with specified name
save_video("desired_folder", images, name="example_binary")
Saving video to folder with specified framerate
save_video("video_file.mp4", images, frame_rate=90.0)
Loading .gif’s
Loading *.gif's uses the load_gif function.
Loading a *.gif
gif = load_gif("gif_file.gif")
Saving .gif’s
Saving your images as a *.gif is as easy as using the save_gif function.
Saving a *.gif
save_gif("gif_file.gif", images, frame_rate=5.0)
Tip
Inserting videos into a presentation as a *.gif is a clever way to avoid technical difficulties (shudder).
Image Pre-Processing
The images module contains functions commonly used in pre-processing imaging data.
Deinterlacing
Resonance-scanning microscopes generate pixel-streams that most microscopy software organize by resonance-scanning
cycles (see explanation). Rather than immediately organizing pixels into individual lines, pixels are collected through
the complete oscillation of the resonant scanner. The data is then split down the center and the latter half reversed
to generate two lines. Because the exact angle of the mirror at any given time is variable, the exact index of the
center pixel is usually defined manually within the software or estimated in real-time by a software algorithm.
Never-the-less, variations related to temperature, drive fluctuations, and murphy’s law–as well as
poor signal-to-noise–often result in images with interlacing artifacts. CalSciPy provides a convenient
deinterlace function to correct for these artifacts.
Deinterlacing images
from CalSciPy.images import deinterlace
import numpy as np
# standard case
images = deinterlace(images)
# correcting noisy data with external reference
y_pixels, x_pixels = images.shape[1:]
reference = np.std(images, axis=0).reshape(1, y_pixels, x_pixels) # get z-projected standard deviation
images = deinterlace(images, reference=reference)
Unfortunately, the fast-fourier transform methods that underlie the implementation of the deinterlacing algorithm have poor spatial complexity (i.e., large memory constraints). This weakness is particularly problematic when using GPU-parallelization. To mitigate these issues, deinterlacing can be performed in-place and batch-wise while maintaining numerically identical results.
Deinterlacing images (memory constrained)
from CalSCiPy.images import deinterlace
# In-place
deinterlace(images, in_place=True)
# Batch-wise
images = deinterlace(images, batch_size=5000)
Explanation: Resonance Scanning
Resonance scanning microscopes achieve fast image acquisition by pairing a slow-galvometric mirror with a fast-resonance scanning mirror: the fast-resonant scanning mirror rapidly scans a single-axis of the field-of-view (i.e., a horizontal line), while the slow-galvometric mirror moves the line along a second-axis. Further gains in acquisition speed are achieved by acquiring images during both the forward and backward scans of the resonant mirror. That is, lines are scanned in alternating directions instead of returning to the origin after each scan in a left-right, left-right, left-right strategy.
While galvometric scanning mirrors can be tightly-controlled using servos, resonant scanning mirrors lack such granular control. Resonance scanning mirrors achieve their rapid motion by vibrating at a fixed frequency. Resonance scanning mirrors are extremely under-dampened harmonic oscillators: while their frequency is tightly distributed, they are prone to large variations in amplitude given very small deviations in their drive. Resonance scanning mirrors also display a smooth cosinusoidal velocity through their entire range-of-motion–the scanner moves slower near the limits of its range–that further complicates synchronization to an external frequency or other means of fine control. Therefore, the entire microscope is typically aligned to the motion of the resonance scanner.
Cycle of a Resonant Scanning Mirror
Filtering
Filtering imaging stacks
from CalSciPy.images import gaussian_filter
# standard deviation of gaussian kernel
sigma = 1.0
filtered_images = gaussian_filter(images, sigma=sigma)
In some situations you may be under memory-constraints. CalScipy supports both in-place and blockwise filtering in these scenarios: simply utilize the in_place or block_size keywords.
Memory-constrained filtering
from CalSciPy.images import median_filter
# size of median filter
window = (3, 3, 3)
# 7000 frame blocks
filtered_images = median_filter(images, window=window, block_size=7000)
# 7000 frame blocks with 3500 frame overlap
filtered_images = median_filter(images, window=window, block_size=7000, block_buffer=3500)
# in-place calculation
filtered_images = median_filter(images, window=window, in_place=True)
Available Multi-dimensional Filters
Note
Using gpu-parallelization is recommended to quickly process imaging stacks. Being said, using gpu parallelization requires that the dataset fit within your GPU’s VRAM. In most cases, this requires breaking the dataset down into smaller blocks. This can be done automatically by using the block_size keyword.
Tip
Median filtering is particularly beneficial for denoising images with mild-to-moderate, signal-independent noise (e.g., the speckle that can occur on images collecting while supplying photomultiplier tubes with high-voltage). It tends to cause less temporal distortion than gaussian filtering as it simply replaces outliers with common datapoints. It also tends to induce less blurring of edges in the image (e.g., spikes, cell borders), though in the worst-case both filters are equally problematic.
Trace Processing
Calculating Δf/f0
CalSciPy supports a variety of methods for calculating the fold fluorescence over baseline through a single function.
from CalScipy.traces import calculate_dfof
dFoF = calculate_dfof(traces, method="mean")
Supported baseline calculation methods
“low-pass”: x-th percentile of the :func:` low-pass <CalSciPy.traces.baseline.low_pass_baseline>` filtered trace
“mean”:
meanof the trace“median”:
medianof the trace“moving_mean”:
moving meanof the trace using a specified window length“percentile”:
x-th percentileof the trace“sliding_mean”:
sliding meancalculated using a sliding window of specified length“sliding_median:
sliding mediancalculated using a sliding window of specified length“sliding_percentile:
sliding percentilecalculated using a sliding window of specified length
Baseline Corrections
CalSciPy currently provides a function for polynomial detrending to correct for a drifting baseline due to time-dependent degradation of signal-to-noise
from CalScipy.traces import detrend_polynomial
detrended_traces = detrend_polynomial(traces, frame_rate=frame_rate)
Assessing Trace Quality
CalSciPy supports assessment of trace quality using a standardized noise metric first defined in the publication associated with the spike-inference software package CASCADE.
from CalScipy.traces import calculate_standardized_noise
# acquisition frequency
frame_rate = 30
standardized_noise = calculate_standardized_noise(dFoF, frame_rate=frame_rate)
Events and Spiking
Note
Inferring spikes from calcium activity is a complex problem. Whether your “spikes” represent a single discrete spike–or a multi-spike event–depends on your choice of inference algorithm, the indicator, sampling rate, imaging quality, and innumerable other factors. It follows that the term event and spike are used interchangeably in CalSciPy.
Generating Rasters
To generate a raster, utilize the generate_raster function.
Firing Rates
CalSciPy provides several small functions that abstract firing rate calculations and improve the readability of your scripts.
calc_firing_rates: Calculate instantaneous firing rates from spike (event) probabilities.calc_mean_firing_rates: Calculate mean firing rates from instantaneous firing rates.normalize_firing_rates: Normalize firing rates by scaling to a max of 1.0.
Reorganizing Data
CalSciPy contains several simple utility functions for converting data between several common forms.
Converting Matrices and Tensors
Often times one might want convert a matrix into a tensor. For example, perhaps one wants to convert a neurons x sample matrix consisting an entire session into a trials x neurons x trial samples tensor.
from CalSciPy.conversion import matrix_to_tensor, tensor_to_matrix
data = np.ones((5, 3600))
samples_per_trial = 600
tensor_form = matrix_to_tensor(data, samples_per_trial)
>>>print(f"{tensor_form.shape=})
tensor_form.shape=(6, 5, 600)
matrix_form = tensor_to_matrix(tensor_form)
>>>print(f"{matrix_form.shape=})
matrix_form.shape=(5, 3600)
Regions of Interest
The roi_tools module provided a standardized ROI class
to represent regions of interest in your imaging data. By having this standardized class, we can import ROIs from any
arbitrary data structure. That is, we can load ROIs from any analysis software we want. A dictionary of ROIs can be
easily formed using the default or custom ROI Handlers. Software-specific
ROI Handlers are available for many major calcium imaging software packages
and a metaclass is provided for customizing your own.
ROI instances
Each ROI instance contains the basic characteristics & properties of an ROI.
Constructing an ROI requires only knowledge of its pixel coordinates. From these coordinates, a variety of attributes
are calculated, including its centroid, radius, and approximate outline. Importantly, these attributes are only
calculated at instantiation and then permanently cached for performance benefits. Therefore, changing the pixels of an
existing ROI instance is not permitted. A properties attribute is provided to allow
users to incorporate arbitrary information for a particular ROI (e.g., SNR). All keyword arguments will be passed to
the properties attribute.
Generating an ROI instance
import numpy as np
from CalSciPy.roi_tools import ROI
x_pixels = [31, 31, 31, 32, 32, 32, 33, 33, 33]
y_pixels = [31, 32, 33, 31, 32, 33, 31, 32, 33]
reference_shape = (64, 64)
properties = {"cool_roi": True}
roi = ROI(x_pixels, y_pixels, reference_shape, properties, special_roi=True)
>>>print(f"{roi.radius=}")
roi.radius=1
>>>print(f"{roi.centroid=}")
roi.centroid=(32, 32)
>>>print(f"{roi.properties}")
roi.properties={'special_roi': True, 'cool_roi': True}
Importing ROIs from File
The ROI Handler metaclass describes a standardized approach to loading
ROIs and can be adapted to handle any arbitrary data structure your ROIs may be stored in. Calling the
load class method for any ROIHandler will generate a tuple
containing a dictionary of ROIs and a reference image in the form of
a numpy array with shape height x width. Each key of the dictionary is an integer indexing
the ROI and its value-pair is the associated ROI.
ROI Handlers for common analysis software are provided.
Suite2P
The Suite2P Handler is provided for loading
suite2p data. It requires the path to a folder containing suite2p data
as its only argument. The folder ought to contain at least the stat.npy and ops.npy files, although the
iscell.npy file is also recommended.
Using the Suite2P Handler
from CalSciPy.roi_tools import Suite2PHandler
rois, reference_image = Suite2PHandler("file_location")
Optogenetics
Style
CalSciPy Style
Access to the styles utilized by CalSciPy is provided through COLORS,
TERM_SCHEME, ColorSpectrum, and through
matplotlib's style contexts.
Color Scheme
Access to COLORS permits retrieval of the associated RGB tuple for a desired
color through its name, index, or directly through its attribute implementation. More information on which colors are
available can be found here.
Using CalSciPy’s Color Scheme
from CalSciPy.color_scheme import COLORS
red = COLORS.RED
red = COLORS("red")
red = COLORS(0)
red, green, blue = [COLORS(i) for i in range(3)]
Matplotlib Style
Users can utilize CalSciPy’s matplotlib style through
matplotlib's style context manager.
Using CalSciPy’s Matplotlib Style
import matplotlib
from matplotlib import pyplot as plt
with plt.style.context("CalSciPy.main"):
fig = plt.figure()
Terminal Style
Users can utilize CalSciPy’s terminal printing style by using TERM_SCHEME.
Calling TERM_SCHEME with a specific property or attribute string alongside
a string message will return your original message with the requested formatting. More information on available
formatting can be found here.
Using CalSciPy’s Terminal Style
from CalSciPy.color_scheme import TERM_SCHEME
message = "Hello World!"
formatted_message = TERM_SCHEME(message, "header")
Dependencies
CalSciPy leverages a variety of excellent dependencies!
Main Dependencies
Optional Dependencies
Optics Dependencies
Organization Dependencies
Contributions
Save me from myself! Contributors welcome!
API Reference
CalSciPy.traces.baseline
- CalSciPy.traces.baseline.baseline_calculation(method)[source]
Retrieves appropriate baseline calculation function
- CalSciPy.traces.baseline.low_pass_baseline(traces, frame_rate=30.0, frequency=1.0, percentile=5.0)[source]
Calculates baseline as the x-th percentile of fluorescence after low-pass filtering using a Hamming window.
- Parameters:
- Return type:
- Returns:
Matrix of n neurons x m samples where each element is the sample’s baseline value for the associated neuron
- CalSciPy.traces.baseline.mean_baseline(traces)[source]
Calculates baseline as the mean of all observations for each traces
- CalSciPy.traces.baseline.median_baseline(traces)[source]
Calculating baseline as the median fluorescence
- CalSciPy.traces.baseline.moving_mean_baseline(traces, window_length=150)[source]
Calculating baseline as the mean fluorescence calculated using a moving mean.
- CalSciPy.traces.baseline.percentile_baseline(traces, percentile=8)[source]
Calculates the baseline as x-th percentile fluorescence
- CalSciPy.traces.baseline.sliding_mean_baseline(traces, window_length=150)[source]
Calculating baseline as the mean of mean fluorescence calculated using a sliding window.
- CalSciPy.traces.baseline.sliding_median_baseline(traces, window_length=150)[source]
Calculates baseline as the mean of median fluorescence calculated using a sliding window.
- CalSciPy.traces.baseline.sliding_percentile_baseline(traces, percentile=8, window_length=150)[source]
Calculates baseline as the mean of the x-th percentile fluorescence calculated using a sliding window.
- Parameters:
- Return type:
- Returns:
Matrix of n neurons x m samples where each element is the sample’s baseline value for the associated neuron
CalSciPy.bruker
- class CalSciPy.bruker.CONSTANTS(BRUKER_XML_OBJECT_MODULES='.xmlobj', DEFAULT_PRAIRIEVIEW_VERSION='5.5.64.600', FIELD_OF_VIEW_PIXELS=(512, 512), FIELD_OF_VIEW_MICRONS=(452.26666666666665, 452.26666666666665), MAGNIFICATION=1.02, OBJECTIVE='25XB', PIXELS_PER_MICRON=0.8833333333333333, POWER_SCALE=0.005, SPIRAL_SCALE=0.03377483443708609, X_GALVO_VOLTAGE_RANGE=(-7.647058823529411, 7.647058823529411), Y_GALVO_VOLTAGE_RANGE=(-8.333333333333334, 8.333333333333334))[source]
Bases:
objectImmutable dataclass containing constants describing prairieview setup
-
BRUKER_XML_OBJECT_MODULES:
str= '.xmlobj' flag for file containing xml objects (for forward compatibility, ignore)
-
FIELD_OF_VIEW_MICRONS:
Tuple[float,float] = (452.26666666666665, 452.26666666666665) field of view in microns (X, Y)
-
BRUKER_XML_OBJECT_MODULES:
- CalSciPy.bruker.load_bruker_tifs(folder, channel=None, plane=None, verbose=True)[source]
This function loads images that were collected during a t-series by Bruker’s Prairieview software and converted to .ome.tif files. Each channel and plane combination is loaded to a separate numpy array. Identification of multiple channels / planes is dependent on
determine_imaging_content <CalSciPy.bruker.parsers.determine_imaging_content(). Images are loaded as unsigned 16-bit (numpy.uint16), though note that raw bruker files are natively could be 12 or 13-bit.- Parameters:
folder (
Union[str,Path]) – folder containing a sequence of single frame tiff fileschannel (
Optional[int], default:None) – specific channel/s to load from dataset (zero-indexed)plane (
Optional[int], default:None) – specific plane/s to load from dataset (zero-indexed)verbose (
bool, default:True) – whether to print progress
- Return type:
- Returns:
a namedtuple of numpy arrays (channel x plane) (frames, y-pixels, x-pixels,
numpy.uint16)
New in version 0.8.0: (experimental)
Warning
Currently untested
See also
repackage_bruker_tifs
- CalSciPy.bruker.load_voltage_recording(path)[source]
Import bruker analog “voltage” recording from a t-series folder or from an individual file.
- Parameters:
path (
Union[str,Path]) – folder or filename containing analog data- Returns:
Dataframe containing time (index, ms) x channel data
- Return type:
Note
By PrairieView naming conventions,these files contain “VoltageRecording” in the name. If your file does not follow these naming convention, you must be explicitly pass the path to your file.
New in version 0.8.0: (experimental)
Warning
Currently untested
CalSciPy.color_scheme
CalSciPy.conversion
- CalSciPy.conversion.align_data(data, reference, interpolate=None, join=False)[source]
Aligns two datasets using their timestamps (index). The timestamps must be in the same units.
- Parameters:
data (
DataFrame) – Data to be aligned to the reference datareference (
DataFrame) – Reference dataset. The function will align the data to its timestamps.interpolate (
Optional[str], default:None) – Whether to interpolate the missing values after alignment. Options include ‘nearest’, ‘slinear’, ‘quadratic’, ‘cubic’, ‘barycentric’, ‘krogh’, ‘piecewise_polynomial’, ‘pchip;, ‘akima’, and ‘cubicspline’join (
bool, default:False) – Whether to join the two datasets instead of returning only the aligned dataset
- Return type:
New in version 0.8.0.
Warning
The timestamps must be in the same units.
- CalSciPy.conversion.external_align(data, samples, reference, interpolate=False, join=False, tag='Feature')[source]
Aligns a numpy array with a reference dataset using the timestamped samples as an intermediary.
- Parameters:
data (
ndarray <numpy.ndarray) – Data to be aligned to the reference datasamples (
DataFrame) – Timestamped samples. Defines the relationship between each data sample and the reference timestamps.reference (
DataFrame) – Reference dataset. The function will align the data to its timestamps.interpolate (
Optional[str], default:None) – Whether to interpolate the missing values after alignment. Options include ‘nearest’, ‘slinear’, ‘quadratic’, ‘cubic’, ‘barycentric’, ‘krogh’, ‘piecewise_polynomial’, ‘pchip;, ‘akima’, and ‘cubicspline’join (
bool, default:False) – Whether to join the two datasets instead of returning only the aligned datasettag (
str, default:'Feature') – String prefix for describing each row in the data
- Return type:
New in version 0.8.0.
Warning
The timestamps must be in the same units.
- CalSciPy.conversion.matrix_to_tensor(matrix, chunk_size)[source]
Generates a tensor given chunk / trial indices
CalSciPy.events
- CalSciPy.events.calc_firing_rates(spike_prob, frame_rate=None, bin_duration=None, in_place=False)[source]
Calculate instantaneous firing rates from spike (event) probabilities.
- Parameters:
spike_prob (
ndarray) – Matrix where each element is the probability of a spiking-event having occurred for that particular combination of neuron and sample.frame_rate (
Optional[Number], default:None) – Frame rate of the datasetbin_duration (
Optional[Number], default:None) – Duration of each observation in secondsin_place (
bool, default:False) – Whether to perform calculation in-place
- Return type:
- Returns:
Firing matrix of n neurons x m samples where each element indicating the instantaneous firing (event) rate
Note
The elements of a spike probability matrix ought to be non-negative, but they do not need to be constrained to values of less than or equal to one.
Note
A value can be provided for only one of frame_rate and bin_duration
- CalSciPy.events.calc_mean_firing_rates(firing_rates)[source]
Calculate mean firing rates from instantaneous firing rates.
- Parameters:
firing_rates (
ndarray) – Matrix of n neuron x m samples or tensor of t trials x n neurons x m samples where each element is either a spike or an instantaneous firing rate- Return type:
- Returns:
1-D vector of mean firing rates or 2-D matrix of trial-organized mean firing rates
Warning
If you do not pass firing_rates in the shape of n neurons x m samples or tensor of t trials x n neurons x m samples your result will be incorrect.
- CalSciPy.events.generate_raster(events, samples=None)[source]
Generate raster the spike (event) times for each neuron
- CalSciPy.events.normalize_firing_rates(firing_rates, in_place=False)[source]
Normalize firing rates by scaling to a max of 1.0. Non-negativity constrained.
- Parameters:
- Return type:
- Returns:
Normalized firing rate matrix of n neurons x m samples
Warning
If you do not pass firing_rates in the shape of n neurons x m samples or tensor of t trials x n neurons x m samples your result will be incorrect.
CalSciPy.images
- CalSciPy.images.deinterlace(images, in_place=False, batch_size=None, reference=None)[source]
Deinterlaces or corrects insufficient deinterlacing of images. This function corrects the insufficient alignment of forward and backward scanned lines. The images are deinterlaced by calculating the translative offset using the phase correlation between forward and backward scanned lines. This offset is then discretized and the backward scans “bumped” by the offset. Specifically, the fourier transforms of forward and backward scanned lines are used to calculate the cross-power spectral density. Thereafter, an inverse fourier transform is used to generate a normalized cross-correlation matrix. The peak of this matrix is the translative offset (phase offset, practically speaking). Fourier transforms implemented through CuPy.If CuPy is not installed NumPy is used as a (slower) replacement.
- Parameters:
images (
ndarray) – Images to deinterlace (frames, y-pixels, x-pixels)in_place (
bool, default:False) – Whether to deinterlace in-placebatch_size (
Optional[int], default:None) – Number of frames included per FFT calculation. Batching decreases memory usage at the cost of performance. Batch calculations are numerically identical to unbatched calculations.reference (
ndarray) – Calculate phase offset using external reference
- Return type:
- Returns:
The deinterlaced images (frames, y-pixels, x-pixels)
Warning
Currently untested.
Warning
The number of frames included in each fourier transform must be several times smaller than the maximum number of frames that fit within your GPU’s VRAM (CuPy) or RAM (NumPy). This function will not automatically revert to the NumPy implementation if there is not sufficient VRAM. Instead, an out of memory error will be raised.
New in version 0.8.0.
- CalSciPy.images.gaussian_filter(images, sigma=1.0, block_size=None, block_buffer=0, in_place=False)[source]
Multidimensional Gaussian filter with GPU-implementation through CuPy. If CuPy is not installed SciPy is used as a (slower) replacement.
- Parameters:
images (
ndarray) – Images to be filtered (frames, y-pixels, x-pixels)sigma (default:
1.0) – Standard deviation for Gaussian kernelblock_size (default:
None) – The number of frames in each processing block.block_buffer (default:
0) – The size of the overlapping region between blockin_place (default:
False) – Whether to calculate in-place
- Returns:
Filtered images (frames, y pixels, x pixels)
- Return type:
- CalSciPy.images.mean_filter(images, window=(1, 0, 0), block_size=None, block_buffer=0, in_place=False)[source]
Multidimensional mean (uniform) filter with GPU-implementation through CuPy. If CuPy is not installed, SciPy is used as a (slower) replacement.
- Parameters:
images (
ndarray) – Images to be filtered (frames, y-pixels, x-pixels)window (
Tuple[int,int,int], default:(1, 0, 0)) – Window of the mean filterblock_size (
Optional[int], default:None) – The number of frames in each processing block.block_buffer (
int, default:0) – The size of the overlapping region between block.in_place (
bool, default:False) – Whether to calculate in-place
- Returns:
Filtered images (frames, y pixels, x pixels)
- Return type:
New in version 0.8.0.
Warning
Currently untested
- CalSciPy.images.median_filter(images, window=(3, 3, 3), block_size=None, block_buffer=0, in_place=False)[source]
Multidimensional median filter with GPU-implementation through CuPy. If CuPy is not installed, SciPy is used as a (slower) replacement.
- Parameters:
images (
ndarray) – Images to be filtered (frames, y-pixels, x-pixels)window (
Tuple[int,int,int], default:(3, 3, 3)) – Window of the median filterblock_size (
Optional[int], default:None) – The number of frames in each processing block.block_buffer (
int, default:0) – The size of the overlapping region between block.in_place (
bool, default:False) – Whether to calculate in-place
- Returns:
Filtered images (frames, y pixels, x pixels)
- Return type:
CalSciPy.io_tools
- CalSciPy.io_tools.load_binary(path, mapped=False, mode='r+', missing_metadata=None)[source]
This function loads images saved in language-agnostic binary format. Ideal for optimal read/write speeds and highly-robust to corruption. However, the downside is that the images and their metadata are split into two separate files. Images are saved with the .bin extension, while metadata is saved with extension .json. If for some reason you lose the metadata, you can still load the binary if you know three of the following: number of frames, y-pixels, x-pixels, and the datatype (
numpy.dtype)- Parameters:
path (
Union[str,Path]) – Folder containing binary file or path to filemapped (
bool, default:False) – Boolean indicating whether to load image using memory-mappingmode (
str, default:'r+') – Indicates the level of access permitted to the original binarymissing_metadata (
Optional[Mapping], default:None) – Enter metadata if you lost or otherwise wish to manually provide it
- Return type:
- Returns:
Images (frames, y-pixels, x-pixels)
- CalSciPy.io_tools.load_gif(path)[source]
Load gif (.mp4)
- Parameters:
- Return type:
- Returns:
Images (frames, y-pixels, x-pixels, color)
New in version 0.8.0.
- CalSciPy.io_tools.load_images(path)[source]
Load images into a numpy array. If path is a folder, all .tif files found non-recursively in the directory will be compiled to a single array.
- CalSciPy.io_tools.save_binary(path, images, name=None)[source]
Save images to language-agnostic binary format. Ideal for optimal read/write speeds and highly-robust to corruption. However, the downside is that the images and their metadata are split into two separate files. Images are saved with the .bin extension, while metadata is saved with extension .json. If for some reason you lose the metadata, you can still load the binary if you know three of the following: number of frames, y-pixels, x-pixels, and the datatype. The datatype is almost always unsigned 16-bit (
numpy.uint16) for all modern imaging systems–even if they are collected at 12 or 13-bit.- Parameters:
path (
Union[str,Path]) – Location to save images in. The path stem is considered the filename if it doesn’t have any extension. If no filename is explicitly provided in the name argument then the default filename is binary_video.images (
ndarray) – Images to save (frames, y-pixels, x-pixels)name (
Optional[str], default:None) – Specify filename for produced files
- Return type:
- Returns:
0 if successful
- CalSciPy.io_tools.save_gif(path, images, frame_rate=30, name=None)[source]
Save a numpy array to a single .gif file.
- Parameters:
- Return type:
- Returns:
0 if successful
New in version 0.8.0.
- CalSciPy.io_tools.save_images(path, images, name=None, size_cap=3.9)[source]
Save a numpy array to a single .tif file. If size > 4GB then saved as a series of files. If path is not a file and a filename is not explicitly provided in the name argument then the default filename will be images.
- CalSciPy.io_tools.save_video(path, images, frame_rate=30, name=None)[source]
Save numpy array as an .mp4. Will be converted to uint8 if any other datatype. If a filename is not explicitly provided in the name argument then the default filename will be video.
- Parameters:
- Return type:
- Returns:
0 if successful
CalSciPy.organization
CalSciPy.organization.files
- class CalSciPy.organization.files.FileMap[source]
Bases:
dictDictionary extension that appends an integer to duplicate keys before storing as a new key-value pair rather than overwriting the existing key-value pair.
New in version 0.8.1.
Warning
Currently untested
- class CalSciPy.organization.files.FileSet(name, parent_directory)[source]
Bases:
objectOrganizing class for a set of files. Contents may be only files or a collection of folders and files. This class is useful in managing coherent sets of data like experimental sessions or a calendar day. It offers several methods for keeping track of datasets.
New in version 0.8.1.
Warning
Currently untested
- property files: dict
- Getter:
Returns the files in the file set (cached)
- Getter Type:
- Setter:
This property cannot be set
- property folders: dict
- Getter:
Returns the folders in the file set (cached)
- Getter Type:
- Setter:
This property cannot be set
- class CalSciPy.organization.files.FileTree(name, base_directory, **kwargs)[source]
Bases:
objectA file tree that organizes experiment data, analyzed results, and figures. For implementation concerns it is not an extension of the built-in dictionary type, but it replicates most of its built-in methods.
New in version 0.8.1.
Warning
Currently untested
- build()[source]
Builds the file-tree by initializing any file sets that do not yet exist
- Return type:
- pop(key)[source]
If the fileset or some other attribute indicated by key is in the filetree, then remove it and return it.
Returns KeyError otherwise
- Return type:
- popitem()[source]
Remove and return a fileset from the filetree. LIFO order guarantee. If the filetree contains no filesets raises KeyError
- Return type:
CalSciPy.optogenetics
- class CalSciPy.optogenetics.Photostimulation(rois, reference_image=None)[source]
Bases:
objectAn instance of this class defines the patterned photostimulation during an optogenetic experiment. It is a standardized class for outlining parameters and selecting targets, and can be used as an argument to other functions for generating visuals, importing to microscopes, and other features.
- add_photostimulation_group(ordered_index, delay=0.0, repetitions=1, point_interval=0.12, name=None)[source]
Method that creates a
StimulationGroupand appends it to thisPhotostimulationinstance’sStimulationSequence(that is, the attributestim_sequence). More information on the arguments can be found in the documentation ofStimulationGroup.- Parameters:
ordered_index (
Sequence[int]) – Identity and order of the ROIs in this groupdelay (
float, default:0.0) – Delay before stimulating grouprepetitions (
int, default:1) – Number of times to repeat the stimulation grouppoint_interval (
float, default:0.12) – Duration between stimulating each target in the stim_sequence (ms)name (
Optional[str], default:None) – Name of the group
- Return type:
- classmethod load_rois(handler=<class 'CalSciPy.roi_tools.suite2p_handler.Suite2PHandler'>, *args, **kwargs)[source]
Class method that builds a
Photostimulationinstance using the specifiedROIHandler. Any additional arguments or keyword arguments are passed to theROIHandler.- Parameters:
handler (
ROIHandler) – desired ROIHandler- Returns:
A photostimulation instance
- Return type:
- property num_groups: int
- Getter:
Number of
StimulationGroup’s in theStimulationSequence- Getter Type:
- Setter:
This property cannot be set
- property num_targets: int
- Getter:
The number of neurons photostimulated
- Getter Type:
- Setter:
This property cannot be set
- property remapped_sequence: int
- Getter:
StimulationGroup’s with indices remapped to only include photostimulatedROI’s- Getter Type:
- Setter:
This property cannot be set
- stim_sequence
StimulationSequence: The sequence ofStimulationGroup's for this experiment
- class CalSciPy.optogenetics.StimulationGroup(rois, ordered_index, delay=0.0, repetitions=1, point_interval=0.12, name=None)[source]
Bases:
object- property num_targets: int
- Getter:
Number of roi targets in group
- Getter Type:
- Setter:
This property cannot be set
- class CalSciPy.optogenetics.StimulationSequence(groups=None, delay=0.0, repetitions=1, interval=0.0)[source]
Bases:
UserListThe sequence of
StimulationGroup’s to be stimulated in thePhotostimulationexperiment.
- CalSciPy.optogenetics.randomize_targets(potential_targets, targets_per_group=1, num_groups=1, spatial_bin_size=None, trials=1)[source]
Randomly select targets to stimulate
- Parameters:
potential_targets (
Union[Iterable,ndarray]) – The rois available to targettargets_per_group (
int, default:1) – The number of neurons to include in each groupnum_groups (
int, default:1) – The number of groups per trialspatial_bin_size (
Optional[int], default:None) – Not implementedtrials (
int, default:1) – The number of trials to
- Return type:
CalSciPy.roi_tools
- class CalSciPy.roi_tools.ApproximateROI(roi, method='literal')[source]
Bases:
_ROIBaseAn approximation of an ROI. The approximated ROI is formed by generating an ellipse at the specified centroid with a radius calculated by the method parameter. Like
ROI, contains the base characteristics & properties of an ROI. LikeROI, the properties of this class are only calculated once.- Parameters:
Warning
The properties of the Approximate ROI will only be calculated once and thereafter cached for performance benefits.
- property method: str
- Getter:
Method used for calculating radius (“bound”, “unbound”, “ellipse”, “literal”)
- Getter Type:
- Setter:
This property cannot be set
- class CalSciPy.roi_tools.ROI(pixels, y_pixels=None, reference_shape=(512, 512), method='literal', plane=None, properties=None, z_pixels=None, **kwargs)[source]
Bases:
_ROIBaseAn ROI object containing the base characteristics & properties of an ROI.
- Parameters:
pixels (
Union[ndarray[Any,dtype[int]],Sequence[int]]) – Nx2 array of x and y-pixel pairs strictly in rc form. If this argument is one-dimensional, it will be considered as an ordered stim_sequence of x-pixels. The matching y-pixels must be then be provided as an additional argument.ypixels (
Optional[Union[ndarray[Any,dtype[int],Sequence[int]]], default:None) – The y-pixels of the roi if and only if the first argument is one-dimensional.reference_shape (
Tuple[float,float], default:(512, 512)) – the shape of the reference image from which the roi was generatedmethod (
str, default:'literal') – the method utilized for generating an approximation of the roi (“mean”, “bound”, “unbound”, “ellipse”)`plane (
Optional[int], default:None) – index of the imaging plane (if multi-plane)properties (
Optional[Mapping], default:None) – optional properties to includezpix (
Optional[Union[ndarray[Any,dtype[int]],Sequence[int]]], default:None) – The z-pixels of the roi (if volumetric)
Note
The approximation attribute is dynamic and can be changed by setter the approximation method.
Warning
The properties of the ROI will only be calculated once and thereafter cached for performance benefits.
- approximation
ApproximateROI: An approximation of the roi
- class CalSciPy.roi_tools.ROIHandler[source]
Bases:
objectAbstract object for generating reference images and ROI objects
- abstract static convert_one_roi(roi, reference_shape=(512, 512))[source]
Abstract method for converting one roi in an ROI object
- abstract static from_file(*args, **kwargs)[source]
Abstract method to load the rois_data_structure and reference_image_data_structure from a file or folder
- Returns:
Data structures containing the rois and reference image
- Return type:
Tuple[Any,Any]
- abstract static generate_reference_image(data_structure)[source]
Abstract method to generate a reference image from some data structure
- class CalSciPy.roi_tools.Suite2PHandler[source]
Bases:
ROIHandler- static generate_reference_image(data_structure)[source]
Generates an appropriate reference image from suite2p ops dictionary
- classmethod import_rois(rois, reference_shape=(512, 512))
Abstract method for importing rois
- Parameters:
- Returns:
Dictionary in which the keys are integers indexing the roi and each roi is an ROI object
- Return type:
- CalSciPy.roi_tools.calculate_centroid(pixels, y_pixels=None)[source]
Calculates the centroid of a polygonal roi .The vertices of the roi’s approximate convex hull are calculated (if necessary) and the centroid estimated from these vertices using the shoelace formula.
- Parameters:
pixels (
Union[ndarray[Any,dtype[int],Sequence[int]]) – Nx2 array of x and y-pixel pairs in xy or rc form. If this argument is one-dimensional, it will be considered as an ordered stim_sequence of x-pixels. The matching y-pixels must be then be provided as an additional argument.y_pixels (
Optional[Union[ndarray[Any,dtype[int],Sequence[int]]], default:None) – The y-pixels of the roi if and only if the first argument is one-dimensional.
- Returns:
The centroid of the roi. Whether the centroid is in xy or rc form is dependent on the form of the arguments
- Return type:
- CalSciPy.roi_tools.calculate_mask(centroid, radii, reference_shape, theta=None)[source]
Calculates boolean mask for an elliptical roi constrained to lie within the dimensions imposed by the reference shape.The major-axis is considered to have angle theta with respect to the y-axis of the reference image.
- Parameters:
centroid (
Sequence[Number,Number]]) – Centroid of the roi in row-column form (y, x)radii (
Union[Number,Sequence[Number,Number]]) – Radius of the roi. Only one radius is required if the roi is symmetrical (i.e., circular). For an elliptical roi both a long and short radius can be provided.reference_shape (
Union[Number,Sequence[Number,Number]]) – Dimensions of the reference image the roi lies within. If only one value is provided it is considered symmetrical.theta (
Number, default:None) – Angle of the long-radius with respect to the y-axis of the reference image
- Returns:
Pixels in row-column form (y, x) comprising a boolean mask identifying which pixels contain the roi within the reference image
- Return type:
- CalSciPy.roi_tools.calculate_mask_pixels(centroid, radii, reference_shape=None, theta=None)[source]
Calculates the pixels of a boolean mask for an elliptical roi constrained to lie within the dimensions imposed by the reference shape.The major-axis is considered to have angle theta with respect to the y-axis of the reference image.
- Parameters:
centroid (
Sequence[Number,Number]]) – Centroid of the roi in row-column form (y, x)radii (
Union[Number,Sequence[Number,Number]]) – Radius of the roi. Only one radius is required if the roi is symmetrical (i.e., circular). For an elliptical roi both a long and short radius can be provided.reference_shape (
Union[Number,Sequence[Number,Number]]) – Dimensions of the reference image the roi lies within. If only one value is provided it is considered symmetrical.theta (
Number, default:None) – Angle of the long-radius with respect to the y-axis of the reference image
- Returns:
Pixels in row-column form (y, x) comprising a boolean mask identifying which pixels contain the roi within the reference image
- Return type:
- CalSciPy.roi_tools.calculate_radius(centroid, vertices_coordinates, method='mean')[source]
Calculates the radius of the roi using one of the following methods:
- mean
A symmetrical radius calculated as the average distance between the centroid and the vertices of the approximate convex hull
- bound
A symmetrical radius calculated as the minimum distance between the centroid and the vertices of the approximate convex hull - 1
- unbound
A symmetrical radius calculated as 1 + the maximal distance between the centroid and the vertices of the approximate convex hull
- ellipse
An asymmetrical set of radii whose major-axis radius forms the angle theta with respect to the y-axis of the reference image
- Parameters:
centroid (
Sequence[Number,Number]) – Centroid of the roi in row-column format (y, x)vertices_coordinates (
ndarray[Any,dtype[int]]) – Nx2 array containing the pixels that form the vertices of the approximate convex hullmethod (
str, default:'mean') – Method to use when calculating radius (“mean”, “bound”, “unbound”, “ellipse”)
- Returns:
The radius of the roi for symmetrical radi; the major & minor radii and the value of theta for asymmetrical radii
- Return type:
- CalSciPy.roi_tools.identify_vertices(pixels, y_pixels=None)[source]
Identifies the points of a given polygon which form the vertices of the approximate convex hull. This function wraps
scipy.spatial.ConvexHull, which is an ultimately a wrapper for QHull. It’s a fast and easy alternative to actually determining the true boundary vertices given the assumption that cellular ROIs are convex (i.e., cellular rois ought to be roughly elliptical).- Parameters:
pixels (
Union[ndarray[Any,dtype[int],Sequence[int]]) – Nx2 array of x and y-pixel pairs in xy or rc form. If this argument is one-dimensional, it will be considered as an ordered stim_sequence of x-pixels. The matching y-pixels must be then be provided as an additional argument.y_pixels (
Optional[Union[ndarray[Any,dtype[int],Sequence[int]]], default:None) – The y-pixels of the roi if and only if the first argument is one-dimensional.
- Return type:
- Returns:
Index of which points form the vertices of the approximate convex hull. It may alternatively be considered an index of the smallest set of pixels that are able to demarcate the boundaries of the roi, though this only holds if the polygon doesn’t have any concave portions.
CalSciPy.traces
- CalSciPy.traces.calculate_dfof(traces, method='mean', external_reference=None, offset=0, in_place=False, **kwargs)[source]
Calculates the fold fluorescence over baseline (i.e., Δf/f0). A variety of different methods of calculating the baseline are provided. All keyword-arguments are passed to the baseline calculation.
- Parameters:
traces (
ndarray) – Matrix of n neurons x m samplesmethod (
str, default:'mean') – Method used to calculate baseline fluorescenceexternal_reference (
Optional[ndarray], default:None) – Matrix of n neurons x m samples used to calculate baselineoffset (
Number, default:0) – Used to offset baselines by some constantin_place (
bool, default:False) – Whether to perform calculations in-place
- Return type:
- Returns:
Matrix of n neurons x m samples where each element is the fold fluorescence over baseline for a particular observation.
- CalSciPy.traces.calculate_standardized_noise(dfof, frame_rate=30.0)[source]
- Calculates a frame-rate independent standardized noise as defined as:
- \(v = \frac{\sigma \frac{\Delta F}F}\sqrt{f}\)
It is robust against outliers and approximates the standard deviation of Δf/f0 baseline fluctuations. This metric was first defined in the publication associated with the spike-inference software package CASCADE.
- Parameters:
- Return type:
- Returns:
Standardized noise (1*%Hz^(-1/2) ) for each neuron
Note
For comparison, the more exquisite of the Allen Brain Institute’s public datasets are approximately 1*%Hz^(-1/2)
Warning
If you do not pass traces in the shape of n neurons x m samples your result will be incorrect.
- CalSciPy.traces.detrend_polynomial(traces, in_place=False)[source]
Detrend traces using a fourth-order polynomial. This function is useful to correct for a drifting baseline due to photo-bleaching and other processes that cause time-dependent degradation of signal-to-noise.
- Parameters:
- Return type:
- Returns:
Detrended matrix of n neuron x m samples.
Warning
If you do not pass traces in the shape of n neurons x m samples your result will be incorrect.
- CalSciPy.traces.perona_malik_diffusion(traces, iters=25, kappa=0.15, gamma=0.25, sigma=0, in_place=False)[source]
Edge-preserving smoothing using perona malik diffusion. This is a non-linear smoothing technique that avoids the temporal distortion introduced onto traces by standard gaussian smoothing.
The parameter kappa controls the level of smoothing (“diffusion”) as a function of the derivative of the trace (or “gradient” in the case of 2D images where this algorithm is often used). This function is known as the diffusion coefficient. When the derivative for some portion of the trace is low, the algorithm will encourage smoothing to reduce noise. If the derivative is large like during a burst of activity, the algorithm will discourage smoothing to maintain its structure. Here, the argument kappa is multiplied by the dynamic range to generate the true kappa.
The diffusion coefficient implemented here is e^(-(derivative/kappa)^2).
Perona-Malik diffusion is an iterative process. The parameter gamma controls the rate of diffusion, while parameter iters sets the number of iterations to perform.
This implementation is currently situated to handle 1-D vectors because it gives us some performance benefits.
- Parameters:
traces (
ndarray) – matrix of M neurons by N samplesiters (
int, default:25) – number of iterationskappa (
float, default:0.15) – used to calculate the true kappa, where true kappa = kappa * dynamic range. range 0-1gamma (
float, default:0.25) – rate of diffusion for each iter. range 0-1in_place (
bool, default:False) – whether to calculate in-place
- Return type:
- Returns:
smoothed traces
CalSciPy.version
- CalSciPy.version.author = "Darik A. O'Neil"
str: CalSciPy author
- CalSciPy.version.maintainer = "Darik A. O'Neil"
str: CalSciPy maintainer
- CalSciPy.version.name = 'CalSciPy'
str: CalSciPy
- CalSciPy.version.version = '0.8.3'
str: CalSciPy version