Configuration and Tips#

This guide covers optional configuration settings and best practices to enhance your experience with NeuRodent.

Logging Configuration#

By default, NeuRodent operations run quietly. If you want to monitor progress and see detailed information about what NeuRodent is doing, you can configure Python’s logging system.

This is especially useful for:

  • Long-running analyses to track progress

  • Debugging issues with data loading or processing

  • Understanding what happens during each step of the pipeline

import sys
import logging

# Configure logging to display progress and debug information
logging.basicConfig(
    format="%(asctime)s - %(levelname)s - %(message)s", 
    level=logging.INFO,  # Change to logging.DEBUG for more detailed output
    stream=sys.stdout
)
logger = logging.getLogger()

print("Logging configured! You'll now see progress messages from NeuRodent.")

Logging configured! You'll now see progress messages from NeuRodent.

Logging Levels#

You can adjust the logging level to control how much information is displayed:

  • logging.DEBUG - Most detailed, shows all internal operations

  • logging.INFO - Shows progress and important steps (recommended)

  • logging.WARNING - Only shows warnings and errors

  • logging.ERROR - Only shows errors

Temporary Directory Configuration#

NeuRodent uses a temporary directory for intermediate files during processing. You can configure this location if you need to:

  • Use a location with more disk space

  • Use a faster storage device (e.g., SSD)

  • Keep temporary files in a specific location

from neurodent.core.utils import get_temp_directory, set_temp_directory

# Set a custom temporary directory
# set_temp_directory("/path/to/your/temp/dir")

# Check the current temp directory
print(f"Using temp directory: {get_temp_directory()}")

Using temp directory: /tmp

Performance Tips#

Memory Management#

For large datasets, consider:

  • Processing data in smaller time windows

  • Using the caching features to avoid recomputing results

  • Closing Python and restarting between analyses to free up memory

Adjustable In-Memory Chunk Durations (RAM vs. Throughput Trade-off)#

When using multiprocess_mode="dask" in AnimalAnalyzer.compute_windowed_analysis(), you can control how much data is held in RAM at once with the chunk_duration_s parameter:

  • chunk_duration_s=None — all fragments are loaded into a single NumPy array before being written to the intermediate zarr store. This maximizes throughput on systems with enough RAM to hold the entire recording and its intermediate representations.

  • chunk_duration_s=3600 (default) — data are streamed in 1-hour chunks and processed batch-by-batch. This limits peak RAM usage while still providing good throughput on most workstations.

  • Smaller chunk_duration_s (e.g. 250) — only that many seconds worth of fragments are buffered at a time, dramatically reducing peak RAM. Recommended on laptops or shared machines with limited memory.

  • Larger chunk_duration_s (e.g. 2500+) — fewer I/O round-trips, better throughput, at the cost of higher peak RAM. Suitable for HPC nodes with hundreds of GB of RAM.

war = AnimalAnalyzer(ao).compute_windowed_analysis(
    features=["rms", "psd"],
    multiprocess_mode="dask",
    chunk_duration_s=250,      # buffer ~250 s of windows at a time -> low RAM usage
)

For frequency-domain spike analysis:

  • chunk_duration_s (in compute_frequency_domain_spike_analysis() and detect_spikes_recording()) controls the duration of each processing chunk; the full recording is always analysed in overlapping chunks. Default is 3600 s (1 hour). Set to None to load the full recording at once.

  • chunk_duration_s (in save_fif_and_json() and convert_to_mne()) controls how many seconds of trace data are read into memory per pass (default 60 s). Set chunk_duration_s=10 on constrained machines and chunk_duration_s=600 for faster I/O.

For LOF bad-channel detection:

  • lof_chunk_duration_s controls the duration of each chunk used for pairwise distance computation (default 60 s). Exposed via config and compute_bad_channels(lof_chunk_duration_s=...).

Parallel Processing#

Parallelism is controlled by multiprocess_mode. It accepts "serial" or "dask", and appears on:

  • AnimalAnalyzer.compute_windowed_analysis: one task per time window

  • AnimalAnalyzer.compute_frequency_domain_spike_analysis and FrequencyDomainSpikeDetector.detect_spikes_recording: one task per channel

  • FrequencyDomainSpikeAnalysisResult.convert_to_mne and FrequencyDomainSpikeAnalysisResult.save_fif_and_json: one task per channel

LongRecordingOrganizer with mode="si" will accept "dask", while with mode="mne" files will always be loaded serially.

LongRecordingOrganizer.save_recording accepts an n_jobs parameter, which is forwarded to SpikeInterface’s save() to parallelize writing a recording to disk.

Attaching a Dask cluster#

multiprocess_mode="dask" builds dask.delayed tasks and calls dask.compute(). With no Dask client active, dask.compute() falls back to the default threaded scheduler. Start a cluster and a client to get process-level parallelism:

from dask.distributed import Client, LocalCluster

def main():
    with LocalCluster(n_workers=8, threads_per_worker=1) as cluster, Client(cluster):
        return AnimalAnalyzer(ao).compute_windowed_analysis(
            features=["rms", "psdband"],
            multiprocess_mode="dask",
        )

if __name__ == "__main__":
    war = main()

In a .py file the cluster must be created inside a function guarded by if __name__ == "__main__":, as above, or the script deadlocks. The same applies to save_recording(n_jobs=...). In a notebook neither guard is needed.

When thinking about switching to "dask" keep in mind:

  • It is slower on short recordings. Starting a LocalCluster takes a few seconds.

  • It analyses one fewer window. The Dask path drops the last fragment unconditionally, so it reports one fewer window than "serial" on the same recording. The Snakemake pipeline defaults to "dask", so WARs it generates are affected. For hours of recordings with window_s=5 this is usually negligible.

The Snakemake pipeline sets up a cluster for you, which is described in workflow/scripts/generate_wars.py.

For a runnable "serial" and "dask" comparison on the bundled example data, see the Windowed Analysis tutorial.

HPC and Workflow Management#

For large-scale analyses across many animals and conditions, consider using NeuRodent’s Snakemake pipeline on a high-performance computing (HPC) cluster. This allows you to:

  • Process multiple animals in parallel

  • Automatically manage dependencies between analysis steps

  • Resume interrupted workflows without recomputing completed steps

  • Scale to hundreds of recordings

See the Snakemake Pipeline Tutorial for a complete guide to setting up and running NeuRodent on an HPC cluster.

Next Steps#

Now that you’ve configured NeuRodent, explore the tutorials:

  • Basic Usage - Complete workflow from data loading to visualization

  • Data Loading - Learn about loading data from different formats

  • Tutorials - More advanced analysis techniques