Configuration and Tips#

This guide covers optional configuration settings and best practices to enhance your experience with NeuRodent.

Logging Configuration#

By default, NeuRodent operations run quietly. If you want to monitor progress and see detailed information about what NeuRodent is doing, you can configure Python’s logging system.

This is especially useful for:

  • Long-running analyses to track progress

  • Debugging issues with data loading or processing

  • Understanding what happens during each step of the pipeline

import sys
import logging

# Configure logging to display progress and debug information
logging.basicConfig(
    format="%(asctime)s - %(levelname)s - %(message)s", 
    level=logging.INFO,  # Change to logging.DEBUG for more detailed output
    stream=sys.stdout
)
logger = logging.getLogger()

print("Logging configured! You'll now see progress messages from NeuRodent.")

Logging configured! You'll now see progress messages from NeuRodent.

Logging Levels#

You can adjust the logging level to control how much information is displayed:

  • logging.DEBUG - Most detailed, shows all internal operations

  • logging.INFO - Shows progress and important steps (recommended)

  • logging.WARNING - Only shows warnings and errors

  • logging.ERROR - Only shows errors

Temporary Directory Configuration#

NeuRodent uses a temporary directory for intermediate files during processing. You can configure this location if you need to:

  • Use a location with more disk space

  • Use a faster storage device (e.g., SSD)

  • Keep temporary files in a specific location

from neurodent import core

# Set a custom temporary directory
# core.set_temp_directory("/path/to/your/temp/dir")

# Check the current temp directory
print(f"Using temp directory: {core.get_temp_directory()}")

Using temp directory: /tmp

/home/runner/work/neurodent/neurodent/.venv/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Performance Tips#

Memory Management#

For large datasets, consider:

  • Processing data in smaller time windows

  • Using the caching features to avoid recomputing results

  • Closing Python and restarting between analyses to free up memory

Adjustable In-Memory Chunk Durations (RAM vs. Throughput Trade-off)#

All memory-limiting parameters in NeuRodent use a consistent unit: seconds, with names ending in _s (or _duration_s).

When using multiprocess_mode="dask" in AnimalOrganizer.compute_windowed_analysis(), you can control how much data is held in RAM at once with the chunk_duration_s parameter:

  • chunk_duration_s=None — all fragments are loaded into a single NumPy array before being written to the intermediate zarr store. This maximizes throughput on systems with enough RAM to hold the entire recording and its intermediate representations.

  • chunk_duration_s=3600 (default) — data are streamed in 1-hour chunks and processed batch-by-batch. This limits peak RAM usage while still providing good throughput on most workstations.

  • Smaller chunk_duration_s (e.g. 250) — only that many seconds worth of fragments are buffered at a time, dramatically reducing peak RAM. Recommended on laptops or shared machines with limited memory.

  • Larger chunk_duration_s (e.g. 2500+) — fewer I/O round-trips, better throughput, at the cost of higher peak RAM. Suitable for HPC nodes with hundreds of GB of RAM.

war = ao.compute_windowed_analysis(
    features=["rms", "psd"],
    multiprocess_mode="dask",
    chunk_duration_s=250,      # buffer ~250 s of windows at a time -> low RAM usage
)

For frequency-domain spike analysis:

  • chunk_duration_s (in compute_frequency_domain_spike_analysis() and detect_spikes_recording()) controls the duration of each processing chunk; the full recording is always analysed in overlapping chunks. Default is 3600 s (1 hour). Set to None to load the full recording at once.

  • chunk_duration_s (in save_fif_and_json() and convert_to_mne()) controls how many seconds of trace data are read into memory per pass (default 60 s). Set chunk_duration_s=10 on constrained machines and chunk_duration_s=600 for faster I/O.

For LOF bad-channel detection:

  • lof_chunk_duration_s controls the duration of each chunk used for pairwise distance computation (default 60 s). Exposed via config and compute_bad_channels(lof_chunk_duration_s=...).

Parallel Processing#

Many NeuRodent functions support parallel processing. Check the function documentation for n_jobs parameters that allow you to use multiple CPU cores.

HPC and Workflow Management#

For large-scale analyses across many animals and conditions, consider using NeuRodent’s Snakemake pipeline on a high-performance computing (HPC) cluster. This allows you to:

  • Process multiple animals in parallel

  • Automatically manage dependencies between analysis steps

  • Resume interrupted workflows without recomputing completed steps

  • Scale to hundreds of recordings

See the Snakemake Pipeline Tutorial for a complete guide to setting up and running NeuRodent on an HPC cluster.

Next Steps#

Now that you’ve configured NeuRodent, explore the tutorials:

  • Basic Usage - Complete workflow from data loading to visualization

  • Data Loading - Learn about loading data from different formats

  • Tutorials - More advanced analysis techniques