elektronn2.utils package

Subpackages

• elektronn2.utils.d3viz package
  • Submodules
  • elektronn2.utils.d3viz.formatting module
  • Module contents

Submodules

elektronn2.utils.cnncalculator module

elektronn2.utils.cnncalculator.cnncalculator(filters, poolings, desired_patch_size=None, mfp=False, force_center=False, desired_output=None, ndim=1)

Helper to calculate CNN architectures

This is a function, but it returns an object that has various architecture values as attributes. Useful is also to simply print ‘d’ as in the example.

Parameters:

• filters (list) – Filter shapes (for anisotropic filters the shapes are again a list)
• poolings (list) – Pooling factors
• desired_patch_size (int or list of int) – Desired patch_size size(s). If None a range of suggestions can be found in the attribute valid_patch_sizes
• mfp (list of int/{0,1}) – Whether to apply Max-Fragment-Pooling in this Layer and check compliance with max-fragment-pooling (requires other patch_size sizes than normal pooling)
• force_center (Bool) – Check if output neurons/pixel lie at center of patch_size neurons/pixel (and not in between)
• desired_output (int or list of int) – Alternative to desired_patch_size
• ndim (int) – Dimensionality of CNN

Examples

Calculation for anisotropic “flat” 3d CNN with mfp in the first layers only:

>>> desired_patch_size   = [211, 211, 20]
>>> filters         = [[6,6,1], [4,4,4], [2,2,2], [1,1,1]]
>>> pool            = [[2,2,1], [2,2,2], [2,2,2], [1,1,1]]
>>> mfp             = [1,        1,       0,       0,   ]
>>> ndim=3
>>> d = cnncalculator(filters, pool, desired_patch_size, mfp=mfp, force_center=True, desired_output=None, ndim=ndim)
Info: patch_size (211) changed to (210) (size not possible)
Info: patch_size (211) changed to (210) (size not possible)
Info: patch_size (20) changed to (22) (size too small)
>>> print(d)
patch_size: [210, 210, 22]
Layer/Fragment sizes:     [[102, 49, 24, 24], [102, 49, 24, 24], [22, 9, 4, 4]]
Unpooled Layer sizes:     [[205, 99, 48, 24], [205, 99, 48, 24], [22, 19, 8, 4]]
Receptive fields: [[7, 15, 23, 23], [7, 15, 23, 23], [1, 5, 9, 9]]
Strides:          [[2, 4, 8, 8], [2, 4, 8, 8], [1, 2, 4, 4]]
Overlap:          [[5, 11, 15, 15], [5, 11, 15, 15], [0, 3, 5, 5]]
Offset:           [11.5, 11.5, 4.5].
    If offset is non-int: floor(offset).
    Select labels from within img[offset-x:offset+x]
    (non-int means, output neurons lie centered on patch_size neurons,
    i.e. they have an odd field of view)

elektronn2.utils.cnncalculator.get_cloesest_valid_patch_size(filters, poolings, desired_patch_size=100, mfp=False, ndim=1)

elektronn2.utils.cnncalculator.get_valid_patch_sizes(filters, poolings, desired_patch_size=100, mfp=False, ndim=1)

elektronn2.utils.gpu module

elektronn2.utils.gpu.get_free_gpu(wait=0, nb_gpus=-1)

elektronn2.utils.gpu.initgpu(gpu)

elektronn2.utils.legacy module

elektronn2.utils.legacy.create_cnn(config_file, n_ch, param_file=None, mfp=False, axis_order='theano', constant_weights=False, imposed_input_size=None)

elektronn2.utils.legacy.load_params_into_model(param_file, model)

Loads parameters directly from save file into a graph manager (this requires that the graph is identical to the cnn from the param file :param param_file: :param gm: :return:

elektronn2.utils.plotting module

class elektronn2.utils.plotting.Scroller(axes, images, names, init_z=None)

Bases: object

onscroll(event)

update()

elektronn2.utils.plotting.add_timeticks(ax, times, steps, time_str='mins', num=5)

elektronn2.utils.plotting.embedfilters(filters, border_width=1, normalize=False, output_ratio=1.0, rgb_axis=None)

Embed an nd array into an 2d matrix by tiling. The last two dimensions of a are assumed to be spatial, the others are tiled recursively.

elektronn2.utils.plotting.my_quiver(x, y, img=None, c=None)

first dim of x,y changes along vertical axis second dim changes along horizontal axis x: vertical vector component y: horizontal vector component

elektronn2.utils.plotting.plot_debug(var, debug_output_names, save_name)

elektronn2.utils.plotting.plot_exectimes(exectimes, save_path='~/exectimes.png', max_items=32)

Plot model execution time dict obtained from elektronn2.neuromancer.model.Model.measure_exectimes()

Parameters:

• exectimes – OrderedDict of execution times (output of Model.measure_exectimes())
• save_path – Where to save the plot
• max_items – Only the max_items largest execution times are given names and are plotted independently. Everything else is grouped under ‘(other nodes)’.

elektronn2.utils.plotting.plot_hist(timeline, history, save_name, loss_smoothing_length=200, autoscale=True)

Plot graphical info during Training

elektronn2.utils.plotting.plot_kde(pred, target, save_name, limit=90, scale='same', grid=50, take_last=4000)

elektronn2.utils.plotting.plot_regression(pred, target, save_name, loss_smoothing_length=200, autoscale=True)

Plot graphical info during Training

elektronn2.utils.plotting.plot_trainingtarget(img, lab, stride=1)

Plots raw image vs target to check if valid batches are produced. Raw data is also shown overlaid with targets

Parameters:

• img (2d array) – raw image from batch
• lab (2d array) – targets
• stride (int) – strides of targets

elektronn2.utils.plotting.plot_var(var, save_name)

elektronn2.utils.plotting.scroll_plot(images, names=None, init_z=None)

Creates a plot 1x2 image plot of 3d volume images Scrolling changes the displayed slices

Parameters:

• images (list of arrays (or single)) – Each array of shape (z,y,x) or (z,y,x,RGB)
• names (list of strings (or single)) – Names for each image
• Usage –
• ----- –
• the scroll interation to work, the "scroller" object (For) –
• be returned to the calling scope (must) –
• fig, scroller = _scroll_plot4(images, names) (>>>) –
• fig.show() (>>>) –

elektronn2.utils.plotting.sma(c, n)

Returns box-SMA of c with box length n, the returned array has the same length as c and is const-padded at the beginning

elektronn2.utils.ptk_completions module

Provides completions for the Python language, file sytem paths and a custom list of words

for the ELEKTRONN2/Elektronn prompt_toolkit shell.

This module is mostly based on - https://github.com/jonathanslenders/ptpython/blob/master/ptpython/completer.py - https://github.com/jonathanslenders/ptpython/blob/master/ptpython/utils.py (at git revision 32827385cca65eabefccb06b56e4cf9d2c1e0120), which both are available under the following license (thanks, Jonathan and contributors!):

Copyright (c) 2015, Jonathan Slenders All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

• Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
• Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
• Neither the name of the {organization} nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

class elektronn2.utils.ptk_completions.NumaCompleter(get_globals, get_locals, words=None, words_metastring=u'')

Bases: prompt_toolkit.completion.Completer

Completer for Python, file system paths and custom words

get_completions(document, complete_event)

Get completions.

elektronn2.utils.utils_basic module

elektronn2.utils.utils_basic.get_free_cpu_count()

elektronn2.utils.utils_basic.parallel_accum(func, n_ret, var_args, const_args, proc=-1, debug=False)

class elektronn2.utils.utils_basic.my_jit(*args, **kwargs)

Bases: elektronn2.utils.utils_basic.DecoratorBase

This mock decorator is used as a pure-Python fallback for numba.jit if numba is not availabe.

If numba is available, the decorator is later replaced by the real numba code.

class elektronn2.utils.utils_basic.timeit(*args, **kwargs)

Bases: elektronn2.utils.utils_basic.DecoratorBase

class elektronn2.utils.utils_basic.cache(*args, **kwargs)

Bases: elektronn2.utils.utils_basic.DecoratorBase

static hash_args(args)

class elektronn2.utils.utils_basic.CircularBuffer(buffer_len)

Bases: object

append(data)

data

mean()

setvals(val)

class elektronn2.utils.utils_basic.AccumulationArray(right_shape=(), dtype=<type 'numpy.float32'>, n_init=100, data=None, ema_factor=0.95)

Bases: object

add_offset(off)

append(data)

clear()

data

ema

max()

mean()

min()

sum()

class elektronn2.utils.utils_basic.KDT(n_neighbors=5, radius=1.0, algorithm='auto', leaf_size=30, metric='minkowski', p=2, metric_params=None, n_jobs=1, **kwargs)

Bases: sklearn.neighbors.unsupervised.NearestNeighbors

warning_shown = False

class elektronn2.utils.utils_basic.DynamicKDT(points=None, k=1, n_jobs=-1, rebuild_thresh=100, aniso_scale=None)

Bases: object

append(point)

get_knn(query_points, k=None)

get_radius_nn(query_points, radius)

elektronn2.utils.utils_basic.import_variable_from_file(file_path, class_name)

elektronn2.utils.utils_basic.pickleload(file_name)

Loads all object that are saved in the pickle file. Multiple objects are returned as list.

elektronn2.utils.utils_basic.picklesave(data, file_name)

Writes one or many objects to pickle file

data:

single objects to save or iterable of objects to save. For iterable, all objects are written in this order to the file.

file_name: string

path/name of destination file

elektronn2.utils.utils_basic.h5save(data, file_name, keys=None, compress=True)

Writes one or many arrays to h5 file

data:

single array to save or iterable of arrays to save. For iterable all arrays are written to the file.

file_name: string

path/name of destination file

keys: string / list thereof

For single arrays this is a single string which is used as a name for the data set. For multiple arrays each dataset is named by the corresponding key. If keys is None, the dataset names created by enumeration: data%i

compress: Bool

Whether to use lzf compression, defaults to True. Most useful for label arrays.

elektronn2.utils.utils_basic.h5load(file_name, keys=None)

Loads data sets from h5 file

file_name: string

destination file

keys: string / list thereof

Load only data sets specified in keys and return as list in the order of keys For a single key the data is returned directly - not as list If keys is None all datasets that are listed in the keys-attribute of the h5 file are loaded.

elektronn2.utils.utils_basic.pretty_string_ops(n)

Return a humanized string representation of a large number.

elektronn2.utils.utils_basic.pretty_string_time(t)

Custom printing of elapsed time

elektronn2.utils.utils_basic.makeversiondir(path, dir_name=None, cd=False)

class elektronn2.utils.utils_basic.Timer(silent_all=False)

Bases: object

check(name=None, silent=False)

plot(accum=False)

summary(silent=False, print_func=None)

elektronn2.utils.utils_basic.unique_rows(a)

elektronn2.utils.utils_basic.as_list(var)

Module contents