Visualizers

Below you will find all classes and functions related to the visualization of models and outputs.

InnerEye.ML.visualizers.activation_maps.extract_activation_maps(args: ModelConfigBase) → None

Extracts and saves activation maps of a specific layer of a trained network

Parameters:

args –

Returns:

InnerEye.ML.visualizers.activation_maps.vis_activation_map(activation_map: ndarray) → ndarray

Normalizes the activation map and maps it to RGB range for visualization

Parameters:

activation_map –

Returns:

InnerEye.ML.visualizers.activation_maps.visualize_2d_activation_map(activation_map: ndarray, args: ModelConfigBase, slice_index: int = 0) → None

Saves all feature channels of a 2D activation map as png files

Parameters:

• activation_map –

• args –

• slice_index –

Returns:

InnerEye.ML.visualizers.activation_maps.visualize_3d_activation_map(activation_map: ndarray, args: ModelConfigBase, slices_to_visualize: Optional[List[int]] = None) → None

Saves all feature channels of a 3D activation map as png files

Parameters:

• activation_map –

• args –

• slices_to_visualize –

Returns:

class InnerEye.ML.visualizers.metrics_scatterplot.MetricsScatterplotConfig(should_validate: bool = True, throw_if_unknown_param: bool = False, **params: Any)

Parameters for metrics_scatterplot

max_dice: Optional[float] = None

min_dice: float = 0.0

name = 'MetricsScatterplotConfig'

param = <param.parameterized.Parameters object>

prefixes: List[str] = []

x_name: Optional[str] = None

x_path: Path = PosixPath('.')

y_name: Optional[str] = None

y_path: Path = PosixPath('.')

InnerEye.ML.visualizers.metrics_scatterplot.create_scatterplots(data: Dict[str, Dict[str, Dict[str, float]]], against: Optional[List[str]], max_dice: Optional[float] = None) → Dict[str, Figure]

Parameters:

• data – dictionary such that data[run][structure][seriesId] = dice_score

• against – run names to plot against (as y axis); if None or empty, do all against all

• max_dice – maximum Dice score to expect; if None, either 1.0 or 100.0 will be inferred from the data in code called from here.

InnerEye.ML.visualizers.metrics_scatterplot.inferred_max_dice(dicts: List[Dict[Any, Dict[Any, float]]]) → float

Given one or more two-level dictionaries of Dice scores, return a guess at what the maximum value for the axes should be. If no score is much more than 1.0, we return 1.0, otherwise we guess the numbers are percentages and return 100.0.

InnerEye.ML.visualizers.metrics_scatterplot.main() → None

Main function

InnerEye.ML.visualizers.metrics_scatterplot.metrics_scatterplot(config: MetricsScatterplotConfig) → Figure

Creates and returns a figure showing how two models compare on a test set. For each point (x, y), x is the Dice score of some structure for some patient in the first model (at config.x_path), and y is the Dice score for the same structure and patient in the second (at config.y_path). Different structures are distinguished by colour and shape. Each model is represented by a metrics.csv file as produced during inference.

InnerEye.ML.visualizers.metrics_scatterplot.metrics_scatterplot_from_dicts(config: MetricsScatterplotConfig, x_dct: Dict[str, Dict[str, float]], y_dct: Dict[str, Dict[str, float]]) → Figure

Parameters:

• config – configuration

• x_dct – dictionary of structures to patients to Dice scores, for x axis

• y_dct – dictionary of structures to patients to Dice scores, for y axis

Returns:

figure to display

InnerEye.ML.visualizers.metrics_scatterplot.satisfies_prefixes(prefixes: List[str], structure: str) → bool

Returns whether “structure” starts with any of the elements of “prefixes”, or “prefixes” is empty.

InnerEye.ML.visualizers.metrics_scatterplot.to_dict(data: DataFrame) → Dict[str, Dict[str, float]]

Parameters:

data – data frame with Patient, Structure and Dice columns

Returns:

dictionary from Structure to dictionary from Patient to Dice score.

InnerEye.ML.visualizers.metrics_scatterplot.write_to_scatterplot_directory(root_folder: Path, plots: Dict[str, Figure]) → None

Writes a file root_folder/scatterplots/basename.png for every plot in plots with key “basename”.

Parameters:

• root_folder – path to a folder

• plots – dictionary from plot basenames to plots (plt.Figure objects)

class InnerEye.ML.visualizers.model_hooks.HookBasedFeatureExtractor(model: Module, layer_name: Optional[List[str]])

forward(input)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

forward_hook_fn(module: Module, input: Any, output: Any) → None

Registers a forward hook inside module

Parameters:

• module –

• input –

• output –

Returns:

training: bool

class InnerEye.ML.visualizers.model_summary.LayerSummary(input_shape: list, output_shape: list, n_params: int, n_trainable_params: int, device: Optional[device])

Parameters to layer properties such as shapes of input/output tensors and number of parameters

device: Optional[device]

input_shape: list

n_params: int

n_trainable_params: int

output_memory_megabytes: float

output_shape: list

class InnerEye.ML.visualizers.model_summary.ModelSummary(model: DeviceAwareModule)

static compute_tensor_memory_megabytes(input_size: Union[List[Size], Sequence[Tuple]]) → float

Returns memory requirement of a tensor by deriving from its size. The returned values are in megabytes

Parameters:

input_size – List of input tensor sizes

generate_summary(input_sizes: Optional[Sequence[Tuple]] = None, input_tensors: Optional[List[Tensor]] = None, log_summaries_to_files: bool = False) → OrderedDict

Produces a human readable summary of the model, and prints it via logging.info. The summary is computed by doing forward propagation through the model, with tensors of a given size or a given list of tensors.

Parameters:

• input_sizes – The list of sizes of the input tensors to the model. These sizes must be specifies without the leading batch dimension.

• input_tensors – The tensors to use in model forward propagation.

• log_summaries_to_files – if True, write the summary to a new file under logs/models instead of stdout

Returns:

InnerEye.ML.visualizers.model_summary.forward_preserve_state(module: DeviceAwareModule, inputs: List[Tensor]) → Tensor

Perform forward pass on input module with given list of torch tensors. The function preserves the random state of the backend libraries to avoid reproducibility issues. Additionally, it temporarily sets the model in evaluation mode for inference and then restores its previous state.

Parameters:

• module – Callable torch module

• inputs – List of input torch tensors

Returns:

output: Output torch tensors

class InnerEye.ML.visualizers.patch_sampling.CheckPatchSamplingConfig(should_validate: bool = True, throw_if_unknown_param: bool = False, **params: Any)

Config class to store settings for patch sampling visualization script

local_dataset: str = None

model_name: str = 'Lung'

name = 'CheckPatchSamplingConfig'

number_samples: int = 10

output_folder: Path = PosixPath('patch_sampling_visualisations')

param = <param.parameterized.Parameters object>

InnerEye.ML.visualizers.patch_sampling.visualize_random_crops(sample: Sample, config: SegmentationModelBase, output_folder: Path) → ndarray

Simulate the effect of sampling random crops (as is done for trainig segmentation models), and store the results as a Nifti heatmap and as 3 axial/sagittal/coronal slices. The heatmap and the slices are stored in the given output folder, with filenames that contain the patient ID as the prefix.

Parameters:

• sample – The patient information from the dataset, with scans and ground truth labels.

• config – The model configuration.

• output_folder – The folder into which the heatmap and thumbnails should be written.

Returns:

A numpy array that has the same size as the image, containing how often each voxel was contained in

InnerEye.ML.visualizers.patch_sampling.visualize_random_crops_for_dataset(config: SegmentationModelBase, output_folder: Optional[Path] = None) → None

For segmentation models only: This function generates visualizations of the effect of sampling random patches for training. Visualizations are stored in both Nifti format, and as 3 PNG thumbnail files, in the output folder.

Parameters:

• config – The model configuration.

• output_folder – The folder in which the visualizations should be written. If not provided, use a subfolder “patch_sampling” in the model’s default output folder

InnerEye.ML.visualizers.regression_visualization.plot_variation_error_prediction(labels: ndarray, predictions: ndarray, filename: Optional[str] = None) → None

Plots the absolute prediction errors as well as the predicted values against the ground truth values.

Parameters:

• labels – ground truth labels

• predictions – model outputs

• filename – location to save the plot to. If None show the plot instead.

InnerEye.ML.visualizers.reliability_curve.plot_reliability_curve(y_predict: Union[List[ndarray], ndarray], y_true: Union[List[ndarray], ndarray], num_bins: int = 15, normalise: bool = False) → None

Plots reliability curves for multiple models to observe model calibration errors. Inputs can be either 1-D or a list of 1-D arrays depending on the use case. List elements are intended to be used for different model types, e.g. y_predict: (num_samples, num_models)

Parameters:

• y_predict – Model predictions, either a 1D array (num_samples) or list of 1D arrays (num_samples, num_models)

• y_true – Target values {0, 1} either a 1D array (num_samples) or list of 1D arrays (num_samples, num_models) Assuming a binary classification case

• num_bins – Number of bins used for model prediction probabilities.

• normalise – If set to true, predictions are normalised to range [0, 1]

References [1] Predicting Good Probabilities with Supervised Learning <https://www.cs.cornell.edu/~alexn/papers/calibration.icml05.crc.rev3.pdf>