Smart Mining & Manufacturing: Dealing with “Really Big” Images

Download the code

Over the next few posts, we will be looking at using deep learning for object detection. In the object detection workflow, we will highlight some of the common challenges in each stage and explore MATLAB features that can help deal with these challenges.

Very often we face the challenge of dealing with “really big” images that are too large to load into memory. These images could be satellite, x-ray, and microscope slides. Regardless of the application, the most asked questions when it comes to data preparation and the questions we will be answering in this post, are:

  1. How do I access large data that does not fit in memory?
  2. How do I label my data faster?

In this first post, we will be focusing on the data preparation stage in the workflow. Data preparation is one of the most important steps in the deep learning workflow, because our model can only be as good as the data fed into it.

Figure 1: Deep Learning Based Detection Workflow

We will be building a deep learning model for Crater Detection on the surface of the moon to demonstrate this workflow. Moon surfaces contain hazards such as slopes, craters, steep cliffs, sand dunes etc. which could be critical for robot navigation. Therefore, an avoidance system is important. Using MATLAB, we will show you how to build a deep learning-based crater detector to ensure safe autonomous landing.

Figure 2: Moon surface with Craters

How do I access large data that does not fit in memory?

When training a neural network, we work with large amounts of data and often we cannot load all of it into memory.

MATLAB has developed two ways of handling this challenge:

Datastores

Datastores are a convenient way of working with and representing collections of data that are too large to fit in memory at one time, as they load the data only as you need it. There is a collection of different types of datastores for different file formats and applications that exist in MATLAB such as images, audio, and files. However, if the existing types do not fit your datatype requirements, there is an option to create a custom datastore.

BlockedImage

Starting in R2021a, the new blockedImage object replaces the existing bigimage object. A blockedImage object is an image made from discrete blocks. Blocked images are used when an image or volume is too large to fit into memory. With a blocked image, you can perform processing without running out of memory.

The general workflow for the blockedImage object is outlined:

  • To load an image into MATLAB, the blockedImage function is used, which wraps around very large images and provides information contained at different resolution levels.
  • To visualise an image, the bigimageshow function is used and will be shown at the coarsest level.
  • When processing your image, you can select the resolution level you would like to work on and begin processing on that level which will get replicated to the entire image. These block images can be processed using the same methods discussed in the previous post on Data pre-processing for Computer Vision Applications. 

BlockedImage class capabilities can also be extended to deep learning using the blockedImageDatastore, wherein very large images can be tiled to increase the amount of training data. BlockedImageDatastore can be used to get all tiles from a blockedImage object and directly pass this datastore to train the network.

How do I label my data faster?

Labelling data for deep learning is repetitive, tedious, and time-consuming, but it is necessary. MATLAB has several interactive tools for accelerating the data labelling process:

Video labeler App

The Video labeler App is included in the Image Processing and Computer Vision toolbox in MATLAB and can be initiated by entering ‘videoLabeler’ in the command window.

This app allows you to import and visualise data for labelling. The general workflow in the app is as follows:

  1. You can import data into the app. Depending on the type of data, you will choose from the options shown below. In the Crater Detection example, a sequence of images was imported.

2. Once imported you can visualise the data in the app. For the crater detection, the sequence of images was visualised in the form of a video in the app.

3. You can then define your labels as per your application, there are different kinds of labels that can be used (rectangle, line, pixel label and project cuboid). In this example, rectangular bounding boxes were used to label the craters.

4. Once the data is labelled you can export the data for training.

Big image labeler

The Big Image labeler can be found on MATLAB Central and enables you to label large images that cannot fit into memory interactively. You do not have to worry about extracting patches and labelling each patch and reconstructing the image, instead you can interactively move around the image and label the different parts of it.

Download the code and accompanying data and try this workflow for your own applications. In the next post, we will go into how you can go about building your own deep learning model for object detection.

What Can I Do Next?

Follow us