The authors created a new CORE50: Continuous Object Recognition Dataset, specifically designed for continuous object recognition, and introduce baseline approaches for different continuous learning scenarios. The dataset has been collected in 11 distinct sessions (8 indoor and 3 outdoor) characterized by different backgrounds and lighting. For each session and for each class object, a 15 seconds video (at 20 fps) has been recorded with a Kinect 2.0 sensor delivering 300 RGB-D frames.

Motivation

Datasets like ImageNet and Pascal VOC offer a robust platform for classification and detection methodologies. However, they primarily adhere to “static” evaluation protocols, dividing the dataset into just two parts: a training set for initial learning and a separate test set for accuracy assessment. For continuous learning approaches, which are currently a focal point of research interest, splitting the training set into multiple batches is crucial. Unfortunately, many existing datasets lack a crucial component for this purpose: multiple, unconstrained views of the same objects captured across different sessions, each with varying backgrounds, lighting conditions, poses, occlusions, and so forth. In the realm of Object Recognition, the authors examine three continuous learning scenarios:

• New Instances (NI): new training patterns of the same classes become available in subsequent batches with new poses and conditions (illumination, background, occlusion, etc.). A good model is expected to incrementally consolidate its knowledge about the known classes without compromising what it has learned before.
• New Classes (NC): new training patterns belonging to different classes become available in subsequent batches. In this case the model should be able to deal with the new classes without losing accuracy on the previous ones.
• New Instances and Classes (NIC): new training patterns belonging both to known and new classes become available in subsequent training batches. A good model is expected to consolidate its knowledge about the known classes and to learn the new ones.

It’s evident that addressing such intricate scenarios necessitates datasets and benchmarks explicitly tailored for continuous learning, facilitating the evaluation and comparison of emerging methodologies. An ideal dataset should encompass a large number of classes that can be incrementally added in subsequent batches. More importantly, it should feature a multitude of views for each class acquired across different sessions. The inclusion of temporally coherent sessions, such as videos where objects move smoothly in front of the camera, is crucial. Temporal consistency can simplify object detection, enhance classification accuracy, and address unsupervised scenarios effectively. In the context of real-world continuous learning, assuming the utilization of all past data at each training step (i.e., a cumulative approach) not only diverges significantly from biological learning but is also impractical for engineering applications. This approach would entail storing all previous data streams and retraining the model on the entire dataset each time new data becomes available. Updating a pre-trained model solely with new data is a more viable option, considering computational and memory constraints.

Recent advancements in transfer learning and fine-tuning with deep neural networks have demonstrated the utility of leveraging previously acquired knowledge for tackling new tasks. However, there remains a notable gap in the context of continuous learning, where a single model is tasked with addressing new tasks while maintaining proficiency on previous ones. Preserving previously acquired knowledge without revisiting old patterns poses a significant challenge, often leading to catastrophic forgetting—a phenomenon wherein previously learned information is severely compromised.

Dataset description

CORe50, specifically designed for ©ontinuous (O)bject (Re)cognition, is a collection of 50 domestic objects belonging to 10 categories: plug adapter, mobile phone, scissor, light bulb, can, glasse, ball, marker, cup and remote control.

Example images of the 50 objects in CORe50. Each column denotes one of the 10 categories.

Classification tasks can be conducted either at the object level, involving 50 distinct classes, or at the category level, with 10 broader classes. The former, considered the default task, is notably more demanding due to the inherent challenge of distinguishing between objects within the same category, particularly under varying poses. The dataset was gathered across 11 separate sessions, comprising 8 indoor and 3 outdoor environments, each characterized by diverse backgrounds and lighting conditions. In each session, a 15-second video was recorded using a Kinect 2.0 sensor, capturing 300 RGB-D frames. The operator manually holds the objects while the camera adopts the operator’s point-of-view, providing a subjective, grab-distance perspective ideal for numerous robotic applications. Throughout the sessions, the grabbing hand alternates between left and right, often leading to occlusions of relevant objects caused by the hand itself. The raw data consists of 1024 × 575 RGB frames and 512 × 424 Depth frames, with depth information calibrated to match RGB coordinates. An acquisition interface specifies a central region for object placement, facilitating an initial fixed cropping process that reduces the frame size to 350 × 350 pixels.

Example of 1 second recording (at 20 fps) of object #26 in session #4 (outdoor). Note the smooth movement, pose change and partial occlusion. The 128 × 128 frames here shown have been automatically cropped from 350 × 350 images based on a fully automated tracker.

Given that domestic objects typically span less than 100 × 100 pixels at arm’s length, only a small portion of the frame captures the object of interest. To address this, the authors utilized temporal information to crop a 128 × 128 box around the object from each 350 × 350 frame. They implemented a motion-based tracker, operating solely on RGB data, allowing for a similar approach even in the absence of depth information. While the majority of objects are fully contained within the crop window, some may extend beyond its borders, particularly if the object is too close to the camera or if the tracker momentarily loses track due to rapid movement. No manual correction was applied, as the authors believe that tracking imperfections are inevitable and should be addressed in later processing stages. The final dataset comprises 164,866 128 × 128 RGB-D images, encompassing 11 sessions with 50 objects each, and approximately 300 frames per session. Three of the eleven sessions were earmarked for testing, while the remaining eight sessions were allocated for training. The authors endeavored to balance the difficulty levels of training and test sessions, considering factors such as indoor/outdoor settings, the hand used for holding (left or right), and background complexity.

One frame of the same object throughout the 11 acquisition sessions. Note the variability in terms of background, illumination, blurring, occlusion, pose and scale.