Seagate object storage used in exascale computing projects: blocks and files

Seagate’s CORTX object storage was used for high-performance research projects in the European Union’s SAGE Exascale computing initiative.

SAGE, started in 2015, is one of those weird made up acronyms, and it apparently stands for Percipient yeshillAGmy for meXascale data-centric computing. PSEDCC doesn’t have the same memorable tone. Regardless, the SAGE system, which aimed to merge Big Data and HPC analytics, had a storage-centric approach in that it was intended to store and process large volumes of data on a large scale.

According to a Summary of the ACM document, “The SAGE storage system consists of several types of storage device technologies in a multi-level I/O hierarchy, including flash, disk, and non-volatile memory technologies. The core component of the SAGE software is Seagate Mere Object Storage, which can be accessed through the Clovis API and higher-level interfaces.” [Mero was a prior name for what became CORTX.]

SAGE paper diagram.

A first prototype of the SAGE system was implemented and installed at the Jülich Supercomputing Center in Germany. AN SAGE 2 The project was established in 2018 to validate a next-generation storage system based on SAGE for extreme scale computing and artificial intelligence/deep learning scientific workflows. “Provides a high-performance, high-resiliency, QoS-capable, multi-tier storage system with data tier designs managed by the Mero Object Store, which is capable of handling in-flight/on-premises data processing within the storage system , accessible through the Clovis API”.

SAGE and SAGE 2 have given rise to research work, such as a doctoral thesis by Wei Der Chien, a student at the KTH Royal Institute of Technology in Stockholm, entitled “Large-scale I/O models for traditional and emerging HPC workloads on next-generation HPC storage systems.“ This contemplated the use of an object store for HPC applications. Chien developed a programming interface that can be used to take advantage of Seagate’s Motr object store.

Motor

Engine, according to Github Documentationis a distributed object and key value storage system that sits at the heart of Seagate CORTX object store and uses high-capacity drives. Its design was influenced by Lustre’s distributed and parallel file system, NFS v4.0, and database technology. Motr interfaces directly with block devices and does not overlay a local file system. It provides a file system interface but is not, itself, a file system.

Motr controls a group of network storage nodes that can be disk-based or solid-state, which means flash, faster PCIe-attached flash, battery-backed memory, and phase-change memory. Each Motr node caches a part of the system state. This cache consists of metadata (information about directories, files, their attributes) and data (file content, usually in the form of pages). The cache can be stored in volatile memory or persistent storage.

IO activities result in system state updates that can occur on multiple nodes. Status updates gradually move to more persistent stores. For example, an update to an in-memory page cache could propagate to a cache stored on a flash drive and then to a cache stored on a disk drive.

A Seagate spokesperson told us that Jülich Supercomputing’s SAGE platform was running CORTX Motr, with 22 nodes: 8 clients and 14 storage nodes. Storage nodes had several tiers: NVRAM, SSD, and HDD, served by different Motr pools. They form a single Motr cluster with these multiple levels of performance.

Users specify which pool to use, and there is a user-driven Hierarchical Storage Management (HSM) tool for moving data between pools. This connects to a free interface, just like HPC applications. We are told that the free The interface is more HPC and AI friendly than Amazon’s S3. free it has high-performance options, such as scatter-gather and direct connections through MPI-IO.

Some members of the HPC community prefer to avoid high-level interfaces like S3 and opt for low-level interfaces like freeand APIs that provide greater control.

NoaSci

This month, Wei and others wrote a follow-up article called “NoaSci: A Numerical Object Array Library for I/O Scientific Applications in Object Storage.” We haven’t seen the full document, but its abstract says: “While the POSIX-based I/O model dominates modern HPC storage infrastructure, emerging object storage technology can potentially improve I/O performance by eliminating these bottlenecks. The researchers designed NoaSci, a numeric object array library for scientific applications, which supports different data formats (eg, HDF5, binary) and focuses on supporting node-local burst buffers and object stores.

They then showed how scientific applications can perform parallel I/O on Seagate’s Motr object store through NoaSci.

Seagate technical staff operating in Senior Vice President Ken Claffey’s systems business team participated in this research on the SAGE and SAGE 2 projects, which in turn informed Wei’s research.

The Motr Low-Level Object API was co-designed by Seagate with its EU HPC partners, including Professor Stefano Markidis of KTH. Wei is a student at Markidis. Their google scholar page shows that its 6the The most cited publication is the original SAGE paper co-authored by Sai Narasimhamurthy, Director of Engineering at Seagate UK.

Another article cited, “MPI windows in storage for HPC applications”, was co-authored by Markidis, Narasimhamurthy and others

Seagate told us, “We are honored that CORTX Motor was the object storage system of choice for these projects and greatly benefited from these relationships that drove CORTX Motor interface to be what it is today and it remains the preferred interface for many within this community”.

It has added an S3 interface for enterprise and cloud users who prefer a higher level interface and are generally unwilling to rewrite their applications to achieve very high performance.

The SAGE and SAGE2 projects have ended, but Seagate continues its collaboration with KTH and others in the IO-MAR Y https://www.esiwace.eu/ Projects

Commentary

Minimum I/O has taken advantage of most of the perceived performance by positioning object storage as a primary data store for applications that need fast access to large amounts of data. We now discover that, embedded in European academic HPC research, Seagate’s CORTX object storage software has a low-level interface to its Motr core system, allowing HPC users to enjoy fast access to data as well. object data.

But, to enjoy the high speed, CORTX must be used with the free API interface, which means changes to the application software are required. It would be fascinating to see if CORTX, through freeit is as fast or even faster than MinIO, and whether CORTX could have a future in the commercial arena for fast access object storage.

Leave a Reply

Your email address will not be published.