Blog on PMem.io

Exploring the Software Ecosystem for Compute Express Link (CXL) Memory

Thu, 25 May 2023 13:46:31 +0200

CXL Software ecosystem

The Compute Express Link (CXL) is going to be a transformative new technology in the heterogeneous memory space. While the transition from Persistent Memory (PMem) to CXL.mem may seem challenging at first, developers who have optimized their applications for PMem will find that no significant changes may be required. In this article, we will explore the CXL software ecosystem and its compatibility with the established PMem concepts and libraries.

Dockers usage in PMDK

Tue, 20 Dec 2022 00:00:00 +0000

In this blog post, I’ll describe why we believe dockers are easy to use, time-saving, and valuable for day-to-day programming and debugging. If you have never heard of dockers (or containers in general), please read, for example, this overview. We use dockers in almost all of the repositories in our organization. In this blog post, I will describe how we use dockers based on the PMDK repository. In some of our repositories, like in memkind, we use a bit different approach, but it still relies on docker. The section “Our various solutions” below describes some differences between our repositories. Let’s get to the details!

Update on PMDK and our long term support strategy

Wed, 16 Nov 2022 00:00:00 +0200

Following Intel’s Q2 earnings call announcing the wind-down of Intel’s Optane business, Intel will also be winding down its investment in new feature development on Persistent Memory Development Kit (PMDK) libraries and adjusting long-term support and maintenance plans accordingly.

The PMDK collection of open-source libraries and tools hosted on GitHub will continue to be available to the software community. Documentation and resources via the pmem.io website will also remain available.

Introduction to libpmem2 (part 1)

Thu, 30 Jun 2022 10:00:00 +0000

A new beginning

This blog post will be about the recently created library libpmem2, which is a part of PMDK repository. Before we go into details it’s worth having a short look at the history. About seven years ago, the first persistent memory library known as libpmem was started. The concept of persistent memory was becoming a reality. Initially, libpmem provided simple support for memory memory management on pmem. Over time, the development of hardware, software, as well as changing customer needs made it necessary to evolve the library to include new features such as support for Windows, DevDax, RAS, eADR, and others. The initial API design was no longer sufficient and it wasn’t flexible enough to extend. This led us to create an improved version of libpmem: libpmem2.

Memory Tiering (part 2): Writing Transparent Tiering Solution

Wed, 29 Jun 2022 00:00:00 +0200

This is the second part of the series of articles about memory tiering. The first one explained what memory tiering is and why we need it, the second one will explain some mechanisms behind transparent tiering.

This article is intended for those who would like to learn how tiering/numa balancing or memory profiling solutions work under the hood. We will focus on a high-level overview of how tiering/numa balancing could be designed, instead of dissecting any particular solution. This approach means that the readers can also learn some general knowledge on topics such as Linux API, hardware-accelerated performance monitoring or architecture of Linux apps.

Memory Tiering (part 1)

Wed, 22 Jun 2022 00:00:00 +0000

Extending memory capacity with PMEM

Databases such as Redis (an in-memory key-value open-source database) consume a lot of memory. Since fast access is essential for them, they use DRAM to store their data. DRAM is quite expensive and has limited capacity, so a solution we propose in this blog post is to use PMEM (and in the future other types of memory available through CXL - see a pmem.io blog post about it). PMEM can be used in two modes - Memory Mode and App Direct. In Memory Mode, user sees DRAM and PMEM memory as single, combined memory node. In App Direct mode, when user configures PMEM as KMEM DAX, the PMEM memory will be directly available in the system and visible as a separate NUMA node without a CPU.

Similarity Search - opportunity for PMEM

Tue, 21 Jun 2022 00:00:00 +0200

At a high level, computing solves problems. These problems, even though different and individual, are sometimes somehow related. A new algorithmic challenge can usually be solved by bringing it down to a well-defined problem with an existing solution. Today, we will be talking about one of such universal solutions - similarity search, which has found its application in various areas of life, from search engines that tell us what we want to find, through recommendation engines that tell us what to watch, where to eat and what to buy, all the way to data science that provides valuable input to business decisions.

Basic asynchronous hashmap with Miniasync library

Fri, 03 Jun 2022 00:00:00 +0000

Miniasync library provides a framework for the composition and execution of asynchronous tasks in C. To accommodate different user-defined tasks and various types of data that they take in, libminiasync makes use of macros.

Using libminiasync for the first time can be challenging. There are multiple examples on the miniasync repo to make it easier. One of them is a hashmap example.

Hashmap example overview

The hashmap example on the Miniasync repository presents a hashmap with a fixed size that allocates memory upon key-value pair insertion. This particular implementation uses linear probing for hashmap entry lookup. Linear probing means that whenever we encounter a collision when inserting a key-value pair, we will move on to the next hashmap entry until we find an unoccupied entry. For the hash function, we use the Austin Appleby MurmurHash3 64-bit finalizer.

Upcoming asynchronous interfaces in PMDK libraries

Wed, 11 May 2022 10:00:00 +0000

In the previous article, I wrote about a new upcoming Xeon platform feature, Data Streaming Accelerator (DSA) - a memory-to-memory DMA engine, and what opportunities and challenges it presents. I outlined the approach we are taking in Persistent Memory Development Kit (PMDK) to expose asynchronous APIs that can take advantage of this new hardware. Lastly, I introduced libminiasync, which is our framework for abstracting asynchronous operations.

This time, I will discuss how miniasync is being used in libpmem2 and our plans for its integration into the rest of PMDK libraries.

Leveraging asynchronous hardware accelerators for fun and profit

Mon, 28 Feb 2022 10:00:00 +0000

One of the greatest benefits of Persistent Memory is that it’s directly accessible by the CPU. But that can also be one of its downsides for specific use cases. For example, if you want to use PMem as an ultra-fast storage device with low access latency.

PMem as storage impedance mismatch

The reason for that is simple - block storage I/O is typically asynchronous due to its relatively high latency and high queue depths required to reach optimal throughputs. This led to developers optimizing software for concurrent access to the storage devices, either directly through asynchronous APIs like io_uring or indirectly by relying on kernel’s built-in mechanisms such as page caching or buffering. And now enter Persistent Memory, where the most natural way of accessing it is synchronously with the CPU. This creates a mismatch in storage use cases between what the software expects, an offloaded background I/O operation, and what is actually happening, a synchronous memory copy.

Introduction to pmemstream

Wed, 26 Jan 2022 00:00:00 +0000

What is pmemstream?

Libpmemstream implements a pmem-optimized log data structure and provides stream-like access to data. It presents a contiguous logical address space, divided into regions, with log entries of arbitrary size. We intend for this library to be a foundation for various, more complex higher-level solutions. Read on to learn about a few example use cases we have in mind. Like most libraries in the PMDK family, this one also focuses on delivering a generic, easy-to-use set of functions.

Disaggregated Memory - In pursuit of scale and efficiency

Fri, 21 Jan 2022 10:00:00 +0100

A software person perspective on new upcoming interconnect technologies.

Existing Server Landscape

Servers are expensive. And difficult to maintain properly. That’s why most people turn to the public cloud for their hosting and computing needs. Dynamic virtual server instances have been key to unlocking efficiency gains for both Cloud Service Providers (CSPs) and their users. CSPs can leverage virtualization to colocate many workloads on fewer physical servers. And cloud users have access to a huge pool of on-demand processing power, only having to pay for what they use. It’s a win-win scenario in terms of efficiency. More people share fewer resources.

How to Contribute to pmem.io

Thu, 07 Oct 2021 00:00:00 +0000

Contributing to this repository

[This entry was edited on 2022-07-22 to update Code of Conduct link and add a link to Contributing guide]

Please note that this blog post is a snapshot of our CONTRIBUTING file. For always up-to-date information, please see guideline file on repository with content of this website.

Getting started

Before you begin:

The pmem.io website is powered by the Hugo static site generator and hosted on GitHub Pages.
Have you read the code of conduct?
Review the existing issues and see if we accept contributions for your type of issue.

Types of contributions

You can contribute to the pmem.io content and website in several ways. This GitHub repository is a place to discuss and collaborate on pmem.io! A small, but mighty team is maintaining this repo. To preserve our bandwidth, off topic conversations will be closed.

Libpmemobj-cpp - lessons learned

Thu, 30 Sep 2021 00:00:00 +0000

Introduction

We’ve been working on C++ bindings for libpmemobj since around 2016 - see our very first tutorial for libpmemobj-cpp. We’ve come a long way since then. A lot has changed - we’ve gained more experience and knowledge, added new features, fixed quite a few bugs, and created at least half a dozen new containers. It’s fair to state this product is now far more mature and well-developed. Over time, we’ve learned at least several lessons about designing and overcoming issues in C++ applications for persistent memory. We want to share them with you to learn from our mistakes and not waste any time.

Concurrency considerations in libpmemobj-cpp

Fri, 17 Sep 2021 00:00:00 +0000

Introduction

Ensuring data consistency on pmem is a challenging task. It gets even more complicated if data is modified concurrently. This blog post describes several challenges related to data visibility, using transactions in multi-threaded environments, and memory leaks.

Lock-free programming on pmem

A fundamental issue (if eADR is not used) is data visibility. When a thread issues a temporal (e.g., MOV) store instruction, the modification might be visible to other threads before it is persistent (data can still be in a CPU cache). Consider the following scenario with one thread writing to a variable using atomic operations and the second one reading from it:

Memkind support for heterogeneous memory attributes

Wed, 12 May 2021 19:55:17 -0700

Introduction

Memkind is a library mostly associated with enabling Persistent Memory. However, it is not the only type of memory that memkind supports. The library is a general solution designed for platforms with heterogeneous memory.

But before we delve into heterogeneous memory itself, let’s start with a short recap about NUMA. The NUMA concept solved the problem of a dynamic extension of the CPU count per socket and system memory. Before NUMA, Uniform Memory Access (UMA) was a common model, in which all processors shared physical memory uniformly. With the UMA approach and a rapid growth of a number of processors and memory in multi-socket machines, platforms faced scalability problems. A single shared bus connecting all CPU(s) and DRAM memory became a bottleneck, as each CPU decreased the available bandwidth. To handle each new device, the length of the bus had to be extended, which in turn negatively impacted the latency.

Using Memkind in Hazelcast

Thu, 11 Feb 2021 19:55:17 -0700

This blog post is published on the Hazelcast blog as well. If interested in Hazelcast, check the other posts there too.

Introduction

The mission of the PMDK team has always been and will always be to make programming persistent memory easier for the community of software developers. One of our goals is to help simplify the integration of persistent memory into software solutions by making it transparent as possible. Adopting ground-breaking and disruptive technology creates a chasm, which is challenging to cross at first. To help with that, PMDK offers a multi-layered stack of solutions so that developers can choose a library that is the best fit for their desired level of abstraction and purpose.

API overview of pmemkv-java binding

Fri, 30 Oct 2020 19:55:17 -0700

Pmemkv is a key-value data store written in C and C++, however, it also opens up a way to leverage persistent memory by developers who prefer high-level languages - such as Java. For more information about other bindings please read Language bindings for pmemkv article and pmemkv README

We built an API for pmemkv-java binding on top of libpmemkv 1.0 API, but java binding is also compatible with newer versions of libpmemkv. You don’t have to worry about the dependencies, because packages from the OS vendor’s repositories suffice.

TieredMemDB - Redis with Persistent Memory

Fri, 25 Sep 2020 19:55:17 -0700

[14-Jan-2022 Note: this blog post has been updated due to the renaming of our database from MemKeyDB to TieredMemDB]

Context

Redis is an in-memory database that supports various data-structures and stores them in main memory. To support data durability, Redis relies on creating periodical snapshots of data or logging all commands that reach the server.

When Persistent Memory was first introduced, we’ve started working on various approaches of using it in Redis. Apart from using its persistence, we also had its huge capacity at our disposal.

Static code analysis of the PMDK

Thu, 20 Aug 2020 19:55:17 -0700

Introduction

In the PMDK team, we focus on the quality of our codebase. One of the standard practices in the software development is a static code analysis, which improves the overall project quality and fixes bugs in the early stage of development. Since there is no silver bullet for avoiding bugs, we already use two different static analysis tools and many runtime checkers e.g. valgrind. Improving static analysis effectiveness is a separate academic problem. What is worth mentioning is the fact that different tools take a various approach. With combined advantages of Open Source projects such as transparency and reliability, as well as a new static analysis tool, we get a new bag of tricks and a fresh look to an old problem :).

Introduction to LLPL

Wed, 27 May 2020 19:55:17 -0700

There are several ways to program with persistent memory from Java. A recent pmem.io blog article described the pmemkv library, a persistent key-value store, which can be used from multiple languages. For Java, pmemkv supports three ubiquitous Java types: String, byte[], and ByteBuffer.

Another PMDK component, the Low-Level Persistence Library (LLPL) is an open source Java library (https://github.com/pmem/llpl) that gives Java developers access to persistent memory in a very fast and flexible way. This article will introduce LLPL by way of Java examples that show how to manage and access persistent memory using the library.

300 nanoseconds (2 of 2)

Thu, 26 Mar 2020 19:55:17 -0700

At the end of the first part of this blog series, I posed three fundamental questions regarding the design of failure atomic data structures for persistent memory.

What does it mean to allocate persistent memory?
How to do fail-safe atomic updates?
Are all data structures suitable for persistent memory?

This time around, I will try to answer these questions to the best of my ability. We will return to the doubly-linked list example to see how it can be modified for PMEM. Finally, I will briefly discuss the evolution of libpmemobj as we refined its internal algorithms for the best performance.

Language bindings for pmemkv

Wed, 04 Mar 2020 19:55:17 -0700

pmemkv is a local/embedded key-value datastore optimized for persistent memory. It is written in C and C++, but to satisfy a wider audience it comes with several bindings for high-level languages. Currently: Java (with JNI), Node.js, Python and Ruby.

The picture below illustrates architecture and software stack of pmemkv and its bindings.

The most up-to-date information about pmemkv and its bindings is located in pmemkv’s README file.

Common for bindings

There are few common characteristics of all bindings:

Introduction to libmemkind

Mon, 20 Jan 2020 19:55:17 -0700

Introduction

Memkind is the library that simplify usage of persistent memory in a volatile mode. There are NVDIMMs technologies, such as Intel Optane DCPMM, that provides persistency, byte-addressability, and also a high capacity when compared with DRAM modules. They can be used as an expansion of main memory and utilized by applications which consume a large amount of memory and do not require persistency, such as in-memory databases, caching engines and scientific simulations.

Memkind support for KMEM DAX option

Mon, 20 Jan 2020 19:55:17 -0700

Introduction

Linux kernel version 5.1 brings in support for the volatile-use of persistent memory as a hotplugged memory region (KMEM DAX). When this feature is enabled, persistent memory is seen as a separate memory-only NUMA node(s). libmemkind API was extended to include new kinds that allow for automatic detection and allocation from these new persistent memory NUMA nodes.

Requirements

1. Kernel 5.1 with KMEM DAX driver enabled.

If support of KMEM DAX driver isn’t enabled in your kernel you will have to configure proper driver installation by run nconfig and enable driver.

300 nanoseconds (1 of 2)

Thu, 19 Dec 2019 19:55:17 -0700

Educating people has always been a challenge for me. I tend to skip over things I wrongly consider obvious, or do large leaps in reasoning when explaining a solution to a problem. And so, when faced with an attempt to explain a complex topic, I tend to ramble on and on, hoping that the audience knows when to interrupt me if I go too fast. However, this doesn’t hold true for blog posts, such as the one I’m currently writing. This is why I have a request to you, the reader, please do let me know if I went too fast on this one - thanks!

Vmem is split out of PMDK

Thu, 31 Oct 2019 19:55:17 -0700

Introduction

We have just split libvmem and its companion libvmmalloc out of the PMDK tree. They now live in a separate repository, and will follow their own release cadence. And, as these libraries are considered mature and finished, no new releases are planned once the split has been tested and tagged – except for defects and new requirements of underlying platforms.

Further development

libvmem remains the only way to use filesystem-managed persistent memory for volatile allocations on Windows.

New release of PMDK

Fri, 11 Oct 2019 19:55:17 -0700

If you are following our mailing group, you’ve probably noticed a stream of release announcements for libraries that are a part of PMDK. Here’s a recap of the most important new features and additions.

libpmemkv 1.0

The primary goal of PMDK is enabling adoption of Persistent Memory. We do so by creating the building blocks that applications can utilize to support PMEM. So far, our work was mostly concentrated on important base functionality such as memory allocation or transactions, and only recently we’ve started to build on that foundation with C++ containers - making persistent memory programming easier and easier.

C++ standard limitations and Persistent Memory

Fri, 04 Oct 2019 19:55:17 -0700

Introduction

C++ language restrictions and the persistent memory programming paradigm imply serious restrictions on objects which may be stored on persistent medium. A user can access persistent memory with memory mapped files to take advantage of its byte addressability thanks to libpmemobj and Storage Networking Industry Association non-volatile memory programming model. No serialization takes place here, thus applications must be able to read and modify directly from the medium even after application was closed and reopened or after the event of power loss.

Multi-level vmemcache

Wed, 12 Jun 2019 19:55:17 -0700

Introduction

vmemcache which we have recently described performs close to optimum when either all keys are approximately equally likely to be queried, or when all key:value pairs fit completely into the fastest form of memory you are willing to use. But, in many workloads, some keys are “hot” and queried over and over again, while the rest, “cold”, may comfortably reside on slower medium. This calls for multiple linked instances of vmemcache, each residing in a different tier.

libvmemcache - buffer-based LRU cache

Tue, 07 May 2019 19:55:17 -0700

Introduction

libvmemcache is a volatile key-value store optimized for operating on NVDIMM based space. However, it can work with any filesystem whether it is stored in memory (tmpfs) or on any storage device. Consequently, libvmemcache will be significantly less performant if it is stored on the storage device other than NVDIMMs.

libvmemcache is an embeddable and lightweight in-memory caching solution. It is designed to fully take advantage of large capacity memory, such as persistent memory with DAX through memory mapping in an efficient and scalable way.

C++ persistent containers - vector

Wed, 20 Feb 2019 19:55:17 -0700

[Note: pmem::obj::vector<> is no longer experimental. The rest of the information in this blog post is still accurate.]

Introduction

The main idea behind pmem containers is to fully exploit persistent memory potential by designing optimized on-media layouts and algorithms for persistent memory programming. On November, we published a blog post about pmem containers. If you haven’t read it yet, I encourage you to do that now.

We have recently added pmem::obj:vector container to libpmemobj-cpp library. This container is currently placed in experimental namespace and folder - this means that both API and layout may change. It provides API similar to std::vector from C++11 but guarantees full exception safety via commit or rollback semantics and allocates data in persistent memory.

Pmreorder basics

Mon, 04 Feb 2019 19:55:17 -0700

Introduction

It’s good practice to run persistent memory application under pmemcheck - a tool which is described here and here.

In this post, we are going to learn about another tool for persistence correctness checking. As you might already know if you’ve read posts linked above, pmemcheck verifies if all stores are made persistent in a proper manner. Our new tool, pmreorder, extends this functionality. It traverses the sequences of stores between flush-fence barriers made by the application, and then replays these memory operations many times in different combinations, to simulate the various possible ways the stores to the NVDIMM could be ordered by the system.

Pool conversion tool

Fri, 01 Feb 2019 19:55:17 -0700

Introduction

When we published the first PMDK stable release, we committed to maintaining stable on-media layout. This means that all future PMDK changes have to be backward compatible. Unfortunately, we weren’t successful in adhering to the strict requirements which would be needed to maintain compatibility, mostly because we made changes whose benefit far outweighed the costs. For this reason, we created the pmempool convert command. This tool was used to convert pools which were created with old PMDK versions to the newer on-media layout. In 1.5 release this functionality was refactored and moved to a separate tool called pmdk-convert.

Extended memcpy in PMDK 1.5

Tue, 22 Jan 2019 19:55:17 -0700

In PMDK 1.5 we added new APIs for bulk persistent memory modifications. In short, we did this to:

give applications the ability to perform low-level performance optimizations
clean up the naming scheme

In order to understand what exactly and why we did that, let’s review the old API. In PMDK 1.4 we had these functions:

void *pmem_memmove_persist(void *pmemdest, const void *src, size_t len);
void *pmem_memcpy_persist (void *pmemdest, const void *src, size_t len);
void *pmem_memset_persist (void *pmemdest, int c, size_t len);

void *pmem_memmove_nodrain(void *pmemdest, const void *src, size_t len);
void *pmem_memcpy_nodrain (void *pmemdest, const void *src, size_t len);
void *pmem_memset_nodrain (void *pmemdest, int c, size_t len);

void *pmemobj_memcpy_persist(PMEMobjpool *pop, void *pmemdest, const void *src, size_t len);
void *pmemobj_memset_persist(PMEMobjpool *pop, void *pmemdest, int c, size_t len);

As you can see, there are two variants of each API - one with _persist and another one with _nodrain suffix. Both variants modify pmemdest argument just as their libc’s equivalent would, but they also make sure that the copied data are flushed. The only difference is that the _persist variants also wait for the flush to finish. As I mentioned earlier, there’s one unfortunate aspect of this naming - it doesn’t follow the scheme used by other libpmem functions, where pmem_flush flushes the data, pmem_drain waits for all flushes to finish and pmem_persist does both. In line with this scheme, _nodrain functions should have used _flush suffix.

Pool features

Wed, 05 Dec 2018 19:55:17 -0700

Introduction

Since the very first release, PMDK pools had internal feature flags. They were mostly a hidden implementation detail. The more observant users might have noticed pool features listing in pmempool info output, but that’s about it.

Release 1.5 introduced a set of new feature flags. And since it’s imperative that system administrators have the ability to manage enabled features, we’ve added functionality that helps with that:

pmempool tool commands for managing features
libpmempool API for managing features
CTL namespace for setting a SHUTDOWN_STATE initial value

Taxonomy of pool features

Each of the toggleable features can be disabled or enabled for the given pool and the support for these features might vary depending on the library version. All features are divided into two groups:

Bad blocks

Mon, 26 Nov 2018 19:55:17 -0700

Introduction

Over time, storage devices can acquire uncorrectable media errors often called “bad blocks”. A bad block is a part of a storage media that is either inaccessible or unwritable due to a permanent physical damage. In case of memory mapped I/O, if a process tries to access (read or write) the corrupted block, it will be terminated by the SIGBUS signal.

Handling bad blocks in PMDK libraries

PMDK libraries can handle bad blocks if the CHECK_BAD_BLOCKS compat feature is turned on. Currently (PMDK v1.5) it is disabled by default because it requires super user privileges. It can be turned on using pmempool-feature.

C++ persistent containers

Tue, 20 Nov 2018 19:55:17 -0700

PMEM containers

Our goal for the libpmemobj C++ bindings is to create a friendly and less error prone API for persistent memory programming. Even with persistent memory pool allocators, convenient interface for creating and managing transactions, auto-snapshotting class templates and smart persistent pointers, designing an application with persistent memory usage may still prove challenging without a plethora of niceties that the C++ programmers are used to.

The natural step forward to make persistent programming easier, is to provide programmers with efficient and useful containers.

C++ persistent containers - array

Fri, 02 Nov 2018 19:55:17 -0700

[Note: pmem::obj::array<> is no longer experimental. The rest of the information in this blog post is still accurate.]

Introduction

Until now, our C++ bindings were missing one important component - persistent containers. In 1.5 release we have introduced the first one - pmem::obj::array. This container is currently placed in experimental namespace and folder (this means that both API and layout can change). It has almost the same functionality as std::array from C++11 but takes care of adding elements to a transaction. Once experimental status will be dropped, it will also guarantee a stable in-memory layout (it will be the same for all compilers). API for pmem::obj::array and std::array is the same, except for the following:

New release of PMDK

Mon, 22 Oct 2018 19:55:17 -0700

We’ve been very quiet on this blog as of late, mostly because of the amount of work that we needed to put into our very ambitiously planned 1.5 release. But we’ve made it, and there’s finally time to get back to discussing the technical minutiae of our work. In this post, we will go over the major library changes that have been introduced in 1.5.

Release planning in the open

But first, I’d like to highlight a change to our release planning process. Instead of using a proprietary walled-off tool, we are slowly moving towards doing our features planning entirely in the open, directly on GitHub. It’s not all the way there yet, but the 1.5 release, and the upcoming one, are already using this new open process. When you enter one of the releases, you will see that most of the planned features have their pages where they can be discussed.

Running FIO with pmem engines

Mon, 25 Jun 2018 19:55:17 -0700

When we, the PMDK team, want to check performance of our library, either to see if there was any regression or if our tweaks did a good job, we run benchmarks. One of them is FIO. It helps us simulate synthetic traffic of reads and writes to a pmem device. In this blog post I will introduce this tool and explain how we commonly use it.

Preparing the environment

For starters, links to needed software:

Using Persistent Memory Devices with the Linux Device Mapper

Tue, 15 May 2018 19:55:17 -0700

Introduction

X86/X64 systems do not typically interleave Persistent Memory Devices (also referred to as ‘modules’ or ‘DIMMs’) across sockets, so a two-socket system will have two separate interleave sets. To use these interleave sets as a single device requires using a software device mapper or volume manager.

This article focuses on using the ‘striped’ (dm-stripe) and ’linear’ (dm-linear) target drivers with persistent memory devices to create virtual devices on which direct access (DAX) enabled filesystems can be created. Both XFS and EXT4 have native DAX support.

Benchmarking with different storage engines using pmemkv

Wed, 27 Dec 2017 19:55:17 -0700

We’re closing out 2017 with two big improvements to pmemkv: support for multiple storage engines, and an improved benchmarking utility based on db_bench. These changes set the stage for some interesting experiments to come next year, as we continue to add new features and tune performance of pmemkv and its utilities and bindings.

Multiple storage engines

A new virtual interface (KVEngine) was recently introduced that allows pmemkv to provide multiple storage engine implementations, without changes to utilities or language bindings or applications using the pmemkv API.

Announcing the Persistent Memory Development Kit

Mon, 11 Dec 2017 19:55:17 -0700

This is to announce a name change: The NVML project is now known as PMDK, the Persistent Memory Development Kit.

Why the name change?

The old name, NVML, made it sound like the project produced a single library that applied to Non-Volatile Memory. In reality, the project currently supports ten libraries, targeted at various use cases for persistent memory, along with language support for C, C++, Java, and Python, tools like the pmemcheck plug-in for valgrind, and an increasing body of documentation, code examples, tutorials, and blog entries. The libraries are tuned and validated to production quality and issued with a license that allows their use in both open- and closed-source products. And the project continues to grow as we learn about new use cases. So the term Development Kit seems much more appropriate. The new name, PMDK, also follows the pattern of similar development kits like the Data Plane Development Kit (DPDK) and the Storage Performance Development Kit (SPDK).

Progress Report Q3 2017

Thu, 05 Oct 2017 19:55:17 -0700

The last quarter was rather… peaceful. But nevertheless there were a few noteworthy things.

FreeBSD & ARM

We always asserted that our library is multi-platform and hardware agnostic… as long as your platform is a recent distribution of Linux (or Windows) on x86 hardware :)

Two things happened that intend to change the current status quo:

There’s an active and ongoing effort of porting the linux-specific parts of the library to FreeBSD, mostly led by @gaweinbergi. Our code is naturally very linux centric due to being developed alongside the kernel support for NVDIMMs, and there are quite a few hurdles to overcome, but everything seems to be going along quite nicely.
Patch that ports the low-level library (libpmem) to ARM has been submitted to our repository by @vvenkates27. This is also a place where there’s a lot of platform specific code due to the novelty of the topic and the fact that the library implementation requires a hefty amount of ISA specific intrinsics.

Allocation classes merged

I’ve talked about this previously, so I won’t repeat myself here, but the allocation classes were finally merged last quarter.

Using Standard Library Containers with Persistent Memory

Mon, 10 Jul 2017 19:55:17 -0700

Introduction

Somewhere along the road, when we were doing the C++ bindings for libpmemobj, we found the need for some kind of containers. We were faced with two viable solutions: write everything from scratch or adapt an existing implementation. The obvious choice was NOT to implement from scratch. We would have to implement at least the basic containers from the C++ standard: vector, list, set, map and their multi- companions. That would be a lot of work, not to mention the testing and maintenance effort. I’d say it would be the last resort, should all of our other options fail.

Progress Report Q2 2017

Tue, 04 Jul 2017 19:55:17 -0700

It’s summer already (at least in my hemisphere) ! :) And it’s time for the next progress report.

Control interface

After a very long in the oven, we’ve finally finalized and merged the CTL API which allows for introspection and modification of the internal state of the library.

This feature has been shaped after the mallctl() available in jemalloc.

The way it works is very simple. Developer defines a function-like entry point, decides whether or not the entry point allows for reading, writing or both, and finally specifies the argument type. All of this happens in a very streamlined fashion.

Apache Kudu Persistent Memory Enabled Block Cache

Mon, 03 Apr 2017 19:55:17 -0700

Using NVM Libraries To Add Persistent Memory Support to Apache Kudu Block Cache

An early project done with the NVM libraries was adding persistent memory support, both volatile and persistent mode, into the Apache Kudu storage engine block cache. This project required modification of existing code.

Apache Kudu: https://github.com/apache/kudu

My repository with the modified code: https://github.com/sarahjelinek/kudu, branch: sarah_kudu_pmem

The volatile mode support for persistent memory has been fully integrated into the Kudu source base. The persistent mode support is not integrated but is ready, has been reviewed and is waiting for official integration.

Progress Report Q1 2017

Wed, 29 Mar 2017 19:55:17 -0700

It’s been three months already since the last time I wrote something ;) Time really flies by quickly when you are doing interesting stuff.

We’ve been very busy with lots of improvements to the library as well as A LOT of tiny fixes: over the last three months the team eliminated virtually every bug found by various static analysis tools and valgrind.

But no-one wants to hear about that, so here’s the meaty stuff:

Zero-copy leaf splits in pmemkv

Thu, 09 Mar 2017 19:55:17 -0700

In a B+ tree, splitting a full leaf into two leaves is one of its slowest operations, but pmemkv optimizes this using a zero-copy strategy. Rather then copying any key/value data between full and new leaf, pmemkv splits leaves by swapping persistent structures in place. This minimizes write amplification and increases performance compared to copying, especially for larger key/value sizes.

The diagram below illustrates a persistent leaf being split. A ‘slot’ is the term used in pmemkv for the grouping of a key, the Pearson hash for the key, and the value for the key. During the split, slots having keys that sort below or equal to the split key are untouched, but those slots in the new leaf are marked unused. Slots where keys sort above the split key have to be relocated to the new leaf, and those original slots in the full leaf are marked unused.

Introducing pmemkv

Tue, 21 Feb 2017 19:55:17 -0700

We’ve blogged before about building and optimizing key-value stores for persistent memory, and we’re excited to put these ideas to the test in a more formal way.

Our new pmemkv project is an open-source key-value store that is optimized for read-heavy workloads on persistent memory. Compared with key-value stores based on the LSM algorithm, pmemkv offers higher read performance and lower write amplification. But our intent is not to deter use of LSM, only to expand the choices developers and architects have for aligning workloads to backing stores.

What's coming in libpmemobj

Wed, 25 Jan 2017 19:55:17 -0700

In my last post I’ve made a promise to share our plans for the near future. So here it is: 4 ideas that we are planning to ship with the upcoming version of libpmemobj.

Please note that most of our plans related to libpmemobj are available on our github issues page with the “Feature” label. Feel free to join the discussion!

Reserve/Initialize/Publish work-flow

Many of persistent memory programming models proposed by researchers [1, 2] provide a different transactional semantics compared to libpmemobj. In the simplest terms, they allow the programmer to reserve and initialize memory anywhere in the code and publish that object inside of a transaction.

Modeling strings with libpmemobj C++ bindings

Mon, 23 Jan 2017 19:55:17 -0700

C++ developers using libpmemobj have more than one option for modeling strings, depending on the size of the strings and whether they are fixed or varying in length. In this post we’ll review the representations that work, known variations to avoid, and finally present a persistent string class that implements these best practices.

Avoid wrapping fixed-size arrays

You might expect (like I did at first!) that p<char[size]> is a proper way to simply model a fixed-size string, but actually this is not correct. The p<> template does not implement the subscript operator[] and so p<char[size]> won’t even compile.

PMDK for Windows

Wed, 21 Dec 2016 19:55:17 -0700

Throughout 2016 a team of engineers from Microsoft, Intel, HPE and HPI have been working to port the PMDK project to Windows and we are happy to announce that Technical Preview release is now available!

Our main goal in porting the library was to make it as easy as possible for developers to use the library in both Linux and Windows environments. To meet this goal, we focused on these elements as we undertook the effort:

libpmemobj - a year in review

Tue, 20 Dec 2016 19:55:17 -0700

It’s been a while since the last post on our blog, but we’ve been busy with the recently released 1.2 version of the library. It comes packed with improvements all throughout the code base and it also brings a handful of new features that we hope will end up being useful.

With the year coming to an end, it’s a good time to look back and discuss the things we’ve learned and accomplished. While the 2015 was spent mostly on designing and implementing the core of the library, with code being furiously written in heavy volumes, this year was much calmer and naturally focused on refinements and iterative improvements. The experience we’ve gathered during the creation of the library was invaluable and allowed us to think about the problems we are tackling from a different, broader, perspective. This is reflected in the things we are focused on and ship with new versions of the library.

C++ bindings for libpmemobj (epilogue) - converting existing applications

Thu, 02 Jun 2016 19:55:17 -0700

During the development of the C++ bindings, we wrote a couple of examples and even more tests. But these are new applications written from scratch to understand persistence. While this approach is OK for newly developed apps, there is a lot of existing code out there that is not designed for persistent memory. It would be a real shame, if the existing solutions couldn’t benefit from the existence of persistent memory because of the amount of work needed to redesign and change them. This was one of the main pain points that we wanted to address with the C++ bindings. We tried to make them as seamless to use and as easily introducible to existing software as possible. In this blog post I will show you exactly how much work is necessary to change an implementation of a ctree from oblivious to fully persistent memory aware.

C++ bindings for libpmemobj (part 7) - synchronization primitives

Tue, 31 May 2016 19:55:17 -0700

To finish off the C++ bindings to libpmemobj blog marathon, I will introduce to you the synchronization mechanisms we implemented. They are mostly C++11-like implementations of different kinds of mutexes and the condition variable. They satisfy their respective concepts (Mutex, SharedMutex and so on), the difference is that they are based on the persistent memory resident synchronization primitives provided by libpmemobj.

Mutex

The pmem::obj::mutex class satisfies the requirements of the Mutex and StandardLayoutType concepts. The usage of this class should be really straightforward for anyone who has ever used the std::mutex. The only difference is that the pmem::obj::mutex has to be placed in persistent memory, within a libpmemobj pool. This is because the implementation needs to be able to reset the mutex the next time the pool is opened after a power failure/crash. In persistent memory, the mutex would not be reinitialized automatically in such case.

C++ bindings for libpmemobj (part 6) - transactions

Wed, 25 May 2016 19:55:17 -0700

As I mentioned in my previous blog post, transactions are the heart of libpmemobj. That is why we had to take utmost care while designing their C++ versions, so that they are as easy to use as possible. There are, however, a couple of compromises we had to make due to the inadequacies of the C++11 standard. That is why we encourage using the lambda, until the C++17 standard is more widely implemented. Frankly, even after C++17, the lambda transactions are more user friendly.

C++ bindings for libpmemobj (part 5) - make_persistent

Thu, 19 May 2016 19:55:17 -0700

One of the most important features of the C++ bindings to libpmemobj is the persistent_ptr smart pointer template. While using it is fairly straightforward, the allocation and object construction with the use of the C API is hard to get right. So like it’s C++ standard’s counterparts, it needed an allocation mechanism with appropriate object construction. This is exactly what this post will try to explain.

Transactional allocations

Probably the most common usage of the allocating functions is within pmemobj transactions. The easiest way to allocate an object into a persistent_ptr is by doing the following:

C++ bindings for libpmemobj (part 4) - pool handle wrapper

Tue, 10 May 2016 19:55:17 -0700

One of the necessary steps in developing the C++ libpmemobj bindings was the introduction of an abstraction of the C pool handle. We decided to do a very simple hierarchy where the pool template inherits from a generic pool_base. This was necessary to be able to have functions/methods which do not depend on the pool’s template argument. Please note that this makes both of these handles impossible to keep in persistent memory, due to the presence of a vtable. It wouldn’t make much sense either way, because the pool handle is runtime and not persistent data.

Persistent allocator design - fragmentation

Thu, 25 Feb 2016 19:55:17 -0700

Implementing a memory allocator is a balance between numerous properties with the two most important being time and space constraints. Making the malloc/free routines reasonably fast is a must for the implementation to be considered usable at all. The algorithm also mustn’t waste excessive amounts of memory.

During development of the library we quickly realized that the performance characteristics will be dominated by the number of cache line flushes that will be required to perform operations in a fail-safe atomic way. In other words, how many times does the on-medium memory layout transitions from one consistent state to another. This lead us to measuring performance of low-level pmalloc in a function of pmem_persist calls. At the time of writing, fail-safe atomic memory allocation in our library requires on average 3 cache line flushes.

How to emulate Persistent Memory

Mon, 22 Feb 2016 19:55:17 -0700

Data allocated with PMDK is put to the virtual memory address space, and concrete ranges are relying on result of mmap(2) operation performed on the user defined files. Such files can exist on any storage media, however data consistency assurance embedded within PMDK requires frequent synchronisation of data that is being modified. Depending on platform capabilities, and underlying device where the files are, a different set of commands is used to facilitate synchronisation. It might be msync(2) for the regular hard drives, or combination of cache flushing instructions followed by memory fence instruction for the real persistent memory.

C++ bindings for libpmemobj (part 0)

Tue, 12 Jan 2016 19:55:17 -0700

Our goal for the C pmemobj library was to make a fully featured implementation of persistent memory programming model without modifying the compiler. It’s meant for authors of very low-level system software and language creators. It’s not particularly pretty nor easy to use. The amount of macros, as well as the trickery inside them, might ‘amaze’ even the biggest preprocessor fans ;)

The natural next step is to leverage the high-level languages features to create a more friendly, less error prone and generally nicer API.

C++ bindings for libpmemobj (part 1) - pmem resident variables

Tue, 12 Jan 2016 19:55:17 -0700

One of the biggest hurdles and error prone things about our C API is that the user has to manually keep track of modifications to the persistent memory resident variables while in a transaction. A special semi-transparent template property class has been implemented to automatically add variable modifications to the transaction undo log.

pmem::obj::p

Let’s start with the vector example from the previous tutorial series. It looked like this:

struct vector {
 int x;
 int y;
 int z;
}

PMEMoid root = pmemobj_root(pop, sizeof (struct vector));

struct vector *vectorp = pmemobj_direct(root);
TX_BEGIN(pop) {
 pmemobj_tx_add_range(root, 0, sizeof (struct vector));
 vectorp->x = 5;
 vectorp->y = 10;
 vectorp->z = 15;
} TX_END

As you can see, the programmer has to remember to call pmemobj_tx_add_range function before any modifications to the memory. In a simple case like this one it might not be such a big deal, but once the code gets complex it may lead to some difficult to find consistency issues.

C++ bindings for libpmemobj (part 2) - persistent smart pointer

Tue, 12 Jan 2016 19:55:17 -0700

In our C API the programmer has to deal with custom pointers represented by the PMEMoid structure. Thanks to some macro magic we made it so that those PMEMoids are somewhat usable. C++ allows us to evolve this concept.

pmem::obj::persistent_ptr

Almost everyone who ever touched a C++ code knows the idea behind smart pointers (for example, std::shared_ptr). Our persistent pointer works in the same way. It wraps around a type and provides implementation of operator*, operator-> and operator[].

C++ bindings for libpmemobj (part 3) - persistent queue example

Tue, 12 Jan 2016 19:55:17 -0700

The best way to learn to code is usually by implementing an example. We are going to be creating a linked-list based queue data structure using the the pmem::obj::p and pmem::obj::persistent_ptr classes and libpmemobj C API. But first, a little bit of CS 101 :)

Linked-list queue

Queue is a collection of elements with two important operations:

push - adds element to the tail of the structure
pop - removes element from the head of the structure

This makes the queue a First-In-First-Out (FIFO) data structure.

Performance improvements

Tue, 15 Dec 2015 19:55:17 -0700

I would like to inform you about the performance improvements that have been going on in PMDK and libpmemobj in particular. We have not been standing still and we are trying out a couple of ideas on how to make our libraries even faster. Some of the improvements are smaller, some are larger. Some of them have already made it to the master branch and some are just ideas on how to rework the internals of libpmemobj to make it even faster. All measurements were made on a two socket, sixteen core machine.

An introduction to replication

Mon, 23 Nov 2015 19:55:17 -0700

Replication is a means for raising the reliability of your pmemobj based applications. You can basically think of it as RAID 1 within PMDK. What happens is, when you write to your pool using the pmemobj_* (memcpy, persist, and so on) primitives, it gets copied to your replicas. Yes, you can have more than one replica. In fact you can have as many as you want, but you have to keep in mind the performance penalty.

Evaluation of a better object container

Tue, 20 Oct 2015 19:55:17 -0700

During performance evaluation of our library, I asked myself a following question:

Which data structure has computational complexity of “insert at end” and “remove given element” operations no worse than a doubly-linked list, but with a smaller constant?

The point of that mental exercise was to come up with a persistent data structure that could replace doubly-linked list in object stores (right now a linked lists of every single user-allocated object) and undo logs.

pmemobjfs - The simple FUSE based on libpmemobj

Tue, 29 Sep 2015 19:55:17 -0700

How to use it

The sources of the pmemobjfs file system are available here. Please refer to README file for instructions on how to create a file system layout and mount it.

NOTE: This is just an example implementation of file system in user space using the libpmemobj library and it is not considered to be production quality. Please do not use this file system to store your data you care about because it may be lost.

Challenges of multi-threaded transactions

Wed, 16 Sep 2015 19:55:17 -0700

Our library currently does not support threads cooperating (writing) within a single transaction. It does shift a lot of work from the library onto the user who now has to think about different parallelization solutions.

This was a conscious decision with iterative approach to creating the library in mind. It was far easier to implement the current transaction support strategy and say that it works with relatively good performance than it would be to implement multi-threaded transactions straight up and say that they work and scale linearly with CPUs thrown at them (yea right). The required validation effort alone would be tremendous.

KV-store improved & measured

Thu, 10 Sep 2015 19:55:17 -0700

As promised in the previous post about the kv-store implementation I’m back with new results after implementing the optimizations I devised a month ago. As a bonus I implemented a red-black tree to have a fair comparison between two data structures that allocate similar number of nodes.

tl;dr: I was right about crit-bit :)

Test platform

The same server was used to run the benchmarks but with the latest 4.2 kernel that contains numerous DAX improvements.

Transactional key-value store using libpmemobj - DIY

Fri, 31 Jul 2015 19:55:17 -0700

Our library often gets compared to NoSQL databases because it stores things on storage in unstructured manner. Which is true, but, when you think about it, the pmemobj library is not technically a database, but can be used to implement one - like the MySQL storage engine example. In this post I’ll describe an example implementation of transactional kv-store, that has two different backends, which I’ll then compare. To make things more interesting, it’s not going to be your typical kv-store, since the data structure behind it won’t be a hashmap. Instead, I’ve selected two different tree algorithms, namely b-tree and crit-bit tree.

An introduction to pmemcheck (part 2) - transactions

Mon, 20 Jul 2015 19:55:17 -0700

In my previous blog post I described the key features of the new persistent memory analysis tool we created - pmemcheck. You should now be aware of the main pitfalls of persistent memory programming and of ways pmemcheck informs you about possible misuses of PMEM. We should now dive into a more general approach of using persistent memory in a failsafe manner - transactions. This shouldn’t be an alien concept for anybody who had anything to do with databases. The idea of transactions in persistent memory is very similar. You enclose a set of operations, which are to be performed as a whole, inside a transaction. The transaction should ensure that on transaction commit you get a durable and consistent state. By durable I mean that all changes will be made persistent and by consistent I mean you will get either the state from before the transaction or after all of the modifications were made. I mentioned databases, because the concept is similar and everybody should be familiar with it, but you have to remember that this is on a slightly different level of abstraction. This is raw access of memory and not database inserts. This of course depends on the implementation of transactions, but it might prove very helpful not to think, that all persistent memory transactions will have ACID properties. Or at least they won’t be on a level you expect them to be.

An introduction to pmemcheck (part 1) - basics

Fri, 17 Jul 2015 19:55:17 -0700

As you probably noticed from the previous posts, persistent memory programming isn’t really that easy. There are a couple of things you have to consider - data consistency being the most important one. The contemporary x86_64 architecture supports at most 8-byte atomic stores. You probably know by now, that by atomic I mean non-torn and not thread-safe. This means that you can be confident that you will not get 4 out of the 8 bytes with the new value and the rest will not be updated. However, there aren’t many real-life programs that think in 8 byte chunks of data. Therefore PMDK resorts to non-trivial tricks to ensure larger portions of memory can be updated in a failsafe manner.

An introduction to pmemobj (part 7) - persistent lists

Fri, 19 Jun 2015 19:55:17 -0700

The pmemobj library provides non-transactional persistent atomic circular doubly-linked lists (or NTPACDLL for short) API with an interface familiar to anyone who have ever included sys/queue.h header file - it’s in fact so similar that I considered not writing this post at all, you can just search the web for CIRCLEQ example.

Fun fact: The exact same list code is used internally by libpmemobj in the transaction undo log implementation.

An introduction to pmemobj (part 5) - atomic dynamic memory allocation

Thu, 18 Jun 2015 19:55:17 -0700

In the previous post I talked about using transactions for allocating new objects, which is fine and is the most similar approach to the standard POSIX way. But it does add an overhead of maintaining an undo log of changes. A more optimal memory management can be achieved using the non-transactional atomic API the pmemobj library provides.

Fail-safe atomic allocations

This API is not similar to the APIs most programmers are used to when it comes to handling memory. First of all, the functions are either allocating to, or freeing from a pointer. The modification of the destination pointer is done in an atomic way so that it is always valid - it either points to a valid and initialized memory block or an OID_NULL. The functions/macros also force you to create the objects in a known state - either by zeroing them (POBJ_ZNEW, POBJ_ZALLOC), or providing a constructor for you to initialize the pointer (POBJ_NEW, POBJ_ALLOC).

An introduction to pmemobj (part 6) - threading

Thu, 18 Jun 2015 19:55:17 -0700

All of the pmemobj library functions are thread-safe, with following two exceptions: pool management functions (open, close and friends) and pmemobj_root when providing different sizes in different threads - so as long as you are using this function the way it’s meant to be used you don’t have to worry about it. As for macros - generally only the FOREACH macros are not thread-safe for obvious reasons.

Synchronization

If you need to put a lock inside a structure that resides on persistent memory, our library provides pthread-like API for that purpose. There’s no need to initialize those locks or to verify their state. When an application crashes they are all automatically unlocked. As for the reason why something like this is needed, consider following code:

An introduction to pmemobj (part 4) - transactional dynamic memory allocation

Wed, 17 Jun 2015 19:55:17 -0700

This is a topic I intentionally avoided not to introduce too much complexity too fast. The pmemobj library contains an implemented from scratch memory allocator, that was designed with persistent memory in mind. There are two separate APIs: non-transactional and transactional.

Transactional allocations

Let’s start with a simple snippet of volatile code:

struct rectangle {
 int a;
 int b;
};

int area_calc(const struct rectangle *rect) {
 return rect->a * rect->b;
}

...
struct rectangle *rect = malloc(sizeof *rect);
if (rect == NULL) return;
rect->a = 5;
rect->b = 10;
int p = area_calc(rect);
/* busy work */
free(rect):

You should know how to modify this code for persistent memory by now, with two exceptions - malloc and free. Let’s start by declaring the layout:

An introduction to pmemobj (part 3) - types

Tue, 16 Jun 2015 19:55:17 -0700

In all of the previous post the code snippets and examples had persistent pointers (PMEMoid) without any type information - they were simple C structures. Very early in the development of the library we discovered that using something like that was extremely error-prone and generally difficult. That’s why considerable effort was put into encapsulating the PMEMoids with type-safe container. The end result can be compared with how shared_ptr and the like are done in C++11. All posts after this one will solely use the type-safety features.

An introduction to pmemobj (part 2) - transactions

Mon, 15 Jun 2015 19:55:17 -0700

By now you should be fairly familiar with the basics persistent memory programming, but to make sure the application is always in a consistent state you had to rely on your own solutions and tricks - like the length of a buffer in the previous example. Now, we will learn a generic solution provided by pmemobj to this type of problems - transactions. For now we will focus on a single-threaded applications with no locking.

An introduction to pmemobj (part 1) - accessing the persistent memory

Sat, 13 Jun 2015 19:55:17 -0700

In the previous post, you learned a little bit about the general concept of the persistent memory programming model, now it’s time to start the coding ;)

Memory pools

If you’ve read the overview you know that persistent memory is exposed by the OS as memory-mapped files, we call them pools.

The pmemobj library provides an interface to easily manage those pools, so that you don’t have to manually create the files or mmap them. Creating a pool is done using the pmemobj_create API function, which takes the usual parameters you would expect for a function creating a file plus a layout, which is a string of your choosing that identifies the pool. It is required that the layout you pass to pmemobj_open matches the one the pool was created with. As with any other OS resource, you have to release the pool using pmemobj_close when the persistent memory pool is no longer needed, usually at the end of the application. To verify the integrity of the pool there’s a pmemobj_check function that verifies if all the required metadata is consistent.

An introduction to pmemobj (part 0) - new programming model

Fri, 12 Jun 2015 19:55:17 -0700

The aim of this tutorial series is to introduce you to programming with persistent, byte-addressable memory using the pmemobj library. We will go over all the available features, implement an example application and learn something about the inner workings of libpmemobj. If you haven’t read the NVM Library overview I encourage you to do that now.

When designing the library API, we’ve put a heavy emphasis on ease of use and “management explainability”, as well as flexibility and performance. Our hope is that the most intuitive solution to a problem is the correct one, but usually there is more than one way of implementing something. In the course of this tutorial I’ll make it clear when that’s the case and explain what are the pros and cons of the different approaches.

Type safety macros in libpmemobj

Thu, 11 Jun 2015 19:55:17 -0700

The PMEMoid plays the role of a persistent pointer in a pmemobj pool. It consist of a shortened UUID of the pool which the object comes from and an offset relative to the beginning of the pool:

typedef struct pmemoid {
 uint64_t pool_uuid_lo;
 uint64_t off;
} PMEMoid;

Operating on such persistent pointers is equivalent to operating on raw pointers to volatile objects represented by void *. This approach is error prone and such errors are very hard to find.

Implementing (simple) MySQL storage engine with libpmemobj

Tue, 02 Jun 2015 19:55:17 -0700

The focus of the pmemobj library, like the name suggests, is storing objects on a persistent medium. A different, but very common, approach of doing exactly the same is to use a database with a specialized interface to manipulate the collection of data. MySQL is one such database, it processes SQL queries by calling (usually) multiple methods of the storage engine used for the data tables that query operates on. This tutorial covers basics of the pmemobj library, including non-transactional allocations and very simple transactions.

Using the Block Translation Table for sector atomicity

Tue, 23 Sep 2014 19:55:17 -0700

Persistent memory based storage is able to perform IO at byte (or more accurately, cache line) granularity. However, we often want to expose such storage as traditional block devices. The block drivers for persistent memory will do exactly this. However, they do not provide any atomicity guarantees. Traditional SSDs typically provide protection against torn sectors in hardware, using stored energy in capacitors to complete in-flight block writes, or perhaps in firmware. We don’t have this luxury with persistent memory - if a write is in progress, and we experience a power failure, the block will contain a mix of old and new data. Applications may not be prepared to handle such a scenario.

Git Workflow

Tue, 09 Sep 2014 19:55:17 -0700

Now that we’ve created the GitHub Repository for the PMDK, here’s a more detailed description of the git workflow we’ve chosen. The basic idea is:

Current development happens on the master branch
Releases are created periodically by tagging
After a major release, a stable-1.x branch is created. All bug fixes should be committed to the oldest affected branch which is currently supported. Stable branches will be merged periodically to later branches and master.
All changes are done using the common GitHub fork and merge model, where GitHub users fork the repository, make their changes, and submit pull requests.

Here’s a walk-through of the common development workflow, where changes end up on the master branch.

NVM Library Overview

Mon, 01 Sep 2014 19:55:17 -0700

[Edit on 2017-12-11: In the years since this entry was written, this work has evolved into PMDK.]

Why are we building an NVM Library? Where does it live? How does it work? This blog entry provides some answers, which refer to this picture showing the overall library architecture:

Why?

The operating system exposes persistent memory to applications as a memory-mapped file, using a persistent memory aware file system as shown in the picture. But that’s a very raw form of access. Stopping there would be like telling applications they have all the dynamic memory they need because the OS provides the sbrk(2) system call. Just as libc and most other language run-time environments provide memory management APIs like malloc(3) and free(3), which build on the raw interfaces like sbrk(), we need to do something similar for persistent memory. Memory-mapping a pmem file gets you direct access, but then you will want to carve it up into data structures and update it in a way that remains consistent across system interruptions.

Linux Examples

Fri, 29 Aug 2014 19:55:17 -0700

[Edit on 2017-12-11: The Linux examples are outdated now, look at PMDK instead.]

The basic architecture. for exposing persistent memory gives applications a very raw type of access. Applications can load/store directly to the persistence, but then what. What are the interesting problems facing an application developer and what would some solutions look like?

To help describe the issues and potential solutions, we’ve published a set of Linux examples around persistent memory. These examples start with a simple “hello, world!” style program for storing a string in persistent memory, and end with a full binary tree sort example which maintains a consistent tree data structure even in the face of power failure or other types of system interruption. The examples include a mini fault injection framework to illustrate the type of tools that are necessary for demonstrating programs correctly survive system interruption.

Crawl, Walk, Run...

Wed, 27 Aug 2014 19:55:17 -0700

If you can’t fly then run, if you can’t run then walk, if you can’t walk then crawl, but whatever you do you have to keep moving forward.

Martin Luther King Jr.

This project, as well as the support for persistent memory in various operating systems, can be thought of as a crawl, walk, run approach. As byte-addressable persistence enters the market, modified system BIOS images and device drivers expose it to other modules in the kernel. Despite the byte-addressability, many of the initial usages are based on block storage device interfaces. But you have to start somewhere, so this is the crawl phase. The move beyond that, we need a software architecture for exposing persistent memory up the stack.

References

Tue, 26 Aug 2014 19:55:17 -0700

Some very interesting research has been happening in the area of Persistent Memory and more is emerging. While the examples provided here are meant as introductory and simple, some publications cover the topic in much more depth and include complete transaction systems, compiler/language enhancements, etc. Here are some of the most important publications in this space (please send links to more and we’ll include them).

One of the most impressive bodies of work in this area is Mnemosyne. Read the paper:

Creating pmem.io

Mon, 25 Aug 2014 19:55:17 -0700

The pmem project in GitHub has been created as an open source project focused on persistent memory programming. Everything on this web site and the associated GitHub repositories is open source under the “three-clause” BSD license.

Some educational Linux examples are included, which demonstrate the SNIA NVM programming model and some of the interesting features and challenges associated with persistent memory.

The team’s initial focus is the Linux NVM Library which will provide useful APIs for memory allocation, transactions, etc. built on top of the NVM programming model.