12. Pmemcheck: persistent memory analyzer

Table of Contents

12.1. Overview
12.1.1. Functionality
12.1.2. Basic Usage
12.1.3. Pmemcheck error types
12.1.4. Pmemcheck transaction support
12.2. Pmemcheck Command-line Options
12.3. Pmemcheck Client Requests
12.4. Pmemcheck Monitor Commands

To use this tool, you must specify--tool=pmemcheck on the Valgrind command line.

12.1. Overview

Pmemcheck is a persistent memory profiler, which helps analyze the way user applications utilize persistent memory. It helps recognize whether the correct store scheme is used and that data is made persistent. With additional options it can also be used to log all writes to a specific memory region and to identify whether persistent memory is used as volatile memory. Pmemcheck is loosely based on Memcheck.

12.1.1. Functionality

Pmemcheck instruments client code and tracks all stores made to a user provided range of memory addresses. Writing applications which make use of persistent memory prove to be a challenge, because memory is not reclaimed automatically on reboot. Because of the specific nature of persistent memory, the store scheme has to follow a special pattern. To make the data persistent, preferably in a failsafe manner, you have to do a STORE->FLUSH->SFENCE sequence. Pmemcheck checks if this particular sequence of operations is made to any of the registered persistent memory regions. On program exit pmemcheck informs of all improperly made stores by providing the address, size and state of the store. Additionally if debug information is available, the stack trace of the offending write will be shown. The available states are:

  • DIRTY - a write has been made

  • FLUSHED - the write has been flushed

For more information on persistent memory programming, please visit pmem.io.

The tool, when run with the --mult-stores option, can also track multiple stores to the same location before the data is made persistent - goes through all the states in the right order. This might indicate that there is something wrong with your application, because you are using persistent memory as volatile memory. If this is the desired use, you can remove the volatile region from analysis using the VALGRIND_PMC_REMOVE_PMEM_MAPPING macro.

12.1.2. Basic Usage

As with most other Valgrind tools, compiling your program with debugging info (the -g option) provides the most valuable output.

To run your program under pmemcheck execute:

valgrind --tool=pmemcheck [pmemcheck options] your_program [program options]

For the tool to be able to correctly inspect your program, you should inform it about the persistent memory specific operations your application performs. At the moment, pmemcheck does not automatically recognize persistent memory regions, appropriate flushes or fences. Your code has to use special macros specified in Pmemcheck Client Requests.

Pmemcheck also has a logging functionality, which when enabled, logs all persistent memory relevant macros as well as all stores made to the registered persistent memory regions. This is especially useful for any kind of postprocessing. To turn this option on, run pmemcheck with --log-stores.

12.1.3. Pmemcheck error types

Pmemcheck is able to catch a couple of possible errors related to programming using persistent memory. The most probable is that some stores are never made persistent, which together with power failures, may leave the persistent memory in an inconsistent state. If debugging symbols are available, the output of such an error looks like this:

Number of stores not made persistent: 3
Stores not made persistent properly:
[0]    at 0x400A73: main (state_machine2.c:29)
	Address: 0x4fec000	size: 1	state: FENCED
[1]    at 0x400B56: main (state_machine2.c:32)
	Address: 0x4fec008	size: 2	state: FLUSHED
[2]    at 0x400BB8: main (state_machine2.c:34)
	Address: 0x4fec010	size: 4	state: DIRTY
Total memory not made persistent: 7

This indicates that three stores made to persistent memory are not guaranteed to be durable in case of power failure. The offending stores are identified by a full stack trace and the state of the store at program exit is shown. However, be aware that having a clean output does NOT make your program fail-safe. External tools have to be used to verify whether your use of persistent memory is correct in case of unforeseen failures.

Pmemcheck, when run with the --mult-stores option, can also detect multiple stores made to the same persistent memory address before the first store has been made persistent. This can indicate an issue with your algorithm, where you overwrite a value multiple times, possibly unintentionally. The use of this option is connected with setting the --indiff value. Some implementations of for example memcpy(), write the same data multiple times to the same location. The --indiff option tells pmemcheck between how many Valgrind SuperBlocks, multiple stores with the same value, size and address are to be ignored. An example output:

Number of stores not made persistent: 1
Stores not made persistent properly:
[0]    at 0x400A89: main (multiple_stores.c:33)
	Address: 0x4fec000	size: 8	state: DIRTY
Total memory not made persistent: 8

Number of overwritten stores: 3
Overwritten stores before they were made persistent:
[0]    at 0x400A6F: main (multiple_stores.c:30)
	Address: 0x4fec000	size: 1	state: DIRTY
[1]    at 0x400A76: main (multiple_stores.c:31)
	Address: 0x4fec000	size: 2	state: DIRTY
[2]    at 0x400A7F: main (multiple_stores.c:32)
	Address: 0x4fec000	size: 4	state: DIRTY

If the values or sizes are different, or the stores overlap somehow, the event will be recorded regardless of the --indiff value.

The third type of errors reported by pmemcheck are related to probably inefficient use of flushes. This is strictly a performance issue and does not have impact on data durability. As such it is disabled by default, use the --flush-check option to turn it on. It reports all occurrences of flushes made on non-dirty cache lines. The output from the flush checking analysis can look like this:

Number of redundantly flushed stores: 3
Stores flushed multiple times:
[0]    at 0x400A5B: main (flush_check.c:27)
	Address: 0x4fec000	size: 8	state: FLUSHED
[1]    at 0x400A5C: main (flush_check.c:28)
	Address: 0x4fec000	size: 8	state: FLUSHED
[2]    at 0x400A5D: main (flush_check.c:29)
	Address: 0x4fec000	size: 8	state: FLUSHED

It indicates that the same store has been flushed multiple times, which is a potential performance issue.

In case when no stores were made to the flushed region, the program will yield the following output:

Number of unnecessary flushes: 3
[0]    at 0x400AB2: main (missed_flush.c:27)
	Address: 0x4fec000	size: 64
[1]    at 0x400B18: main (missed_flush.c:29)
	Address: 0x4fec040	size: 128
[2]    at 0x400B71: main (missed_flush.c:31)
	Address: 0x0	size: 64

Pmemcheck also allows tracking of persistent memory transactions. To learn more about the built-in transaction support in pmemcheck see Transactions. The primary type of issue found by pmemcheck are stores made to regions of memory not tracked by transactions. An example output of the out-of-transaction stores has the following form:

Number of stores made outside of transactions: 2
Stores made outside of transactions:
[0]    at 0x400C3E: main (trans_impl_nest.c:39)
	Address: 0x4fec000	size: 1
[1]    at 0x400C58: main (trans_impl_nest.c:42)
	Address: 0x4fec018	size: 8

A different type of error message related to the transactions functionality within pmemcheck, are active transactions on program exit. This kind of error shows that some transactions have not finished properly. This can lead to data corruption and possibly performance degradation. If any transactions are active on exit, a report with the following form is issued:

Number of active transactions: 1
[0]    at 0x400AE5: main (trans_impl_nest.c:32)
	tx_id: 1	 nesting: 1

Another type of consistency error might occur when different transactions are tracking the same memory region. Let's say for example the implementation of your application uses an undo log and two separate transactions add the same memory region to their undo lists. If one of them fails, the changes get rolled back and the resulting state of the whole persistent memory "pool" might be inconsistent. The warning message for this issue has the following form:

Number of overlapping regions registered in different transactions: 1
Overlapping regions:
[0]    at 0x400C9C: make_tx (trans_mt.c:35)
   by 0x4C2F181: start_thread (pthread_create.c:312)
   by 0x4F3F47C: clone (clone.S:111)
	Address: 0x520a010	size: 4	tx_id: 3
   First registered here:
[0]'   at 0x400C9C: make_tx (trans_mt.c:35)
   by 0x4C2F181: start_thread (pthread_create.c:312)
   by 0x4F3F47C: clone (clone.S:111)
	Address: 0x520a010	size: 4	tx_id: 2

12.1.4. Pmemcheck transaction support

Contemporary hardware architectures guarantee that only stores up to a certain size can be made atomically. Here atomicity refers to non-torn writes and not concurrency safety. At the time of writing this documentation the x86-64 architecture supports 8 byte atomic writes. This in most cases is insufficient in real world applications. Therefore the need to introduce transactions, which allow larger ranges of persistent memory to be changed atomically. Whether transactions are rolled-back, redone or employ some other technique is of no importance to pmemcheck. What is important is that pmemcheck knows which regions of persistent memory are tracked by the transaction and can be modified freely while the transaction is active.

Transaction support cannot be turned off, however the runtime overhead is negligible if no transactions are made in the analyzed application. To inform pmemcheck of a new transaction start, use one of two available macros VALGRIND_PMC_START_TX or VALGRIND_PMC_START_TX_N. Depending on whether your application uses flattened, implicit per-thread transactions or explicit thread-independent transactions, use the appropriate macro to ensure the transactions are tracked correctly. Internally each thread has a list of transactions it contributes to and each transactions has a list of persistent memory regions which belong to the transaction. When a new thread starts a transaction, a new entry for the transactions is created with an empty persistent memory regions list. At the same time a new thread entry is created and the id of the transaction is added to its list of active transactions. If different threads are to contribute to the same transaction, pmemcheck has to be informed about the relation between threads and transactions the thread is to contribute to. To do this use the VALGRIND_PMC_ADD_THREAD_TO_TX_N and VALGRIND_PMC_REMOVE_THREAD_FROM_TX_N macros.

Once the transaction is started, pmemcheck has to be informed of the regions which will be tracked by the transaction. To do that call either the VALGRIND_PMC_ADD_TO_TX or the VALGRIND_PMC_ADD_TO_TX_N macro. The two flavors are analogous to the VALGRIND_PMC_START_TX and VALGRIND_PMC_START_TX_N macros. The first one is for implicit transactions and the second for explicit transactions. Both of these macros will add the given region to the list of tracked persistent memory regions within the given transaction (implicit or explicit). Please note that changes made to volatile memory will NOT be reported for obvious reasons. If there is a need to remove any of the regions from the transaction, the VALGRIND_PMC_REMOVE_FROM_TX or the VALGRIND_PMC_REMOVE_FROM_TX_N macros can be used. To add regions of memory which are not to be checked throughout the whole duration of the application run, use the VALGRIND_PMC_ADD_TO_GLOBAL_TX_IGNORE macro.

To end a transaction call either the VALGRIND_PMC_END_TX or VALGRIND_PMC_END_TX_N. As before the _N suffixed versions are for applications using explicit transactions. Once a transaction is ended the transaction entry is removed along with the registered regions. At the same time all active thread entries have the corresponding transaction removed from their index. Once a thread has no more active transactions, it is also removed from the list of active threads. If you nest a transaction with the same id (which is always the case with implicit transactions), you bump up the nesting counter. A transaction will not be removed until the nesting counter hits 0. The counter is decremented on each transaction end.

Once a transaction is started, pmemcheck tracks all stores made by any given thread and checks whether they are made to a persistent memory region that is part of a transaction. If it is not, the store is registered and will be reported on program exit. Changes made outside of transactions are a possible data consistency concern, should the transaction be interrupted in any way. Each such event should be carefully analyzed. Additionally when the tool is run with --tx_only, all changes to persistent memory must be made within a transaction (excluding the regions added by the VALGRIND_PMC_ADD_TO_GLOBAL_TX_IGNORE macro).

12.2. Pmemcheck Command-line Options

Pmemcheck-specific command-line options are:

--indiff=<uint> [default: 0 SBlocks]

This option is only used in conjunction with --mult-stores. Between this many Valgrind SuperBlocks, stores of the same size, address and value will not be counted as an error.

--mult-stores=<yes|no> [default: no]

This option enables analyzing multiple stores to the same region of memory between persistent memory barriers. This might indicate erroneous usage of persistent memory by your application. If you run into problems which are independent of your application (I have seen memcpy() do this on numerous occasions), fine tune the --indiff value to get rid of false-positives.

--log-stores=<yes|no> [default: no]

Turns on logging all persistent memory related events (most Pmemcheck Client Requests and stores to persistent memory) in a human-readable, parsable format. This is useful if some postprocessing or in depth analysis is to be made. Depending on the usage, the log output can be huge - use with caution.

--print-summary=<yes|no> [default: yes]

This option enables or inhibits printing the analysis summary. This option is particularly useful together with --log-stores.

--flush-check=<yes|no> [default: no]

When this option is turned on, pmemcheck checks if your application flushes non-dirty cachelines. Be aware that the tool does not recognize flushes, it depends on your application informing it about the explicit flushes being made. To do that use the VALGRIND_PMC_DO_FLUSH macro.

--flush-align=<yes|no> [default: no]

This flag turns on flush aligning to the native machine's cache line size. It might sometimes prove beneficial to analyze the application with full cache line flushing. Setting this in conjunction with --flush-check might result in false-positives.

--tx_only=<yes|no> [default: no]

This flag turns on strict transaction checks in pmemcheck. When turned on, pmemcheck will track all changes made to memory, which were not registered as being part of a transaction. This is a potential data consistency issue, as these changes will not be rolled-back/redone on transaction recovery, after an interruption. All changes, also those made without a running transaction, will be reported by the tool. For more information on transaction support built into pmemcheck see Transactions.

12.3. Pmemcheck Client Requests

Pmemcheck-specific client requests are:

VALGRIND_PMC_REGISTER_PMEM_MAPPING(addr, size)

Registers in pmemcheck a persistent memory region. This is the basic macro used for persistent memory analysis. Only stores to one of the registered regions are taken into account in pmemcheck.

VALGRIND_PMC_REGISTER_PMEM_FILE(desc, addr_base, size, offset)

Register a persistent memory file. This macro does not register a persistent memory region, it is used to link the regions to a specific file. This is especially useful for any kind of post processing. Returns 1 when the given file could not be found 0 otherwise.

VALGRIND_PMC_REMOVE_PMEM_MAPPING(addr, size)

Remove a region from the pmemcheck persistent memory register. If the removed region overlaps a registered region, it will be spliced. Very useful for carving out parts of persistent memory which are used as volatile memory.

VALGRIND_PMC_CHECK_IS_PMEM_MAPPING(addr, size)

Checks if a given regions is registered as a persistent memory mapping in pmemcheck. Returns 0 if not, 1 if fully within mapping, 2 otherwise.

VALGRIND_PMC_PRINT_PMEM_MAPPINGS

Prints out the registered persistent memory mappings.

VALGRIND_PMC_DO_CLFLUSH(addr, size)

Simulates a flush on all stores to persistent memory within the given address range. Depending on the --flush-align option the flush address might be aligned to the native machine's cache line size.

VALGRIND_PMC_DO_FENCE

Simulates a memory fence for the appropriate stores to registered persistent memory regions.

VALGRIND_PMC_SET_CLEAN

Sets the desired region of persistent memory as clean. This may be useful for discarding changes made to persistent memory.

VALGRIND_PMC_WRITE_STATS

Writes persistent memory analysis statistics.

VALGRIND_PMC_START_TX

Starts a transaction with an implicit transaction id. The id will be the id of the thread which issues the client request. Nesting transactions is allowed - the nesting counter of the transaction is incremented by 1.

VALGRIND_PMC_START_TX_N(txn)

Starts a transaction with the given transaction id. Nesting transactions is allowed - the nesting counter of the transaction is incremented by 1.

VALGRIND_PMC_END_TX

Ends a transaction with an implicit transaction id. The id will be the id of the thread which issues the client request. The nesting counter of the transaction is decremented by 1. If it hits 0, the transaction is no longer tracked by pmemcheck. Returns 1 when no matching and 0 otherwise.

VALGRIND_PMC_END_TX_N(txn)

Ends a transaction with the given transaction id. The nesting counter of the transaction is decremented by 1. If it hits 0, the transaction is no longer tracked by pmemcheck. Returns 1 when no matching and 0 otherwise.

VALGRIND_PMC_ADD_TO_TX(addr, size)

Adds a persistent memory region to the implicit transaction. Stores made during the transaction to persistent memory outside of the registered regions will be reported as a potential data consistency issue. Returns 1 when no matching transaction was found, 2 when the current thread does not participate in this transaction and 0 otherwise. Multiple additions of the same region do not result in multiple region entries. They are registered as contiguous space and not separate enteties.

VALGRIND_PMC_ADD_TO_TX_N(txn, addr, size)

Adds a persistent memory region to the given transaction. Stores made during the transaction to persistent memory outside of the registered regions will be reported as a potential data consistency issue. Returns 1 when no matching transaction was found, 2 when the current thread does not participate in this transaction and 0 otherwise. Multiple additions of the same region do not result in multiple region entries. They are registered as contiguous space and not separate enteties.

VALGRIND_PMC_REMOVE_FROM_TX(addr, size)

Removes a persistent memory region from the implicit transaction. Returns 1 when no matching transaction was found, 2 when the current thread does not participate in this transaction and 0 otherwise. Removing a region once removes it from the transaction, regardless how many times it was added.

VALGRIND_PMC_REMOVE_FROM_TX_N(txn, addr, size)

Removes a persistent memory region from the given transaction. Returns 1 when no matching transaction was found, 2 when the current thread does not participate in this transaction and 0 otherwise. Removing a region once removes it from the transaction, regardless how many times it was added.

VALGRIND_PMC_ADD_THREAD_TO_TX_N(txn)

Adds the current thread to the given transaction. Returns 1 when no matching transaction was found, 0 otherwise.

VALGRIND_PMC_REMOVE_THREAD_FROM_TX_N(txn,)

Removes the current thread from the given transaction. Returns 1 when no matching transaction was found, 2 when the current thread does not participate in this transaction and 0 otherwise.

VALGRIND_PMC_ADD_TO_GLOBAL_TX_IGNORE(addr, size)

Adds a persistent memory region to the global ignore list. Once added the memory region cannot be removed until the end of the execution of the application. Out of transaction writes to these regions will not be reported.

12.4. Pmemcheck Monitor Commands

The Pmemcheck tool provides monitor commands handled by the Valgrind gdbserver (see Monitor command handling by the Valgrind gdbserver). Tool specific monitor commands are:

  • help prints monitor help message.

  • print_stats prints registered pmemcheck statistics.

  • print_pmem_regions prints registered persistent memory regions.

  • print_log_regions prints registered loggable persistent memory regions.