CXL Linux Sync: Ground Rules

  • Do not share confidential information
  • Do not share confidential product details
  • Do not disclose CXL consortium confidential information
  • Do discuss any Linux questions about released CXL specifications:
  • Do use IRC as a supplement for this sync meeting for quick questions
    • #cxl on irc.oftc.net
  • Do follow-up on linux-cxl@vger.kernel.org for longer questions / debug
  • https://pmem.io/ndctl/collab/

September 2023

  • Opens:
    • John: CXL memory online by default memhp_default_state=offline not working?
  • QEMU
  • cxl-cli
  • v6.6 Fixes
  • v6.7 Queue

QEMU

  • Merge window induced slowness
  • Round-up of fixlets sent up
  • Multiple HDM Decoder support for endpoints posted
  • Serial number update
  • Maintainer feedback administrivia cleanups
  • Sort out revision numbers for spec version comments
    • advocate with your rep about caching old copies at spec-landing
  • MCTP I2C from NVME
    • Single Aspeed i2c controller driver has support
    • POC quality / out-of-tree support until server class driver arrives
  • DCD Update
    • waiting for kernel-side code resolution
    • Get Extent List for unaccepted memory, track pending state in the implementation
    • cxl-test may need updates too
  • MHD Update
    • Joint effort with SK Hynix, custom command set
    • Proto-DCD
    • Single logical device
    • Software Development Vehicle
  • CPMU, ARM, Compliance, Type-2
  • SPDM Interest
    • WDC looking at library-izing it, still looking to support and external agent
  • FM API (MCTP Mailboxes + Switch CCI + MHD Mailbox)

v6.6 Fixes

v6.7 Queue

  • RCH EH (under)
  • Kernel SPDM
    • WDC showing up to help
    • Invite to CXL sync? Invited to “devsec”

August 2023

QEMU Update

  • Not a huge amount going in this merge, doc, fixes Multiple HDM decoders should be going in this merge.
    • Lot of stuff is backed up by the mailbox rework
    • Jonathans gitlab has DCD preview queued up.
      • Ira did some testing and fixes were merged in latest version
      • Jonathan might have broken it with rebasing. So just a reminder that this is work in progress.
    • MCTP support over I2c… Support is coming from NVME-MI this work is similar to FM-API

cxl-cli update:

v6.5 Fixes Queue

v6.6 Queue

  • RCH Error handling
    • Terry working on it right now. Was waiting on response from Dan which should be there yesterday.
    • Will pick that work back up
  • Type2
    • Davidlohr to submit the fix for type2 init collision. (Merged)
    • Dan rebasing patches. There is conflict here with the Switch CCI work. See below.
  • DCD
    • Ira is reworking the patch set quite a bit.
    • Fan’s QEMU DCD work is being used
    • Cxl-test being added for better regression testing
    • Cxl-test event processing was changed
    • New DAX device work needed to handle sparse extents within the dax region
    • Interleaving is in the back of his head and Navneet has been looking into this. However, interleaving is not slated for this initial work
    • Jonathan - concerned that interleaving should not to be precluded
    • Leave in comments about where interleaving would fit in.
    • Interleaving is the next major feature…
    • QEMU - DCD merge would be at least 6.7 aligned.
  • Switch CCI (Jonathan)
    • Opens around what we do for user space – almost every command is destructive
    • Maybe just CXL raw commands are required?
    • Patch set has been a pain to rebase on type 2 from Dan
    • Would really like review / feedback
    • Davidlohr would like to merge the ‘moving around code’ sooner
    • Would help with the type 2 conflicts
    • It is hard to generalize the code without this second user
    • Not critical for 6.6
    • would like to see an early merge slated for 6.7
    • In the end – Security questions are major gating factor
  • Memory tiring in general
    • CDAT vs HMAT
    • ‘Distance’ calculations vary
    • Patch set: ‘Mem tiring calculating abstract distance from ACPI’ (v6.7 material)

FM general topics

  • We said we would talk about FM things in this meeting…
  • Is there something at plumbers? Yes there is.
  • Plumbers BoF for FM stuff?

Question from discord:

  • John: “numa ratio policy patch?”
    • Jonathan will try and dig in to see where the patches are
    • We are talking at a VMA level.

QEMU Update

cxl-cli update

v6.5 Fixes Queue

v6.6 Queue

July 2023

ndctl / cxl-cli update

  • v78 - minor fixups only - will go out this week
  • v79
    • Firmware update (no outstanding comments)
    • Poison injection (awaiting new rev)
    • Others?

    …further notes not captured.

June 2023

  • Opens:
    • OpenBMC collaboration
    • Labels / Persistent Naming (6.3 issue?)
    • Add a CXL-CLI Item to the Agenda
  • QEMU Update
  • v6.4 Fixes
  • v6.5 Merge Queue
  • Post v6.5 material

QEMU Update

  • QEMU DCD Support?
  • MLD Support
  • CCI layering work for OpenBMC collab
  • I2C ACPI aspeed controller (upstream questionable)

v6.4 Fixes

v6.5 Merge Queue

Post v6.5

May 2023

  • Opens:
    • rasdaemon patches need review
  • LSF/MM takeaways
  • QEMU Update
  • v6.4 pull summary
  • v6.5 Queue

LSF/MM takeaways

  • CXL 3.0 specification update review well received
  • Discussed nodes vs zones and mempolicy vs mmap flags, nodes+mempolicy continues as the path forward
  • Fabric manager: several efforts in flight (one in rust one in golang, OCP and OFA efforts as well)
  • Live migration: CXL as a transport for migration, opportunity for migrate in place

QEMU Update

  • Several patchkits ready and awaiting final merge:
    • volatile memory
    • poison handling
    • events
  • DCD support starting to surface
    • Initial test results of the pre-RFC implementation look good
    • QMP based interface

v6.4 pull summary

v6.5 queue

April 2023

  • Opens:
  • QEMU Update
  • v6.3 Fixes
  • v6.4 Queue
  • v6.5 Queue

v6.3 Fixes

v6.4 Queue

v6.5 Queue

March 2023

v6.4 Queue

Notes

  • Question about kernel code modularity for accelerator drivers
    • Expectation is that it is a bug if CXL core code cannnot be reused for devices outside of the class-device definition
  • DCD Sharing may be the first user of HDM-DB functionality in the kernel, QEMU model for this in scoping
  • Multi-head (not yet MLD) device support in the works for QEMU
  • QEMU gaining a fix for clearing the HDM decoder COMMITTED bit when deactivating decoders
  • Poison
  • Poison inject can be done unconditionally, rely on “injected” indication to delineate real vs simulated hardware problems
    • open question: should the driver taint the kernel on inject? No, ACPI EINJ does not
    • Poison list: emit trace event on inject event? Maybe already covered by another event record

February 2023

  • Opens:
    • CXL DVSEC emulation fixes
    • QEMU Update
  • v6.3 Merge Window
  • v6.4 Queue

v6.3 Merge Window

  • Move tracepoints to cxl_core
  • Export CXL _OSC error control result
  • CXL Events to Linux Trace Events (including interrupts)
  • HDM decoder emulation
  • Default “Soft Reserved” (EFI_MEMORY_SP) handling policy (kernel)
  • Volatile Region Discovery
  • Volatile Region Provisioning
  • Set timestamp

v6.4 Queue

  • Poison inject and clear
  • Forward and reverse address translation (DPA <==> HPA)
  • Poison retrieval
  • memory_failure() for CXL events
  • Dynamic Capacity Device support
    • Sparse DAX Region infrastructure
    • DCD event plumbing
  • CDAT + QTG _DSM integration
  • DOE rework
  • Standalone CXL IDE
    • PCIE SPDM pre-requisite
  • RAS Capability Tracing on RCH AER events
  • Maintenance Feature Support (DRAM PPR)
  • CXL perf monitoring
  • Switch CCI

Notes:

  • QEMU:
    • Several patch kits in flight: https://gitlab.com/jic23/qemu/-/commits/cxl-2023-02-21/
  • AER Discussion:
    • What about CXL Reset for recovery?
      • May be more relevant for future Type-2 devices than Type-3
      • Add another PCI error recovery reset type?
      • Map FLR => CXL Reset?
      • PCI core supports per-device reset methods
  • DCD
    • Look at MLD support before Switch CCI support
  • CXL perf monitoring
    • https://lore.kernel.org/r/20221018121318.22385-1-Jonathan.Cameron@huawei.com
  • FW Update
    • depends on background command support
    • revisit for v6.4
  • Scan Media
    • revisit for v6.4

January 2023

Agenda 01/24

  • Opens:
    • DAX-page request API rework
    • FM Project? LSF/MM topic
    • Type-3 volatile
    • QEMU Update
  • v6.2 Merge Window
  • v6.2-rc Fixes
  • v6.3 Status
  • v6.3+ Future Work

v6.2 Merge Window

  • Cache invalidation for region physical invalidation scenarios
  • DOE kernel/user access collision detection
  • RCH preparation patches
  • RCH Support (including DVSEC Range Register enumeration)
  • Security commands (including background commands)
  • RAS Capability Tracing on VH AER events
  • XOR Interleave Math support
  • cxl_pmem_wq removal
  • EFI CPER record parsing for CXL error records

v6.2-rc Fixes

Merged in cxl/fixes:

  • RAS UE addr mis-assignment

Pending merge:

  • Fix nvdimm unregistration

v6.3 Status

Merged in cxl/next:

  • Move tracepoints to cxl_core
  • Export CXL _OSC error control result

Pending merge:

  • CXL Events to Linux Trace Events (including interrupts)
  • Poison inject and clear
  • Forward and reverse address translation (DPA <==> HPA)
  • Poison retrieval
  • HDM decoder emulation

Awaiting next (or first) posting:

  • RAS Capability Tracing on RCH AER events
  • Volatile Region Discovery
  • Volatile Region Provisioning
  • CDAT + QTG _DSM integration
  • Set timestamp
  • memory_failure() for CXL events
  • DOE rework

v6.3+ Future Work

  • Default “Soft Reserved” (EFI_MEMORY_SP) handling policy (cxl-cli + daxctl)
  • Dynamic Capacity Device support
  • Sparse DAX Region infrastructure
  • DCD event plumbing
  • Standalone CXL IDE
    • PCIE SPDM pre-requisite
  • Maintenance Feature Support (DRAM PPR)
  • CXL perf monitoring

FM Future

  • MLD Mailbox support for DCD event injection
  • Switch mailbox CCI
    • Multi-head device mailbox tunneling

QEMU

  • Start new threads for debug issues not on patches
  • Greg’s volatile region setup testing
  • Passthrough decoder checks
  • SPDM still pending

November 2022

Agenda 11/29

  • Opens:
    • FSDAX ->notify_failure() regression work still pending
    • Others?
  • Fixes merged for v6.1-rc4
  • v6.2 merge window status
  • Post v6.2 Features

v6.1-rc4 Fixes

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tag/?h=v6.1-rc4

Merged:

  • Mailbox input payload fix
  • Decoder commit crash
  • LSA payload handling fix
  • CFMWS NUMA Node setup
  • Fix switch attached to single-port host-bridge
  • BUG in create-region when no more intermediate port decoders available
  • Fix region object memory leak
  • Fix memdev object memory leak
  • cxl_pmem static analysis fix

v6.2 Merge Window Status

Merged:

  • Cache invalidation for region physical invalidation scenarios
  • DOE kernel/user access collision detection
  • RCH preparation patches

In the queue (has review):

  • RCH Support (including DVSEC Range Register enumeration)
  • Security commands (including background commands)
  • CXL Events to Linux Trace Events (including interrupts)
  • RAS Capability Tracing on RCH and VH AER events

In the queue (needs review):

  • XOR Interleave Math support
  • Forward and reverse address translation (DPA <==> HPA)
  • Poison retrieval
  • cxl_pmem_wq removal
  • EFI CPER record parsing for CXL error records

At risk:

  • Volatile Region Discovery
  • Volatile Region Provisioning
    • CDAT + QTG _DSM integration
  • Poison inject and clear
  • CXL perf monitoring

Post v6.2 Features

  • MLD Mailbox support for DCD event injection
  • Dynamic Capacity Device support
    • Sparse DAX Region infrastructure
    • DCD event plumbing
  • Switch mailbox CCI
    • Multi-head device mailbox tunneling
  • Standalone CXL IDE
    • PCIE SPDM pre-requisite
  • Maintenance Feature Support (DRAM PPR)
  • Default “Soft Reserved” (EFI_MEMORY_SP) handling policy (cxl-cli + daxctl)

October 2022

Agenda 10/25

  • Opens:
    • FSDAX page reference counting rework (merged in mm-unstable)
    • FSDAX ->notify_failure() regression work still pending
    • Code First ECR: ‘SP’ attribute in SRAT
    • QEMU emulation status update
    • Others?
  • Fixes pending for v6.1-rc
  • Features in flight for v6.2
  • Rough plans for post v6.2 work

v6.1 Fixes

https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git/log/?h=fixes

Queued:

  • Mailbox input payload fix
  • Decoder commit crash
  • LSA payload handling fix
  • CFMWS NUMA Node setup

Pending:

  • Fix switch attached to single-port host-bridge
  • BUG in create-region when no more intermediate port decoders available

v6.2 Features

In rough priority order, feedback welcome:

  • RCH Support (including DVSEC Range Register enumeration)
  • Cache invalidation for region physical invalidation scenarios
  • RAS Capability Tracing on RCH and VH AER events
  • CXL Events to Linux Trace Events (including interrupts)
  • EFI CPER record parsing for CXL error records
  • Forward and reverse address translation (DPA <==> HPA)
  • Volatile Region Discovery
  • Volatile Region Provisioning
  • Security commands (including background commands)
  • CXL perf monitoring
  • Miscellaneous cleanups and renames

Post v6.2 Features

  • Dynamic Capacity Device support
    • Sparse DAX Region infrastructure
    • DCD event plumbing
  • Maintenance Feature Support (DRAM PPR)
  • Switch mailbox CCI
    • Multi-head device mailbox tunneling
  • Default “Soft Reserved” (EFI_MEMORY_SP) handling policy (cxl-cli + daxctl)

August 2022

Agenda 8/30

  • Opens:
    • FSDAX ->notify_failure() fixes
    • FSDAX page reference counting rework
  • Linux v6.0-rc1 and ndctl (ndctl, daxctl, cxl-cli) v74 released
  • Fix and Feature queue for v6.0-rc, v6.1 and ndctl-v75
  • Rough plans for post v6.1 work for CXL 3.0 enabling

Recently released

  • Kernel:
    • DPA Space Accounting
    • PMEM Region Provisioning
    • DOE Support in PCI core
    • CDAT retrieval (for debug)
  • User tooling:
    • cxl create-region
    • cxl reserve/free-dpa
    • cxl list -vvv

Next fixes and features

  • ‘arch_flush_memregion()’
  • Fix validation of x1 switch topologies
  • Volatile region provisioning
  • Region labels
  • Security commands support
  • Trace events for CXL events (including interrupts)
  • ‘cxl monitor’ command
  • CXL AER handling
  • Address translation

Future work

  • Performance monitoring
  • Maintenance Feature Support (DRAM PPR)
  • Dynamic Capacity Device support
  • Default “Soft Reserved” (EFI_MEMORY_SP) handling policy

July 2022

Agenda 7/26

  • Opens:
    • FSDAX page reference counting rework
  • What is queued for v6.0 (and ndctl-v74)?
  • Late v6.0 updates
  • Post v6.0 work

Queued for v6.0

  • DOE Support in PCI core
  • CDAT retrieval (for debug)
  • DPA Space Accounting
  • PMEM Region Provisioning

In review for v6.0

Post v6.0 material

  • Pre-existing region enumeration
  • Volatile region provisioning
  • XORMAP interleave support
  • Trace Events for CXL Events
  • List Poison
  • Scan Media
  • Address translation
  • Region persistence in labels
  • Region enumeration via labels

June 2022

Agenda: 6/28

  • Opens:
    • CXL Device Tree Support
    • MEM_HWINIT_MODE=0
    • QEMU mainline CXL support is live
  • What is in review for v5.20 (and ndctl-v74)
  • What else might make v5.20?
  • What is post v5.20 material?

v5.20 in review

v5.20 on deck

  • Pre-existing region enumeration
  • Region persistence in labels
  • Region enumeration via labels
  • Address translation foundation

Post v5.20 material

  • List Poison
  • Scan Media
  • XORMAP interleave support
  • Trace Events for CXL Events
  • Address translation (in cxl-cli) for all kernel supported Events, List Poison, and Scan Media

May 2022

Agenda: 5/31

  • What is in v5.19?
  • What is on deck for v5.20?
  • What is post v5.20 material?
  • Opens

v5.19 / ndctl-v73

  • Kernel
    • lockdep annotations
    • CXL _OSC (native CXL hotplug + error “handling”)
    • Disable suspend
    • Mem_enable fixes

v5.20 / ndctl-v74

  • Kernel
    • Region Provisioning
    • DOE Core
    • CXL CDAT Retrieval
    • Event record handling core
      • Scan Media records
      • Event Interrupts
      • Background command timesharing
  • Userpace
    • ‘cxl create-region’
    • Region listing support
    • Scan media / Event records to json
    • Address translation

Post v5.20 / v6.0

  • Kernel
    • SPDM Attestation
    • IDE
    • Security commands
  • Userspace
    • Attestation helper process
    • CXL Device-DAX Policy