How To Write A Device Driver That Doesn’t Break On Partitioned Systems

How To Write A Device
Driver That Doesn’t Break
On Partitioned Systems
Mike Tricker
Program Manager
Windows Kernel
Microsoft Corporation
Bruce Sherwin
Software Design Engineer
Windows Kernel
Microsoft Corporation
Agenda
Introduction
Motivation
Dynamic Partitioning (DP) on Windows
Server codename “Longhorn”
Kernel driver reminders
Driver considerations on
partitionable systems
Call to action
Dynamic Partitioning
Core
Memory
Core
…
…
Core
Cache
Core
Memory
…
Core
Memory
Cache
Core
Core
…
Core
Cache
Cache
Windows Server
Longhorn dynamic
hardware partitioning
features are focused on
improving server RAS
Memory
Service
Processor
IO Bridge
IO Bridge
IO Bridge
IO Bridge
...
...
...
...
Partition Manager
PCI Express
Example Hardware Partitionable Server
1. Partition Manager provides the UI
for partition creation and management
2. Service Processor controls the inter
processor and IO connections
3. Platforms partitionable to the socket level;
virtualization used for sub socket partitioning
4. Support for dynamic partitioning and
socket replacement
Motivation For DP
This is being driven in part by the availability of
multi-core processors
In a year or two a 4-processor system will have the performance
of a 32-processor system today
And will require Enterprise-class RAS features
In Windows Server Longhorn DP is focused primarily
on Reliability, Availability and Serviceability (RAS)
Minimizing unplanned downtime due to failing hardware
Replacing hardware that is showing signs of impending failure
Capacity on demand
Adding system resources as needed
What We Are Doing
Windows Server Longhorn 64-bit will add support
for Hot Add of processors
Together with PCI Express native hot plug of devices
And continues to support Hot Add memory
It will also support Hot Replace of memory
and processors
Transparent to applications
Drivers receive power IRPs
Windows Server Longhorn is planned to support
Hot Add of I/O host bridges
Supporting devices using line-based interrupts
DP will not be available on all Server SKUs
Kernel Driver Reminders
All 64-bit x64 kernel drivers must be signed or
they will not load on Windows Server Longhorn
or Windows Vista
Drivers that manipulate critical kernel data
structures will cause the system to bugcheck,
for example
System service tables
IDT
GDT
Kernel stacks not allocated by the kernel
Patching any part of the kernel
Taking Advantage Of DP
In Your Device Driver
DP-Aware Drivers
Receiving Notifications
Use IoRegisterPlugPlayNotification()
EventCategory is
EventCategoryDeviceInterfaceChange
The GUID used in the
DEVICE_INTERFACE_CHANGE_NOTIFICATION
structure will be
For Hot Add memory: GUID_DEVICE_MEMORY
For Hot Add processor: GUID_DEVICE_PROCESSOR
Include WDM.H and POCLASS.H
DP-Aware Drivers
Memory Changes
Note: Hot Add memory does not affect
either paged or non-paged pool sizes in
Windows Server Longhorn
Drivers may attempt to allocate more
physical memory after receiving the Hot
Add memory notification
Since memory usage is fundamentally
competitive they should handle the case
of an allocation failure as they (should)
do already
DP-Aware Drivers
Processor Changes
Drivers that care about the number of active processors
should call
KeQueryActiveProcessorCount
to update their count
Do not use KeNumberProcessors – it’s undocumented and is
not static
If they receive a Hot Add processor notification
they must call this DDI again to see what
changed
Which gives them both the updated count and the
affinity mask
So they can update their internal count
Affinity Mask Manipulation
Avoid affinity manipulation “by hand”; use
appropriate RTL APIs for affinity manipulation
when absolutely required
As mentioned on the previous slide
KeQueryActiveProcessorCount
should do exactly what you need
As well as returning the count of active processors it
also returns the affinity mask:
Count = KeQueryActiveProcessorCount(
&AffinityMask);
This will provide you with a current mask that
you can use as you do today
Processor Scaling
If a driver uses per-processor worker threads
Create new ones per newly added processor
Load balancing algorithms should be Hot Add
processor-aware to ensure scalability
As processors are added workloads may need to
rebalance between threads
Miniport developers should ask the class driver
owners about possible behavior changes on
Hot Add
How the miniport could benefit from the
additional resources
Per-Processor Data
Using arrays (or “slotted” data structures) with one entry
per processor works well
Either allocate enough memory to handle the maximum number
of processors possible for that architecture when creating the
data structure
And accept the overhead when the additional processors
are not present
Or use a data structure that can be grown dynamically as
processors are added
So long as your driver gets notified when processors are Hot Added
Provide wrapper functions to hide the complexity of
accessing the entries using the processor number as
the index
E.g., GetData(Processor) and SetData(Processor)
Using Per-Processor Data
// Register for PnP notification
status = IoRegisterPlugPlayNotification(
EventCategoryDeviceInterfaceChange,
0,
&GUID_DEVICE_PROCESSOR,
DriverObject,
CpuDeviceArrivalNotificationCallback,
NULL,
(PVOID) PerProcessorData
);
NTSTATUS
CpuDeviceArrivalNotificationCallback(
IN PVOID NotificationStructure,
IN PVOID Context
)
{
MY_DATA_STRUCTURE *PerProcessorData = (MY_DATA_STRUCTURE *)Context;
// Get current processor count and affinity
newNumberOfProcessors =
KeQueryActiveProcessorCount(&newActiveProcessorMask)
Using Per-Processor Data
if (newNumberOfProcessors > oldNumberOfProcessors) {
// Expand existing per-processor data structures to be aware of new
//processors
InitializeMyDataStructureWithNewProcessorCount(
PerProcessorData,
newNumberOfProcessors);
// Create a thread that runs with the new processor affinity
PsCreateSystemThread(&thread,
THREAD_ALL_ACCESS,
NULL,
NULL,
NULL,
(PKSTART_ROUTINE) MyWorkerThread,
(PVOID) &newActiveProcessorMask);
...
}
}
VOID
MyWorkerThread(IN OUT PVOID Context)
{
PKAFFINITY affinityMask = (PKAFFINITY)Context;
// Set thread affinity
KeSetSystemAffinityThread(*affinityMask);
...
}
Interrupt Targeting
To a New Processor
System triggers a tree-wide rebalance
Allows interrupt-consuming device
drivers to connect interrupts to newly
added processors
DPCs queued from an ISR will run on the
same processor by default
Please don’t fail QUERY_STOP or you
will prevent the system from distributing
interrupts to new processors
DP-Aware Drivers
Resource Rebalance
Started State
IRP_MN_QUERY_STOP_DEVICE (1)
IRP_MN_CANCEL_STOP_DEVICE (4)
Stop-Pending State
IRP_MN_START_DEVICE (3)
IRP_MN_STOP_DEVICE (2)
Stopped State
Failed Restart (5)
Connecting Interrupts
Windows will include new processor in
the affinity for interrupt resource assigned
during rebalance
New resources passed in
IRP_MN_START_DEVICE
Driver should call
IoConnectInterruptEx() using this
new affinity
This will connect the device interrupt
to new processor
Application Implications
Running processes will not change affinity by default
for compatibility
They can receive notifications and change their affinity if desired
RegisterDeviceNotification()
GUID_DEVICE_MEMORY
GUID_DEVICE_PROCESSOR
SetProcessAffinityMask()
SetThreadAffinityMask()
GlobalMemoryStatusEx()
New processes will take advantage of new processor(s)
System process affinity will be changed to include the
newly added processors
Worker threads run in system process
User Mode IOCTLs
Applications may issue custom IOCTLs to a partner
device driver
~40% of “tier 1” applications we test today install drivers to
handle specific tasks
In this case both need to register for Hot Add processor
notifications otherwise they may get out of sync
For example the application may issue IOCTLs on
a thread running on a processor the driver doesn’t
know about
If the driver uses per-processor data, then it needs to register
for hot add processor notification and update its data structures
Hot Replace Flow
System state is migrated in stages
1. Paged memory is copied
2. System is quiesced
Stop DMA
Stop interrupts
Devices moved to D3
Quiesce duration ~1 second
3. Non-paged memory is copied
Processor state is migrated
4. System is resumed
DMA and interrupts resumed
Drivers return to D0
Hot Replace Implications
For Drivers
Hot Replace uses a pseudo-S4 state
where we temporarily “hibernate” the
system to quiesce devices (stop DMA
operations and stop interrupts)
But without actually using a hiberfile
Your driver therefore needs to handle
power S IRP and D IRP requests
IRP_MN_SET_POWER, S4
IRP_MN_SET_POWER, D3
Note: This includes IA-64
System Implications
For Hot Replace
Applications do not see a system change
when a Hot Replace operation occurs
Amount of physical memory is unchanged
Number of logical processor is unchanged
This is deliberate to avoid application
compatibility issues
Hot Replace is treated as an
atomic operation
Hot Add Processor
Bruce Sherwin
Software Design Engineer
Windows Kernel
Related Issues
NUMA Behavior
NUMA is becoming widespread and
should be transparent in most cases
If the platform supports the _PXM ACPI
method you will automatically use NUMA
local memory at AddDevice and
StartDevice time
This ensures your device extensions and
common buffers are allocated out of NUMA
local memory
Testing
Today's DP-capable systems are typically large,
restricting access for driver developers
We expect that to change in the next few years
In the meantime if your device needs to work
with DP we expect you to work with the OEM on
whose system you’ll ship to ensure a great
customer experience
Also work with Microsoft, since we’re also testing
these systems
If you’re already in-box and can run on these
systems you should be covered
Future Technologies
Indicating DP Support
Detecting If DP Is Available
We’ve been asked how to determine if a system
actually supports DP
Since OEMs may ship very similar systems with DP
as a high profile RAS feature
And thus not enabled on lower cost systems
Firmware mechanism to report
platform capabilities
Kernel and user mode API to query
this capability
Exact mechanism TBD
Supporting Hot Remove
Windows Server Longhorn will not support Hot
Remove of memory or processors
When we do we’re concerned about drivers
and applications
Sparse affinity mask and drivers parsing the mask
until finding a missing entry
Memory that’s been pinned for DMA
Drivers using KeNumberProcessors
We’ll have to be very careful about what drivers
we load on such a platform
And consider what may need to be blocked
Call To Action
If you want your 64-bit x64 drivers to load on
Windows Server Longhorn (or Windows Vista)
ensure they’re signed
Ensure that your 64-bit x64 drivers avoid manipulating
undocumented private kernel data structures as they
will bugcheck the system
Please follow the guidelines for good behavior outlined
in this presentation
Realize that DP won’t be “just a high end feature” for
much longer
That will change during the life of Windows Server Longhorn
Ensure that any driver targeting future Windows Server
releases is DP (and NUMA) aware
Additional Resources
Web resources
Dynamic Partitioning home page
http://www.microsoft.com/whdc/system/platform/server/dhp.mspx
White papers
http://www.microsoft.com/whdc/system/vista/kernel-en.mspx
WinHEC 2005 sessions
http://download.microsoft.com/download/9/8/f/98f3fe47-dfc3-4e74-92a3088782200fe7/TWAR05016_WinHEC05.ppt
Kernel patching FAQ
http://www.microsoft.com/whdc/driver/kernel/64bitpatch_FAQ.mspx
Related sessions
Kernel Enhancements for Windows Server Longhorn
How to Use the WDK to Develop, Sign and Test Drivers
For feedback on all things pertaining to Dynamic Partitioning please
send mail to: dpfb @ microsoft.com
© 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market
conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation.
MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.