aoe Linux Proving Grounds

for 2.6.31-rc8: allocate unused request queue for sysfs

Description:

In 2.6.31, sysfs exposes the request queue of block drivers that don't use a request queue.  The aoe driver was not allocating its request queue using the block layer's functions like blk_alloc_queue or blk_init_queue, and so an oops was being generated.

This patch uses blk_alloc_queue to allocate the queue in the expected way.  Jens Axboe says that the aoe driver should have been doing that already, even though the recent changes to the kobject code revealed the problem.

Status:

This patch is now (20090910) being pushed from Jens Axboe to Linus Torvalds.

Link:

http://support.coraid.com/support/linux/pre/aoe-prereleases/2.6.31-rc8-sleepalloc.diff

for 2.6.31-rc3: ensure that AoE minor address is supported

Description:

This change improves diagnostic messages in the mainline kernel, where the driver is still limited to 16 slot addresses per shelf address.  Once dynamic system minor device numbers are put into the mainline aoe driver, this restriction will be eliminated.

Status:

This change has been announced on the Linux Kernel Mailing List and pushed to the linux-next tree.

Link:

2.6.31-rc3_minor-address-supported.diff

aoe6-72pre2: support 2.6.30

Description:

This patch allows the aoe driver to be built for a 2.6.30 kernel.

In 2.6.30, the hdreg.h macros like WIN_READ, etc., are only defined outside of the kernel. The changes committed to the mainline kernel allow aoe6-72pre2 to build for 2.6.30 but must be reverted during compatibility configuration for backwards compatibility with kernels so old that hdreg.h and ata.h are incompatible.

Link:

The patch can be found here.

aoe6-72pre1: don't include relay when Module.symvers is absent

Description:

I used the relay_fs module (or the relay module plus debugfs) to do some tracing when last working on network congestion avoidance.  I left the tracing code skeleton in, because it could be quite handy during testing, but it had been a major headache for the compatibility system: People running kernels without relay were building aoe modules that depended on relay, and their aoe module wouldn't load.

One reason for that issue was a simple mixup, where a different configuration was used to build the running kernel than the one that was in the kernel sources used to build the aoe driver.  It turns out there was another reason.  If the Module.symvers file is absent, then the "relay_open undefined" message that the compatibility test looks for never appears, even when the function is not defined.

This patch causes the compatibility system to create an aoe driver that doesn't use relay when Module.symvers is absent or when the relay functions are undefined.

Link:

The patch can be found here.

71pre1: preserve timings when moving frames

Description:

This prerelease fixes a bug that was causing premature timeouts of I/O when AoE commands were reassigned from an unresponsive destination MAC address to a new one.

It also includes support for an easier way to specify local network interfaces to the aoe_iflist module parameter: Using commas to separate them means that fancy quoting is no longer needed in scripts.

Other changes are described in the NEWS file.

Link:

The patch can be found here.

70zds_tgtrtt_sticky: for challenging networks

Description:

This patch helps the aoe driver handle complex networks with serious congestion problems.  In our tests, this patch increased throughput from 36 MB/s to 186 MB/s in a scenario with a theoretical limit of about 200 MB/s.  It decreased throughput for a best case network by only about 3%.  Examples of networks that could benefit from this patch include...

• networks where the AoE targets have a greater number of ports than the AoE initiator does, or vice versa,
• networks where the different network paths have different capacities,
• networks with hardware problems that cause packet loss.

Link:

The patch can be found here.

Tag:

The three parts of this patch reflected in its name are,

• "zds": for testing, support a setting of zero aoe_deadsecs that disables device failure (also disabling network path failure, so don't use this in production, since packets will be retransmitted to an unresponsive destination forever),
• "tgtrtt": the round trip time (RTT) tracking statistics are kept for each AoE device's remote MAC address instead of just per AoE device,
• "sticky": before the (exponentially fast) slow start is allowed to quickly increase the congestion window size, there is a "cooling off" period where it "sticks" near a smaller value.

Meta: about the aoe Linux Proving Grounds

This blog describes patches for the Linux aoe driver, the in-kernel AoE initiator.  The patches change an aoe driver to fix problems or to add features.  They are considered "unstable" because they are experimental and not thoroughly tested, but you can try them if you have a test environment.  If you do, please leave feedback here by commenting.  You can also email ecashin at coraid dot com directly.

A patch can be applied to the aoe driver source code using the "patch" command while working in the aoe source directory, e.g.,

    cd aoe6-70
    patch -p1 < /tmp/aoe6-69-70pre1.diff

There are two places to find the aoe driver.  One is at CORAID's website, and the other is in the Linux kernel distributed at kernel.org or through the various Linux distributions (like Fedora and debian).

Unless otherwise specified, the patches here apply to the aoe driver at the CORAID website.  There is ongoing work to bring the kernel.org driver up to the level of development that the CORAID-website driver has reached.

70maxioc_qpcpu: processor affinity with work_queues

Description:

In aoe6-70, two kernel threads, aoe_tx and aoe_ktio, handle AoE packet transmission and I/O completion respectively.  Each of these threads can run on any processor.

The CPU caches should provide the most benefit when the CPU running the process that is doing the I/O is the only CPU that interacts with that data.   Specifically, the skb_copy_bits function called during I/O completion would ideally be called on the CPU that will use the data.

This patch eliminates the two special purpose threads, replacing each with a work queue and a set of per-CPU queues.  The schedule_work function always schedules work to run on the current CPU.  Scheduling work when work is already pending is OK, because the per-CPU queues that hold the data keep track of what work really needs to be done.

Caveats:

This patch works with aoe built for 2.6.28.y kernels.  It uses the for_each_cpu macro that does not exist in earlier kernels.

Testing has been minimal.  On module load, the target did not fully initialize immediately, as there appeared to be I/O failures during the block layer's reading of the partition table.  After that, however, I/O seemed to proceed normally.

Link:

The patch can be found here.

Tag: 70maxioc_qpcpu

This patch applies to aoe6-70, adding two module parameters, aoe_maxioc and aoe_maxtx, to cap the number of I/O completions or transmissions respectively, that the driver does before releasing the CPU and allowing the scheduler to run another process.

The "qpcpu" suffix indicates that this patch causes the driver to use a separate queue for each CPU, so that I/O completions and transmissions have CPU affinity.