aoe Linux Proving Grounds

aoe6-75pre1: avoid system time jumps and support 2.6.34

Description

This prerelease supports the 2.6.33 and 2.6.34 kernels and adds a workaround for an oddity in the SLES 10 kernel source.

This patch also includes the safety feature from aoe6-74tjump2: Handling Time Jumps that helps the driver avoid mistakenly timing out packets when the system time changes.

Link

aoe6-74-75pre1.diff

aoe6-74tjump2: Handling Time Jumps

Description

The 74tjump patch described earlier has an overflow problem, and I think that it's not really a simple enough fix.

The 74tjump2 fix is conceptually easier to grasp: for packets sent more than 250 milliseconds ago, according to the gettimeofday time, instead use a jiffies-based time.  This change requires an extra datum per "frame", the data structure that tracks an AoE command, to record the sent time in jiffies.

By subtracting unsigned integers, the jiffies time calculation avoids overflow as long as the time difference is not more than 2^32 jiffies.  That's about 1193 hours on a machine with HZ set to 1000.

Link

aoe6-74-74tjump2.diff
aoe6-74tjump2-74tjump2p.diff  

aoe6-74tjump: Handling Time Jumps

Description

In order to best accomodate both congested, non-congested, and changing networks, the aoe driver started using high-resolution timing information for tracking round trip times in version 65.

A survey of the facilities in the Linux kernel was performed to identify a suitable method for obtaining high-resolution time, where suitability was determined by efficiency, portability across different kernel versions, and ease of use.  The do_gettimeofday function was the best choice.

Naively using gettimeofday for tracking the times when packets were sent, though, exposes the aoe driver to anomalies when the system time is set, since the time might suddenly shift by some arbitrary amount.

This patch detects and accommodates such time shifts, updating the information about when packets were sent whenever a time jump is detected.

It includes some extra debugging diagnostics and is a bit eager, detecting time jumps of only a fraction of a second.

Link

http://support.coraid.com/support/linux/pre/aoe-prereleases/aoe6-74-74tjump.diff

for 2.6.31-rc8: allocate unused request queue for sysfs

Description:

In 2.6.31, sysfs exposes the request queue of block drivers that don't use a request queue.  The aoe driver was not allocating its request queue using the block layer's functions like blk_alloc_queue or blk_init_queue, and so an oops was being generated.

This patch uses blk_alloc_queue to allocate the queue in the expected way.  Jens Axboe says that the aoe driver should have been doing that already, even though the recent changes to the kobject code revealed the problem.

Status:

This patch is now (20090910) being pushed from Jens Axboe to Linus Torvalds.

Link:

http://support.coraid.com/support/linux/pre/aoe-prereleases/2.6.31-rc8-sleepalloc.diff

for 2.6.31-rc3: ensure that AoE minor address is supported

Description:

This change improves diagnostic messages in the mainline kernel, where the driver is still limited to 16 slot addresses per shelf address.  Once dynamic system minor device numbers are put into the mainline aoe driver, this restriction will be eliminated.

Status:

This change has been announced on the Linux Kernel Mailing List and pushed to the linux-next tree.

Link:

2.6.31-rc3_minor-address-supported.diff

aoe6-72pre2: support 2.6.30

Description:

This patch allows the aoe driver to be built for a 2.6.30 kernel.

In 2.6.30, the hdreg.h macros like WIN_READ, etc., are only defined outside of the kernel. The changes committed to the mainline kernel allow aoe6-72pre2 to build for 2.6.30 but must be reverted during compatibility configuration for backwards compatibility with kernels so old that hdreg.h and ata.h are incompatible.

Link:

The patch can be found here.

aoe6-72pre1: don't include relay when Module.symvers is absent

Description:

I used the relay_fs module (or the relay module plus debugfs) to do some tracing when last working on network congestion avoidance.  I left the tracing code skeleton in, because it could be quite handy during testing, but it had been a major headache for the compatibility system: People running kernels without relay were building aoe modules that depended on relay, and their aoe module wouldn't load.

One reason for that issue was a simple mixup, where a different configuration was used to build the running kernel than the one that was in the kernel sources used to build the aoe driver.  It turns out there was another reason.  If the Module.symvers file is absent, then the "relay_open undefined" message that the compatibility test looks for never appears, even when the function is not defined.

This patch causes the compatibility system to create an aoe driver that doesn't use relay when Module.symvers is absent or when the relay functions are undefined.

Link:

The patch can be found here.

71pre1: preserve timings when moving frames

Description:

This prerelease fixes a bug that was causing premature timeouts of I/O when AoE commands were reassigned from an unresponsive destination MAC address to a new one.

It also includes support for an easier way to specify local network interfaces to the aoe_iflist module parameter: Using commas to separate them means that fancy quoting is no longer needed in scripts.

Other changes are described in the NEWS file.

Link:

The patch can be found here.

70zds_tgtrtt_sticky: for challenging networks

Description:

This patch helps the aoe driver handle complex networks with serious congestion problems.  In our tests, this patch increased throughput from 36 MB/s to 186 MB/s in a scenario with a theoretical limit of about 200 MB/s.  It decreased throughput for a best case network by only about 3%.  Examples of networks that could benefit from this patch include...

• networks where the AoE targets have a greater number of ports than the AoE initiator does, or vice versa,
• networks where the different network paths have different capacities,
• networks with hardware problems that cause packet loss.

Link:

The patch can be found here.

Tag:

The three parts of this patch reflected in its name are,

• "zds": for testing, support a setting of zero aoe_deadsecs that disables device failure (also disabling network path failure, so don't use this in production, since packets will be retransmitted to an unresponsive destination forever),
• "tgtrtt": the round trip time (RTT) tracking statistics are kept for each AoE device's remote MAC address instead of just per AoE device,
• "sticky": before the (exponentially fast) slow start is allowed to quickly increase the congestion window size, there is a "cooling off" period where it "sticks" near a smaller value.

Meta: about the aoe Linux Proving Grounds

This blog describes patches for the Linux aoe driver, the in-kernel AoE initiator.  The patches change an aoe driver to fix problems or to add features.  They are considered "unstable" because they are experimental and not thoroughly tested, but you can try them if you have a test environment.  If you do, please leave feedback here by commenting.  You can also email ecashin at coraid dot com directly.

A patch can be applied to the aoe driver source code using the "patch" command while working in the aoe source directory, e.g.,

    cd aoe6-70
    patch -p1 < /tmp/aoe6-69-70pre1.diff

There are two places to find the aoe driver.  One is at CORAID's website, and the other is in the Linux kernel distributed at kernel.org or through the various Linux distributions (like Fedora and debian).

Unless otherwise specified, the patches here apply to the aoe driver at the CORAID website.  There is ongoing work to bring the kernel.org driver up to the level of development that the CORAID-website driver has reached.