aoe Linux Proving Grounds

70zds_tgtrtt_sticky: for challenging networks

Description:

This patch helps the aoe driver handle complex networks with serious congestion problems.  In our tests, this patch increased throughput from 36 MB/s to 186 MB/s in a scenario with a theoretical limit of about 200 MB/s.  It decreased throughput for a best case network by only about 3%.  Examples of networks that could benefit from this patch include...

• networks where the AoE targets have a greater number of ports than the AoE initiator does, or vice versa,
• networks where the different network paths have different capacities,
• networks with hardware problems that cause packet loss.

Link:

The patch can be found here.

Tag:

The three parts of this patch reflected in its name are,

• "zds": for testing, support a setting of zero aoe_deadsecs that disables device failure (also disabling network path failure, so don't use this in production, since packets will be retransmitted to an unresponsive destination forever),
• "tgtrtt": the round trip time (RTT) tracking statistics are kept for each AoE device's remote MAC address instead of just per AoE device,
• "sticky": before the (exponentially fast) slow start is allowed to quickly increase the congestion window size, there is a "cooling off" period where it "sticks" near a smaller value.

Meta: about the aoe Linux Proving Grounds

This blog describes patches for the Linux aoe driver, the in-kernel AoE initiator.  The patches change an aoe driver to fix problems or to add features.  They are considered "unstable" because they are experimental and not thoroughly tested, but you can try them if you have a test environment.  If you do, please leave feedback here by commenting.  You can also email ecashin at coraid dot com directly.

A patch can be applied to the aoe driver source code using the "patch" command while working in the aoe source directory, e.g.,

    cd aoe6-70
    patch -p1 < /tmp/aoe6-69-70pre1.diff

There are two places to find the aoe driver.  One is at CORAID's website, and the other is in the Linux kernel distributed at kernel.org or through the various Linux distributions (like Fedora and debian).

Unless otherwise specified, the patches here apply to the aoe driver at the CORAID website.  There is ongoing work to bring the kernel.org driver up to the level of development that the CORAID-website driver has reached.

70maxioc_qpcpu: processor affinity with work_queues

Description:

In aoe6-70, two kernel threads, aoe_tx and aoe_ktio, handle AoE packet transmission and I/O completion respectively.  Each of these threads can run on any processor.

The CPU caches should provide the most benefit when the CPU running the process that is doing the I/O is the only CPU that interacts with that data.   Specifically, the skb_copy_bits function called during I/O completion would ideally be called on the CPU that will use the data.

This patch eliminates the two special purpose threads, replacing each with a work queue and a set of per-CPU queues.  The schedule_work function always schedules work to run on the current CPU.  Scheduling work when work is already pending is OK, because the per-CPU queues that hold the data keep track of what work really needs to be done.

Caveats:

This patch works with aoe built for 2.6.28.y kernels.  It uses the for_each_cpu macro that does not exist in earlier kernels.

Testing has been minimal.  On module load, the target did not fully initialize immediately, as there appeared to be I/O failures during the block layer's reading of the partition table.  After that, however, I/O seemed to proceed normally.

Link:

The patch can be found here.

Tag: 70maxioc_qpcpu

This patch applies to aoe6-70, adding two module parameters, aoe_maxioc and aoe_maxtx, to cap the number of I/O completions or transmissions respectively, that the driver does before releasing the CPU and allowing the scheduler to run another process.

The "qpcpu" suffix indicates that this patch causes the driver to use a separate queue for each CPU, so that I/O completions and transmissions have CPU affinity.