[PLUG] megaraid_sas write caching?

[Aaron Ten Clay]

On Dec 15, 2007, at 10:24 AM, Aaron Burt wrote:

> On Sat, Dec 15, 2007 at 02:34:32AM -0800, Aaron Ten Clay wrote:
>> Hey all,
>>
>> Just got Gentoo installed on a shiny new Dell PowerEdge 2950 server
>> with an LSI Logic MegaRAID SAS RAID controller. Everything is great
>> except this:
>>
>> [   81.028864] sd 0:2:0:0: [sda] 1951399936 512-byte hardware sectors
>> (999117 MB)
>> [   81.028991] sd 0:2:0:0: [sda] Write Protect is off
>> [   81.029059] sd 0:2:0:0: [sda] Mode Sense: 1f 00 10 08
>> [   81.029121] sd 0:2:0:0: [sda] Write cache: disabled, read cache:
>> disabled, supports DPO and FUA
>>
>> As the kernel reports, both write- and read-caching are disabled.  
>> This
>> is grossly obvious during file copies of a few GB or more - the  
>> entire
>> system grinds to a halt for 5+ minutes per GB. Even the console is
>> unresponsive.
>
> One usually disables write-caching with RAID.  It makes it much less
> likely that you'll end up with an inconsistent RAID volume if  
> there's a
> power-cut.
>
> That said, the little caches on the drives won't help you much,  
> anyway.
> Depending on what technique is used for the copying, you're running  
> into
> a problem with some implementations of the block caching, readahead
> and/or scheduler systems.  Basically, it tries to read as many files
> as it can into RAM, then swaps madly, while writing files out, while
> it's trying to read more, and trips over its own feet.  I don't know
> what kernel config ad optimizations you're using in Gentoo, but poor
> choices there could make it worse.
>
> The cp command, I understand, does things in little chunks to prevent
> this.
>
> _______________________________________________
> PLUG mailing list
> PLUG at lists.pdxlinux.org
> http://lists.pdxlinux.org/mailman/listinfo/plug

Thanks for the reply. The server is UPS-backed and the RAID controller  
has it's own battery backup unit. I don't do anything special with the  
kernel, I just like Gentoo's package management. I've tried the copy  
with both cp and rsync, in both cases the servers console becomes  
unusable. I've never seen this before - I have several systems running  
with 3ware raid controllers that respond just fine under heavy disk  
utilization. The kernel also doesn't say write caching is disabled on  
those boxes.

I think what's happening is the kernel isn't buffering writes to the  
RAID controller - irrespective of whether the RAID controller is  
disabling it's own write cache or the caches on each disk.

-Aaron