3par 7400 linux & oracle

[steveg]

Hi,

We have just got our 1st 3par 7400 to run a oracle DB.

during our testing we never see any write merges on the multipath devices ( using deadline scheduler), for example:

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
dm-3 0.00 0.00 0.00 2018.00 0.00 8.38 8.50 0.37 0.18 0.14 28.00
dm-5 0.00 0.00 0.00 1801.00 0.00 8.39 9.54 0.37 0.20 0.15 27.50
dm-11 0.00 0.00 0.00 3818.00 0.00 16.77 8.99 0.74 0.19 0.14 51.90

Has anyone else seen this behaviour ?

Thanks

Steve

[Richard Siemers]

Linux is foreign to me, I suspect/believe "write merging" is the same thing as "write coalescing" in other vocabularies... which 3PAR ports have settings to enable/disable interupt coalescence, and in recent versions/patches have changed the default to be off. Maybe thats related?

Code: Select all

showport -par  

N:S:P Connmode ConnType CfgRate MaxRate Class2   UniqNodeWwn VCN      IntCoal
4:0:1 disk     loop     auto    4Gbps   disabled disabled    disabled enabled
4:0:2 disk     loop     auto    4Gbps   disabled disabled    disabled enabled
4:0:3 disk     loop     auto    4Gbps   disabled disabled    disabled enabled
4:0:4 disk     loop     auto    4Gbps   disabled disabled    disabled enabled
4:5:1 host     point    auto    4Gbps   disabled disabled    disabled disabled
4:5:2 host     point    auto    4Gbps   disabled disabled    disabled disabled
4:5:3 host     point    auto    4Gbps   disabled disabled    disabled disabled
4:5:4 host     point    auto    4Gbps   disabled disabled    disabled disabled

[steveg]

Hi,

thanks, i'll take a look. my current thinking is that the linux kernel & device mapper is now obeying FLUSH and FUA commands and sendning them down to the 3par where on our old EVA devices and kernels it was being ignored somewhere along the OS I/O path.

Cheers

Steve

[steveg]

Hi,

just to let everyone know in case they stumble into the same problem. HP upgraded the 3par from 3.1.3 to 3.2.1 P04 and that disabled the WCE bit on the presented SCSI devices and now we are seeing write merges and its looking quite a bit better from early results

Issue ID 104168 in the 3.2.1 release notes.

[afidel]

Hi,

just to let everyone know in case they stumble into the same problem. HP upgraded the 3par from 3.1.3 to 3.2.1 P04 and that disabled the WCE bit on the presented SCSI devices and now we are seeing write merges and its looking quite a bit better from early results

Issue ID 104168 in the 3.2.1 release notes.

Hmm, so they lie to the OS and say that they aren't caching writes. Not sure I'm a fan, sure it improves performance but it might lead to issues. I mean the write cache is supposed to be nonvolatile, but bugs happen and if the OS has failed to issue a flush because it's assuming there is no cache them that could lead to data corruption.

[steveg]

I dont know if the particular version we had on the 3par was different to prior versions so it might just be a bug in that particular version of the 3par OS we hit.

Prior to 3.2.1 when running sdparm -g WCE against a 3par volume

WCE 1 [cha: n, def: 0, sav: 0]

now its

WCE 0 [cha: n, def: 0, sav: 0]

Our HP EVA's all report write cache disabled as well so maybe all vendors do it.

[JohnMH]

All vendors with a proper writeback cache solution will do this. The need to be able to Implement this is where the storage can't guarantee data and cache integrity through end to end integrity checks and a minimum of mirrored cache with destage to permanent media. i.e a non engineered roll your own solution.
If you were to obey this then you undermine much of 3PAR's design goal and may as well deploy some cheap JBOD storage.