SUMMARY - Configuring J4200 with SAS multipathing and ZFS (without CAM)

From: Chris Hoogendyk <hoogendyk_at_bio.umass.edu>
Date: Fri Sep 18 2009 - 17:30:27 EDT
This is a very late summary which subsumes a couple of questions, but 
all relates to the same setup. In the end, it worked beautifully (really 
fast) and simply. The documentation and release notes just have to catch 
up and not be tangled in the forest of alternate products and options.

---------------


The micro summary is that the hardware documentation clearly shows how 
to make dual SAS connections between a T5220 and the J4200, and a 
reconfigure reboot just brings it up. Format shows double entries for 
all the drives. So you get 24 drives where you have 12. If, at this 
point, you did `stmsboot -e`, it would coalesce those back down to 12 
and assign longer GUID's for each drive (see below). Format shows those, 
and then a `zpool create  . . . ` sets it all up and you're in business. 
I chose to set up a single raidz2 with one hot spare. So, 9 data drives, 
2 parity drives, 1 hot spare. Performance tests were blazing.

Since I wanted to exclude my internal drives from multipathing, I had to 
edit  /kernel/drv/mpt.conf (not fp.conf, which is the only thing the 
documentation explains, and which is only for fiber channel), and then 
use `stmsboot -u`. Details below.

Meanwhile, the StorageTek Common Array Manager (CAM) software is not 
required. I didn't use it at all, nor install it. Some respondents 
indicated it might be used for monitoring the health of the array.

---------------


Thanks to Bryan Allen, Scott Lawson, Matt R. Clausen, Deborah Crocker, 
and, most especially, to James McPherson, Senior Kernel Software 
Engineer for Solaris at Sun. 8-)

My original messages and their replies are all posted at the end. In the 
interest of helping anyone else who encounters the uncertainties I did, 
I'll summarize a bit more of the actual steps I went through with some 
comments on what's missing in the documentation.

[Note: The part numbers in the following might be edu part numbers. We 
got the chassis & drives on special edu pricing.]
I started with a T5220. Added a J4200 chassis (XTA4200R00A10DISK) filled 
out with SAS drives (XTA6SS1NJ-300G15K) and a second SAS controller 
(XTA-4200-IOM). Rack kit (XTA-4200-2URK-19U) is a pretty basic design. 
You could build one yourself like this one. Then put a dual SAS PCI card 
(SG-XPCIE8SAS-E-Z) in the T5200. Connected both HBA SAS ports on the 
T5220 to the SIM boards on the back of the J4200 according to the 
diagram on p.53 of the J4200 System Overview.pdf (downloaded all pdf 
docs from http://docs.sun.com/app/docs/prod/j4200.array~j4200?l=en#hic). 
Did a `touch /reconfigure` before taking the T5220 down to install the 
SAS HBA and connect everything. When it booted back up, it was just 
ready to fly, except I had to deal with the multipathing.

Patches and updates for SAS multipathing. Solaris 10 should be at U6. I 
had mine at U7 (5/09) by the time I tackled this. The J4200 Release 
Notes, on page 4, indicates specific versions of CAM and patches that 
you must use if you configure multipathing and use the Common Array 
Manager (CAM) software. Just to put a plug in for Live Update, it was a 
breeze to get from U6 to U7 installing from a mounted ISO, and I only 
had a couple of minutes of down time when I restarted with the new boot 
environment.

Getting the multipathing right was the real hangup, although it was 
simple once I was confident what to do. The documentation 
(http://docs.sun.com/app/docs/doc/820-1931/gfpva?a=view) refers to SAS 
several times, but the explicit example of "Per-Port" control of 
multipathing only mentions /kernel/drv/fp.conf and apparently assumes 
that you know that fp.conf refers to fiber channel and mpt.conf refers 
to SAS. The only references to mpt.conf are in the section on automatic 
configuration with `stmsboot -e`, where it lists several system 
configuration files that you do not have to edit (stmsboot does it for 
you). Also, the example listing of the entries in /dev/cfg don't match 
the details in the edit of fp.conf, so it is not clear how the parent 
name and port number are derived. Ultimately, in googling around, I 
stumbled on a Sun engineer who had answered some questions for somone, 
googled some more to find his email address, and sent a message directly 
to him. He answered my questions in detail (see below) and provided both 
some excellent information and reassurances.

In looking back over the J4200 documentation, and specifically searching 
for mpt.conf, I found a one sentance reference (in the 92 page J4200 
System Overview document) saying, "To enable multipathing on specific 
ports and disable multipathing on others, edit the /kernel/drv/mpt.conf 
file." No details or examples.

So. My internal drives had entries like (full list down in one of the 
messages):

/pci@0/pci@0/pci@2/scsi@0/sd@2,0

And my external drives (on the J4200) had entries like:

/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@5,0

I wanted the SAS multipathing to ignore the internal drives and 
automatically take care of the external drives. So, I edited 
/kernel/drv/mpt.conf, adding the line to disable mpxio on the internal 
drives and setting the default to not disable mpxio.

name="mpt" parent="/pci@0/pci@0/pci@2" unit-address="0" mpxio-disable="yes";
mpxio-disable="no";
 

Then I ran `stmsboot -u`. After that format showed the drives in the 
J4200 as just 12 with new long GUID's assigned. See 
http://www.jmcp.homeunix.com/~jmcp/WhatIsAGuid.pdf. That's an excellent 
document by James McPherson. Also, he pointed out that the parent 
property is documented in the driver.conf (4) man page, although, I 
found that to be a bit abstruse. Anyway, format showed me the GUID's, 
which I then could use in a zpool command.

# zpool create -f jpool raidz2 c5t5000C5000F5D37E3d0 
c5t5000C5000F5D529Bd0 c5t5000C5000F6CFBFFd0 \      
 > c5t5000C5000F520E4Fd0 c5t5000C5000F524B4Fd0 c5t5000C5000F523647d0 
c5t5000C50004B4B3CFd0 c5t5000C50004B4B7C7d0 \
 > c5t5000C50004B4FA1Fd0 c5t5000C50004B49B27d0 c5t5000C50004B49BABd0 
spare c5t5000C50004B49973d0

It was interesting that with that many drives and the long GUID's, if I 
just tried to put it all on one line, I hit some sort of limit and the 
command wouldn't take any more. Don't know if that was a ksh limit, a 
term limit, or what. Anyway, using the "\" continuations worked just fine.

That's the whole story. After that it's just a ZFS snap. ;-)


-- 
---------------

Chris Hoogendyk

-
   O__  ---- Systems Administrator
  c/ /'_ --- Biology & Geology Departments
 (*) \(*) -- 140 Morrill Science Center
~~~~~~~~~~ - University of Massachusetts, Amherst 

<hoogendyk@bio.umass.edu>

--------------- 

Erdvs 4



Messages below
     Original Message #1 to SunManagers: Configuring J4200 with ZFS & 
not CAM?
          Reply from Bryan Allen
          Reply from Scott Lawson
          Reply from Matt R. Clausen
          Reply from Tomasz Ostrowski
     Original Message #2 to SunManagers: stmsboot on t5220 with J4200
          Reply from Deborah Crocker
     Original Message #3 to SunManagers: Re: stmsboot on t5220 with J4200
     Follow up to Deborah Crocker: Re: stmsboot on t5220 with J4200
          2nd Reply from Deborah Crocker
     Original Message to James McPherson at Sun: SAS Multipathing 
configuration in Solaris 10
          Reply from James McPherson at Sun
               Reply back to James McPherson at Sun


-------- Original Message #1 to SunManagers --------
Subject: 	Configuring J4200 with ZFS & not CAM?
Date: 	Fri, 24 Apr 2009 11:59:36 -0400



We've just gotten a J4200 to attach to a T5220. My intention is to
configure it with ZFS, which I'm already using for the extra internal
drives on the T5220 (I used hardware mirroring for the boot drive). My
understanding of the J4200 is that it is supposed to be a JBOD, almost
plug and play. However, when I opened the box, it came with the manual
and CD for CAM - the StorageTek Common Array Manager software.

I am still waiting for some drives, so I'm not setting it up yet. I have
a SAS HBA and will be setting it up with SAS drives. If it is just a
matter of plugging it all in and doing a reconfigure reboot, whereupon
the drives show up in format and can then be configured with zpool and
zfs commands, then I'm all set.

Can anyone tell me the relationship/difference between CAM and ZFS?

Am I going to need CAM at all? Or is it sort of like you don't need
raidctl if you are using zfs?

Thanks,

Chris Hoogendyk



-------- Reply from Bryan Allen --------
Subject: 	Re: Configuring J4200 with ZFS & not CAM?
Date: 	Fri, 24 Apr 2009 13:12:13 -0400


I have a J4200 attached to an X4100 M2 using the LSI SAS card from Sun. No
issues. Very happy with it.

If you are using a RAID card, you'll need to configure CAM/RAID/et al. 
If you
are using a plain HBA, you don't need to mess with CAM.

Note that the Adaptec SAS RAID card you get from Sun cannot do JBOD. It 
cannot
be flashed with Adaptec's firmware (which supports JBOD) and Sun support
wouldn't tell me if an update was forthcoming. You also cannot configure 
each
LUN as RAID0, thus exporting each drive to ZFS individually. You would 
have to
create two-disk RAID0s and export the pairs.

I returned that card and got the LSI HBA. There was too much complexity 
for no
reason in the RAID configuration given how I wanted to use ZFS.

You don't need to do a reconfigure reboot, either:

devfsadm -C
cfgadm -al
zpool create ...
-- 
bda
cyberpunk is dead. long live cyberpunk.



-------- Reply from Scott Lawson --------
Subject: 	Re: Configuring J4200 with ZFS & not CAM?
Date: 	Sat, 25 Apr 2009 09:19:00 +1200


Chris,

It should just be a matter of inserting the SAS controller and doing a
reconfigure boot.

I have a T5220 with a J4500 attached in the same way. Works fine for me
and was simple
to set up.

I am not sure why they included CAM at all, it may be that it can be
used as an expansion chassis
for one of the entry level Storagetek arrays. CAM is used for managing
the Storagetek 25xx and
6xxx series hardware RAID arrays. The J4200 is also used with the new
open storage appliances
but I can't see why it would need CAM.. I don't recall receiving CAM
with my J4500. I was also
one of the beta testers for the J4500.

'raidctl' is part of the OS and is used for internal controllers. There
is also a host hardware RAID controller
available for the J series arrays which is made by Adaptec. But if you
use ZFS then you really have no need
for it. It's more for people that use these JBOD's with Windows or Linux.

/Scott.



-------- Reply from Matt R. Clausen --------
Subject: 	RE: Configuring J4200 with ZFS & not CAM?
Date: 	Tue, 28 Apr 2009 17:03:28 -0500


CAM is strictly used for monitoring the J4X00 series arrays. They are 
nothing more then a JBOD so there's nothing for CAM to configure (you 
don't even have to install it though it's a good thing to have because 
it can monitor the array health).

Once you get it all hooked up, you will see all the individual drives 
appear to the operating system from which you can then put them into 
your respective ZFS pools/volumes/etc.

Hope this helps.



-------- Reply from Tomasz Ostrowski --------
Subject: 	Configuring J4200 with ZFS & not CAM?
Date: 	Wed, 24 Jun 2009 08:14:03 +0200


Hello Chris,

Did you manage to configure the disk array withou CAM?
I'm starting with iscsi tech. and now i wonder how to assing an ip
addres tho the SIM controlem on the disk array? Or this is wrong step?
Do you have any guide lines?

Best Regards
Tomek

Tomasz Ostrowski
NSN Preconfiguration Engineer



-------- Original Message #2 to SunManagers --------
Subject: 	stmsboot on t5220 with J4200
Date: 	Mon, 03 Aug 2009 17:39:06 -0400


I have a T5220 with an attached J4200. I added a dual SAS PCI card to
the T5220 and added a second SAS controller to the J4200. It has 12 300G
SAS drives in it.

On the T5220, I used hardware raid to mirror the boot drive, have one
plain drive, and configured 4 drives with zfs raidz. Those are all in use.

When I check format, I see all the internal drives and 2 instances of
all of the drives in the J4200 (see at end).

I want to configure the J4200 with zpool.

As I understand it, I need to do `stmsboot -e` first to set up
multiplathing and reboot. Then I should see the renamed drive devices in
format and can just feed them in to a zpool create.

My problem is that I have never done this before, and I'm afraid of
messing up my existing setup on the T5220. What I get is

# stmsboot -L

stmsboot: MPxIO is not enabled

# stmsboot -e

WARNING: stmsboot operates on each supported multipath-capable controller
detected in a host. In your system, these controllers are

/pci@0/pci@0/pci@2/scsi@0
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0

If you do NOT wish to operate on these controllers, please quit stmsboot
and re-invoke with -D { fp | mpt } to specify which controllers you wish
to modify your multipathing configuration for.

Do you wish to continue? [y/n] (default: y) n

#

It's showing me both the internal scsi (SAS) systems and the external
SAS, and I don't see any way to restrict it to just do the external system.

The `stmsboot -D mpt -e` option doesn't seem to offer anything
different, since it's all SAS. But, since I am telling it to use the mpt
driver, it doesn't do the same sort of autodiscovery informing me what
it is going to do. It just does

# stmsboot -D mpt -e

WARNING: This operation will require a reboot.
Do you want to continue ? [y/n] (default: y) n

#

I can imagine that it might look at all the drives and see that I have
duplicate instances on all the external ones and multipath them, leaving
the others alone; but, I'm just guessing. And, search as I might, I
can't find any examples that explain it in enough detail or that show
what happens if you say no and then configure something differently (or
even continue the example out through the format showing all the
internal and external drives without excerpting the list).

If anyone who has actually done this could give me some encouragement
and/or guidance, I would certainly appreciate it.

TIA


Chris Hoogendyk



AVAILABLE DISK SELECTIONS:
0. c1t0d0 <LSILOGIC-LogicalVolume-3000 cyl 65533 alt 2 hd 16 sec 273>
/pci@0/pci@0/pci@2/scsi@0/sd@0,0
1. c1t2d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848>
/pci@0/pci@0/pci@2/scsi@0/sd@2,0
2. c1t4d0 <SEAGATE-ST914602SSUN146G-0603-136.73GB>
/pci@0/pci@0/pci@2/scsi@0/sd@4,0
3. c1t5d0 <SEAGATE-ST914602SSUN146G-0603-136.73GB>
/pci@0/pci@0/pci@2/scsi@0/sd@5,0
4. c1t6d0 <SEAGATE-ST914602SSUN146G-0603-136.73GB>
/pci@0/pci@0/pci@2/scsi@0/sd@6,0
5. c1t7d0 <SEAGATE-ST914602SSUN146G-0603-136.73GB>
/pci@0/pci@0/pci@2/scsi@0/sd@7,0
6. c4t0d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625>
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@0,0
7. c4t1d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625>
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@1,0
8. c4t2d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625>
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@2,0
9. c4t3d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625>
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@3,0
10. c4t4d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625>
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@4,0
11. c4t5d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625>
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@5,0
12. c4t6d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625>
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@6,0
13. c4t7d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625>
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@7,0
14. c4t8d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625>
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@8,0
15. c4t9d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625>
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@9,0
16. c4t10d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625>
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@a,0
17. c4t11d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625>
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@b,0
18. c4t13d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625>
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@d,0
19. c4t14d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625>
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@e,0
20. c4t15d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625>
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@f,0
21. c4t16d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625>
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@10,0
22. c4t17d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625>
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@11,0
23. c4t18d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625>
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@12,0
24. c4t19d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625>
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@13,0
25. c4t20d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625>
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@14,0
26. c4t21d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625>
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@15,0
27. c4t22d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625>
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@16,0
28. c4t23d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625>
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@17,0
29. c4t24d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625>
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@18,0



-------- Reply from Deborah Crocker --------
Subject: 	Re: stmsboot on t5220 with J4200
Date: 	Tue, 04 Aug 2009 09:08:02 -0500


Review /kernel/drv/fp.conf. You can exempt controllers from 
multipathing. For instance, on our T5240 we have:

mpxio-disable="no"; [This turns multipathing on]
name="fp" parent="/pci@500/pci@0/pci@c/SUNW,qlc@0" port=0 
mpxio-disable="yes";

You'll see more comments on configuring this in the fp.conf file

Deb C.



-------- Original Message #3 to SunManagers --------
Subject: 	Re: stmsboot on t5220 with J4200
Date: 	Wed, 12 Aug 2009 17:32:20 -0400


Sorry, I'm getting a bit frustrated with this. Production system. Can't 
play around. Has to be right. (I got one response from Deborah Crocker 
indicating that I need to edit the fp.conf, but then ... )

So, from http://docs.sun.com/app/docs/doc/820-1931/gfpva?a=view , 
halfway down, it describes how to "Configure Multipathing by Port." It 
says to use `parent="parent name" port=port number`, but the explicit 
example it gives does not match with the /dev/cfg entries that it starts 
the explanation with. So, it's not clear how to derive the parent name 
and the port number, and they don't explain it.

I'm guessing that in my case `parent="/pci@0/pci@0/pci@2/scsi@0" port=0` 
would designate my internal drives. I also presume that since they have 
folded SAS in with FC, that the `name="fp"` designation would apply for 
the internal drives. So, I could exclude them and be ready to go, with:

name="fp" parent="/pci@0/pci@0/pci@2/scsi@0" port=0 mpxio-disable="no";

I've put some output at the very end of this that I extracted from
format, /dev/cfg, and prtdiag for reference.

again, TIA.


Chris Hoogendyk



# format


/pci@0/pci@0/pci@2/scsi@0/sd@0,0
/pci@0/pci@0/pci@2/scsi@0/sd@2,0
/pci@0/pci@0/pci@2/scsi@0/sd@4,0
/pci@0/pci@0/pci@2/scsi@0/sd@5,0
/pci@0/pci@0/pci@2/scsi@0/sd@6,0
/pci@0/pci@0/pci@2/scsi@0/sd@7,0


/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@0,0
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@1,0
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@2,0
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@3,0
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@4,0
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@5,0
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@6,0
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@7,0
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@8,0
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@9,0
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@a,0
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@b,0
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@d,0
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@e,0
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@f,0
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@10,0
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@11,0
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@12,0
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@13,0
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@14,0
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@15,0
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@16,0
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@17,0
/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@18,0



# ls -l /dev/cfg

c1 -> ../../devices/pci@0/pci@0/pci@2/scsi@0:scsi <-- internal drives

c2 -> ../../devices/pci@0/pci@0/pci@9/pci@0/scsi@8:scsi
c3 -> ../../devices/pci@0/pci@0/pci@9/pci@0/scsi@8,1:scsi <-- tape library?

c4 -> ../../devices/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0:scsi 
<-- J4200



# prtdiag -v


================================ IO Devices ================================
Slot +           Bus   Name +                
Status           Type  Path
----------------------------------------------------------------------------
MB/NET0          PCIE  network-pciex8086,105e
                       /pci@0/pci@0/pci@1/pci@0/pci@2/network@0
MB/NET1          PCIE  network-pciex8086,105e
                       /pci@0/pci@0/pci@1/pci@0/pci@2/network@0,1
MB/NET2          PCIE  network-pciex8086,105e
                       /pci@0/pci@0/pci@1/pci@0/pci@3/network@0
MB/NET3          PCIE  network-pciex8086,105e
                       /pci@0/pci@0/pci@1/pci@0/pci@3/network@0,1

MB/SASHBA        PCIE  scsi-pciex1000,58
                       /pci@0/pci@0/pci@2/scsi@0 <-- internal drives

MB/RISER1/PCIE1  PCIE  LSILogic,sas-pciex1000,58
                       /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0 <-- 
J4200

MB/RISER0/PCIE0  PCIE  network-pciex8086,105e
                       /pci@0/pci@0/pci@8/pci@0/pci@9/network@0
MB/RISER0/PCIE0  PCIE  network-pciex8086,105e
                       /pci@0/pci@0/pci@8/pci@0/pci@9/network@0,1

MB               PCIX  usb-pciclass,0c0310
                       /pci@0/pci@0/pci@1/pci@0/pci@1/pci@0/usb@0
MB               PCIX  usb-pciclass,0c0310
                       /pci@0/pci@0/pci@1/pci@0/pci@1/pci@0/usb@0,1
MB               PCIX  usb-pciclass,0c0320
                       /pci@0/pci@0/pci@1/pci@0/pci@1/pci@0/usb@0,2

MB/RISER2/PCIE2  PCIE  scsi-pci1000,30
                       /pci@0/pci@0/pci@9/pci@0/scsi@8
MB/RISER2/PCIE2  PCIE  scsi-pci1000,30
                       /pci@0/pci@0/pci@9/pci@0/scsi@8,1 <-- tape library




-------- Follow up to Deborah Crocker --------
Subject: 	Re: stmsboot on t5220 with J4200
Date: 	Fri, 14 Aug 2009 13:26:59 -0400


Just a quick followup -- why is it fp.conf and 'name="fp"' rather than
mpt.conf?

If I do a stmsboot -e (and then say no to stop it), it lists my
controllers, but doesn't indicate fp or mpt. If I do stmsboot -D mpt -e,
it doesn't list them first, but asks directly if I want to continue. I
said no. So I don't know what it would have come up with. Same with -D fp.

Also, how do you know port=0?

I don't know if you saw my followup on sunmanagers, but the examples in
the Sun documents on multiplathing don't follow through from one item to
the next. Their listing of /dev/cfg does not match up with their example
of editing fp.conf, and there's no explanation of how to derive those
items. So, I'm not sure/confident that the following will be correct for
my system:

mpxio-disable="no";
name="fp" parent="/pci@0/pci@0/pci@2/scsi@0" port=0 mpxio-disable="yes";

Sorry to email you directly, but no one else has responded to either of
my postings on sunmanagers.

I pasted some system output at the bottom.

Thank you very much.


Chris Hoogendyk



[Repeat of output that was at the end of the preceding message #3 to 
SunManagers]




-------- 2nd Reply from Deborah Crocker --------
Subject: 	Re: stmsboot on t5220 with J4200
Date: 	Fri, 14 Aug 2009 18:16:11 -0500


Sorry for my hastily written note from before. I wanted to give you
something to look at to get going but was running between meetings. We
use a lot of fiber around here and only turn on multipathing for that,
not the mpt controller. So I'll admit, our situation is different.

With your SAS driver, you would use mpt.conf. I believe for all your
internal devices the port is "0". Only for the external dual port
card do you have "0" and "1".

It worries me that the drives on the LSI don't show up on different
controllers (with 12 on each). You noted that drives on the LSI (SAS)
card are all c4. It seems like there has to be a pair of controllers
there for mpxio to understand what to multipath over. For example, here
is of our cfgadm output (again, for fiber)

c2 fc-fabric connected configured unknown
c2::50060e8014436901 disk connected configured unknown
c4 fc-fabric connected configured unknown
c4::50060e8014436911 disk connected configured unknown

The disks, however, are in the system under mpxio as c5.

What kind of downtime window can you get? You may just have to test this.

Deb C.


-------- Original Message to James McPherson at Sun --------
Subject: 	SAS Multipathing configuration in Solaris 10
Date: 	Mon, 24 Aug 2009 14:24:00 -0400



While researching how to set up SAS multipathing, I've seen a number of 
your postings on various Solaris related lists. I hope you don't mind my 
asking you a question directly. Once I've resolved it, I will post a 
summary back to the Sun Managers list where I have asked for help and 
received essentially no response.

I have a T5220 with a J4200. I added a dual SAS card to the T5220 and 
connected everything according to page 53 of the J4200 System Overview. 
I also updated Solaris to u7 (Solaris 10 5/09 s10s_u7wos_08 SPARC) and 
applied the latest patches, checking to see that I had what was required 
for SAS multipathing (Live Update is sooo cool).

While I see various places where things have been updated to handle SAS 
multipathing, all the documentation I've been able to find talks about 
multipathing in terms of fiber channel. I've gotten a few hints that I 
should be looking at mpt.conf rather than fp.conf, and found one post 
that specifically referred to configuring SAS multipathing by editing 
mpt.conf.

I would like to exclude the internal drives and have the J4200 
multipathed. Then I will configure it with zpool and zfs, probably 9 
data drives, 2 parity drives and 1 hot spare (that gives me more safety 
than 2 raidz, and more data than 2 raidz2). They are all 300G 1500rpm SAS.

I gather that what I need is to replace 'mpxio-disable="yes"' in 
/kernel/drv/mpt.conf with:

name="mpt" parent="/pci@0/pci@0/pci@2/scsi" unit-address="0" 
mpxio-disable="yes";
mpxio-disable="no";

and then do a `stmsboot -u`.

Before doing this, the information from format is as it appears at the 
end of this email. For the internal drives, I've used hardware raid to 
mirror my boot drive, I have some plain drives for Amanda holding disk 
space, and the rest are configured as a zpool raidz. I'm slightly 
concerned that the 12 drives in the J4200 appear as 24 instances 
numbered sequentially without any distinction of 12 on one port and 12 
on another. I presume the multipath software will figure it out. Sorry 
if I seem too cautious. I don't like breaking production systems, and I 
don't have room for error in a situation where fixing it could get hairy.

Just a confirmation that all of this is correct would be cool.

A pointer to any updated documentation that covers these topics would be 
helpful as well. The documentation on fiber channel, for example, uses 
port instead of unit-address, and there is no explanation for the 
derivation of parent and port. In fact, the example showing the devices 
doesn't correspond to the details of the fp.conf. Apparently someone 
updated one without bothering to keep the information in sync throughout 
the example (see 
http://docs.sun.com/app/docs/doc/820-1931/gfpva?a=view). That makes it 
difficult to deduce (absent an explanation) how the parent and port are 
derived.

Thanks in advance. I appreciate your consideration.


Chris Hoogendyk



AVAILABLE DISK SELECTIONS:
0. c1t0d0 <LSILOGIC-LogicalVolume-3000 cyl 65533 alt 2 hd 16 sec 273>
/pci@0/pci@0/pci@2/scsi@0/sd@0,0

[etc. repeating format output from message #2 on stmsboot to SunManagers 
above]




-------- Reply from James McPherson at Sun --------
Subject: 	Re: SAS Multipathing configuration in Solaris 10
Date: 	Tue, 25 Aug 2009 11:30:41 +1000


Gday Chris,

On Mon, 24 Aug 2009 14:24:00 -0400
Chris Hoogendyk <hoogendyk@bio.umass.edu> wrote:
 > While researching how to set up SAS multipathing, I've seen a number of
 > your postings on various Solaris related lists. I hope you don't mind my
 > asking you a question directly. Once I've resolved it, I will post a
 > summary back to the Sun Managers list where I have asked for help and
 > received essentially no response.
 >
 > I have a T5220 with a J4200. I added a dual SAS card to the T5220 and
 > connected everything according to page 53 of the J4200 System Overview.
 > I also updated Solaris to u7 (Solaris 10 5/09 s10s_u7wos_08 SPARC) and
 > applied the latest patches, checking to see that I had what was required
 > for SAS multipathing (Live Update is sooo cool).

It is. But pkg image-update is even cooler, gotta say :-)

 > While I see various places where things have been updated to handle SAS
 > multipathing, all the documentation I've been able to find talks about
 > multipathing in terms of fiber channel. I've gotten a few hints that I
 > should be looking at mpt.conf rather than fp.conf, and found one post
 > that specifically referred to configuring SAS multipathing by editing
 > mpt.conf.

Yes, that's correct - SAS multipathing at present uses the mpt.conf 
file. It's never used fp.conf, which is Fibre Channel-specific.

The general case, where you turn on MPxIO for all attached devices which 
are capable, is to run "/usr/sbin/stmsboot -e" - which should definitely 
be all that you need.


 > I would like to exclude the internal drives and have the J4200
 > multipathed. Then I will configure it with zpool and zfs, probably 9
 > data drives, 2 parity drives and 1 hot spare (that gives me more safety
 > than 2 raidz, and more data than 2 raidz2). They are all 300G 1500rpm 
SAS.
 >
 > I gather that what I need is to replace 'mpxio-disable="yes"' in
 > /kernel/drv/mpt.conf with:
 >
 > name="mpt" parent="/pci@0/pci@0/pci@2/scsi" unit-address="0"
 > mpxio-disable="yes";
 > mpxio-disable="no";
 >
 > and then do a `stmsboot -u`.

Not quite - your mpt.conf is incorrect.

The example given in /kernel/drv/mpt.conf is

# You can also enable or disable MPxIO on a per HBA basis.
# Per HBA settings override the global setting for the specified HBAs.
# To disable MPxIO on a controller whose parent is /pci@7c0/pci@0/pci@9
# and the unit-address is "0" set:
# name="mpt" parent="/pci@7c0/pci@0/pci@9" unit-address="0" 
mpxio-disable="yes";

So what you would need to do (based on your info above) is to use

name="mpt" parent="/pci@0/pci@0/pci@2" unit-address="0" mpxio-disable="yes";
mpxio-disable="no";

You do, however, need to run "/usr/sbin/stmsboot -u"

 > Before doing this, the information from format is as it appears at the
 > end of this email. For the internal drives, I've used hardware raid to
 > mirror my boot drive, I have some plain drives for Amanda holding disk
 > space, and the rest are configured as a zpool raidz. I'm slightly
 > concerned that the 12 drives in the J4200 appear as 24 instances
 > numbered sequentially without any distinction of 12 on one port and 12
 > on another. I presume the multipath software will figure it out. Sorry
 > if I seem too cautious. I don't like breaking production systems, and I
 > don't have room for error in a situation where fixing it could get hairy.

Yes, MPxIO will figure it out. While you've got two physical connectors 
on the one card, they are not actually two separate controllers. The way 
that the LSISAS1068 chip is designed, those two physical connectors 
(PHYs) are each an x4 SAS lane. When you connect a cable to both of 
them, then they become a single x8 SAS lane instead.

You might it helpful to have a look at the info in my What Is A Guid? 
presentation - http://www.jmcp.homeunix.com/~jmcp/WhatIsAGuid.pdf


 > Just a confirmation that all of this is correct would be cool.
 >
 > A pointer to any updated documentation that covers these topics would be
 > helpful as well. The documentation on fiber channel, for example, uses
 > port instead of unit-address, and there is no explanation for the
 > derivation of parent and port. In fact, the example showing the devices
 > doesn't correspond to the details of the fp.conf. Apparently someone
 > updated one without bothering to keep the information in sync throughout
 > the example (see
 > http://docs.sun.com/app/docs/doc/820-1931/gfpva?a=view). That makes it
 > difficult to deduce (absent an explanation) how the parent and port are
 > derived.


That's really quite annoying about the docs.sun.com link, because I was 
heavily involved with re-writing that exact doc to bring it up to date 
for SAS MPxIO (and make the FC MPxIO text better).

Try the OpenSolaris version instead - 
http://docs.sun.com/app/docs/doc/820-3070?l=en

The parent property is covered in the driver.conf(4) manpage, but I do 
agree that the description needs an update, especially for the port 
property. If you'd care to log a bug against the doco, using 
http://bugs.opensolaris.org, that would be appreciated.


Best regards,
James C. McPherson
--
Senior Kernel Software Engineer, Solaris
Sun Microsystems
http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog



-------- Reply back to James McPherson at Sun --------
Subject: 	Re: SAS Multipathing configuration in Solaris 10
Date: 	Thu, 27 Aug 2009 17:18:24 -0400


Thank you! It worked beautifully. And, it is *really* fast.

I've been delayed a little in responding. We have the overload just
before school starts. Then on Tuesday we were notified at 4:45pm that
there would be a power outage in our server room for 8 hours beginning
at 6:30am the next morning. Apparently something in the new building was
not wired to code, and they were going to have to re-do it. That was an
opportunity to do the `stmsboot -u`, but then I didn't get to play with
it for another day.

Now I need to post a summary back to the sunmanagers list, have a look
at the opensolaris documentation, and post a bug report to encourage an
update of that.


Chris Hoogendyk
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers
Received on Fri Sep 18 17:32:10 2009

This archive was generated by hypermail 2.1.8 : Thu Mar 03 2016 - 06:44:14 EST