This is a very late summary which subsumes a couple of questions, but all relates to the same setup. In the end, it worked beautifully (really fast) and simply. The documentation and release notes just have to catch up and not be tangled in the forest of alternate products and options. --------------- The micro summary is that the hardware documentation clearly shows how to make dual SAS connections between a T5220 and the J4200, and a reconfigure reboot just brings it up. Format shows double entries for all the drives. So you get 24 drives where you have 12. If, at this point, you did `stmsboot -e`, it would coalesce those back down to 12 and assign longer GUID's for each drive (see below). Format shows those, and then a `zpool create . . . ` sets it all up and you're in business. I chose to set up a single raidz2 with one hot spare. So, 9 data drives, 2 parity drives, 1 hot spare. Performance tests were blazing. Since I wanted to exclude my internal drives from multipathing, I had to edit /kernel/drv/mpt.conf (not fp.conf, which is the only thing the documentation explains, and which is only for fiber channel), and then use `stmsboot -u`. Details below. Meanwhile, the StorageTek Common Array Manager (CAM) software is not required. I didn't use it at all, nor install it. Some respondents indicated it might be used for monitoring the health of the array. --------------- Thanks to Bryan Allen, Scott Lawson, Matt R. Clausen, Deborah Crocker, and, most especially, to James McPherson, Senior Kernel Software Engineer for Solaris at Sun. 8-) My original messages and their replies are all posted at the end. In the interest of helping anyone else who encounters the uncertainties I did, I'll summarize a bit more of the actual steps I went through with some comments on what's missing in the documentation. [Note: The part numbers in the following might be edu part numbers. We got the chassis & drives on special edu pricing.] I started with a T5220. Added a J4200 chassis (XTA4200R00A10DISK) filled out with SAS drives (XTA6SS1NJ-300G15K) and a second SAS controller (XTA-4200-IOM). Rack kit (XTA-4200-2URK-19U) is a pretty basic design. You could build one yourself like this one. Then put a dual SAS PCI card (SG-XPCIE8SAS-E-Z) in the T5200. Connected both HBA SAS ports on the T5220 to the SIM boards on the back of the J4200 according to the diagram on p.53 of the J4200 System Overview.pdf (downloaded all pdf docs from http://docs.sun.com/app/docs/prod/j4200.array~j4200?l=en#hic). Did a `touch /reconfigure` before taking the T5220 down to install the SAS HBA and connect everything. When it booted back up, it was just ready to fly, except I had to deal with the multipathing. Patches and updates for SAS multipathing. Solaris 10 should be at U6. I had mine at U7 (5/09) by the time I tackled this. The J4200 Release Notes, on page 4, indicates specific versions of CAM and patches that you must use if you configure multipathing and use the Common Array Manager (CAM) software. Just to put a plug in for Live Update, it was a breeze to get from U6 to U7 installing from a mounted ISO, and I only had a couple of minutes of down time when I restarted with the new boot environment. Getting the multipathing right was the real hangup, although it was simple once I was confident what to do. The documentation (http://docs.sun.com/app/docs/doc/820-1931/gfpva?a=view) refers to SAS several times, but the explicit example of "Per-Port" control of multipathing only mentions /kernel/drv/fp.conf and apparently assumes that you know that fp.conf refers to fiber channel and mpt.conf refers to SAS. The only references to mpt.conf are in the section on automatic configuration with `stmsboot -e`, where it lists several system configuration files that you do not have to edit (stmsboot does it for you). Also, the example listing of the entries in /dev/cfg don't match the details in the edit of fp.conf, so it is not clear how the parent name and port number are derived. Ultimately, in googling around, I stumbled on a Sun engineer who had answered some questions for somone, googled some more to find his email address, and sent a message directly to him. He answered my questions in detail (see below) and provided both some excellent information and reassurances. In looking back over the J4200 documentation, and specifically searching for mpt.conf, I found a one sentance reference (in the 92 page J4200 System Overview document) saying, "To enable multipathing on specific ports and disable multipathing on others, edit the /kernel/drv/mpt.conf file." No details or examples. So. My internal drives had entries like (full list down in one of the messages): /pci@0/pci@0/pci@2/scsi@0/sd@2,0 And my external drives (on the J4200) had entries like: /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@5,0 I wanted the SAS multipathing to ignore the internal drives and automatically take care of the external drives. So, I edited /kernel/drv/mpt.conf, adding the line to disable mpxio on the internal drives and setting the default to not disable mpxio. name="mpt" parent="/pci@0/pci@0/pci@2" unit-address="0" mpxio-disable="yes"; mpxio-disable="no"; Then I ran `stmsboot -u`. After that format showed the drives in the J4200 as just 12 with new long GUID's assigned. See http://www.jmcp.homeunix.com/~jmcp/WhatIsAGuid.pdf. That's an excellent document by James McPherson. Also, he pointed out that the parent property is documented in the driver.conf (4) man page, although, I found that to be a bit abstruse. Anyway, format showed me the GUID's, which I then could use in a zpool command. # zpool create -f jpool raidz2 c5t5000C5000F5D37E3d0 c5t5000C5000F5D529Bd0 c5t5000C5000F6CFBFFd0 \ > c5t5000C5000F520E4Fd0 c5t5000C5000F524B4Fd0 c5t5000C5000F523647d0 c5t5000C50004B4B3CFd0 c5t5000C50004B4B7C7d0 \ > c5t5000C50004B4FA1Fd0 c5t5000C50004B49B27d0 c5t5000C50004B49BABd0 spare c5t5000C50004B49973d0 It was interesting that with that many drives and the long GUID's, if I just tried to put it all on one line, I hit some sort of limit and the command wouldn't take any more. Don't know if that was a ksh limit, a term limit, or what. Anyway, using the "\" continuations worked just fine. That's the whole story. After that it's just a ZFS snap. ;-) -- --------------- Chris Hoogendyk - O__ ---- Systems Administrator c/ /'_ --- Biology & Geology Departments (*) \(*) -- 140 Morrill Science Center ~~~~~~~~~~ - University of Massachusetts, Amherst <hoogendyk@bio.umass.edu> --------------- Erdvs 4 Messages below Original Message #1 to SunManagers: Configuring J4200 with ZFS & not CAM? Reply from Bryan Allen Reply from Scott Lawson Reply from Matt R. Clausen Reply from Tomasz Ostrowski Original Message #2 to SunManagers: stmsboot on t5220 with J4200 Reply from Deborah Crocker Original Message #3 to SunManagers: Re: stmsboot on t5220 with J4200 Follow up to Deborah Crocker: Re: stmsboot on t5220 with J4200 2nd Reply from Deborah Crocker Original Message to James McPherson at Sun: SAS Multipathing configuration in Solaris 10 Reply from James McPherson at Sun Reply back to James McPherson at Sun -------- Original Message #1 to SunManagers -------- Subject: Configuring J4200 with ZFS & not CAM? Date: Fri, 24 Apr 2009 11:59:36 -0400 We've just gotten a J4200 to attach to a T5220. My intention is to configure it with ZFS, which I'm already using for the extra internal drives on the T5220 (I used hardware mirroring for the boot drive). My understanding of the J4200 is that it is supposed to be a JBOD, almost plug and play. However, when I opened the box, it came with the manual and CD for CAM - the StorageTek Common Array Manager software. I am still waiting for some drives, so I'm not setting it up yet. I have a SAS HBA and will be setting it up with SAS drives. If it is just a matter of plugging it all in and doing a reconfigure reboot, whereupon the drives show up in format and can then be configured with zpool and zfs commands, then I'm all set. Can anyone tell me the relationship/difference between CAM and ZFS? Am I going to need CAM at all? Or is it sort of like you don't need raidctl if you are using zfs? Thanks, Chris Hoogendyk -------- Reply from Bryan Allen -------- Subject: Re: Configuring J4200 with ZFS & not CAM? Date: Fri, 24 Apr 2009 13:12:13 -0400 I have a J4200 attached to an X4100 M2 using the LSI SAS card from Sun. No issues. Very happy with it. If you are using a RAID card, you'll need to configure CAM/RAID/et al. If you are using a plain HBA, you don't need to mess with CAM. Note that the Adaptec SAS RAID card you get from Sun cannot do JBOD. It cannot be flashed with Adaptec's firmware (which supports JBOD) and Sun support wouldn't tell me if an update was forthcoming. You also cannot configure each LUN as RAID0, thus exporting each drive to ZFS individually. You would have to create two-disk RAID0s and export the pairs. I returned that card and got the LSI HBA. There was too much complexity for no reason in the RAID configuration given how I wanted to use ZFS. You don't need to do a reconfigure reboot, either: devfsadm -C cfgadm -al zpool create ... -- bda cyberpunk is dead. long live cyberpunk. -------- Reply from Scott Lawson -------- Subject: Re: Configuring J4200 with ZFS & not CAM? Date: Sat, 25 Apr 2009 09:19:00 +1200 Chris, It should just be a matter of inserting the SAS controller and doing a reconfigure boot. I have a T5220 with a J4500 attached in the same way. Works fine for me and was simple to set up. I am not sure why they included CAM at all, it may be that it can be used as an expansion chassis for one of the entry level Storagetek arrays. CAM is used for managing the Storagetek 25xx and 6xxx series hardware RAID arrays. The J4200 is also used with the new open storage appliances but I can't see why it would need CAM.. I don't recall receiving CAM with my J4500. I was also one of the beta testers for the J4500. 'raidctl' is part of the OS and is used for internal controllers. There is also a host hardware RAID controller available for the J series arrays which is made by Adaptec. But if you use ZFS then you really have no need for it. It's more for people that use these JBOD's with Windows or Linux. /Scott. -------- Reply from Matt R. Clausen -------- Subject: RE: Configuring J4200 with ZFS & not CAM? Date: Tue, 28 Apr 2009 17:03:28 -0500 CAM is strictly used for monitoring the J4X00 series arrays. They are nothing more then a JBOD so there's nothing for CAM to configure (you don't even have to install it though it's a good thing to have because it can monitor the array health). Once you get it all hooked up, you will see all the individual drives appear to the operating system from which you can then put them into your respective ZFS pools/volumes/etc. Hope this helps. -------- Reply from Tomasz Ostrowski -------- Subject: Configuring J4200 with ZFS & not CAM? Date: Wed, 24 Jun 2009 08:14:03 +0200 Hello Chris, Did you manage to configure the disk array withou CAM? I'm starting with iscsi tech. and now i wonder how to assing an ip addres tho the SIM controlem on the disk array? Or this is wrong step? Do you have any guide lines? Best Regards Tomek Tomasz Ostrowski NSN Preconfiguration Engineer -------- Original Message #2 to SunManagers -------- Subject: stmsboot on t5220 with J4200 Date: Mon, 03 Aug 2009 17:39:06 -0400 I have a T5220 with an attached J4200. I added a dual SAS PCI card to the T5220 and added a second SAS controller to the J4200. It has 12 300G SAS drives in it. On the T5220, I used hardware raid to mirror the boot drive, have one plain drive, and configured 4 drives with zfs raidz. Those are all in use. When I check format, I see all the internal drives and 2 instances of all of the drives in the J4200 (see at end). I want to configure the J4200 with zpool. As I understand it, I need to do `stmsboot -e` first to set up multiplathing and reboot. Then I should see the renamed drive devices in format and can just feed them in to a zpool create. My problem is that I have never done this before, and I'm afraid of messing up my existing setup on the T5220. What I get is # stmsboot -L stmsboot: MPxIO is not enabled # stmsboot -e WARNING: stmsboot operates on each supported multipath-capable controller detected in a host. In your system, these controllers are /pci@0/pci@0/pci@2/scsi@0 /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0 If you do NOT wish to operate on these controllers, please quit stmsboot and re-invoke with -D { fp | mpt } to specify which controllers you wish to modify your multipathing configuration for. Do you wish to continue? [y/n] (default: y) n # It's showing me both the internal scsi (SAS) systems and the external SAS, and I don't see any way to restrict it to just do the external system. The `stmsboot -D mpt -e` option doesn't seem to offer anything different, since it's all SAS. But, since I am telling it to use the mpt driver, it doesn't do the same sort of autodiscovery informing me what it is going to do. It just does # stmsboot -D mpt -e WARNING: This operation will require a reboot. Do you want to continue ? [y/n] (default: y) n # I can imagine that it might look at all the drives and see that I have duplicate instances on all the external ones and multipath them, leaving the others alone; but, I'm just guessing. And, search as I might, I can't find any examples that explain it in enough detail or that show what happens if you say no and then configure something differently (or even continue the example out through the format showing all the internal and external drives without excerpting the list). If anyone who has actually done this could give me some encouragement and/or guidance, I would certainly appreciate it. TIA Chris Hoogendyk AVAILABLE DISK SELECTIONS: 0. c1t0d0 <LSILOGIC-LogicalVolume-3000 cyl 65533 alt 2 hd 16 sec 273> /pci@0/pci@0/pci@2/scsi@0/sd@0,0 1. c1t2d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848> /pci@0/pci@0/pci@2/scsi@0/sd@2,0 2. c1t4d0 <SEAGATE-ST914602SSUN146G-0603-136.73GB> /pci@0/pci@0/pci@2/scsi@0/sd@4,0 3. c1t5d0 <SEAGATE-ST914602SSUN146G-0603-136.73GB> /pci@0/pci@0/pci@2/scsi@0/sd@5,0 4. c1t6d0 <SEAGATE-ST914602SSUN146G-0603-136.73GB> /pci@0/pci@0/pci@2/scsi@0/sd@6,0 5. c1t7d0 <SEAGATE-ST914602SSUN146G-0603-136.73GB> /pci@0/pci@0/pci@2/scsi@0/sd@7,0 6. c4t0d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625> /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@0,0 7. c4t1d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625> /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@1,0 8. c4t2d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625> /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@2,0 9. c4t3d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625> /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@3,0 10. c4t4d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625> /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@4,0 11. c4t5d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625> /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@5,0 12. c4t6d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625> /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@6,0 13. c4t7d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625> /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@7,0 14. c4t8d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625> /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@8,0 15. c4t9d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625> /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@9,0 16. c4t10d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625> /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@a,0 17. c4t11d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625> /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@b,0 18. c4t13d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625> /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@d,0 19. c4t14d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625> /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@e,0 20. c4t15d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625> /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@f,0 21. c4t16d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625> /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@10,0 22. c4t17d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625> /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@11,0 23. c4t18d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625> /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@12,0 24. c4t19d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625> /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@13,0 25. c4t20d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625> /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@14,0 26. c4t21d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625> /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@15,0 27. c4t22d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625> /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@16,0 28. c4t23d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625> /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@17,0 29. c4t24d0 <SUN300G cyl 46873 alt 2 hd 20 sec 625> /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@18,0 -------- Reply from Deborah Crocker -------- Subject: Re: stmsboot on t5220 with J4200 Date: Tue, 04 Aug 2009 09:08:02 -0500 Review /kernel/drv/fp.conf. You can exempt controllers from multipathing. For instance, on our T5240 we have: mpxio-disable="no"; [This turns multipathing on] name="fp" parent="/pci@500/pci@0/pci@c/SUNW,qlc@0" port=0 mpxio-disable="yes"; You'll see more comments on configuring this in the fp.conf file Deb C. -------- Original Message #3 to SunManagers -------- Subject: Re: stmsboot on t5220 with J4200 Date: Wed, 12 Aug 2009 17:32:20 -0400 Sorry, I'm getting a bit frustrated with this. Production system. Can't play around. Has to be right. (I got one response from Deborah Crocker indicating that I need to edit the fp.conf, but then ... ) So, from http://docs.sun.com/app/docs/doc/820-1931/gfpva?a=view , halfway down, it describes how to "Configure Multipathing by Port." It says to use `parent="parent name" port=port number`, but the explicit example it gives does not match with the /dev/cfg entries that it starts the explanation with. So, it's not clear how to derive the parent name and the port number, and they don't explain it. I'm guessing that in my case `parent="/pci@0/pci@0/pci@2/scsi@0" port=0` would designate my internal drives. I also presume that since they have folded SAS in with FC, that the `name="fp"` designation would apply for the internal drives. So, I could exclude them and be ready to go, with: name="fp" parent="/pci@0/pci@0/pci@2/scsi@0" port=0 mpxio-disable="no"; I've put some output at the very end of this that I extracted from format, /dev/cfg, and prtdiag for reference. again, TIA. Chris Hoogendyk # format /pci@0/pci@0/pci@2/scsi@0/sd@0,0 /pci@0/pci@0/pci@2/scsi@0/sd@2,0 /pci@0/pci@0/pci@2/scsi@0/sd@4,0 /pci@0/pci@0/pci@2/scsi@0/sd@5,0 /pci@0/pci@0/pci@2/scsi@0/sd@6,0 /pci@0/pci@0/pci@2/scsi@0/sd@7,0 /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@0,0 /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@1,0 /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@2,0 /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@3,0 /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@4,0 /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@5,0 /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@6,0 /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@7,0 /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@8,0 /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@9,0 /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@a,0 /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@b,0 /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@d,0 /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@e,0 /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@f,0 /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@10,0 /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@11,0 /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@12,0 /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@13,0 /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@14,0 /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@15,0 /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@16,0 /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@17,0 /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0/sd@18,0 # ls -l /dev/cfg c1 -> ../../devices/pci@0/pci@0/pci@2/scsi@0:scsi <-- internal drives c2 -> ../../devices/pci@0/pci@0/pci@9/pci@0/scsi@8:scsi c3 -> ../../devices/pci@0/pci@0/pci@9/pci@0/scsi@8,1:scsi <-- tape library? c4 -> ../../devices/pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0:scsi <-- J4200 # prtdiag -v ================================ IO Devices ================================ Slot + Bus Name + Status Type Path ---------------------------------------------------------------------------- MB/NET0 PCIE network-pciex8086,105e /pci@0/pci@0/pci@1/pci@0/pci@2/network@0 MB/NET1 PCIE network-pciex8086,105e /pci@0/pci@0/pci@1/pci@0/pci@2/network@0,1 MB/NET2 PCIE network-pciex8086,105e /pci@0/pci@0/pci@1/pci@0/pci@3/network@0 MB/NET3 PCIE network-pciex8086,105e /pci@0/pci@0/pci@1/pci@0/pci@3/network@0,1 MB/SASHBA PCIE scsi-pciex1000,58 /pci@0/pci@0/pci@2/scsi@0 <-- internal drives MB/RISER1/PCIE1 PCIE LSILogic,sas-pciex1000,58 /pci@0/pci@0/pci@8/pci@0/pci@1/LSILogic,sas@0 <-- J4200 MB/RISER0/PCIE0 PCIE network-pciex8086,105e /pci@0/pci@0/pci@8/pci@0/pci@9/network@0 MB/RISER0/PCIE0 PCIE network-pciex8086,105e /pci@0/pci@0/pci@8/pci@0/pci@9/network@0,1 MB PCIX usb-pciclass,0c0310 /pci@0/pci@0/pci@1/pci@0/pci@1/pci@0/usb@0 MB PCIX usb-pciclass,0c0310 /pci@0/pci@0/pci@1/pci@0/pci@1/pci@0/usb@0,1 MB PCIX usb-pciclass,0c0320 /pci@0/pci@0/pci@1/pci@0/pci@1/pci@0/usb@0,2 MB/RISER2/PCIE2 PCIE scsi-pci1000,30 /pci@0/pci@0/pci@9/pci@0/scsi@8 MB/RISER2/PCIE2 PCIE scsi-pci1000,30 /pci@0/pci@0/pci@9/pci@0/scsi@8,1 <-- tape library -------- Follow up to Deborah Crocker -------- Subject: Re: stmsboot on t5220 with J4200 Date: Fri, 14 Aug 2009 13:26:59 -0400 Just a quick followup -- why is it fp.conf and 'name="fp"' rather than mpt.conf? If I do a stmsboot -e (and then say no to stop it), it lists my controllers, but doesn't indicate fp or mpt. If I do stmsboot -D mpt -e, it doesn't list them first, but asks directly if I want to continue. I said no. So I don't know what it would have come up with. Same with -D fp. Also, how do you know port=0? I don't know if you saw my followup on sunmanagers, but the examples in the Sun documents on multiplathing don't follow through from one item to the next. Their listing of /dev/cfg does not match up with their example of editing fp.conf, and there's no explanation of how to derive those items. So, I'm not sure/confident that the following will be correct for my system: mpxio-disable="no"; name="fp" parent="/pci@0/pci@0/pci@2/scsi@0" port=0 mpxio-disable="yes"; Sorry to email you directly, but no one else has responded to either of my postings on sunmanagers. I pasted some system output at the bottom. Thank you very much. Chris Hoogendyk [Repeat of output that was at the end of the preceding message #3 to SunManagers] -------- 2nd Reply from Deborah Crocker -------- Subject: Re: stmsboot on t5220 with J4200 Date: Fri, 14 Aug 2009 18:16:11 -0500 Sorry for my hastily written note from before. I wanted to give you something to look at to get going but was running between meetings. We use a lot of fiber around here and only turn on multipathing for that, not the mpt controller. So I'll admit, our situation is different. With your SAS driver, you would use mpt.conf. I believe for all your internal devices the port is "0". Only for the external dual port card do you have "0" and "1". It worries me that the drives on the LSI don't show up on different controllers (with 12 on each). You noted that drives on the LSI (SAS) card are all c4. It seems like there has to be a pair of controllers there for mpxio to understand what to multipath over. For example, here is of our cfgadm output (again, for fiber) c2 fc-fabric connected configured unknown c2::50060e8014436901 disk connected configured unknown c4 fc-fabric connected configured unknown c4::50060e8014436911 disk connected configured unknown The disks, however, are in the system under mpxio as c5. What kind of downtime window can you get? You may just have to test this. Deb C. -------- Original Message to James McPherson at Sun -------- Subject: SAS Multipathing configuration in Solaris 10 Date: Mon, 24 Aug 2009 14:24:00 -0400 While researching how to set up SAS multipathing, I've seen a number of your postings on various Solaris related lists. I hope you don't mind my asking you a question directly. Once I've resolved it, I will post a summary back to the Sun Managers list where I have asked for help and received essentially no response. I have a T5220 with a J4200. I added a dual SAS card to the T5220 and connected everything according to page 53 of the J4200 System Overview. I also updated Solaris to u7 (Solaris 10 5/09 s10s_u7wos_08 SPARC) and applied the latest patches, checking to see that I had what was required for SAS multipathing (Live Update is sooo cool). While I see various places where things have been updated to handle SAS multipathing, all the documentation I've been able to find talks about multipathing in terms of fiber channel. I've gotten a few hints that I should be looking at mpt.conf rather than fp.conf, and found one post that specifically referred to configuring SAS multipathing by editing mpt.conf. I would like to exclude the internal drives and have the J4200 multipathed. Then I will configure it with zpool and zfs, probably 9 data drives, 2 parity drives and 1 hot spare (that gives me more safety than 2 raidz, and more data than 2 raidz2). They are all 300G 1500rpm SAS. I gather that what I need is to replace 'mpxio-disable="yes"' in /kernel/drv/mpt.conf with: name="mpt" parent="/pci@0/pci@0/pci@2/scsi" unit-address="0" mpxio-disable="yes"; mpxio-disable="no"; and then do a `stmsboot -u`. Before doing this, the information from format is as it appears at the end of this email. For the internal drives, I've used hardware raid to mirror my boot drive, I have some plain drives for Amanda holding disk space, and the rest are configured as a zpool raidz. I'm slightly concerned that the 12 drives in the J4200 appear as 24 instances numbered sequentially without any distinction of 12 on one port and 12 on another. I presume the multipath software will figure it out. Sorry if I seem too cautious. I don't like breaking production systems, and I don't have room for error in a situation where fixing it could get hairy. Just a confirmation that all of this is correct would be cool. A pointer to any updated documentation that covers these topics would be helpful as well. The documentation on fiber channel, for example, uses port instead of unit-address, and there is no explanation for the derivation of parent and port. In fact, the example showing the devices doesn't correspond to the details of the fp.conf. Apparently someone updated one without bothering to keep the information in sync throughout the example (see http://docs.sun.com/app/docs/doc/820-1931/gfpva?a=view). That makes it difficult to deduce (absent an explanation) how the parent and port are derived. Thanks in advance. I appreciate your consideration. Chris Hoogendyk AVAILABLE DISK SELECTIONS: 0. c1t0d0 <LSILOGIC-LogicalVolume-3000 cyl 65533 alt 2 hd 16 sec 273> /pci@0/pci@0/pci@2/scsi@0/sd@0,0 [etc. repeating format output from message #2 on stmsboot to SunManagers above] -------- Reply from James McPherson at Sun -------- Subject: Re: SAS Multipathing configuration in Solaris 10 Date: Tue, 25 Aug 2009 11:30:41 +1000 Gday Chris, On Mon, 24 Aug 2009 14:24:00 -0400 Chris Hoogendyk <hoogendyk@bio.umass.edu> wrote: > While researching how to set up SAS multipathing, I've seen a number of > your postings on various Solaris related lists. I hope you don't mind my > asking you a question directly. Once I've resolved it, I will post a > summary back to the Sun Managers list where I have asked for help and > received essentially no response. > > I have a T5220 with a J4200. I added a dual SAS card to the T5220 and > connected everything according to page 53 of the J4200 System Overview. > I also updated Solaris to u7 (Solaris 10 5/09 s10s_u7wos_08 SPARC) and > applied the latest patches, checking to see that I had what was required > for SAS multipathing (Live Update is sooo cool). It is. But pkg image-update is even cooler, gotta say :-) > While I see various places where things have been updated to handle SAS > multipathing, all the documentation I've been able to find talks about > multipathing in terms of fiber channel. I've gotten a few hints that I > should be looking at mpt.conf rather than fp.conf, and found one post > that specifically referred to configuring SAS multipathing by editing > mpt.conf. Yes, that's correct - SAS multipathing at present uses the mpt.conf file. It's never used fp.conf, which is Fibre Channel-specific. The general case, where you turn on MPxIO for all attached devices which are capable, is to run "/usr/sbin/stmsboot -e" - which should definitely be all that you need. > I would like to exclude the internal drives and have the J4200 > multipathed. Then I will configure it with zpool and zfs, probably 9 > data drives, 2 parity drives and 1 hot spare (that gives me more safety > than 2 raidz, and more data than 2 raidz2). They are all 300G 1500rpm SAS. > > I gather that what I need is to replace 'mpxio-disable="yes"' in > /kernel/drv/mpt.conf with: > > name="mpt" parent="/pci@0/pci@0/pci@2/scsi" unit-address="0" > mpxio-disable="yes"; > mpxio-disable="no"; > > and then do a `stmsboot -u`. Not quite - your mpt.conf is incorrect. The example given in /kernel/drv/mpt.conf is # You can also enable or disable MPxIO on a per HBA basis. # Per HBA settings override the global setting for the specified HBAs. # To disable MPxIO on a controller whose parent is /pci@7c0/pci@0/pci@9 # and the unit-address is "0" set: # name="mpt" parent="/pci@7c0/pci@0/pci@9" unit-address="0" mpxio-disable="yes"; So what you would need to do (based on your info above) is to use name="mpt" parent="/pci@0/pci@0/pci@2" unit-address="0" mpxio-disable="yes"; mpxio-disable="no"; You do, however, need to run "/usr/sbin/stmsboot -u" > Before doing this, the information from format is as it appears at the > end of this email. For the internal drives, I've used hardware raid to > mirror my boot drive, I have some plain drives for Amanda holding disk > space, and the rest are configured as a zpool raidz. I'm slightly > concerned that the 12 drives in the J4200 appear as 24 instances > numbered sequentially without any distinction of 12 on one port and 12 > on another. I presume the multipath software will figure it out. Sorry > if I seem too cautious. I don't like breaking production systems, and I > don't have room for error in a situation where fixing it could get hairy. Yes, MPxIO will figure it out. While you've got two physical connectors on the one card, they are not actually two separate controllers. The way that the LSISAS1068 chip is designed, those two physical connectors (PHYs) are each an x4 SAS lane. When you connect a cable to both of them, then they become a single x8 SAS lane instead. You might it helpful to have a look at the info in my What Is A Guid? presentation - http://www.jmcp.homeunix.com/~jmcp/WhatIsAGuid.pdf > Just a confirmation that all of this is correct would be cool. > > A pointer to any updated documentation that covers these topics would be > helpful as well. The documentation on fiber channel, for example, uses > port instead of unit-address, and there is no explanation for the > derivation of parent and port. In fact, the example showing the devices > doesn't correspond to the details of the fp.conf. Apparently someone > updated one without bothering to keep the information in sync throughout > the example (see > http://docs.sun.com/app/docs/doc/820-1931/gfpva?a=view). That makes it > difficult to deduce (absent an explanation) how the parent and port are > derived. That's really quite annoying about the docs.sun.com link, because I was heavily involved with re-writing that exact doc to bring it up to date for SAS MPxIO (and make the FC MPxIO text better). Try the OpenSolaris version instead - http://docs.sun.com/app/docs/doc/820-3070?l=en The parent property is covered in the driver.conf(4) manpage, but I do agree that the description needs an update, especially for the port property. If you'd care to log a bug against the doco, using http://bugs.opensolaris.org, that would be appreciated. Best regards, James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog -------- Reply back to James McPherson at Sun -------- Subject: Re: SAS Multipathing configuration in Solaris 10 Date: Thu, 27 Aug 2009 17:18:24 -0400 Thank you! It worked beautifully. And, it is *really* fast. I've been delayed a little in responding. We have the overload just before school starts. Then on Tuesday we were notified at 4:45pm that there would be a power outage in our server room for 8 hours beginning at 6:30am the next morning. Apparently something in the new building was not wired to code, and they were going to have to re-do it. That was an opportunity to do the `stmsboot -u`, but then I didn't get to play with it for another day. Now I need to post a summary back to the sunmanagers list, have a look at the opensolaris documentation, and post a bug report to encourage an update of that. Chris Hoogendyk _______________________________________________ sunmanagers mailing list sunmanagers@sunmanagers.org http://www.sunmanagers.org/mailman/listinfo/sunmanagersReceived on Fri Sep 18 17:32:10 2009
This archive was generated by hypermail 2.1.8 : Thu Mar 03 2016 - 06:44:14 EST