Please see the OpenSolaris power FAQ [[http://hub.opensolaris.org/bin/view/Community+Group+pm/FAQ]]

Yeah, I've seen that. I wish it said more than "More information on updated Storage Power Management will come later." Practical info is what I'm missing here...

Anyway, CPU power management seems to be working just fine. All 4 cores of my L3426 are at 1.2GHz when the system is idle.

What I can't seem to get working is the storage aspect. I've read the page on power.conf and have typed in (as per the paths given by 'format'):

device-thresholds /scsi_vhci/disk@g50014ee2592e3e55 5m
device-thresholds /scsi_vhci/disk@g50014ee2ae83efdd 5m
device-thresholds /scsi_vhci/disk@g50014ee2ae83ec05 5m
device-thresholds /scsi_vhci/disk@g50014ee2ae8421fb 5m
device-thresholds /scsi_vhci/disk@g50014ee2ae83ec83 5m

... but after 5 minutes the watt reading on my Kill-A-Watt doesn't drop like it does under CentOS (54W) or Windows (47W). I'd like to get this figured out before I order another (10) 2TB disks (there are 5 now, will be 15 total, plus a couple SSDs once Intel releases their next-gen SLC models) and take it home for the wife to see.

For reference, if it matters, the system is as follows:

• SuperMicro X8SIL-F
• Intel Xeon L3426 1.86GHz Quad-Core w/ HT, 45W
• 4GB Kingston ECC DDR3-1333 (2x 2GB)
• Intel PRO/1000 ET Dual-Port (82576EB controller)
• (2) LSI 9211-8i 6.0Gb/s SAS Controllers
• (5) Western Digital 2TB Green 64MB Cache ("EARS" model with 4K sectors)

BTW, as an aside, the built-in NICs on that motherboard, 2x Intel 82574L, don't work correctly under OpenSolaris or Nexenta, a random one (though always the same one on any given board) shows as misconfigured 98% of the time when you boot and if you try to use the other, the system will lock up after a few minutes of network transfer; I returned two boards before I figured out this wasn't a defect and that it worked under CentOS & Windows Server 2008 R2. Should I file a bug report somewhere on that?

Hmm

I have set "autopm enable" in /etc/power.conf and it spins down harddrives after a few minutes - never counted how many though.

( a 6 drive RAIDZ2 which takes forever to spinup again since it starts each disk individually :) )

That did it. Once I set "autopm enable" rather than "autopm default" the lines I wrote above took effect. I'm now sitting at 49W. Makes me wonder what Windows is doing to get those extra 2W... Thanks for you help.

hello

there is a thread about nic problems with 2xintel nic + sas controller at www.opensolaris.org forum

maybee it is the same problem with the sas controller

gea

Guenther Alka wrote: duplicated post...

I just pulled both SAS controllers and I've still got the same problem. Only one port shows up on this board and if I try and use it the system locks up.

I don't actually think this is the same issue. I've tried both the Intel PRO/1000 PT Dual-Port Server Adapter (82571EB, using e1000g) and the newer Intel PRO/1000 ET Dual-Port Server Adapter (82576EB, using igb) and both work fine. I'm more inclined to believe that this is a problem with the 3420 PCH (or rather, something SuperMicro has done to it, more later). The two PCI-E 2.0 x8 ports used by the SAS cards use all 16 lanes that are embedded in the CPU. That means that the (2) x1 lanes used by the (2) 82574L NICs are attached to the PCH.

All three boards that I've had show the following in 'fmadm faulty -a"

--------------- ------------------------------------  -------------- ---------
TIME            EVENT-ID                              MSG-ID         SEVERITY
--------------- ------------------------------------  -------------- ---------
Jan 29 13:36:54 c8057977-5214-e497-b950-c691880de997  PCIEX-8000-0A  Critical
Host        : nexenta
Platform    : X8SIL     Chassis_id  : 0123456789
Product_sn  :
Fault class : fault.io.pciex.device-interr
Affects     : dev:////pci@0,0/pci8086,3b4a@1c,4/pci15d9,605@0
faulted and taken out of service
FRU         : "MB" (hc://:product-id=X8SIL:server-id=nexenta:chassis-id=0123456789/motherboard=0)
faulty
Description : A problem was detected for a PCIEX device.
Refer to http://sun.com/msg/PCIEX-8000-0A for more information.
Response    : One or more device instances may be disabled
Impact      : Loss of services provided by the device instances associated with
this fault
Action      : Schedule a repair procedure to replace the affected device.  Use
fmadm faulty to identify the device or contact Sun for support.

... and pci8086,3b4a is part of the 3400-series chipset (specifically, "5 Series/3400 Series Chipset PCI Express Root Port 5") and pci15d9,605 is something belonging to SuperMicro. The 3420 PCH is supposed to come with a 82578DM integrated into the PCH and an 82574L attached to a PCI-E x1 lane. My guess is that this issue is related to something funky SuperMicro did to cram (2) 82574L NICs in there...

For reference, this is not just an issue with OpenSolaris & Nexenta. The HCL for this board on SuperMicro's site is very specific about what operating systems this board works with. CentOS/RHEL are covered, as are newer versions of Windows Server. If you try Fedora (which is listed on the HCL but not checked off as compatible) you get the same issues with the NICs as you get under Nexenta: one port typically shows up and if you try to use it the system locks up. Under CentOS 5.4 x64 & Windows Server 2008 R2 x64 the NICs work flawlessly.

Jason, have you reported this to Supermicro? They usually take very good care of bug/error reports...

BTW, I am experiencing the same problems as you are.

No, I haven't. I kind of figured they'd ignore me since I'm using an unsupported OS (actually, it's not only unsupported, it's not even on the chart). I'll give it a shot though. If you report the same issues they might take it more seriously since it would be more than one user with the same issue trying to do the same unsupported thing.

BTW, in the short-term, regarding your issues, both the PRO/1000 PT & ET cards work fine. I'd be perfectly satisfied with a solution that allowed me to disable the onboard NICs so that they didn't show up to the OS (and maybe it would save me a watt or two) as both the 82571EB and 82576EB are better performers than the 82574L If you're going for low power consumption or are looking for the absolute best performance, regardless of price (ET NICs are about $20 more for the cheapest vendor), pickup the ET rather than the PT. It's got both lower power usage (2.9W vs. 4.95W, doesn't even need a heat sink) and more Rx/Tx queues per port.

PT - http://www.intel.com/Assets/PDF/prodbrief/pro1000ptdualportserveradapter.pdf

ET - http://www.intel.com/Assets/PDF/prodbrief/320116.pdf

I'll support you and will file my own report. Hopefully we can gather enough attention to get it fixed in the 2009.03 release...

Thanks a lot for the Intel info. Will try to find the ET-model ASAP! Lower wattage is definitely my way!

Actually, Amazon has them (that's where I got mine). They're one of the few vendors that does. Dell told me it would be a 2-3 week wait.

http://www.amazon.com/dp/B001KU90BO/

Another thing to keep in mind when trying to build a low-power system is that not all hardware is created equal. RAM is a great example of this and most people don't think twice about it. They just grab the cheapest name-brand that works in their board at whatever capacity they think they need.

The SuperMicro X8SIL-F with a Xeon CPU can use either Unbuffered ECC or Registered ECC DDR3 RAM, running at speeds up to 1333MHz. Most people know that Registered RAM has a higher power draw than Unbuffered, but did you know that it's close to double for a 2GB stick of Kingston (3.480W vs 2.025W)? Beyond that, and this is really going to blow your mind, did you know that capacity has basically zero to do with relative power consumption? Using Kingston RAM again, based on their spec sheets, a 1GB stick of Unbuffered ECC DDR3-1333 draws 1.215W, a 2GB stick 2.025W, but a 4GB stick only 1.552W. That means that with Unbuffered ECC you can have 8GB with 25% lower power consumption than 4GB and, even more dramatic, you can have 16GB (4x4GB) of Unbuffered ECC with less power consumption than 4GB (2x2GB) of Registered ECC.

I know all that sounds like nitpicking, but when you're trying to build a low-power system, 1W here, 1.5W there, etc. can really add up. Also, in the above case, as well as with the Intel ET vs PT NICs, low power doesn't mean low performance as you can dramatically increase performance by going with 2x 4GB rather than 2x 2GB and lower power consumption at the same time.

Thanks a lot for all the info. Very interesting!

I have filed a bug report to Supermicro. Let's see what they can do about it!

I ended up with an Intel PRO 1000 GT adapter because it was cheap and easily available here in Sweden. Works like a charm!

BTW, what kind of power supply are you using? I took an old 500W but it seems a bit to powerful and surely wastes energy (I'm idling at around 61W right now with five Samsung 1,5TB disks).

I'm using a Seasonic X750. The wattage rating is overkill but it was selected because it's modular (no extra cables obstructing airflow), because it's got a single 12V rail (which means that drive cages can be split-plane powered by multiple cables), and because it's 80 Plus Gold certified (87% efficient or higher at any load between 20-100%). As an added benefit, the fan turns completely off unless it's needed (which, in my case, it's typically not) so it's also very quiet.

As to power consumption, heh, I didn't quite hit my goal of 50W at maximum power savings (CPU throttled back, drives in standby, etc.). Then again, that was a completely arbitrary number, and I came REAL close.

Here's what I ended up with:

• SuperMicro X8SIL-F
• Intel Xeon L3426 1.86GHz Quad-Core w/ HT, 45W
• 8GB Kingston ECC DDR3-1333 (2x 4GB)
• Intel PRO/1000 ET Dual-Port (82576EB controller)
• (2) LSI 9211-8i 6.0Gb/s SAS Controllers
• (15) Western Digital 2TB Green 64MB Cache ("EARS" model with 4K sectors)
• (2) 16GB PicoUSB thumb drives attached to motherboard header (not the USB socket on the board)
• Seasonic X750 PSU
• (3) Nexus 92mm Silent PWM fans (SuperMicro drive cages, stock fans were horrendously noisy)
• (1) Xilence Red Wing 120mm PWM fan (rear of case)
• Thermaltake Silent 1156 CPU cooler w/ 92mm PWM fan
• All PWM fans set in system BIOS to 50%, system is basically silent.

... and the numbers you've been waiting for, drum roll please...

• 107W w/ all 15 drives idle
• 55W w/ all 15 drives in standby

As an FYI, if I unplug the 3 drive cage fans I drop to 50W. The system can actually be used without them as long as you don't constantly thrash it. In my office (with an ambient of around 76F) I can run dd and write across an array of all 15 for about 90 minutes before they start to overheat and alarms go off. Under "normal" usage they'd be fine without fans.

Oscar Forsstrom wrote:

Thanks a lot for all the info. Very interesting!

I have filed a bug report to Supermicro. Let's see what they can do about it!

I ended up with an Intel PRO 1000 GT adapter because it was cheap and easily available here in Sweden. Works like a charm!

BTW, what kind of power supply are you using? I took an old 500W but it seems a bit to powerful and surely wastes energy (I'm idling at around 61W right now with five Samsung 1,5TB disks).

Any word on your bug report with supermicro ? I recently got exactly the same board with the same problem. (thus making nexenta unusable at the moment)

Would love to here how it turned out.

I had similar problems with a X8SIL-F Board w. BIOS 1.0.c and its NICs under Windows 2008 R2 and Debian lenny - so indeed it's not Nexenta/OpenSolaris related only.

The BIOS is rather badly set, my resolution was that you need to a) disable Watchdog functionality (otherwise all 5-10' reboot with Windows and Linux) and b) most speficially disable: "Active State Power Management". Since I disabled this option, the Server, a SuperMicro SC826 pushed more than 300GB over CIFS without hickups - so it looks stable.

Besides I recommend you to download very latest IPMI Firmware 1.37 only available via Thomas Krenn Germany (until now not from SuperMicro) which also solves lots of issues with Remote console and newest Sun (Windows) JVMs: http://www.thomas-krenn.com/en/download.html

Hope sharing this helps others as we had quite a lot of hassles with this board (changed it 3 times with always same symptoms)

Ah yes, and if you get stable BIOS Settings, save/dump them with SMCMOS.exe from SuperMicro :-)

Mat Simon wrote:

I had similar problems with a X8SIL-F Board w. BIOS 1.0.c and its NICs under Windows 2008 R2 and Debian lenny - so indeed it's not Nexenta/OpenSolaris related only.

The BIOS is rather badly set, my resolution was that you need to a) disable Watchdog functionality (otherwise all 5-10' reboot with Windows and Linux) and b) most speficially disable: "Active State Power Management". Since I disabled this option, the Server, a SuperMicro SC826 pushed more than 300GB over CIFS without hickups - so it looks stable.

Besides I recommend you to download very latest IPMI Firmware 1.37 only available via Thomas Krenn Germany (until now not from SuperMicro) which also solves lots of issues with Remote console and newest Sun (Windows) JVMs: http://www.thomas-krenn.com/en/download.html

Hope sharing this helps others as we had quite a lot of hassles with this board (changed it 3 times with always same symptoms)

Ah yes, and if you get stable BIOS Settings, save/dump them with SMCMOS.exe from SuperMicro :-)

This is very helpful information thanks!

I also disabled/switches a couple of options in the BIOS and it appears to run fine now. Think I hit some of the same as you have since my experience is exactly the same :)

Also thanks for the IPMI Firmware link, I was really looking for a firmware update for it. Since there are a lot of little nuisances with it; specially the long delays on both physical hardware and ipmi-kvm and the often disconnects from the vnc server.

We should stay in touch and provide this information, together with the 'perfect' bios settings to the rest of the world :)

P.S. Direct link to IPMI firmware: https://www.thomas-krenn.com/en/download.html?manufacturer=5&category=82&product=6809

Hmmm... That's interesting. If disabling ASPM fixes the issues with the built-in NICs then that means that OpenSolaris/Nexenta/Fedora 12 don't properly support it whereas Windows & CentOS do.

I've physically disabled (jumper) the 82574L NICs in my system so I can't test this right now, but I'll give it a try on Monday or Tuesday.

Regarding my support ticket, I guess the ASPM issue makes it invalid. Sounds like an OS issue rather than a hardware one. I had also included requests for more flexible fan control and a couple other things but I haven't heard back on any of that.

EDIT: Oh, and just to update since this thread has my server specs, My system now has a 160GB Intel X25-M as a read cache and has been upgraded to 16GB of RAM (4x 4GB). Max power savings puts it at 58W.

Another slight update about ASPM: Stay away with this advanced power management option in your BIOS when using *Solaris. Have a look at /var/fm/fmd and you will maybe see an endless stream of errlog-Files which may or may not fill up your syspool sooner or later.

You can read the fmd's output with fmdump (-V / -eV) and you will see that there are lots and lots of recoverable errors. ASPM can reduce PCIe power consumption at cost of some latency which OpenSolaris Kernel apparently doesn't like. And it's not only Solaris, after searching the web, ASPM is rather considered causing more headache than good - also on some Linux Distros. The idea behind it is good but it seems to be too new to be stable enough.

For your readings: http://www.h-online.com/open/news/item/Kernel-Log-Coming-in-2-6-32-Part-6-Infrastructure-869973.html

It would. And you should actually try this -- the strength of the braking force is VERY strong. It'll really surprise you! Acai Optimum