Supermicro IPMI BIOS upgrade fail [Solved!]
Sunday, August 24. 2014
I tried to upgrade my Supermicro SuperServer 5015A-EHF-D525 IPMI BIOS to have the Heartbleed fixed in it. It failed on me. Badly. When I run:
lUpdate -f SMT_316.bin -i kcs -r y
The not-so-friendly response is:
If the FW update fails,PLEASE TRY AGAIN
update part 0, the size is 0x800000 bytes
Transfer data ................
40K bytes 1%ERROR !! BMC did not in correct state
ERROR:SEND "ReceiveFWData" COMMAND TO BMC FAILED
It looks like Supermicro's Linux upgrade tool is the culprit. It enters BMC upgrade mode, starts pushing bits to FlashROM, and then segfaults. I tried couple of BIOS-versions, but to make things worse, I was going from version 2.x to 3.x and there was no downgrade possibility anymore. The BMC was semi-concious, but it really couldn't do much. For example, it didn't have a proper MAC-address, and its networking was effectively out of play.
The worst part of this failing upgrade is, that to get the BMC upgrade mode disabled, you need to pull the plug. If there is electricity connected to the machine, the BMC will stay on.
Update:
Also the ipmitool bmc reset cold helps.
Luckily, there are couple of options for accessing the BMC directly from OS-side. One of them is IPMItool, but it didn't yield any results. The BMC was stuck somehow. Same story with manufacturer's IPMICFG.
I was almost going to give on on this and was planning to RMA it to Supermicro support in Netherlands. Then a newer version of IPMI BIOS was released and I attempted to upgrade into it. Same story, Linux utility crashes badly causing havoc. As the last move, I USB-booted the hardware into DOS-mode. There is a flash-utility in their ZIP-file for DOS. IT WORKED!! How lucky was that!
The new BIOS-version was sane, it knew its own MAC-address and started operating properly. I was so happy!
Who would have thought, that DOS was abandoned almost 20 years ago, and it once more saves the day.
Howard on :
Sadly, i upgraded via the Webinterface, so i can't check if that was the culprit with the linux command line utility.
It didn't segfault for me, but it failed with the above message.
So if anybody googles for the error and finds this page, try lUpdate -f SMT_316.bin -i kcs -r n instead.
And if it works, please leave a comment for the rest of the world to know and find this page
Jari Turkia on :
I completely missed that one.
My background story was, that I got a complaint from my ISP, that my IP-address has a heartbleed flaw in it. Closer inspection revealed, that it was the IPMI web interface. I turned to Supermicro support, who sent me an upgraded firmware, which bricked the entire thing. Thay forgot to mention the not-to-preserve for 2.x to 3.x also.
It actually took me couple of months to get the device from co-location to have a go with this one.
Alex on :
I'll try upgrading using DOS the next time i'll go to the datacenter...
Alex on :
Jari Turkia on :
Currently there is no newer IPMI BIOS for my board, but I'll definitely try this the next time.
Running lUpdate the it says:
VERSION : 1.21
BUILD DATE : Dec 3 2012
My previous version was:
VERSION : 1.15
BUILD DATE : Jan 7 2010
Not especially new. Either one of those.
Dalimil Gala on :
I managed to upgrade fw in X7SPA-HF board from v2.64 to v3.16 using lUpdate 1.21 on 64bit Debian 6.0 Squeeze.
Xeno on :
Jari Turkia on :
Sven on :
I had the same problem when updating the IPMI firmware. Support forgot to mention the "preserve configuration" option and it failed. The only option was to RMA the board but the warranty was expired. I disabled IPMI and used the board without it.
Recently I moved all my files to a NAS and I decided to give it a try to fix the IPMI. I searched the internet and found this website. First I try to fix it in Windows Server 2008 R2, but that didn't work. As they mentioned using lUpdate, I decided to install Debian 9.2.1 and give it a try. I used lUpdate 1.15 and it worked like a charm.
Jari Turkia on :
I love Supermicro hardware, but both their software engineering and support surely falls very short on IPMI upgrades. To me it would make sense that they would address this problem on their support site and improve the Linux-tool for not to crash on a trivial operation. But no, DOS to the rescue!