Hacker's ramblings

Solar panel, photovoltaic system, solar generator. These babies have many names.

There are three of 505 watt panels on my roof. Besides the ones in the picture, I do have more panels. This is just my 1500W on sunrise side. Having micro-inverters works well for east-west -installations.

Conversation around solar panels is constantly bubbling. Lot of discussion, not so much facts. Plenty of opinions back and forth. Topics being reviewed include:

• "Installation is expensive. Is this profitable?"
• "Installation is expensive. What's the breakeven in years?"
• "Sun doesn't shine in Finland! Will this make any sense?"

All of those are valid questions. As I was tempted to find out, I went and ordered an installation last year. In this blog post summarize my experience with solar power since last summer.

Summary

Here's the thing briefly with spoilers:

• No, the thing isn't profitable. It doesn't make any sense as breakeven in cost savings to cover the installation fees is many many many years.
• When it goes to electricity, I wanted some security of supply by having those panels on my roof. That went completely sideways! Those things won't do anything, unless there is a functioning electicity network.
  • This is because inverter needs a place to feed excess electricity into.
  • If no such sink exists, inverter chooses to go silent.
  • This behavior can be altered by going for a more expensive offgrid installation. I do not have such thing.
• Sun does shine in Finland. Savings in electricity is real and tangible.

Measuring stuff

Last summer I wrote a piece about HAN/P1 -port. This is the basis. It is imperative to get exact readings on electricity consumption. Also, as important are readings on exported excess energy. Second thing to measure is the solar production. I'm skipping a ton of details, this is what a Grafana-visualization would look like:

Key:

• Blue area: total electricity consumption of my house
• Black line: amount of imported electricity
• Purple area: amount of exported electricity

Findings:

• When there is solar production indicated by purple excess export, black line for imported energy drops nicely below total consumption.
  • Panels do reduce my electricity bill by providing some of my consumption from own production.
  • There are cases where import actually reaches zero. Momentarily, I'm not paying anything for my electricity. Nice!
• There are many cases where black line for import isn't at zero while there is plenty of export.
  • This is the design flaw with solar panel installation.
  • The only real way of keeping all the solar harvest is to have a battery where export would go to at all times.
  • Also, this is how electricity works. Panel must have a destination where harvested energy goes into each millisecond. If there isn't one, it goes to export.

Results of Measurement

For a period of 10 months, staring from 3rd August 2023 to 29th May 2024 is a reference. No own electricity production. From 29th May 2024 to 3rd August 2025 is the "new normal", panel assisted consumption. As a weekly average, a year with and without solar panels look like this:

Key:

• Orange bars indicate weekly average of produced energy.
• Blue line indicates difference between year without panels and year with panels. There are occasions where saving exceeds produced amount.
  • Years are not comparable. Weather tends to do whatever it likes.

Findings:

• It would have been really nice to have a "clean" reference data for entire 12 month period. Unfortunately, this wasn't possible. Still, 10 months of a year is still a good reference material to measure improvement. The improvement is there.
• There is miniscule amount of solar production in November, December and January.
  • Indeed. Sun does shine occasionally during nothern hemisphere dark months.
• Energy savings are real
  • On the right hand side of the graph, months June and July indicate no energy saving as both years have solar panels. This is the flaw in refenrence consumption data.

The Important Stuff

Lots of graphs and details. What' the key takeaway here? Can we summarize all this somehow?

What Others Say on Profitability

There is a Master's Thesis from Feb 2024: Techno-economic analysis on optimizing the value of photovoltaic electricity in a high-latitude location.

Gist of the thesis is twofold: First, to maximize the profitability, consume your own production. Second, (this is self-evident) installation direction is a factor. In plain words, eat your own dog food and on northern hemisphere, install your panels to southernly direction. The design of PV system installation must be to capture as much sun as possible and consume your own production as much as possible.

Measuring Self-consumption

My Home Assistant setup comes with Energy-dashboard as default:

It has a reading for "self-consumed solar energy". Exactly what I should be monitoring!

As this is important, there is a discussion on calculation "Computation of self-sufficiency/autarkie and self-usage of PV". From the doscussion, following useful math can be found:

• energy_used = energy_imported + energy_production - energy_exported
• net_returned_to_grid = energy_exported - energy_imported
• self_sufficiency_perc = (energy_used - energy_imported ) / energy_used 100.0
• self_consumed_solar_energy_perc = (energy_production - energy_exported ) / energy_production 100.0

Doing the same with Home Assistant is rather simple. Btw. I'm using VictoriaMetrics add-on. Doing something like this in MetricsQL shows following data for the past year:

• delta(sensor.active_energy_import[1y]) = 9000 kWh
• delta(sensor.active_energy_export[1y]) = 1700 kWh
• delta(sensor.total_production[1y]) = 3400 kWh

This is exactly what I'll need to get: self_consumed_solar_energy_perc = 49%
I can self-consume roughly half of my production and rest is exported as excess.

This is a vital metric, as indicated by the thesis on profitability. To increase this percentage, I'd need to store the production in a battery. This battery might be in a car. Little bit of Home Assistant -tinkering and I'd be able to charge the car on excess export energy. Alternative is to double my investment and go for a (expensive) solar battery storage. On those two, I'd might choose the EV.

Measuring Profitability

All the relevant numbers are there, let's convert all this into time and money. More math:

• energy_saved = energy_production - energy_exported
• energy_saved = 3400 - 1700 = 1700 kWh

That is the amount I used of my own production, but didn't have to pay for import. My local network provider takes 4 cents / kWh for transfer. Assuming my energy costs 7 cents / kWh for the entire period, then 1700 kWh would have cost me 190€.

Assuming I'd have a good year and save 200€ on my electricity bill. Further, assuming my PV system cost after tax deductions would be 5000€. A simple 5000/200 division gives the breakeven. That's 25 years! No way, this is profitable nor sensible. Well, at least I have security of supply, when there is an outage .... oh, wait! As mentioned earlier, the panels turn off when there is no electricity. A complete bust!

Finally

On financial perspective, this is not sane. Without batteries, I simply cannot reach the self-consumption numbers needed.

Tinkering with these is a fun hobby, but that's as far it goes.

This is what's happening in my system tray:

Blue is an optional update indicator. Yellow or orange indicate security patches.

This is what happens when I click the update icon:

Pretty good advertisement space! Also, additional updates include:

I have zero intention to upgrade. If Windows 11 had anything better or something I'd ever need, I'd already be running the darned thing. As it's just a Windows 10 with pretty bad user interface, I'll stick with my 10 for time being.

Thanks, Microsoft!

On my Arch, I was doing the basic update with pacman -Syu. There was an announcement on linux-6.15.5.arch1-1. Nice!

Aaaaaand it failed.

(75/75) checking for file conflicts                [####################] 100%
error: failed to commit transaction (conflicting files)
linux-firmware-nvidia: /usr/lib/firmware/nvidia/ad103 exists in filesystem
linux-firmware-nvidia: /usr/lib/firmware/nvidia/ad104 exists in filesystem
linux-firmware-nvidia: /usr/lib/firmware/nvidia/ad106 exists in filesystem
linux-firmware-nvidia: /usr/lib/firmware/nvidia/ad107 exists in filesystem
Errors occurred, no packages were upgraded.

Oh. How unfortunate, that. How to get past that obstacle? I tried all kinds of pacman -S --overwrite "*" linux-firmware-nvidia and such, but kept failing. I was just getting error messages spat at me. That's a weird package as it contains number of subpackages, which in reality don't exist at all. Confusing!

The winning sequence was to first let the thing go pacman --remove linux-firmware and follow that with install pacman -S linux-firmware.

Before: Linux version 6.14.4-arch1-2
A reboot later: Linux version 6.15.5-arch1-1

Maybe that's why people think Arch isn't for regular users. Its only for nerds.

Long time no post. I've been busy doing tons of stuff at work, at home and elsewhere. Had very little time to post. Now that summer vacation is there, I finally have availability to do some posting.

Four years ago I got a really fast connection. The reason why I moved ages ago was to get a fiber Internet connection (also there was the obvious need for more space).

Now that my beloved ISP has more offering also in this region, I chose to go for a speedup. There is an article in Finnish about symmetric speeds: Kuituliittymien lähetysnopeudet nousee!

Before:

After:

Observations:

1. Download speed measured in 2021 was 878 Mbit/s, now 940 and something. Nice improvement there.
2. Then the obvious. 2021 upload was limited to 400 Mbit/s, measuring 393. The speedup is significant as we're nearing 950. Very nice!

Next:
Going above 1 Gbit. I have no idea how long that should take. Years? A decade? 10 Gbit/s connections are available in Finland, 100€ / month. As I'd love to have one, unfortunately, given geography and ISP's turf wars such thing isn't an option for me.

Also: Back-in-the-days I got lots of comments saying something like "Who needs 1 Gbit/s connection!" Note how that wasn't a question. As the Elisa article says, lot of ISP's customers got their gigabit symmetric before me, it is fair to say such speeds are commonly seen. No more luddites commenting on speeds.

For messaging, there are plenty of choices. I must admit, my thinking is similar to criminals: the less any government knows about me and my messaging, the better. Today, full anonymity is gone. Really, really bad actors were staying below law enforcement radar, and now those really good ones for messaging are gone.

So, I'm doing what Mr. Snowden does and am using Signal daily.

For governments to keep track on me, Signal works via phone number. My issue with using a phone number as identifying factor is, in any country, there are "like five" different phone numbers (in reality an area code has roughly 10 million different numbers).

As you must feel confused, let me clarify. The reason, I say "5" is because when numbering scheme was designed, 10 million amounted roughtly to infinite. Today, anybody can dial 1000 numbers per second and exhaust the entire number space in less than three hours. Obviously, there are multiple area codes and prefixes, so we have multiple sets of 10 million numbers. So, it would take a day to dial all possible numbers. With single computer. What if somebody could obtain two computers? Or three?

Let's face it phone number as a technical invention has been obsoleted for years. It should NOT be used to identify me in any messaging app. It's convenient to do so. Governments have been tracking phones for many decades and they can demand messaging protocol operators to enforce phone identification. Still, by any measure Signal is the safest option.

Moving on. Phone numbers: bad. Signal: good.
This is what happened the other day:

The text said:

Open Signal on your phone
Your account will be deleted soon unless you open Signal on your phone.
This message will go away if you're done it successfully.

For the past two years, I've never used Signal on a mobile device. To me typing messages with a non-keyboard is madness! So, I'm just using messaging from my computer(s).
It seems there is a limit to it.
Government wants to track you, so you must verify the existence of your phone number for every 2 years. Fair.

Last summer I wrote about an attempt to get 802.11ax / Wifi 6 to work on a Linux. Spoiler: It didn't.

A week ago, the author of many Realtek-drivers, Nick Morrow contacted me to inform of a new driver version for RTL8832BU and RTL8852BU Chipsets.

After ./install-driver.sh, the kernel module 8852bu is installed. Dmesg will still display product as 802.11ac WLAN Adapter, however incorrect information that will be. After couple of retries, I managed to get WPA3 authentication working.

Ah joy. The USB-stick works! Performs quite fast also.
Very quirky driver, still. I can't seem to get the thing working on every plugin. Need to try multiple times. Typical failure is "No secrets were provided" -error with "state change: need-auth -> failed (reason 'no-secrets', managed-type: 'full')" in message log. I have absolutely no idea why this is happening, the built-in Realtek works every time.

I have to confess: I'm a server hugger. Everything is in cloud or going there. My work is 100% in the clouds, my home pretty much is not.

There are drawbacks.

5.58am, fast asleep, there is a faint beeping waking you. It's relentless and won't go way. Not loud one to alarm you on fire, but not silent one to convince you to go back to sleep. Yup. RAID-controller.

What I have is a LSI MegaRAID SAS 9260-4i. The controller is from 2013. Year later LSI ceased to exist by aquisition. Also the product is rather extinct, Broadcom isn't known for their end user support. As there is proper Linux-driver and tooling after 11 years, I'm still running the thing.

A trivial MegaCli64 -AdpSetProp -AlarmSilence -aALL makes the annoying beep go silent. Next, status of the volume: MegaCli64 -LDInfo -Lall -aALL reveals the source for alarm:

Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name                :
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 7.276 TB
Sector Size         : 512
Mirror Data         : 7.276 TB
State               : Degraded
Strip Size          : 64 KB
Number Of Drives    : 2

Darn! Degraded. Uh/oh. One out of two drives in a RAID-1 mirror is gone.

In detail, drive list MegaCli64 -PDList -a0 (for clarity, I'm omitting a LOT of details here):

Adapter #0

Enclosure Device ID: 252
Slot Number: 0
Drive's position: DiskGroup: 0, Span: 0, Arm: 1
Device Id: 7
PD Type: SATA

Raw Size: 7.277 TB [0x3a3812ab0 Sectors]
Firmware state: Online, Spun Up
Connected Port Number: 1(path0) 
Inquiry Data:             ZR14F8DXST8000DM004-2U9188                      0001    
Port status: Active
Port's Linkspeed: 6.0Gb/s 
Drive has flagged a S.M.A.R.T alert : No


Enclosure Device ID: 252
Slot Number: 1
Drive's position: DiskGroup: 0, Span: 0, Arm: 0
Device Id: 6
PD Type: SATA

Raw Size: 7.277 TB [0x3a3812ab0 Sectors]
Firmware state: Failed
Connected Port Number: 0(path0) 
Inquiry Data:             ZR14F8PSST8000DM004-2U9188                      0001    
Port's Linkspeed: 6.0Gb/s 
Drive has flagged a S.M.A.R.T alert : No

For slots 0-3, the one connected to cable #1 is off-line. I've never go the idea why ports have different numbering to slots. When doing the mechanical installation with physical devices, it is easy to verify cables matching the slot numbers, not port numbers.

From this point on, everything became clear. Need to replace the 8 TB Seagate BarraCudas with a fresh pair of drives. Time was of the essence, and 6 TB WD Reds were instantly available.

To recovery!

New Reds where in their allotted trays. BarraCudas where on my floor hanging from the cables.
Btw. for those interested, case is Fractal Define R6. Rack servers are NOISY! and I really cannot have them inside the house.

Creating a new array: MegaCli64 -CfgLdAdd -r1 [252:2,252:3] WT RA Direct NoCachedBadBBU -a0. Verify the result: MegaCli64 -LDInfo -L1 -a0

Virtual Drive: 1 (Target Id: 1)
Name                :
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 5.457 TB
Sector Size         : 512
Mirror Data         : 5.457 TB
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 2
Span Depth          : 1
Default Cache Policy: WriteThrough, ReadAhead, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteThrough, ReadAhead, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Disk's Default
Encryption Type     : None
Is VD Cached: No

To my surprise, the RAID-volume hot-plugged into Linux also! ls -l /dev/sdd resulted in a happy:

brw-rw----. 1 root disk 8, 48 Jan  5 09:32 /dev/sdd

Hot-plug was also visible in dmesg:

kernel: scsi 6:2:1:0: Direct-Access     LSI      MR9260-4i        2.13 PQ: 0 ANSI: 5
kernel: sd 6:2:1:0: [sdd] 11719933952 512-byte logical blocks: (6.00 TB/5.46 TiB)
kernel: sd 6:2:1:0: Attached scsi generic sg4 type 0
kernel: sd 6:2:1:0: [sdd] Write Protect is off
kernel: sd 6:2:1:0: [sdd] Write cache: disabled, read cache: enabled, supports DPO and FUA
kernel: sd 6:2:1:0: [sdd] Attached SCSI disk

Next up: Onboarding the new capacity while transferring data out of the old one. With Linux's Logical Volume Manager, or LVM, this is surprisingly easy. Solaris/BSD people are screaming: "It's sooooo much easier with ZFS!" and they would be right. Its capabilities are 2nd to none. However, what I have is Linux, a Fedora Linux, so LVM it is.

Creating LVM partition: parted /dev/sdd

GNU Parted 3.6
Using /dev/sdd
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mktable gpt    

(parted) mkpart LVM 0% 100%
(parted) set 1 lvm on                                                     
(parted) p
Model: LSI MR9260-4i (scsi)
Disk /dev/sdd: 6001GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  6001GB  6001GB               LVM   lvm

(parted) q 

With LVM, inform of a new physical volume: pvcreate /dev/sdd1

  Physical volume "/dev/sdd1" successfully created.
  Not creating system devices file due to existing VGs.

Extend the LVM volume group to the new physical volume: vgextend My_Precious_vg0 /dev/sdd1

Finally, inform LVM to vacate all data from degraded RAID-mirror. As VG has two PVs in it, this effectively copies all the data. On-the-fly. With no downtime. System running all the time. Command is: pvmove /dev/sdb1 /dev/sdd1

Such moving isn't fast. With time, the measured wallclock-time for command execution was 360 minutes. That's 6 hours! Doing more math with lvs -o +seg_pe_ranges,vg_extent_size, indicates PV extent size to be 32 MiB. On the PV, 108480 extents were allocated to VGs. That's 3471360 MiB in total. For 6 hour transfer, that's 161 MiB/s on average. To set that value into real World, my NVMe SSD benchmarks 5X faster on write. To repeat the good side: my system was running all the time, services were on-line without noticeable performance issues.

Before tearing down the hardware, final thing with LVM is to vacate broken array from VG: vgreduce My_Precious_vg0 /dev/sdb1 followed by pvremove /dev/sdb1.

Now that LVM was in The Desired State®, final command to run was to remove degraded volume from LSI: MegaCli64 -CfgLdDel -L0 -a0

To conclude this entire shit-show, it was time to shutdown system, remove BarraCudas and put case back together. After booting the system, annoying beep was gone.

Dust having settled, it was time to take a quick looksie on the old drives. Popping BarraCudas to a USB3.0 S-ATA -bridge revealed both drives being functional. Drives weren't that old, 2+ years of 24/7 runtime on them. Still today, I don't know exactly what happened. I only know LSI stopped loving one of the drives for good.

In this blog, I've established two facts: I own domains and I run my own mail server. When you merge those two together we get to the point where I have mailboxes which completely ignore the left side of user @ domain in an email address. This enables me to use unique email address to each an every possible usage. When I start getting spam from some poor bastard, I can easily identify and attribute blame. This does happen surprisingly often.

Last autumn, such an incident happened. As the "poor bastard" in question was Stiga.com I'm publishing the details here. For a small ones, I'm willing to give them the benefit of a doubt as most organizations don't have that much skills and resources on information security. Anybody having 450 million € sales per year, don't expect me to hold back.

Timeline

September 2024

Spam:

Reply-To: info@cuscmm.com
From: Mossack Fonseca <d33858864@gmail.com>
Date: Mon, 23 Sep 2024 15:02:47 +0100

I hope this email finds you well. On behalf of Jeff Bezos, the CEO of Amazon, I am writing to inform you that you have been randomly selected to receive a donation from his fortune of $194.6 billion usd. Yes, you read that right!

Spamming happened twice. Couple days between the spam. Please note, Google or Gmail has nothing to do with this. They're simply the transport media.

As it was easy to attribute the fraud to Stiga, I instantly send them feedback demanding my (EU) 2016/680 aka. GDPR allotted right to know what was leaked.

October 2024

Sign of life. Stiga is alive!

You can imagine the drill. "No this wasn't us." and "No such thing can happen with us." -style of email exhange. I laid out all the facts from the date I've created my Stiga.com account with every single detail on the timeline, the tone started to shift into "Please, elaborate." and "Can you send us all the details, please." which from my point of view was nice.

Obviously I assisted them with all the information I had.

November 2024

Boom! Announcement:

Notification pursuant to Article 34 of Regulation (EU) 2016/679

What has happened:
On 24.9.2024, STIGA's ICT team detected a security breach affecting our systems.
Specifically, the login details of one of our supplier were used inappropriately.
As a result, some of your personal information was temporarily disseminated without authorization.

Notice how ICT team "detected" this incident a day after I received the first spam.

Finally

I haven't received any further spam on that address. It seems the fallout wasn't especially big.

Meanwhile in Finland ...

Late 2024, a similar incident occurred. Article: Cyber attack hits Valio, putting data of 5,000 at risk.

Pretty much the same story. A vendor got hit. My thinking: a specific person at vendor working remotely from home. Credentials to the customer system got leaked. Subsequently those stolen creds were used to extract a dump of GDPR-protected personnel data.

Prediction

This seems to be a thing nowadays. Corpos are getting better and better at protecting their own data. However, the external parties they hire to maintain systems aren't.

Feel free to call me wrong on that.

Update: January 2025

Getting spam with my Stiga.com -email. What once leaks, can not be un-leaked.

For a couple months, there is an exhibition of "operators and automated data processing designers" in Museum of Technology.

I visited the exhibit as tons of old hardware was shipped from Computer Museum of Finland, Jyväskylä. As there were so many interesting pieces of hardware, I'm presenting a few pictures here. On any typical blog post, I'd blanket my text with links. Here, on purpose, I'm omitting them. "Do your own research" as conspiracy lunatics say!

Funet Cisco AGS+

Back in the day, in 1988, when you visited ftp.funet.fi, your traffic when through this exact router. It was the first ever router manufactured by Cisco Systems.

Psion Revo

I owned one. It was a magnificant piece of pocket-size computing power! As a minus, any kind of transfer required the thing to be plugged into a PC. This minor drawback didn't slow me. Neither did the black&white screen.

Later Nokia purchased Epoc and made it Symbian.

Nokia Communicator 9210

While this wasn't the first ever communicator by Nokia, it was the best one. Proper screen, good keyboard, Symbian 5, ah.

Back-in-the-days, I was poached to a company to write Symbian C++ code for this device. Fun times!

IBM PC

As in - the first one. Ever! From 1981.

For this invention everybody on this globe owes a lot. If IBM had kept the system closed, there wouldn't be no ecosystem for hardware manufacturers nor software crafters. This ecosystem made all the next rounds of evolution possible landing us where we are today.

Ancient: IBM 3380 HDD

Moving to antics. Back in 60s this refridgerator-sized thing was one of the early HDDs invented by IBM. Capacity was 2,5 gigabytes. During that era RAM was in kilobytes, floppy disks barely reached megabyte.

Ancient: PDP-11

This DEC thing pre-dates me. Those things from 60s were so rare, I'm sure not many ever landed shores of Finland. The screen size is something from 2020s. However, the display is 1m x 1m x 1m and has to weigh a ton!

Ancient: DEC VT102 Terminal for the PDP-11

When you open a "terminal" in OS of your choice, it's a software version of that. Funny thing is, VT102 is still a common terminal type to emulate.

Ancient: IBM System/360 Control Panel

Your Windows 7 had a Control Panel. This is the same thing, but for IBM S/360. That's how you'd manage your computer's settings back in the 60s.

With this Nightcafe-created image of Santa Claus' "cloud service" I'd like to wish you all Merry Christmas!

It seems AI took the cloud-part very literally. Totally understandable. How would one draw such an abstract concept.

Last weekend, I had the chance of visiting Superfest 2024. It's an eSports event organized by Supercell. Championship of three games were played:

1. Clash of Clans World Championship Finals
2. Clash Royale Leagure World Finals
3. Brawl Stars World Finals

When entering the event, lounge:

Sunday finals schedule poster:

Saturday game of Clash of Clans. STMN players Darkstar, Fluxxy, Ninj, Synthé and VAD Hawk playing against Millesime MG players KingsMan, Natchoa, Teemper, TryHard and Max:

Sunday game of Clash Royale from lounge, two players from SKCalalas Sub vs Ryley. Ryley became 2nd:

Brawl Stars champions HMBLE players Symantec, BosS and Lukii after their victory:

Well organized event. I wish I would have understood the games bit better. Those games are pretty unknown to me.
Still: GG!

As scheduled to end of October 2024: Announcing Fedora 41 from Red Hat.

Such distro has mechanism to do in-place upgrade: Upgrading Fedora Linux Using DNF System Plugin. It is based on DNF System Upgrade.

Priciple is to run dnf system-upgrade download --releasever=41, tap Yes couple times and then dnf system-upgrade reboot. It works (mostly) and I have used such in-place upgrade many times on a VM running in Hetzner.

If you haven't read between the lines yet, let's state the obvious: I'm posting this in a scenario where everyhing didn't go as planned.

Imagine a virtual machine running in a data center far far away. There is interaction via SSH or if needed, a browser-based console can be used for dire needs. A failed update was indeed such.

41 Upgrade Begins - And Fails

Simultaneously with dnf system-upgrade reboot, I start sending ICMP echo requests to my VM to see the point in time when it begins pinging me back. This is a clear indication of upgrade being finished. I waited roughly 20 minutes without a response. Such a long time is an obvious indicator of a problem. Subsequently I logged in Hetzner's portal to pop open a console. Console showed me an upgraded system in the middle of a reboot-cycle. Stuck doing nothing.

That being unexpected, I chose to go for a Ctrl-Alt-Del. My wish came trough, a request to reboot nicely recycled the stuck system and a login-prompt greeted me on the console. Still, ping didn't. On the console, the only single keyboard layout made available is hard-coded ANSI US. On hardware, all my keyboards have layout ISO Finnish. That makes those elusive often used characters like dash (-), slash (/), pipe (|) or colon (:) be in very very different places slowing the entire process.

On the Console - Missing Package

Poking around the system on console indicated an upgraded VM. Everything else on the system was nice & peachy besides networking. There was no IP-addresses assigned. Actually entire NetworkManager was missing from the system. It did not exist. At all! Also every single bit of configuration at /etc/NetworkManager/ was absent.

Transferrring the muich-needed RPM-package NetworkManager-1.50.0-1.fc41 by eyeballing a rather dumb virtual console is fruitless. A quick analysis of the thing ended with a harsh one: it doesn't support any sensible means of transmitting files. Receiving any sent data with copy/paste or any other low-level means was absent. Something else was needed.

The Fix - Scraping

I opted to fix the network by hand. ip-command was installed in the system and it worked perfectly. That's all I needed. Or, almost all.

In my infinite wisdom, I didn't have any of the IP-details at hand. I reasoned to myself the system upgrade having worked flawlessly multiple times before this. I didn't NEED to save IPv4 or IPv6 -addresses, their routing setup or DNS-resolvers. I knew from tinkering with these boxes that on x86-64 architecture Hetzner VMs  all those details are static, manually set to Hetzner-assigned values. Their modern setup on Arm v8 does utilize DHCP for IPv4. My box was on a traditional rack and I couldn't count on automation to assist on this one.

Scraping all the bits and pieces of information was surprisingly easy. My own DNS-records helped. After the fact, I realized a shortcoming, if I would have looked at the bottom of the web-console, those IP-addresses would have been shown there. At the time I didn't. Routing defaults can be found from documentation such as Static IP configuration.

Now I knew what to set for the values.

The Fix - Manual Labor

Now the "fun" begun. I need to setup IPv4 address complete with routing to restore functionality of dnf-command. This would make it possible to install NetworkManager to get nmcli-command back.

Sequence is as follows:

ip addr add 192.0.2.1/32 dev eth0
ip route add 172.31.1.1 dev eth0 src 192.0.2.1
ip route add default via 173.31.1.1 src 192.0.2.1

Btw. see RFC5737 for IPv4-addresses and RFC3849 for IPv6-addresses reserved for documentation. I'm not going to post my box's coordinates here.

Fedora DNS-setup is via systemd-resolved, checking file /etc/systemd/resolved.conf. File had survived the update intact. It still had the content of:

DNS=185.12.64.1 185.12.64.2 2a01:4ff:ff00::add:1

A perfect & valid solution.

The Fix - Managing Network

Ta-daaa! Ping worked. dnf worked! Everything worked! The joy!

At this point running dnf install NetworkManager wasn't much. Trying to figure out what was wrong proved to be a headache.

On initial glance nmcli conn show:

NAME  UUID                                  TYPE      DEVICE
eth0  12345678-1234-1234-1234-123456789abc  ethernet  --

What!? Why isn't my eth0-connection associated with a device! No amount of attempts, tinkering, cursing nor yelling helped. I could not associate a device with the connection. My only option was to drop the hammer on the thing: nmcli conn del eth0

Now my eth0 didn't work as it didn't exist. A delete made sure of it. Next, getting it back:

nmcli conn add type ethernet ifname eth0 con-name eth0 ipv4.method manual ipv4.addr 192.0.2.1
nmcli conn modify eth0 ipv4.gateway 172.31.1.1
nmcli conn modify eth0 ipv6.addr 2001:db8::1/64
nmcli conn modify eth0 ipv6.gateway fe80::1

Final twist was to make those changes effective: nmcli device reapply eth0

IPv6 begun operating, IPv4 was unchanged. Everything was where it should have been after the upgrade.
That was it for NetworkManager, moving on.

Outcome

The only true measure of a possible success is a system reboot. If my tinkering survived a normal cycle, then all was good. Nothing special to report on that. Everything stuck and survived a rinse-cycle. File eth0.nmconnection stored NetworkManager configs as expected.

Why this thing exploded remains unknown. Missing any critical pieces of a system is always a disaster. Surviving this one with very little damage was lucky. I may need to increase my odds and re-image the VM. My guess is, this box survives only so-and-so many upgrades. I may have used all of the lives it has.

My employer opted to host a meetup. As they needed somebody to give a presentation there, obviously, I stepped up.

Thanks for all the participants!

For those interested, my presentation on Microsoft Fabric Real-Time Intelligence.

On 26th of September, PostgreSQL Global Development Group announced the release of version 17.

Here is an easy one: Can you guess at which point I made the upgrade?

The slope is a maintenance break. Datadog wasn't measuring HTTP-performance while I was tinkering to make the actual upgrade.

What worries me is the performance being itsy-bitsy worse with version 17. Graph is smooth as silk. However, crunching the numbers to smooth the zig-zag, 16 seems to have better performance on average. Difference isn't big, but it is there. Maybe I'm missing a new setting to improve cache performance or something?

It took "couple" iterations on Apple to realize they'll must add a password manager to their mobile operating system.

Next Monday, iOS 18 will be released with one. It's called "Passwords":

As I already have a password manager, this isn't a big deal. However, this new password-thingie extends functionality beyond passwords and has something The World has never seen: Possibility to edit Passkeys. (Anti-climatic, huh!)

Taking a closer look, this is what the password-app contains:

Further dill-down on (the never-seen-before) Passkeys:

See how many dupes I have for https://webauthn.io/!
Three years back, I wrote a piece about Passkeys and it required a lot of research, testing and snooping around. Toying around throw-away Passkeys resulted in lot of junk, it seems.

While muttering "my precious", Yubico declared all information on existing passkeys a state secret. After the first one's example all other major Passkeys-providers like Appe and Google followed suit. They hid Passkeys well out of your reach.
I really never got the logic behind that. Why is it not allowed to see your own passkeys? Maybe if your security device is lost, also the criminals can see the list of your passkeys? Puzzling.

Finally I can get rid of all test junk. It took unbelievebly long to get this feature. However, I love the fact it is there.