trelane @trelane@librem.one

Warning, long text

Power Outage in Spain – An Analysis

Solar energy comes out of your panels as direct current (DC). That’s all well and good, but homes and grids run on alternating current (AC). Enter the inverter – the humble box that turns solar wizardry into household juice.

Now, inverters aren’t just fancy plug adapters. They have to sync up with the grid – which means they generate exactly the same frequency as the rest of the system. No grid? No syncing. In that case, the inverter goes into what’s called island mode and produces power only for local use. So, if my solar system isn’t connected to the external grid, it can’t run the house – but it can still power two little emergency sockets. Cheers, I guess.

Normally, the grid runs at 50 Hz – that’s hertz, not some obscure Scandinavian metal band. But this frequency can wobble a bit. Physically and technically speaking, it rises when there’s too much power and not enough consumption, and falls when there’s a hungry grid and not enough electricity to feed it.

To keep the grid safe, inverters have an emergency shutdown feature: if the frequency goes over a set limit (apparently around 50.2 Hz), they also jump ship and go into island mode.

Spain’s energy mix is a bit unusual: lots of nuclear, lots of renewables – and a large chunk of those renewables are solar. Makes perfect sense in a country where “cloudy” means three fluffy cotton balls drifted by.

Now, nuclear energy comes with two charming quirks. First, you can’t change its output quickly – it’s not a dimmer switch, more like a cruise ship rudder. Second, nuclear plants cost nearly the same to run at half speed as they do at full throttle. So, naturally, you want to keep them purring along at max capacity.

Then came Monday, with weather conditions perfect enough to make a solar engineer weep with joy: loads of sun, plenty of wind. By 9 a.m., Spain’s energy needs were entirely met by nuclear and renewables. In fact, they had surplus electricity and began exporting it by the bucketload. They shut down everything easy to shut down – but nuclear? No chance. It stayed full steam ahead.

Then, two unfortunate things happened: one transmission line to France caught fire (as you do), and another developed resonances due to meteorological oddities.

So far, this is all well documented. Now we step into speculation territory.

These instabilities meant Spain couldn’t get rid of its excess electricity. The grid frequency rose past that critical 50.2 Hz mark – and boom: many solar systems switched to island mode. At that moment, they were providing nearly 15 gigawatts – around 60% of the national supply. And just like that, poof – they were gone.

Suddenly, two-thirds of the electricity vanished. Wind, nukes, and batteries couldn’t keep up – quite the opposite, in fact. To prevent damage, the nuclear plants initiated emergency shutdowns. Not great. (More on why that’s bad in a bit.) Within seconds, the entire grid collapsed. The solar systems were poised to help – but there was no grid left to sync with.

Everything went dark.

Portugal and southern France were also knocked offline, as they’d been happily sipping from Spain’s excess power. The European grid wasn’t amused and unceremoniously kicked Spain out of the club. France, with a bit of backup and a stiff upper lip, restored its network fairly quickly. My home automation system even picked up the moment the frequency dipped and France cranked up its own generation.

Portugal got the rough end of the stick. With fewer reserves and being smaller in size, they couldn’t help themselves – and no one else could help either, since Spain’s their only neighbour.

Rebooting the Grid – Why It’s a Right Pain

Restarting a collapsed grid isn’t just a matter of flipping a giant switch. It’s tricky for two reasons:

Generation and consumption have to be in perfect balance. If not, we’re back to square one.
Nuclear power plants can’t just be turned back on. After an emergency shutdown, they suffer from something called xenon poisoning (yes, one of the very same issues that made Chernobyl a household name). You’ve got to wait for that to wear off – which means the reactors were still offline two days later.

The fix? You split the grid into smaller bits. For each chunk, you build up some capacity, bring it online, then move on to the next. Rinse and repeat. This takes hours. Meanwhile, the sun moves across the sky – and even if you do reconnect the solar arrays, they won’t produce nearly as much as before. Come 8 p.m., they’re more or less useless.

So Spain needed outside help. They were gradually reconnected to the European grid – in small, careful steps. Without that assistance, large parts of Spain would probably still be in the dark. That’s why electricity came back first in places like Barcelona, close to the French border, while Portugal endured the longest wait.

Notes & Musings

Considering the scale of the event, the recovery was impressively quick. In San Sebastian, power was back within 2 hours. (For comparison: Wismar in Germany had a 45-minute outage last year because one substation had a wobble.) Portugal got its power back after 23 hours. I had expected one to two days.
This was the largest blackout in Europe in 40 years. If, as suspected, climate-related factors helped spark (pun intended) the situation, then modernising the grid to better handle volatility is absolutely essential. That includes implementing the long-debated power zones in Germany.

Some atomic nuclei are round. Others are football-shaped. Some are even stranger. Some are pear-shaped! Some have a 'halo' of protons and neutrons orbiting a smaller core. They're all just minimizing energy - unless they're in a temporary 'excited state'. But the forces between protons and neutrons are so complex that this can make lots of things happen.

Just as noble gases are exceptionally inert because they have a filled shell of electrons, the roundest and most stable nuclei are those with a filled shell of protons - and, separately, a filled shell of neutrons. Filled shells happen when we hit a 'magic number'. The magic numbers are

2, 8, 20, 28, 50, 82, and 126.

I want to show you some cool patterns in these numbers. But first, some examples:

• Hydrogen, with just one proton, is perfectly round, but that's an exception.

• Helium-4 has 2 protons and 2 neutrons, and that's the second most common element in the universe.

• Oxygen-16 has 8 protons and 8 neutrons, and this is the most common form of oxygen.

• Calcium-40 has 20 protons and 20 neutrons, and this is the most common form of calcium. It's also the heaviest stable nucleus that has the same number of protons and neutrons! Heavier nuclei need more neutrons to be stable, but calcium-40 is stabilized by the fact that 20 is a magic number.

• Calcium-48 has 20 protons and 28 neutrons. It's not stable, because it has too many neutrons - but it has a half-life of 64 quintillion years, so it's damned close to being stable, thanks to the power of magic numbers.

• Lead-208 has 82 protons and 126 neutrons. It's the most common form of lead.

(1/2)

Twenty years ago while a young grad student, I was asked to implement an experimental annealing algorithm some boffins in the U cooked up. Along the way I stumbled into one of the cleanest small-project architectures I've ever designed. On its strength, I was invited to speak at CodeCon 2006.

For its twenty-year anniversary I dusted it off & gave it serious attention for the first time in fifteen years. Way back I put a lot of work into making it as C++98 standards conformant as possible. I was pleasantly stunned how little code rot had set in. Overhauling to C++23 only took a few hours.

If you've found my political commentaries useful or my _Paranoia_ humor funny, well — it would please me a lot if you'd just take a quick look at the README.

I'm good with law and government, but I'm not proud of being good with it.

But I *am* an engineer, and I'm proud of some of my pet projects.

Drop a boost or favorite on this comment, or star/follow/fork on GitHub, if you think this kind of high-quality hackery is your bag. And as always, constructive criticism, useful bug reports, and high quality PRs are the best praise there is.

https://github.com/rjhansen/djinni

Computer science has had a bad case of identity disorder since the mid-'90s. I was in undergrad at the time and saw it take root. When I began my degree the only people taking CS were nerds: by graduation the web was beginning to take off, every company wanted to be Netscape.

In '98 I was a senior sitting in on an interview with a high school senior who openly said he wanted to study CS because his parents were insistent it would land him a good job, and computers were "fun enough," he guessed.

By '00, most of my alma mater's CS students were like that.

*There's nothing wrong with wanting a good job.* But CS is a hell of a way to get a good job. If you want to reach the serious bucks you have to have a fanatical commitment to *always* studying, because your tech skills are becoming obsolescent faster than you can learn new ones, for starters.

And if your commitment wavers, then no matter how rockstar you are now you are at most five years from being a mediocrity with an out-of-date skillset.

And the psychological toll this takes cannot be overstated, and that's on top of all the other psych tolls you have to pay.

Hot tip: if you're interested in learning the details of world sanctions against Russia, you should attend this free webinar.

I know some of the people involved. They're knowledgeable and violently allergic to bullshit.

https://instituteforfinancialintegrity.registration.goldcast.io/webinar/7a670fc9-a42e-41ec-a8d7-569c387c3f9a#Registration

Still using OpenOffice? It has unfixed security issues over a year old: https://en.wikipedia.org/wiki/Apache_OpenOffice#Security – So all users are strongly recommended to update to one of the actively maintained successor projects, like LibreOffice. (Please share and help raise awareness about this!)

LWN in EPUB format

https://lwn.net/Articles/1006562/ #LWN

Our dev blog helps new #LibreOffice developers to get started and explains core concepts. Here's what we'll be covering in 2025: https://dev.blog.documentfoundation.org/2025/01/16/outlook-for-the-new-year-2025/

So here is a weird one ... the LWN site has been seeing a steady stream of login attempts, all using weird yahoo addresses as the username. By "weird" I mean things like lllbnwidgqeerdyi@yahoo.com and other equally unlikely strings.

These do not correspond to LWN accounts, but somebody has looked at our login form for long enough to post the login attempts directly, without loading the form first. The attempts come from all over the Internet, suggesting that some sort of botnet is doing this.

I don't suppose anybody else has seen this sort of pattern, or has any idea what it is that they may be trying to accomplish?

[$] The OpenWrt One system

OpenWrt is, despite its relatively low profile, one of our community's most important distributions; it runs untold numbers of network routers and has served as the base on which a [...]

https://lwn.net/Articles/994961/ #LWN

If you were ever curious what working for me would be like, you have an opportunity to come experience it.

https://boards.greenhouse.io/tenstorrent/jobs/4485951007

Lots going on, lots of low level distro, packaging, ci/cd pipeline stuff, and in a very open source centric company.

#FediHire #FediJobs

There's a series of books/engineer notes on electronics by Forrest Mims which I think is fantastic.

https://www.ersbiomedical.com/Forrest-Mims-Series_ep_126.html

[Question] If you were buying a well-supported, software-developer #Linux laptop for work, paid for by the company, either in the UK or the USA, what would you buy?

🤔

(pls share for eyeballs and opinions)

Since the #xz incident started, I've been maintaining an FAQ/living document on what we know at https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27.

I think most people in my extended circle either already seen it but posting about it given there's likely to be a lot more questions as we go into the working week.

Thank you to everyone who has contributed tips, suggestions, and edits. Thanks especially to @cadey who has helped a lot with editing.

This is the best timeline I've seen so far on what we know about the Xz backdoor. Some good info here for researchers: https://boehs.org/node/everything-i-know-about-the-xz-backdoor

My story about how telematics data from people's cars unexpectedly raised their insurance rates is on the front page today...

https://www.nytimes.com/2024/03/11/technology/carmakers-driver-tracking-insurance.html?unlocked_article_code=1.b00.DzhQ.GXkvg-kgWebx&smid=url-share

... and this is where it started: me lurking on car forums and seeing comments like this.

If this story doesn't convince lawmakers we need a strong federal privacy law, I'm not sure what will.

Blogged: “I don’t think the cheapest APC Back-UPS units can be monitored except in Windows”
https://strugglers.net/~andy/blog/2024/02/23/i-dont-think-the-cheapest-apc-back-ups-units-can-be-monitored-except-in-windows/

Another Chrome 0-day patched

https://therecord.media/google-chrome-zero-day-patch-webrtc

36 years ago today was the Max Headroom TV STL hijacking. https://en.wikipedia.org/wiki/Max_Headroom_signal_hijacking

I regard this incident as the best and purest hacking prank in modern history. The combination of sophisticated, meticulous technical planning and execution, utterly juvenile content, essential harmlessness, lack of financial motive, and never getting caught or identified (or later taking credit) remains, in my opinion, unmatched to this day.

Pure art.