Its the start of my vacation time.. time to let the fun begin!!! I’m finally off of work; whoop, whoop! 😏

So I slept in but after I woke up I realized that all our servers went offline – its stormy at my work and it knocked the power off. The power came back on and I got the servers back up, only for them to go back down much harder the second time around.

My work closed down due to the outages but be that all as it may, the servers went down hard and they wouldn’t come back up. Honestly, that scared me a little bit, but I prayed about it and got to work. I’ve spent a fair amount of time preparing for disaster recovery. I won’t go into excessive detail so this is the 30,000 feet overview.

Below is the basic setup.

[Secure remote access to router] -> [Special Interface to OOB management network] -> [iDRAC Enterprise Controller]

From there I was able to access the virtual console which gives me access to send the host OS commands and power off and power on, etc. I was able to bounce the server and I saw the VM’s come back online but we still had issues with the ADDC (domain controller) coming up. I used PowerShell to stop all the VM’s and brought up the DC first, which brought DNS (domain name server) and DHCP (Dynamic Host Configuration Protocol) services (among others) back up again.

Once I could see in our online dashboard (single pane of glass for visibility into our network) that the Host server was marked as “online” again, along with the DC, I brought the other VM’s back up in order of importance and verified that they are all running again. Below is the “green screen” from the host server I was working on:

Its kind of funny, that as a little kid, around 6 or so, I was fascinated by green screen computers, and in a way, I still am.. I was coding on video games (peek and poke – anyone remember those commands?) and text adventures before most kids were really even playing computer games. There was many low spots in my childhood but this was one of the bright ones. I don’t care if I’m considered a nerd – I am so grateful that God put computers in my life, computers and dogs, those are a few of my favorite things.

My childhood was a mixture of working in the gravel pit, playing with my dog in the bush, and nights were spent tuning the satellite dish (does anyone remember Anik 1), and writing programs on my computer which had to be saved to cassette tape and reloaded from tape because there was no persistent storage (hard drive) – only volatile RAM memory which cleared when you turned off the computer. Back in the day, you had to “CLOAD” and type in the name of the file, connect the cassette, rewind, press play, and it would slowly load the file back into the computer.. it was a very, very slow process but somehow less tedious than typing out the entire source code of your game (but not by much 😁). Radio Shack was my fantasy store back then – I just was crazy about going to Clearspring mall, playing games at Radio Shack, then head to the red square in the middle of the mall where I would walk back and forth on my hands for an extended period of time. There are a lot of great memories.

Hildebrandt’s Music was also one of my favorite places back then.. I remember going downstairs, hearing the music, the kids would get orange Kool-Aid and sugar wafers, and you’d have music and 25 snotty kids screaming and running back and forth with the parents wondering the kids couldn’t sit still – gee, I don’t know 😂😂😂.. sugar rush much?

There was another funny memory – we used to have “Penner Foods” in the Clearspring mall, in the section close to the back of the frozen isle, they used to have these white drink fountains where you could fill up 2 liter plastic bottles with every kind of soda, mixing and match as you wished. The mall was pretty cool back in the day but it never had a proper food court, now we have all the fast food restaurants outside the mall so I think that was a bit of a missed opportunity.

Have a great weekend!

Jason