tfw I accidentally fucked up the main wetfish server

  • 16 Replies
  • 18338 Views
tfw I accidentally fucked up the main wetfish server
« on: December 28, 2016, 02:14:47 pm »
main irc server is down
irc services are down
fishy is down
mumble is down
wetfish login is down
teeworlds servers are down
irc bouncer is down

fuck me

I'm supposed to be sleeping right now

edit: realized more services were down than I originally thought
« Last Edit: December 29, 2016, 06:42:12 am by rachel »
*spork*

Re: tfw I accidentally fucked up the main wetfish server
« Reply #1 on: December 28, 2016, 03:02:04 pm »
congrats rachel!
no don't

Re: tfw I accidentally fucked up the main wetfish server
« Reply #2 on: December 29, 2016, 04:45:48 am »
thanks!!

update:
main irc server was back up first
mumble is back up now too

still working on bringing services and fishy back online
*spork*

Re: tfw I accidentally fucked up the main wetfish server
« Reply #3 on: December 29, 2016, 08:51:37 am »
oops I did it again
*spork*

lemonrising's Avatar

lemonrising

Re: tfw I accidentally fucked up the main wetfish server
« Reply #4 on: December 29, 2016, 12:08:49 pm »
What a glorious fuck up
but ty for working hard to fix <3

Re: tfw I accidentally fucked up the main wetfish server
« Reply #5 on: December 29, 2016, 01:55:28 pm »
Alright so here's what happened.

There's been an intermittent network issue on the colocated wetfish server. Yesterday I opened a support ticket with traceroutes and a description of the issue. While I was in there looking at the server configuration, I decided to upgrade the packages on the system. Debian 8 has been out for years now so I figured it'd be good to upgrade from version 7. Unfortunately the VM software I use on the server isn't compatible with Debian 8 so I had to revert the updates. While rolling back the updates, apt decided to remove my VM software. You can imagine my horror as I saw "Virtual container 100 shutting down... Virtual container 101 shutting down... Virtual container 102 shutting down..." scrolling past on my screen.

I literally yelled FUCK FUCK FUCK at my computer screen. I think I was channeling dsockwell.

After about an hour of reading through documentation I figured out how to reinstall all of the software that was removed and that fixed most of the problems. Unfortunately I had also decided to upgrade three of the VMs to use Debian 8 months ago. They had been running fine, but after being rebooted the network was no longer functional. Turns out systemd is incompatible with the kernel version I was using on the hypervisor. Ugh.

I decided it was too risky to try upgrading the kernel, rebooting, and potentially bricking the entire box; so instead I spun up three new VMs and copied all of the files over from the old ones.

Everything is back up and running now except for fishy's databases. So that means greetings, !quote and !metronome are broken at the moment. I'll fix that tomorrow.
*spork*

Re: tfw I accidentally fucked up the main wetfish server
« Reply #6 on: December 30, 2016, 08:54:35 am »
Ironic that only a few weeks ago, a post I made about Linux not being the greatest thing ever was CENSORED by RACHEL

HMMMMMMM...


Karma is a great thing
Shit Ass

redheron's Avatar

redheron

Re: tfw I accidentally fucked up the main wetfish server
« Reply #7 on: December 30, 2016, 12:18:55 pm »
Ironic that only a few weeks ago, a post I made about Linux not being the greatest thing ever was CENSORED by RACHEL

HMMMMMMM...


Karma is a great thing
*rips a wet fart in what's mouth*

our posts are equally helpful

Re: tfw I accidentally fucked up the main wetfish server
« Reply #8 on: December 30, 2016, 12:29:42 pm »
Ironic that only a few weeks ago, a post I made about Linux not being the greatest thing ever was CENSORED by RACHEL

HMMMMMMM...


Karma is a great thing

*spork*

Re: tfw I accidentally fucked up the main wetfish server
« Reply #9 on: December 30, 2016, 04:34:28 pm »
Honk services remained online during the outage.
o bb

Re: tfw I accidentally fucked up the main wetfish server
« Reply #10 on: January 19, 2017, 01:22:13 am »
Everything is back up and running now except for fishy's databases. So that means greetings, !quote and !metronome are broken at the moment. I'll fix that tomorrow.

So Anomaly reminded me that it's been over 2 weeks and I still never brought fishy's databases back online. So I went in there and copied all of the database files and BOOM. Unable to load the database, shit's fuckin corrupted. I dig deeper and realize it's because I was using mysql 5.6 on the old server, but the default package available is only 5.5.

Fortunately MySQL has a nice APT repository that lets you control what version you want and I upgraded to 5.6. Once I deleted the old innodb logfiles everything returned to normal! Quotes and greetings are back :)

In retrospect it seems pretty simple... but damn I just wasted 3 fucking hours figuring that shit out.
*spork*

Re: tfw I accidentally fucked up the main wetfish server
« Reply #11 on: January 19, 2017, 06:52:25 am »
Gurl 3 hours. That's like 3 minutes compared to pstraw learning time. Also what is tfw mean? 😆
« Last Edit: January 19, 2017, 07:38:06 am by pstrawberrie »
i'm a twat

Re: tfw I accidentally fucked up the main wetfish server
« Reply #12 on: January 19, 2017, 10:26:20 am »
Also what is tfw mean? 😆

that face when / that feel when
*spork*

Re: tfw I accidentally fucked up the main wetfish server
« Reply #13 on: January 19, 2017, 01:11:42 pm »
yay now i get to spam people with my greeting
no don't

redheron's Avatar

redheron

Re: tfw I accidentally fucked up the main wetfish server
« Reply #14 on: January 19, 2017, 02:01:36 pm »
OH SHIT, FISHY'S BACK AND READY FOR ACTION