Wetfish Online
Discussion Forums => Shitpost Central => Topic started by: rachel on December 28, 2016, 02:14:47 pm
-
main irc server is down
irc services are down
fishy is down
mumble is down
wetfish login is down
teeworlds servers are down
irc bouncer is down
fuck me
I'm supposed to be sleeping right now
edit: realized more services were down than I originally thought
-
congrats rachel!
-
thanks!!
update:
main irc server was back up first
mumble is back up now too
still working on bringing services and fishy back online
-
oops I did it again
-
What a glorious fuck up
but ty for working hard to fix <3
-
Alright so here's what happened.
There's been an intermittent network issue on the colocated wetfish server. Yesterday I opened a support ticket with traceroutes and a description of the issue. While I was in there looking at the server configuration, I decided to upgrade the packages on the system. Debian 8 has been out for years now so I figured it'd be good to upgrade from version 7. Unfortunately the VM software I use on the server isn't compatible with Debian 8 so I had to revert the updates. While rolling back the updates, apt decided to remove my VM software. You can imagine my horror as I saw "Virtual container 100 shutting down... Virtual container 101 shutting down... Virtual container 102 shutting down..." scrolling past on my screen.
I literally yelled FUCK FUCK FUCK at my computer screen. I think I was channeling dsockwell.
After about an hour of reading through documentation I figured out how to reinstall all of the software that was removed and that fixed most of the problems. Unfortunately I had also decided to upgrade three of the VMs to use Debian 8 months ago. They had been running fine, but after being rebooted the network was no longer functional. Turns out systemd is incompatible with the kernel version I was using on the hypervisor. Ugh.
I decided it was too risky to try upgrading the kernel, rebooting, and potentially bricking the entire box; so instead I spun up three new VMs and copied all of the files over from the old ones.
Everything is back up and running now except for fishy's databases. So that means greetings, !quote and !metronome are broken at the moment. I'll fix that tomorrow.
-
Ironic that only a few weeks ago, a post I made about Linux not being the greatest thing ever was CENSORED by RACHEL
HMMMMMMM...
Karma is a great thing
-
Ironic that only a few weeks ago, a post I made about Linux not being the greatest thing ever was CENSORED by RACHEL
HMMMMMMM...
Karma is a great thing
*rips a wet fart in what's mouth*
our posts are equally helpful
-
Ironic that only a few weeks ago, a post I made about Linux not being the greatest thing ever was CENSORED by RACHEL
HMMMMMMM...
Karma is a great thing
(https://charlyshades.files.wordpress.com/2013/03/laughing.gif)
-
Honk services remained online during the outage.
-
Everything is back up and running now except for fishy's databases. So that means greetings, !quote and !metronome are broken at the moment. I'll fix that tomorrow.
So Anomaly reminded me that it's been over 2 weeks and I still never brought fishy's databases back online. So I went in there and copied all of the database files and BOOM. Unable to load the database, shit's fuckin corrupted. I dig deeper and realize it's because I was using mysql 5.6 on the old server, but the default package available is only 5.5.
Fortunately MySQL has a nice APT repository (https://dev.mysql.com/doc/mysql-apt-repo-quick-guide/en/) that lets you control what version you want and I upgraded to 5.6. Once I deleted the old innodb logfiles everything returned to normal! Quotes and greetings are back :)
In retrospect it seems pretty simple... but damn I just wasted 3 fucking hours figuring that shit out.
-
Gurl 3 hours. That's like 3 minutes compared to pstraw learning time. Also what is tfw mean? 😆
-
Also what is tfw mean? 😆
that face when / that feel when
-
yay now i get to spam people with my greeting
-
OH SHIT, FISHY'S BACK AND READY FOR ACTION
(http://i.imgur.com/c9PLQ6P.png)
-
OH SHIT, FISHY'S BACK AND READY FOR ACTION
(http://i.imgur.com/c9PLQ6P.png)
Whew that's a nice fish. I'm kind of jealous I don't have any cats like that.
-
OH SHIT, FISHY'S BACK AND READY FOR ACTION
(http://i.imgur.com/c9PLQ6P.png)
MEME CANNONS LOCKED AND LOADED