Wetfish Matrix Server Upgrades

The Wetfish matrix server is being migrated from a DigitalOcean VPS to a dedicated server. The current VPS for the matrix server is at 95% disk usage and DigitalOcean is crazy expensive for disk space. We're moving to Dedicated.com and getting 2x the RAM 12x the disk storage space for only $10 a month more.

The old matrix server was shut down at 8:17 PM (mountain time) and as of writing the migration is approximately 30% complete. Service should be restored within the next couple hours.

i was depressed and losing it ngl

Unfortunately we ran into some permission issues during the migration and had to roll back our DNS to the old server. The matrix server is running again on the DigitalOcean VPS and we will complete the migration after Christmas.

Dec 24 05:38:43 us-chicago-01 matrix-synapse[630794]: Starting synapse with args -m synapse.app.homeserver -c /data/homeserver.yaml
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]: Traceback (most recent call last):
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     return _run_code(code, main_globals, None,
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     exec(code, run_globals)
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/site-packages/synapse/app/homeserver.py", line 421, in <module>
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     main()
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/site-packages/synapse/app/homeserver.py", line 411, in main
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     hs = setup(sys.argv[1:])
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/site-packages/synapse/app/homeserver.py", line 319, in setup
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     config = HomeServerConfig.load_or_generate_config(
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/site-packages/synapse/config/_base.py", line 773, in load_or_generate_config
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     obj.parse_config_dict(
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/site-packages/synapse/config/_base.py", line 794, in parse_config_dict
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     self.invoke_all(
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/site-packages/synapse/config/_base.py", line 393, in invoke_all
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     res[config_class.section] = getattr(config, func_name)(*args, **kwargs)
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/site-packages/synapse/config/repository.py", line 141, in read_config
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     self.media_store_path = self.ensure_directory(
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/site-packages/synapse/config/_base.py", line 242, in ensure_directory
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     os.makedirs(dir_path, exist_ok=True)
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/os.py", line 225, in makedirs
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     mkdir(name, mode)
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]: PermissionError: [Errno 13] Permission denied: '/matrix-media-store-parent/media-store'
Dec 24 05:38:47 us-chicago-01 systemd[1]: matrix-synapse.service: Main process exited, code=exited, status=1/FAILURE

After rolling back the DNS to the old server we were able to determine the root cause of the issue.

In order to complete the migration, there will be another maintenance window in the morning of Tuesday December 27th.

Maintenance window is scheduled for 6pm - 10pm mountain time (8pm - midnight eastern)

We're doing another round of matrix upgrades, trying to get to the bottom of the weird latency spikes we've been seeing when sending messages from Matrix to IRC.

We're upgrading to the latest version of synapse using the matrix ansible github project.

Maintenance window tonight, Wednesday March 8th, from 6 PM until 10 PM (Mountain time)

It's that time again! We're doing some more periodic maintenance of our matrix server.

Maintenance Window: 7 - 9 AM EST, Friday July 21st

The primary change we're making this time is re-enabling public user registration with email address verification enabled (and maybe a captcha?). Previously we allowed users to register accounts on our server without doing any verification which was abused by spammers who registered thousands of accounts and then flooded channels on other homeservers. To prevent the spam attack we had to conduct emergency maintenance to prevent new user registration a couple months ago.

In addition to re-enabling user registration we'll also be pulling a bunch of updates from the upstream matrix-docker-ansible github repo. Here's the changelog if you're curious - https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/CHANGELOG.md

The primary change we're making this time is re-enabling public user registration with email address verification enabled (and maybe a captcha?).

Yeah turns out you can't setup email verification with the ansible playbook??



It appears to only support recaptcha (which we now have enabled along with registration itself).

There's two issues (that we knew about from last time) that I'm probably going to open against the playbook:
- appservice-irc metrics aren't scraped (prom config snippet doesn't exist in the playbook)
- prometheus scrape intervals are crazy fast (like 5 seconds) and are not configurable though the playbook

It's that time of year again. The last time we upgraded the matrix server was in October 2023 so we're going to do another server upgrade. This should hopefully improve some of the performance issues users have been reporting with slow sync times. If the issue persists we may need to schedule an additional maintenance where we migrate uploaded media files to s3 so that we can use a server with less disk space but better performance.

Maintenance scheduled: Tuesday December 10th

What to expect: Matrix will be unavailable and a bunch of users will disconnect and reconnect from IRC. Should only take a couple hours

If the issue persists we may need to schedule an additional maintenance where we migrate uploaded media files to s3 so that we can use a server with less disk space but better performance.

We may well do this anyway as a cost saving + overall-goodness-improving measure.

see yall on the other side

Maintenance ran into some issues tonight, the database migration kept getting killed half way through (at different points every time) and we had to roll back the server to our previous configuration.

The current theory is that the performance of our oldschool magnetic harddrives are causing slow write speeds and the database queries are timing out, causing the upgrade to fail.

We are going to try upgrading the hardware to use SSDs instead and hopefully that will fix the database issue as well as the slow syncing speeds that users have been reporting when viewing channels.

The deed is done! The matrix server has been updated to the latest version with help from cyba and goos 🎉