Wetfish Matrix Server Upgrades

The Wetfish matrix server is being migrated from a DigitalOcean VPS to a dedicated server. The current VPS for the matrix server is at 95% disk usage and DigitalOcean is crazy expensive for disk space. We're moving to Dedicated.com and getting 2x the RAM 12x the disk storage space for only $10 a month more.

The old matrix server was shut down at 8:17 PM (mountain time) and as of writing the migration is approximately 30% complete. Service should be restored within the next couple hours.

i was depressed and losing it ngl

Unfortunately we ran into some permission issues during the migration and had to roll back our DNS to the old server. The matrix server is running again on the DigitalOcean VPS and we will complete the migration after Christmas.

Dec 24 05:38:43 us-chicago-01 matrix-synapse[630794]: Starting synapse with args -m synapse.app.homeserver -c /data/homeserver.yaml
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]: Traceback (most recent call last):
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     return _run_code(code, main_globals, None,
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     exec(code, run_globals)
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/site-packages/synapse/app/homeserver.py", line 421, in <module>
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     main()
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/site-packages/synapse/app/homeserver.py", line 411, in main
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     hs = setup(sys.argv[1:])
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/site-packages/synapse/app/homeserver.py", line 319, in setup
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     config = HomeServerConfig.load_or_generate_config(
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/site-packages/synapse/config/_base.py", line 773, in load_or_generate_config
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     obj.parse_config_dict(
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/site-packages/synapse/config/_base.py", line 794, in parse_config_dict
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     self.invoke_all(
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/site-packages/synapse/config/_base.py", line 393, in invoke_all
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     res[config_class.section] = getattr(config, func_name)(*args, **kwargs)
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/site-packages/synapse/config/repository.py", line 141, in read_config
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     self.media_store_path = self.ensure_directory(
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/site-packages/synapse/config/_base.py", line 242, in ensure_directory
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     os.makedirs(dir_path, exist_ok=True)
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/os.py", line 225, in makedirs
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     mkdir(name, mode)
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]: PermissionError: [Errno 13] Permission denied: '/matrix-media-store-parent/media-store'
Dec 24 05:38:47 us-chicago-01 systemd[1]: matrix-synapse.service: Main process exited, code=exited, status=1/FAILURE

After rolling back the DNS to the old server we were able to determine the root cause of the issue.

In order to complete the migration, there will be another maintenance window in the morning of Tuesday December 27th.

Maintenance window is scheduled for 6pm - 10pm mountain time (8pm - midnight eastern)

We're doing another round of matrix upgrades, trying to get to the bottom of the weird latency spikes we've been seeing when sending messages from Matrix to IRC.

We're upgrading to the latest version of synapse using the matrix ansible github project.

Maintenance window tonight, Wednesday March 8th, from 6 PM until 10 PM (Mountain time)

It's that time again! We're doing some more periodic maintenance of our matrix server.

Maintenance Window: 7 - 9 AM EST, Friday July 21st

The primary change we're making this time is re-enabling public user registration with email address verification enabled (and maybe a captcha?). Previously we allowed users to register accounts on our server without doing any verification which was abused by spammers who registered thousands of accounts and then flooded channels on other homeservers. To prevent the spam attack we had to conduct emergency maintenance to prevent new user registration a couple months ago.

In addition to re-enabling user registration we'll also be pulling a bunch of updates from the upstream matrix-docker-ansible github repo. Here's the changelog if you're curious - https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/CHANGELOG.md

The primary change we're making this time is re-enabling public user registration with email address verification enabled (and maybe a captcha?).

Yeah turns out you can't setup email verification with the ansible playbook??



It appears to only support recaptcha (which we now have enabled along with registration itself).

There's two issues (that we knew about from last time) that I'm probably going to open against the playbook:
- appservice-irc metrics aren't scraped (prom config snippet doesn't exist in the playbook)
- prometheus scrape intervals are crazy fast (like 5 seconds) and are not configurable though the playbook