Wetfish Matrix Server Upgrades

The Wetfish matrix server is being migrated from a DigitalOcean VPS to a dedicated server. The current VPS for the matrix server is at 95% disk usage and DigitalOcean is crazy expensive for disk space. We're moving to Dedicated.com and getting 2x the RAM 12x the disk storage space for only $10 a month more.

The old matrix server was shut down at 8:17 PM (mountain time) and as of writing the migration is approximately 30% complete. Service should be restored within the next couple hours.

i was depressed and losing it ngl

Unfortunately we ran into some permission issues during the migration and had to roll back our DNS to the old server. The matrix server is running again on the DigitalOcean VPS and we will complete the migration after Christmas.

Dec 24 05:38:43 us-chicago-01 matrix-synapse[630794]: Starting synapse with args -m synapse.app.homeserver -c /data/homeserver.yaml
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]: Traceback (most recent call last):
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     return _run_code(code, main_globals, None,
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     exec(code, run_globals)
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/site-packages/synapse/app/homeserver.py", line 421, in <module>
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     main()
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/site-packages/synapse/app/homeserver.py", line 411, in main
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     hs = setup(sys.argv[1:])
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/site-packages/synapse/app/homeserver.py", line 319, in setup
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     config = HomeServerConfig.load_or_generate_config(
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/site-packages/synapse/config/_base.py", line 773, in load_or_generate_config
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     obj.parse_config_dict(
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/site-packages/synapse/config/_base.py", line 794, in parse_config_dict
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     self.invoke_all(
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/site-packages/synapse/config/_base.py", line 393, in invoke_all
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     res[config_class.section] = getattr(config, func_name)(*args, **kwargs)
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/site-packages/synapse/config/repository.py", line 141, in read_config
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     self.media_store_path = self.ensure_directory(
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/site-packages/synapse/config/_base.py", line 242, in ensure_directory
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     os.makedirs(dir_path, exist_ok=True)
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:   File "/usr/local/lib/python3.9/os.py", line 225, in makedirs
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]:     mkdir(name, mode)
Dec 24 05:38:45 us-chicago-01 matrix-synapse[630794]: PermissionError: [Errno 13] Permission denied: '/matrix-media-store-parent/media-store'
Dec 24 05:38:47 us-chicago-01 systemd[1]: matrix-synapse.service: Main process exited, code=exited, status=1/FAILURE

After rolling back the DNS to the old server we were able to determine the root cause of the issue.

In order to complete the migration, there will be another maintenance window in the morning of Tuesday December 27th.

Maintenance window is scheduled for 6pm - 10pm mountain time (8pm - midnight eastern)

We're doing another round of matrix upgrades, trying to get to the bottom of the weird latency spikes we've been seeing when sending messages from Matrix to IRC.

We're upgrading to the latest version of synapse using the matrix ansible github project.

Maintenance window tonight, Wednesday March 8th, from 6 PM until 10 PM (Mountain time)