Sympl issueshttps://gitlab.com/sympl.io/sympl/-/issues2019-04-16T22:28:40Zhttps://gitlab.com/sympl.io/sympl/-/issues/180Symbiosis: symbiosis-email cron.d entry points to incorrect binary2019-04-16T22:28:40ZPaul CammishSymbiosis: symbiosis-email cron.d entry points to incorrect binaryImported from https://www.github.com/BytemarkHosting/symbiosis/issues/72
`/usr/sbin/symbiosis-encrypt-mailpass` should be `/usr/sbin/symbiosis-email-encrypt-passwords`.
```
@hourly root [ -x /usr/sbin/symbiosis-encrypt-mailpass ] && /...Imported from https://www.github.com/BytemarkHosting/symbiosis/issues/72
`/usr/sbin/symbiosis-encrypt-mailpass` should be `/usr/sbin/symbiosis-email-encrypt-passwords`.
```
@hourly root [ -x /usr/sbin/symbiosis-encrypt-mailpass ] && /usr/sbin/symbiosis-encrypt-mailpass
```https://gitlab.com/sympl.io/sympl/-/issues/181Symbiosis: symbiosis-email-encrypt-passwords --verbose command is not recognised2020-08-22T16:07:25ZPaul CammishSymbiosis: symbiosis-email-encrypt-passwords --verbose command is not recognisedImported from https://www.github.com/BytemarkHosting/symbiosis/issues/65
Using Symbiosis Wheezy.
I need to encrypt a users email account password. Although I remember the password file for a user usually is updated with an encrypted ve...Imported from https://www.github.com/BytemarkHosting/symbiosis/issues/65
Using Symbiosis Wheezy.
I need to encrypt a users email account password. Although I remember the password file for a user usually is updated with an encrypted version of same password this doesnt appear to be working for me at the moment.
I tried the below command from /srv as admin user
symbiosis-email-encrypt-passwords --verbose
but just get
-bash: symbiosis-email: command not foundhttps://gitlab.com/sympl.io/sympl/-/issues/183Symbiosis: symbiosis-httpd-configure --diff-only option2019-04-17T20:05:54ZPaul CammishSymbiosis: symbiosis-httpd-configure --diff-only optionImported from https://www.github.com/BytemarkHosting/symbiosis/issues/48
Sometimes the configuration has been manually edited but it'd be nice to go back to the factory one, however the only way to see what would change other than check...Imported from https://www.github.com/BytemarkHosting/symbiosis/issues/48
Sometimes the configuration has been manually edited but it'd be nice to go back to the factory one, however the only way to see what would change other than checking manually, is to move the old hand-edited configuration out of the way, and then to run the symbiosis-httpd-configure (which reloads the site) and then compare the changes afterwards.
Would be nice if you could ask symbiosis-httpd-configure just to give you a diff of what would change if you asked it to take over the config for a particular site.https://gitlab.com/sympl.io/sympl/-/issues/221Symbiosis: symbiosis-httpd-logger is run where it's not really needed2019-06-20T13:24:05ZPaul CammishSymbiosis: symbiosis-httpd-logger is run where it's not really neededThe HTTP and HTTPS templates for sites both run the symbiosis-httpd-logger process (aka sypl-web-logger) which does little other than write logs owned by the admin user.
This is useful for the zz-mass-hosting configuration, as it then w...The HTTP and HTTPS templates for sites both run the symbiosis-httpd-logger process (aka sypl-web-logger) which does little other than write logs owned by the admin user.
This is useful for the zz-mass-hosting configuration, as it then writes logs to the relevant locations, but it's wasted resources when you have a lot of sites running.
If #219 happens, then the templates should just write the files directly via the normal apache method.Sympl v9.0 (for Debian Stretch)https://gitlab.com/sympl.io/sympl/-/issues/185Symbiosis: symbiosis-ssl can generate SSL config for sites that have no certi...2019-04-17T20:11:54ZPaul CammishSymbiosis: symbiosis-ssl can generate SSL config for sites that have no certificateImported from https://www.github.com/BytemarkHosting/symbiosis/issues/44
symbiosis-ssl can generate SSL config for sites that have no certificate returned by Lets Encrypt. This can lead to invalid configuration, and Apache being unable ...Imported from https://www.github.com/BytemarkHosting/symbiosis/issues/44
symbiosis-ssl can generate SSL config for sites that have no certificate returned by Lets Encrypt. This can lead to invalid configuration, and Apache being unable to re-start.
This has been observed both in terms of missing certs that were never returned successfully from Lets Encrypt, or where symbiosis-ssl didn't have permission to write the certificate, but still wrote the SSL config.https://gitlab.com/sympl.io/sympl/-/issues/186Symbiosis: Testing in gitlab2019-04-22T17:57:10ZPaul CammishSymbiosis: Testing in gitlabImported from https://www.github.com/BytemarkHosting/symbiosis/issues/53
At the moment the install / dist-upgrade / upgrade tests get weirdly-far in gitlab-ci then fails. Here's a quick summary of how the tests used to work on maker2 (a...Imported from https://www.github.com/BytemarkHosting/symbiosis/issues/53
At the moment the install / dist-upgrade / upgrade tests get weirdly-far in gitlab-ci then fails. Here's a quick summary of how the tests used to work on maker2 (as I understand it), and then later I'll go into detail on why the tests fail in gitlab-ci
The Current Situation
================
1. autotest sets up a VM using schroot magic i don't fully understand
2. the VM boots up with systemd and all that jazz, uses DHCP & SLAAC to configure its networking, and automatically runs all the scripts in the autotest folder, keeping all the output as a log file. Once done, it shuts down
3. autotest cracks open the VM's filesystem and reads the logfile. Somehow it detects failures and fails if there was a failure then it exits with a nonzero error code so that maker2 knows
Some Feelings About The Current Situation
===============================
Patrick said something about autotest using the console to talk to the tests, and something else much scarier about the VM sshing into the host to run something.
This doesn't work on gitlab-ci, and is also kinda hacky, for a few reasons.
* the scripts in the autotest folder aren't particularly focussed. In addition to actually running tests, they do these and probably others:
* add an admin user
* install all the packages needed by symbiosis from a big list of packages
* install symbiosis
* opening up the filesystem of the VM so you can prod it is pretty gross
On the plus side it works, and it would only take a bit of effort to port the whole schroot setup over to gitlab-ci (but would have to run using a shell runner)
Why the tests fail in gitlab-ci
=====================
When gitlab-ci runs a container it starts bash in the context of the container. Effectively, bash is PID 1 for the container. There's no init-system to talk to to get stuff going. I believe the apt-get install step for some packages starts them using /etc/init.d (probably something about the package detecting a lack of systemd and putting a proper sysvinit script in) which would explain why a lot of the tests actually succeed. BUT SOME OF THEM FAIL, and we should really be doing a much more realistic test than running our symbiosis full-system tests in a docker container that isn't a full symbiosis system.
With that in mind:
A More Realistic Test Proposal
======================
We're still going to want to run symbiosis in a VM, I think. To do a realistic full-install / dist-upgrade test we need to have a realistic system, which the docker container environment isn't. We need a systemd to talk to so we can schedule restarts, that sort of thing.
We will need some test-specific configurations (particularly repo URLs) too. And we'll need to be able to orchestrate the testing and fail the build when the tests fail.
We could create an image prior to the testing which would have a user account with passwordless sudo and a .ssh/authorized_keys . The private key would be kept in the [secret variables](https://gitlab.bytemark.co.uk/open-source/symbiosis/settings/ci_cd) section of the project on gitlab, and so would be presented to the gitlab-ci script as an env var.
In the gitlab-ci script we'd start the VM with qemu, as we do for bytemark/bytemark-packer-templates, then use ansible to copy over the tests, install the symbiosis packages, and run the tests. We could write our ansible playbook so that it captures the logs and copies them back to the runner and have the gitlab-ci script spit the logs out, then exit with ansible's exit code.
This would make our test output more readable and shorter, not be quite as weird the current autotest setup on maker2, probably not require also running a DHCP server.
The work we'd need to do:
* add an ansible layer to docker-images/layers
* rewrite the `autotest/` scripts as ansible playbooks
* make a base VM image with the necessary networking & ssh setup
Thoughts @pcherry , @jcarter ?Testing SuitePaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/187Symbiosis: testing sites don't get DocumentRoot set2019-04-14T21:44:45ZPaul CammishSymbiosis: testing sites don't get DocumentRoot setImported from https://www.github.com/BytemarkHosting/symbiosis/issues/43
testing sites - e.g. example.com.testing.server.group.user.uk0.bigv.io don't have DocumentRoot set. Not being set by mod_rewriteImported from https://www.github.com/BytemarkHosting/symbiosis/issues/43
testing sites - e.g. example.com.testing.server.group.user.uk0.bigv.io don't have DocumentRoot set. Not being set by mod_rewritehttps://gitlab.com/sympl.io/sympl/-/issues/333symbiosis_utmp.so fails to load2023-05-02T15:52:19ZPaul Cammishsymbiosis_utmp.so fails to loadsymbiosis_utmp.so is failing to load, possibly because it's currently built with the wrong toolset (bullseye not bookworm).symbiosis_utmp.so is failing to load, possibly because it's currently built with the wrong toolset (bullseye not bookworm).Sympl 12 (bookworm)https://gitlab.com/sympl.io/sympl/-/issues/306Sympl 11: Installing sympl-mysql doesnt write the password to /home/sympl2021-02-15T11:33:44ZPaul CammishSympl 11: Installing sympl-mysql doesnt write the password to /home/symplThis is currently causing the testing to fail, and will need looking into.This is currently causing the testing to fail, and will need looking into.Sympl 11 for Debian BullseyePaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/307Sympl 11: sympl-mail - Update exim configurations (historic)2021-02-12T18:21:16ZPaul CammishSympl 11: sympl-mail - Update exim configurations (historic)Changes to the Exim configuration were needed to pass the existing test suite.
These were done in 7dc9c294 15c8c20f 5a1b47ae 33d97665 6b4fbe1c
See also #304 which is related as it involved a workaround.Changes to the Exim configuration were needed to pass the existing test suite.
These were done in 7dc9c294 15c8c20f 5a1b47ae 33d97665 6b4fbe1c
See also #304 which is related as it involved a workaround.Sympl 11 for Debian Bullseyehttps://gitlab.com/sympl.io/sympl/-/issues/232Sympl determines host name incorrectly during install2022-04-26T09:50:34ZPaul CammishSympl determines host name incorrectly during installDuring the install, sympl creates a 'default' directory based on the hostname of the machine. However, it incorrectly uses the domain 'localdomain' when creating this directory.
On a clean debian machine, the /etc/hostname file contains...During the install, sympl creates a 'default' directory based on the hostname of the machine. However, it incorrectly uses the domain 'localdomain' when creating this directory.
On a clean debian machine, the /etc/hostname file contains a bare hostname. Code in core/debian/postinst uses this file as the hostname, and if it sees a 'bare' hostname, appends 'localdomain' to the hostname read from the file.
The debian installation had a full hostname specified, and typing
hostname -f
retrieves this full host name correctly.
The postinst script will also fall back to using hostname -f if /etc/hostname exists.Sympl v9.0 (for Debian Stretch)Paul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/265sympl-backup triggers `tar` warnings2019-09-17T13:45:19ZPaul Cammishsympl-backup triggers `tar` warningshttps://forum.sympl.host/t/backups-tar-warning-about-non-optional-arguments/44
## Problem Description
When doing backups, the following message is shown, with the backup succeeding:
```
Creating archive using 'DRIVER_TAR_GZ'...
tar:...https://forum.sympl.host/t/backups-tar-warning-about-non-optional-arguments/44
## Problem Description
When doing backups, the following message is shown, with the backup succeeding:
```
Creating archive using 'DRIVER_TAR_GZ'...
tar: The following options were used after any non-optional arguments in archive create or update mode. These options are positional and affect only arguments that follow them. Please, rearrange them properly.
tar: --no-recursion has no effect
tar: Exiting with failure status due to previous errors
Checking TOC of archive file (< real file, > archive entry)...
```
This is due to changes to `tar` in Buster.
Paul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/297sympl-backup: `backup2l -r <regexp>` in Buster only restores directories, and...2020-07-06T12:45:46ZPaul Cammishsympl-backup: `backup2l -r <regexp>` in Buster only restores directories, and no filesFrom: https://forum.sympl.host/t/problem-restoring-with-backup2l/138/7
In short, the 'extract' functionality is missing from the TAR driver for backup2l, meaning it can do everything apart from actually extract the relevant files.
The ...From: https://forum.sympl.host/t/problem-restoring-with-backup2l/138/7
In short, the 'extract' functionality is missing from the TAR driver for backup2l, meaning it can do everything apart from actually extract the relevant files.
The files are backed up okay, but the automatic restore functionality is broken.Paul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/217sympl-backup: Pre/post backup scripts need updating2019-06-13T23:36:45ZPaul Cammishsympl-backup: Pre/post backup scripts need updatingThey do 3 things:
1. Sync a copy of any existing backups from the backup space.
2. Dump MySQL and Postgres(!?) databases, although not particularly well.
3. Sync the result of the backups to the backup space once complete.
This uses the...They do 3 things:
1. Sync a copy of any existing backups from the backup space.
2. Dump MySQL and Postgres(!?) databases, although not particularly well.
3. Sync the result of the backups to the backup space once complete.
This uses the old deprecated Bytemark backup space, determining the destination server via the hostname of the local server, although this can be configured.
It's probably worth replacing the backup sync functionality with a couple of popular options and replacing the SQL dump script with something more modern which doesn't lock tables when dumping.Sympl v9.0 (for Debian Stretch)Paul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/295sympl-cli: running some commands as root doesn't ensure result has the right ...2020-09-09T17:23:53ZPaul Cammishsympl-cli: running some commands as root doesn't ensure result has the right ownerExample: `sudo sympl web create example.com` creates the directory in /srv with the owner as root.
https://forum.sympl.host/t/sympl-cli-feature-discussion/30/8Example: `sudo sympl web create example.com` creates the directory in /srv with the owner as root.
https://forum.sympl.host/t/sympl-cli-feature-discussion/30/8Paul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/318sympl-core: Cross signed Let's Encrypt bundle flags all LE certs as expired2021-10-04T10:11:53ZPaul Cammishsympl-core: Cross signed Let's Encrypt bundle flags all LE certs as expiredThis is caused by the current Ruby codebase which uses the OpenSSL library to build a certificate store, used to validate certificates.
The bundle now includes an extra cert with a copy of the normal intermediate signed by the now-expir...This is caused by the current Ruby codebase which uses the OpenSSL library to build a certificate store, used to validate certificates.
The bundle now includes an extra cert with a copy of the normal intermediate signed by the now-expired DST X3 Root certificate (used as a workaround for old devices which don't have the new X1 root cert), meaning the bundle is effectively signed twice.
This is fine in the vast majority of cases, but in this instance, the presence of an intermediate signed by an expired root raises an error, which then means sympl-ssl.rb considers the whole chain invalid, leading to it retrieving new certs on every run.
A workaround has been put together in sympl-ssl to remove the expired intermediate from the ssl.bundle and ssl.combined when preceded by the normal cert in !243 !244 !245.
Longer-term, the existing sympl-ssl will be replaced by the new version in development.Paul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/271sympl-core: On each install, check the user is in the right groups2020-01-28T00:25:25ZPaul Cammishsympl-core: On each install, check the user is in the right groupsAt the moment, the `sympl` user is only added to the relevant groups (notably www-data) when the user is created, rather than on installation of `sympl-core`.
This can cause some issues if the sympl user already exists (from a removed i...At the moment, the `sympl` user is only added to the relevant groups (notably www-data) when the user is created, rather than on installation of `sympl-core`.
This can cause some issues if the sympl user already exists (from a removed install, or it was created before installing), so it would be safer to check each time `sympl-core` is installed.Paul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/280sympl-core: sympl-filesystem-security breaks access to config/stats-htaccess2020-04-20T10:41:34ZPaul Cammishsympl-core: sympl-filesystem-security breaks access to config/stats-htaccessReported by a user, the `config/stats-htaccess` file has it's permissions reset by `sympl-filesystem-security` to a configuration which prevents access by www-data, and therefore Apache denied all access to example.com/statsReported by a user, the `config/stats-htaccess` file has it's permissions reset by `sympl-filesystem-security` to a configuration which prevents access by www-data, and therefore Apache denied all access to example.com/statsPaul CammishPaul Cammish2020-04-20https://gitlab.com/sympl.io/sympl/-/issues/290sympl-core: sympl-filesystem-security removes +x flag from /etc/sympl/firewal...2020-04-27T17:06:12ZPaul Cammishsympl-core: sympl-filesystem-security removes +x flag from /etc/sympl/firewall/local.d/*The directory contains scripts run at the end of sympl-firewall, which need to be executable, but `sympl-filesystem-security` currently removes that flag.The directory contains scripts run at the end of sympl-firewall, which need to be executable, but `sympl-filesystem-security` currently removes that flag.Paul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/299sympl-core: sympl-filesystem-security reset permissions on public/cgi-bin2020-09-09T17:23:53ZPaul Cammishsympl-core: sympl-filesystem-security reset permissions on public/cgi-binThis causes cgi-bin scripts to fail, and various headaches for anyone with older stuff.This causes cgi-bin scripts to fail, and various headaches for anyone with older stuff.Paul CammishPaul Cammish