Sympl issueshttps://gitlab.com/sympl.io/sympl/-/issues2019-06-20T22:41:04Zhttps://gitlab.com/sympl.io/sympl/-/issues/233An installer script would be nice...2019-06-20T22:41:04ZPaul CammishAn installer script would be nice...This would allow us to run a single command which would install Sympl and set the relevant option so the user is not prompted at all.
This would also be able to point the user to documentation and make them aware of things like the `sym...This would allow us to run a single command which would install Sympl and set the relevant option so the user is not prompted at all.
This would also be able to point the user to documentation and make them aware of things like the `sympl` user using the `root` users password (which may not be secure) and/or force them to set a new one and include the root users authorized keys file.Sympl v9.0 (for Debian Stretch)Paul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/231sympl-filesystem-security: /srv/example.com/public is not set 27752019-06-12T13:11:10ZPaul Cammishsympl-filesystem-security: /srv/example.com/public is not set 2775Looks like I missed this when I was putting the script together, should be a simple fix:
`find "${domain}/public" ! -type l ! \( -type f ! -perm 664 -exec chmod 664 {} \; -o -type d -perm 2775 -exec chmod 2775 {} \; \)`
sympl-filesyste...Looks like I missed this when I was putting the script together, should be a simple fix:
`find "${domain}/public" ! -type l ! \( -type f ! -perm 664 -exec chmod 664 {} \; -o -type d -perm 2775 -exec chmod 2775 {} \; \)`
sympl-filesystem-security should also check config/ssl/sets exists before trying to do anything with it
Sympl v9.0 (for Debian Stretch)Paul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/230sympl-web: Logs directory is not automatically created2019-06-12T13:11:07ZPaul Cammishsympl-web: Logs directory is not automatically createdThis looks to happen when the directory is not owned by a non-system user, and is likely in `sympl-web-logger`
Adding this to sympl-web-configure in a relevant place should fix it:
```ruby
dirname = File.dirname("#{domain.directory}...This looks to happen when the directory is not owned by a non-system user, and is likely in `sympl-web-logger`
Adding this to sympl-web-configure in a relevant place should fix it:
```ruby
dirname = File.dirname("#{domain.directory}/public/logs/.")
unless File.directory?(dirname)
verbose "\tCReating log directory #{dirname}"
FileUtils.mkdir_p(dirname)
FileUtils.chown_R 'sympl', 'sympl', dirname, :verbose => true
end
```Sympl v9.0 (for Debian Stretch)Paul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/220Web stats are insecure and need updating2019-06-12T13:10:49ZPaul CammishWeb stats are insecure and need updatingIt's unclear if the stats stuff even gets used, as it's not mentioned much in the old Symbiosis docs.
However, some time ago it was supposed to be disabled by default, but that's not the case, so it's automatically generated for each si...It's unclear if the stats stuff even gets used, as it's not mentioned much in the old Symbiosis docs.
However, some time ago it was supposed to be disabled by default, but that's not the case, so it's automatically generated for each site at /stats, and doesn't require any auth at all.
This should either be secured properly, or replaced with something a bit more up to date, like goaccess which has a package and is realtime.Sympl v9.0 (for Debian Stretch)Paul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/222A new 'theres no site here yet' page is needed2019-06-07T10:37:50ZPaul CammishA new 'theres no site here yet' page is neededThe old one had 2012-eta Bytemark branding, but I should be able to do something better - just need a proper logo for Sympl created.The old one had 2012-eta Bytemark branding, but I should be able to do something better - just need a proper logo for Sympl created.Rebranding Symbiosis to SymplPaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/216The auth/helper processes don't seem to be running after a reboot2019-06-02T21:18:53ZPaul CammishThe auth/helper processes don't seem to be running after a rebootspecifically:
```
/usr/sbin/pure-authd --run /usr/sbin/symbiosis-ftpd-check-password --socket /var/run/pure-ftpd/pure-authd.sock
/usr/bin/ruby /usr/sbin/symbiosis-email-poppassd
/usr/bin/ruby /usr/sbin/symbiosis-email-dict-proxy
```specifically:
```
/usr/sbin/pure-authd --run /usr/sbin/symbiosis-ftpd-check-password --socket /var/run/pure-ftpd/pure-authd.sock
/usr/bin/ruby /usr/sbin/symbiosis-email-poppassd
/usr/bin/ruby /usr/sbin/symbiosis-email-dict-proxy
```Rebranding Symbiosis to SymplPaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/215Command line scripts need to be renamed, references to symbiosis in filesyste...2019-06-06T11:01:27ZPaul CammishCommand line scripts need to be renamed, references to symbiosis in filesystem removedWIP in https://gitlab.mythic-beasts.com/sympl/sympl_stretch/merge_requests/19
Any existing command line scripts should have symlinks/helpers created to them from the old names for compatibility.
Ruby Libraries can stay where they are f...WIP in https://gitlab.mythic-beasts.com/sympl/sympl_stretch/merge_requests/19
Any existing command line scripts should have symlinks/helpers created to them from the old names for compatibility.
Ruby Libraries can stay where they are for now (this can be tackled later), but things like /etc/symbiosis and dpkg copies need to be moved/renamed (and symlinks created).
* [x] package:core
* [x] package:backup
* [x] package:common
* [x] package:cron
* [x] package:dns
* [x] package:email
* [x] package:firewall
* [x] package:ftpd
* [x] package:httpd
* [x] package:monit
* [x] package:mysql
* [x] package:phpmyadmin
* [x] package:updater
* [x] package:webmail
----
* [x] update all version numbers
* [x] double-check all copyright filesRebranding Symbiosis to SymplPaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/214Packages needed to be renamed2019-05-31T17:05:57ZPaul CammishPackages needed to be renamedbytemark-symbiosis, symbiosis-* packages needed to be renamed to match the new sympl naming.
bytemark-symbiosis -> sympl-core
symbiosis-* -> sympl-*bytemark-symbiosis, symbiosis-* packages needed to be renamed to match the new sympl naming.
bytemark-symbiosis -> sympl-core
symbiosis-* -> sympl-*Rebranding Symbiosis to SymplPaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/206symbiosis-test skips phpmyadmin tests2019-05-28T11:58:10ZPaul Cammishsymbiosis-test skips phpmyadmin testsIt looks like due to the changes to MariaDB, the tests which expect to log in to phpmyadmin as root/debian-sys-maint are failing.
```
Skipping phpmyadmin debian-sys-maint auth test - password not found.
Skipping phpmyadmin root auth tes...It looks like due to the changes to MariaDB, the tests which expect to log in to phpmyadmin as root/debian-sys-maint are failing.
```
Skipping phpmyadmin debian-sys-maint auth test - password not found.
Skipping phpmyadmin root auth test - password not found.
```
This should be fairly simple to fix to use the generated 'admin' username/password, and ensure the passwordless logins fail.Testing SuitePaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/205"Quota exceeded (mailbox for user is full)"2019-05-28T11:58:11ZPaul Cammish"Quota exceeded (mailbox for user is full)"Symbiosis-test outputs `Quota exceeded (mailbox for user is full)` twice while running. This may be a bug, or it may be operating normally. Either way it should be fixed or supressed.Symbiosis-test outputs `Quota exceeded (mailbox for user is full)` twice while running. This may be a bug, or it may be operating normally. Either way it should be fixed or supressed.Testing SuitePaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/204"Not running MySQL backup tests, since not all the requirements are in place."2019-05-28T11:58:08ZPaul Cammish"Not running MySQL backup tests, since not all the requirements are in place."It looks like the relevant ruby libraries are missing for symbiosis-test from the repo/install (and would have been on the build box), but an attempt to track the relevant version down didn't come up with a perfect match.
This can proba...It looks like the relevant ruby libraries are missing for symbiosis-test from the repo/install (and would have been on the build box), but an attempt to track the relevant version down didn't come up with a perfect match.
This can probably just be rewritten in bash, as it's some simple SQL queries.Testing SuitePaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/227Sympl parser2019-07-05T12:19:35ZPaul CammishSympl parserA basic version of the Sympl parser should be created, covering the most common things:
Creating domains, sites, mailboxes, ftp accounts, etc.
This can then be used in the new documentation, and expanded on further.A basic version of the Sympl parser should be created, covering the most common things:
Creating domains, sites, mailboxes, ftp accounts, etc.
This can then be used in the new documentation, and expanded on further.BacklogPaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/170Symbiosis: Request: Write mysql root credentials to /root/.my.cnf when imaging.2019-06-09T23:33:49ZPaul CammishSymbiosis: Request: Write mysql root credentials to /root/.my.cnf when imaging.Imported from https://www.github.com/BytemarkHosting/symbiosis/issues/56
(Note: This may be something for imager, or Stretch, but applied to Symbiosis *images* only)
It's never clear that the `root`, `admin` and mysql `root@localhost` ...Imported from https://www.github.com/BytemarkHosting/symbiosis/issues/56
(Note: This may be something for imager, or Stretch, but applied to Symbiosis *images* only)
It's never clear that the `root`, `admin` and mysql `root@localhost` users all have the same password in a newly imaged machine, which leads to users likely changing the root/admin passwords like they should, and not making note of the `root@localhost` password we set for mysql.
Simply writing the below to `/root/.my.cnf` (with relevant permissions) would make password recovery simpler, and allow the user to log in directly.
```config
[client]
user=root
password="<example>"
```
There's a small outside risk to this, by keeping it in `/root` would negate most of this, and make things simpler for users.Sympl v9.0 (for Debian Stretch)Paul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/219Administrative user should be named something unique / permissions tidyup2019-06-09T23:34:46ZPaul CammishAdministrative user should be named something unique / permissions tidyupAs is, Symbiosis has an 'admin' user which is used for most functions, and (in theory) some level of security, although this is weakly enforced, and there is a possibility of a name collision.
A better alternative would probably be to h...As is, Symbiosis has an 'admin' user which is used for most functions, and (in theory) some level of security, although this is weakly enforced, and there is a possibility of a name collision.
A better alternative would probably be to have a user called 'sympl' or similar which would own the configs, files and so on, and then it would be safe to chown/chmod items in the config directory to prevent access to things which should be secure.
This looks to be a fairly simple change, and would go with giving the user a proper home directory (/home/sympl) rather than forcing them to use /srv, which becomes untidy quickly, and gives us the opportunity to pre-populate some basic settings (prompt, etc) for ease of use.Sympl v9.0 (for Debian Stretch)Paul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/186Symbiosis: Testing in gitlab2019-04-22T17:57:10ZPaul CammishSymbiosis: Testing in gitlabImported from https://www.github.com/BytemarkHosting/symbiosis/issues/53
At the moment the install / dist-upgrade / upgrade tests get weirdly-far in gitlab-ci then fails. Here's a quick summary of how the tests used to work on maker2 (a...Imported from https://www.github.com/BytemarkHosting/symbiosis/issues/53
At the moment the install / dist-upgrade / upgrade tests get weirdly-far in gitlab-ci then fails. Here's a quick summary of how the tests used to work on maker2 (as I understand it), and then later I'll go into detail on why the tests fail in gitlab-ci
The Current Situation
================
1. autotest sets up a VM using schroot magic i don't fully understand
2. the VM boots up with systemd and all that jazz, uses DHCP & SLAAC to configure its networking, and automatically runs all the scripts in the autotest folder, keeping all the output as a log file. Once done, it shuts down
3. autotest cracks open the VM's filesystem and reads the logfile. Somehow it detects failures and fails if there was a failure then it exits with a nonzero error code so that maker2 knows
Some Feelings About The Current Situation
===============================
Patrick said something about autotest using the console to talk to the tests, and something else much scarier about the VM sshing into the host to run something.
This doesn't work on gitlab-ci, and is also kinda hacky, for a few reasons.
* the scripts in the autotest folder aren't particularly focussed. In addition to actually running tests, they do these and probably others:
* add an admin user
* install all the packages needed by symbiosis from a big list of packages
* install symbiosis
* opening up the filesystem of the VM so you can prod it is pretty gross
On the plus side it works, and it would only take a bit of effort to port the whole schroot setup over to gitlab-ci (but would have to run using a shell runner)
Why the tests fail in gitlab-ci
=====================
When gitlab-ci runs a container it starts bash in the context of the container. Effectively, bash is PID 1 for the container. There's no init-system to talk to to get stuff going. I believe the apt-get install step for some packages starts them using /etc/init.d (probably something about the package detecting a lack of systemd and putting a proper sysvinit script in) which would explain why a lot of the tests actually succeed. BUT SOME OF THEM FAIL, and we should really be doing a much more realistic test than running our symbiosis full-system tests in a docker container that isn't a full symbiosis system.
With that in mind:
A More Realistic Test Proposal
======================
We're still going to want to run symbiosis in a VM, I think. To do a realistic full-install / dist-upgrade test we need to have a realistic system, which the docker container environment isn't. We need a systemd to talk to so we can schedule restarts, that sort of thing.
We will need some test-specific configurations (particularly repo URLs) too. And we'll need to be able to orchestrate the testing and fail the build when the tests fail.
We could create an image prior to the testing which would have a user account with passwordless sudo and a .ssh/authorized_keys . The private key would be kept in the [secret variables](https://gitlab.bytemark.co.uk/open-source/symbiosis/settings/ci_cd) section of the project on gitlab, and so would be presented to the gitlab-ci script as an env var.
In the gitlab-ci script we'd start the VM with qemu, as we do for bytemark/bytemark-packer-templates, then use ansible to copy over the tests, install the symbiosis packages, and run the tests. We could write our ansible playbook so that it captures the logs and copies them back to the runner and have the gitlab-ci script spit the logs out, then exit with ansible's exit code.
This would make our test output more readable and shorter, not be quite as weird the current autotest setup on maker2, probably not require also running a DHCP server.
The work we'd need to do:
* add an ansible layer to docker-images/layers
* rewrite the `autotest/` scripts as ansible playbooks
* make a base VM image with the necessary networking & ssh setup
Thoughts @pcherry , @jcarter ?Testing SuitePaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/201`sympl-ssl` does not support Let's Encrypt v2 API2019-10-30T09:16:52ZPaul Cammish`sympl-ssl` does not support Let's Encrypt v2 APIAt present, as it's using an old Ruby library, `symbiosis-ssl` does not support the updated version of the Let's Encrypt API, meaning that as per [this notice](https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430), it wi...At present, as it's using an old Ruby library, `symbiosis-ssl` does not support the updated version of the Let's Encrypt API, meaning that as per [this notice](https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430), it will begin to stop working in November of 2019 for new installs, and through the next year, slowly stop working.
With this in mind, it would make sense to refactor this element of Sympl into a wrapper around existing Let's Encrypt tools, such as certbot or acmetool, rather than using a third party library, retaining the existing generation of self-signed certs and general cert management.Paul CammishPaul Cammish2019-10-31https://gitlab.com/sympl.io/sympl/-/issues/4Full test suite is needed2019-05-28T11:58:07ZPaul CammishFull test suite is neededThe basics should cover (off the top of my head)...
* backup
* [x] Backups should write sizable data into /var/backup
* [x] MySQL backups should happen okay
* common
* [x] Let's Encrypt certs should be able to be issued okay
* [x] Exi...The basics should cover (off the top of my head)...
* backup
* [x] Backups should write sizable data into /var/backup
* [x] MySQL backups should happen okay
* common
* [x] Let's Encrypt certs should be able to be issued okay
* [x] Existing legacy certs (config/ssl.key etc) should work and override
* cron
* [x] Individual cron jobs should run on each site.
* [x] Normal functional crons should happen as expected.
* dns
* [x] DNS Should be generated as per template, based on changes
* [x] Modified files should not be changed
* doc
* [x] Man pages should load okay for each command
* email
* [x] Email should be accepted and delivered for a specific mailbox
* [x] Email should be forwarded as expected
* [x] Default forwards should work as expected
* [x] Sieve filters should run normally.
* firewall
* [x] Changes to the files in /etc/symbiosis/firewall should be echoed in iptables
* ftpd
* [x] FTP connections should be accepted and end up in the relevant directory
* httpd
* [x] Sites should serve content
* [x] Certs added should enable HTTPS
* [x] Logs should be written to public/logs
* monit
* [x] Stopped services should restart when needed
* [x] Unneeded services should stop when not needed
* mysql
* [x] MySQL connections should operate normally and allow DB creation/changes/deletion.
* phpmyadmin
* [x] Users should be able to log in via basic auth over HTTPS only
* webmail
* [x] Existing users should be able to log in via webmail
* xmpp
* [x] Existing users should be able to log in via XMPPTesting SuitePaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/350sympl-filesystem-security: Play nicer with composer-based setups2024-03-22T16:34:14ZPaul Cammishsympl-filesystem-security: Play nicer with composer-based setupsComposer tends to put things in public/vendor, which it expects to be executable (copmoser itself, drush, etc), and currently `sympl-filesystem-security` resets these permissions.
A simple fix is to just exclude the contents of public/v...Composer tends to put things in public/vendor, which it expects to be executable (copmoser itself, drush, etc), and currently `sympl-filesystem-security` resets these permissions.
A simple fix is to just exclude the contents of public/vendor when we also exclude public/cgi-binPaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/349sympl12 - sympl-php-configure - open_basedir inherits other variables when no...2024-03-15T14:32:24ZPaul Cammishsympl12 - sympl-php-configure - open_basedir inherits other variables when not setWhen `open_basedir` isn't set in an FPM (ie: `disable-php-security` is enabled), it inherits the last setting it had for another site which doe have it set, which will likely break the site.
A workaround for this is to either use a sepa...When `open_basedir` isn't set in an FPM (ie: `disable-php-security` is enabled), it inherits the last setting it had for another site which doe have it set, which will likely break the site.
A workaround for this is to either use a separate pool, or edit the apache config and manually set `open_basedir` to `/`.Paul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/327letsencrypt initialisation uses incorrect e-mail address2023-05-12T15:40:23ZPaul Cammishletsencrypt initialisation uses incorrect e-mail address# Summary
When letsencrypt is initialised, if a second website has already been created, that site's domain is used to register with letsencrypt rather than the system's hostname domain.
# Steps to reproduce
1. Automatically install ...# Summary
When letsencrypt is initialised, if a second website has already been created, that site's domain is used to register with letsencrypt rather than the system's hostname domain.
# Steps to reproduce
1. Automatically install sympl on Debian 11.
2. 'sympl web create banana.DOMAIN'
3. Follow wiki instructions to rename system from localhost.localdomain to apple.DOMAIN
4. 'echo "letsencrypt" > /srv/apple.DOMAIN/config/ssl-provider'
5. 'sudo sympl-ssl --verbose --force $newhost'
# What is the current bug behavior?
When letsencrypt is run for the first time, if a website other than the default one has already been created, the wrong domain is used to register with letsencrypt
# What is the expected correct behavior?
The system hostname domain should be used
# Relevant logs and/or screenshots
```
* Examining certificates for apple.DOMAIN
SSL set 0: The certificate subject is not valid for this domain apple.DOMAIN.
SSL set 0: The certificate subject is not valid for this domain apple.DOMAIN.
No valid certificate sets found.
Fetching a new certificate from LetsEncrypt.
Created new account with email address: root@banana.DOMAIN
Requesting verification for apple.DOMAIN from https://acme-v02.api.letsencrypt.org/directory
Successfully verified apple.DOMAIN
Requesting verification for www.apple.DOMAIN from https://acme-v02.api.letsencrypt.org/directory
!! Unable to verify www.apple.DOMAIN (status: invalid)
!! Check http://www.apple.DOMAIN/.well-known/acme-challenge/V45LrunGXuYPgAU8fnsLSvQDZReL0DemhcFc0Nf0APY works.
Successfully fetched new certificate and created set 1
Rolled over to SSL set 1
```
You can see that while the correct certificate is requested (apple.DOMAIN), the wrong e-mail address (root@banana.DOMAIN) is used to register with letsencrypt.
# Possible fixes
Sorry, no idea.
/cc @kelduum