Online service provision

Here are some notes on provision of online services such as email, XMPP, shell accounts, DVCS and general file/rsync hosting, etc; the focus is on properly set software and decent public services. I don't have much of experience with public ones, but the notes are mostly on technologies rather than practices, aiming primarily GNU/Linux systems.

Legal and financial matters, service abuse

Usually service providers are obliged to assist governments with surveillance and/or censorship, and possibly to follow additional laws on user information handling. Which is not necessarily bad, but worse in some cases than in others (that is, getting servers confiscated, engaging into mass surveillance and/or censorship, and/or setting backdoors to enforce laws that don't make sense would be less desirable than just rarely helping with actual crime investigations, once warrant is provided and targeting individual users), so this should be investigated. Apparently the corresponding Russian law is such that it's better to keep services as far away from it as possible (the example with backdoors, mass surveillance, and censorship describes it). Estonia provides "e-residency", which possibly may help to provide services under its laws.

An user agreement should be prepared carefully, yet be readable.

Payment processors tend to be an issue as well, though some of their issues are just inherited from the bank cards (and most of the others – from trying to mitigate those with fraud detection). The options (e.g., PayPal) are bad, but they work sometimes, more or less.

Service abuse is what brings up some of the legal issues (and even when it doesn't, it's highly undesirable), but apparently it can be mitigated by requiring a small payment for confirmation, which is straightforward with regular bills, but viable with donations as well (e.g., as sdf.org does).

SSH

SSH is one of the most widespread protocols with good authentication and software implementations, useful for both regular shell accounts and the ones restricted to provide specific functionality (email and DVCS, for instance).

System user restrictions

Better isolation and restrictions than regular file permissions are desirable in systems shared among strangers. Some of the ways to set such restrictions can be observed in the hashbang/shell-server's "security" task, and here is the list I have collected:

PAM limits and namespaces
OpenSSH's sshd(8) can (and does by default on Debian, see sshd_config(5)) use pam(8), including session management modules such as pam_limits(8) (which sets ulimit and nice, see limits.conf(5)) and pam_namespace(8) (which sets polydirs such as per-user tmp directories, see namespace.conf(5)). These are user-space and not necessarily reliable, "PAM escape" via certain programs is possible -- so those should be limited too.
File systems
Some (pseudo-)filesystem mounting options are useful for access restrictions: hidepid=2 for proc(5), newinstance for devpts (documented in mount(8)), etc. Mounting /tmp/ into memory and avoiding swap can be useful for both performance and security. Disk partition encryption with LUKS/dm-crypt would also be useful to reduce the risk of compromising user data, though that applies to computing in general.
cgroups
With systemd, systemctl(1) can be used to set those (see systemd.resource-control(5)) and limit resource usage for PAM sessions. I wonder why hashbang.sh only seems to set that for non-interactive sessions.
iptables
Among iptables-extensions(8) there's the owner extension, which allows to match outbound packets on local users and groups. This seems useful for limiting user network capabilities without limiting system services.
sshd
An SSH daemon itself should be configured to disable SSH functionality that isn't desirable, such as TCP forwarding.
Other hardening
There are hardening guidelines around, which tend to include both restrictions and additional security measures. For instance, hashbang/hardening, "Hardening" on Debian Wiki.

Specific functionality

For more restricted services, there may be no need in shell access, or in system users altogether, but other SSH uses may still be desired. There are SSH server libraries (e.g., libssh, a Haskell ssh library; libssh2 may be better to avoid, with its rather bad track record and regularly found vulnerabilities) for that, and many per-key restrictions can be defined in authorized_keys files or encoded into certificates with OpenSSH (see sshd(8) for the documentation), including command restrictions. It may be too restrictive for some programs (where the arguments should be dynamic), but wrappers could be used for those.

Non-SSH authentication

PAM authentication may be nice to reuse for everything (possibly via SASL), especially if shell access is provided, but unfortunately it's only usable for plaintext authentication.

SASL is nice for uniform authentication across services. It's usually not tied to system users, though could be used with LDAP (and so can PAM). See the "user authentication" note for more on the topic.

Backups

Applicability of different methods depends on the kinds of data stored.

Scaling

A decent service shouldn't trap users, so horizontal scaling should be as easy as setting identical systems, relying on federated protocols for interoperation. Configuration management systems such as Ansible are useful for that.

DoS protection

It's rather hard to be certain that a complex system would function properly under unexpected loads. Stress testing should be performed, and other iptables extensions could be useful here, such as hashlimit to set per-IP limits.