Online service provision

Here are some notes on provision of online services such as email, XMPP, shell accounts, DVCS and general file/rsync hosting, etc; the focus is on properly set software and decent public services. I don't have much of experience with public ones, but the notes are mostly on technologies rather than practices, aiming primarily GNU/Linux systems. For private ones, see the notes on private server setup.

Usually service providers are obliged to assist governments with surveillance and/or censorship, and possibly to follow additional laws on user information handling. Which is not necessarily bad, but worse in some cases than in others (that is, getting servers confiscated, engaging into mass surveillance and/or censorship, and/or setting backdoors to enforce laws that don't make sense would be less desirable than just rarely helping with actual crime investigations, once warrant is provided and targeting individual users), so this should be investigated. Apparently the corresponding Russian law is such that it's better to keep services as far away from it as possible. Estonia provides "e-residency", which possibly may help to provide services under its laws.

Although perhaps even with an oppressive government it's possible (acceptable) to provide a service to those whom it doesn't affect (i.e., pretty much anyone outside), while following all the regulations, and presenting clearly what they are, as opposed to the common practices of not mentioning it at all, being security- and privacy-oriented but under the radar, in a grey area, and/or (partially) blocked. or boasting security and privacy while in fact following regulations opposing those. A slightly sarcastic presentation picturing benevolent supervisors providing a useful service by filtering "extremist content" and suchlike may also be quite fun. Among features, in place of strict privacy laws it could list some of the local nonsense, but from the point of view of a hypothetical happy citizen (though in order to keep it light, will have to pick something that sounds silly/peculiar, yet not particularly bigoted). Maybe also presenting uncertainty and instability as excitement. Although as of 2021 and in Russia, since many foreign mail servers are being blocked, it seems that delivery failure rate would be unacceptable for a mail service. And then 2022 with Russia's "special operation" happened, at which point having anything to do with Russia became an edgy choice in much of the world.

A relevant discussion (though there's probably plenty more around): "Ask HN: What is the best jurisdiction for internationally distributed teams?".

An user agreement should be prepared carefully, yet be readable.

Payment processors tend to be an issue as well, though some of their issues are just inherited from the bank cards (and most of the others – from trying to mitigate those with fraud detection). The options (e.g., PayPal) are bad, but they work sometimes, more or less.

Service abuse is what brings up some of the legal issues (and even when it doesn't, it's highly undesirable), but apparently it can be mitigated by requiring a small payment for confirmation, which is straightforward with regular bills, but viable with donations as well (e.g., as sdf.org does).

SSH

SSH is one of the most widespread protocols with good authentication and software implementations, useful for both regular shell accounts and the ones restricted to provide specific functionality (email and DVCS, for instance), needed for pubnix-style systems.

System user restrictions

Better isolation and restrictions than regular file permissions are desirable in systems shared among strangers. Some of the ways to set such restrictions can be observed in the hashbang/shell-server's "security" task, and here is the list I have collected:

PAM limits and namespaces
OpenSSH's sshd(8) can (and does by default on Debian, see sshd_config(5)) use pam(8), including session management modules such as pam_limits(8) (which sets ulimit and nice, see limits.conf(5)) and pam_namespace(8) (which sets polydirs such as per-user tmp directories, see namespace.conf(5)). These are user-space and not necessarily reliable, "PAM escape" via certain programs is possible -- so those should be limited too.
File systems
Some (pseudo-)filesystem mounting options are useful for access restrictions: hidepid=2 for proc(5), newinstance for devpts (documented in mount(8)), etc. Mounting /tmp/ into memory and avoiding swap can be useful for both performance and security. Disk partition encryption with LUKS/dm-crypt would also be useful to reduce the risk of compromising user data, though that applies to computing in general.
cgroups
With systemd, systemctl(1) can be used to set those (see systemd.resource-control(5)) and limit resource usage for PAM sessions. I wonder why hashbang.sh only seems to set that for non-interactive sessions.
iptables
Among iptables-extensions(8) there's the owner extension, which allows to match outbound packets on local users and groups. This seems useful for limiting user network capabilities without limiting system services.
sshd
An SSH daemon itself should be configured to disable SSH functionality that isn't desirable, such as TCP forwarding.
Other hardening
There are hardening guidelines around, which tend to include both restrictions and additional security measures. For instance, hashbang/hardening, "Hardening" on Debian Wiki.

Specific functionality

For more restricted services, there may be no need in shell access, or in system users altogether, but other SSH uses may still be desired. There are SSH server libraries (e.g., libssh, a Haskell ssh library; libssh2 may be better to avoid, with its rather bad track record and regularly found vulnerabilities) for that, and many per-key restrictions can be defined in authorized_keys files or encoded into certificates with OpenSSH (see sshd(8) for the documentation), including command restrictions. It may be too restrictive for some programs (where the arguments should be dynamic), but wrappers could be used for those.

Gitea, for instance, forces execution of its own command (via command in ~git/.ssh/authorized_keys for each added user), and disallows everything but command execution (as used by git), manually ensuring that commands are git ones, and checking repository access privileges using its own rules.

Non-SSH authentication

PAM authentication may be nice to reuse for everything (possibly via SASL), especially if shell access is provided, but unfortunately it's only usable for plaintext authentication.

SASL is nice for uniform authentication across services. It's usually not tied to system users, though could be used with LDAP (and so can PAM). See the "user authentication" note for more on the topic.

User directory

To detach users from the underlying operating system, possibly using a shared user directory across multiple servers, LDAP is a common option.

Backups

Applicability of different methods depends on the kinds of data stored.

Scaling

A decent service shouldn't trap users, so horizontal scaling should be as easy as setting identical systems, relying on federated protocols for interoperation. Configuration management systems such as Ansible are useful for that.

DoS protection

It's rather hard to be certain that a complex system would function properly under unexpected loads. Stress testing should be performed, and other iptables extensions could be useful here, such as hashlimit to set per-IP limits.