================= SFTP, VoIP, tea ================= Yet another personal news digest: on the work-related SFTP and globbing adventures, attempts to get voice conferences working over HTTP, and tea. SFTP and glob ============= Had to implement processing of files from a remote server, available over SFTP. Already had local file processing implemented, so thought SSHFS would be a good fit, with a systemd mount unit. Under the load, it worked for a day, and then the processes disappeared without leaving any logs, even with the "reconnect" option. Not completely unexpected, since this sort of thing happens, especially with services that leave no logs. Then learned that SSHFS is not maintained lately. I also had FTP implemented, with curl, which supports SFTP as well, so decided to go that route instead. Did not have globbing implemented for FTP though, simply used a glob library that operates on local files (while the FTP tasks worked fine with directories, rather than glob patterns), but now it is needed, since the files relevant for those tasks are not neatly placed in dedicated directories. Furthermore, having about a thousand source objects (separate processing tasks), traversing all the files (requesting contents of all the directories in which they may be) for each one is slow, especially over SFTP. So I implemented custom glob pattern matching, with matching over multiple patterns at once (in a single file hierarchy traversal), and using a provided directory listing function, so that it can be used for both local and remote files: ``glob :: Monad m => [(String, a)] -> (String -> m [String]) -> m [(String, a)]`` (the ``a`` type is for tagging of the patterns, and then of the found results). It is left to add a dedicated thread that will run it, and some synchronization to queue and block the worker threads until there are results for them. Not a fan of synchronization between threads in general, but threads are used here without needing synchronization usually, as lightweight processes (since those thousands of tasks, with a process per task, could be quite a bit of overhead, especially given that hundreds have to run simultaneously), I probably would have avoided multithreading anyway, if it was not for Haskell libraries including and aiming blocking functions, and use of multithreading: it is hard and mostly unnecessary to fight that. Feels a bit odd to keep adding functionality like that: I also have crontab and systemd.timer-style schedule string parsing there, for custom scheduling, in-place pangocairo bindings for text rendering, some logging and IPC helpers. Trying to keep the codebase relatively small, but having to implement those seemingly generic things anyway. Actually thought of extending an existing glob library or writing a new one, but then the simplified matching (just character sets and "*" are implemented here) will not suffice, and it is easy to write in-place anyway. Likewise with schedule strings: they may look potentially awkward, but are quite simple. VoIP over HTTP ============== In a post two months ago I wondered about using HTTP for voice conferences in web browsers, to avoid complexity, bugs, and connection issues of WebRTC, and tried it out recently, implementing for that. For the chat implementation, decided to use C (``-std=c89 -Wall -Wextra -pedantic``) with an optional libfcgi dependency, and trying to keep the features minimal, making it usable with simpler web browsers and audio players, since most of the web chat, file sharing, and voice call software I am finding otherwise is the opposite of that. Only supporting Ogg with the Opus codec (which can be handled for such stream relaying even without depending on libogg and libopus, at least in a somewhat hacky and potentially unreliable way: simply resending the first Ogg page, containing the identification Opus packet, and detecting the first page in a stream by looking at the corresponding Ogg page header bit). Initially implemented it in a single multithreaded process with ISO C threads (including corresponding mutexes, condition variables), then rewrote it to use shared memory (shm_open, mmap) and process-shared mutexes and condition variables, with pthreads, then had to debug some FastCGI business (tried to use the program in CGI mode, with fcgi-wrap, which apparently fails to detect client disconnects, with write functions always succeeding) and possible deadlocking, and decided that if it causes trouble (at least complicates debugging, makes it less certain where the issue is) already, it is better to avoid threads and their synchronization, simply using a dedicated server to keep track of the state and organize multiple clients. So rewrote it for the third time, adding a server available over Unix domain sockets with SEQPACKET and a few basic commands, simply passing the C structures over it, not bothering with fancy serialization. In the JS part, implemented the AJAX-based textual chat functionality to add on top of regular JS-free chat, and implemented audio upload: using MediaRecorder to obtain Ogg pages, but had to upload those in separate requests, since Firefox does not support a streaming upload. If it did though, could have simply used Icecast to relay such streams: Icecast's source clients are supposed to provide a single stream over HTTP, with the PUT method, and chunked transfer encoding. Could still add relaying to Icecast into the chat though, but it probably will not be very useful. The Ogg pages are relayed without any synchronization and buffering, and I flush the stream after sending those, so that part is fast. Had to disable nginx's FastCGI buffering (by setting the X-Accel-Buffering header, though it can be done in nginx configuration, too). Yet the delays, when listening to it from Firefox with the