I’m pretty new to selfhosting, but one thing that I know to take seriously is log collection. Since there are a lot of different type of logs (kernel log, application logs, etc) and logs come in many different formats (binary, json, strings) - it’s no easy task to collect them centrally and look through them whenever neccessarly.
I’ve looked at grafana and tried the agent briefly, but it wasn’t as easy as I thought (and it might be a too big tool for my needs). So I thought to ask the linuxlemmy community to get some inspiration.
Syslog (rsyslod) is usually the standard answer for the average sysadmin, but it depends a lot on your needs. A lot of newer loggers output as pure JSON, which offer benefits to readability and more approachable search logic/filters/queries (I’m so tired of regex).
When you start venturing down the road of finding the right way to store and forward the output of logging drivers from Docker containers, as one example, rsyslod starts to feel dated.
The easy answers if you want to throw money at the problem are solutions like Splunk, Datadog, or New Relic. If you don’t want to (and most people wouldn’t), then alternatives certainly exist, but some of them are just as heavy on system resources. Greylog has relative feature parity with Splunk Enterprise, but consumes just as much compute and storage if not more, and I found it to be a much larger pain in the butt to administer and keep running.
The likeliest answer to this problem is Grafana Loki, just based on what I’ve read of its capabilities, but I haven’t had a chance to circle back and test it out. Someone here who has might be able to weigh in and speak to its strengths/weaknesses.