* Package name : loki-database
Version : 2.1.0-1
Upstream Author : Grafana Labs
* URL : https://github.com/grafana/loki
* License : AGPLv3
Programming Lang: Go
Description : Like Prometheus, but for logs.
Loki is a horizontally-scalable, highly-available, multi-tenant log
aggregation system inspired by Prometheus (https://prometheus.io/). It
is designed to be very cost effective and easy to operate. It does not
index the contents of the logs, but rather a set of labels for each log
stream.
Compared to other log aggregation systems, Loki:
- does not do full text indexing on logs. By storing compressed,
unstructured logs and only indexing metadata, Loki is simpler to
operate and cheaper to run.
- indexes and groups log streams using the same labels you’re already
using with Prometheus, enabling you to seamlessly switch between
metrics and logs using the same labels that you’re already using with
Prometheus.
- is an especially good fit for storing Kubernetes
(https://kubernetes.io/) Pod logs. Metadata such as Pod labels is
automatically scraped and indexed.
- has native support in Grafana (needs Grafana v6.0).
A Loki-based logging stack consists of 3 components:
- promtail is the agent, responsible for gathering logs and sending them
to Loki.
- loki is the main server, responsible for storing logs and processing
queries.
- Grafana (https://github.com/grafana/grafana) for querying and
displaying the logs.
Loki is like Prometheus, but for logs: we prefer a multidimensional
label-based approach to indexing, and want a single-binary, easy to
operate system with no dependencies. Loki differs from Prometheus by
focusing on logs instead of metrics, and delivering logs via push,
instead of pull.
The upstream package name conflicts with the existing package loki
("MCMC linkage analysis on general pedigrees"). However, that package is
"dead upstream" (according to debian/watch), so perhaps this package can
get the name eventually. Name suggestions are appreciated.
As hinted by the above excerpt from the readme, I plan to package
promtail as a separate binary package. Grafana itself live in another
source package and will be a separate effort.
Since Loki is a relatively generic name used for many different things, personally I think no one package should use the unqualified name. For example the Loki C++ library distinguishes itself as loki-lib. The name loki-database seems a good choice for the software you are packaging. https://en.wikipedia.org/wiki/Loki_(disambiguation) https://en.wikipedia.org/wiki/Loki_(C++) Since Grafana was in Debian before and was removed due to being orphaned, outdated and RC-buggy, please note the extra steps required when reintroducing packages; base your new package on the latest version of the old package (such as from the old now archived VCS), unarchive/reopen and triage bugs closed by the removal and reopen and triage security issues closed by the removal. https://tracker.debian.org/pkg/grafana https://tracker.debian.org/news/994097/removed-260dfsg-3-from-unstable/ https://bugs.debian.org/909592 https://alioth-archive.debian.org/git/collab-maint/grafana.git.tar.xz https://bugs.debian.org/cgi-bin/pkgreport.cgi?archive=both;src=grafana https://www.debian.org/doc/manuals/developers-reference/pkgs.html#reintroducing-pkgs
I will try and do that in the future. I wasn't sure how to add a custom header in my email client, but I think I have it figured out. OK, thanks for the feedback. I will keep that in mind. But I was not planning to package grafana any time soon (perhaps I should have used "would" and not "will"). I think I have my hands full with Loki. Actually, I would view getting just promrus packaged as a good starting point, since it should be useful on its own (as you can run Loki on a non-debian server or use the cloud version).
I should point out that there's already a separate RFP for repackaging Grafana in Debian, as #923872. It's a challenge to package, to say the least, but I think it's such a useful package that it should be properly packaged. Hopefully, however, it should not *block* loki's packaging. We should be able to install loki from Debian packages without requiring us to also package Grafana: the web UI could be run from Docker or upstream packages or whatever... And hopefully loki may be easier to package than Grafana (which doesn't say much), but looking at the dependency list makes me run screaming in horror: https://github.com/grafana/loki/blob/main/go.mod 200+ deps, a dozen "replace" (which, in golang world, means "a fork of an upstream library we vendor in here", basically). From that perspective, it's actually worse than Grafana, which has less dependencies, believe it or not. But Grafana is also a webapp which is where it really hurt us, because there you also need to package 300+ nodejs dependencies (!). At least loki spares us that horror. Thanks for working on that difficult package, and let us know of your progress! a.
On 2022-03-31 10:06:40, Antoine Beaupré wrote: [...] For what it's worth, I looked at those deps and it seems we're missing about 28 immediate dependencies (which themselves might have their own missing deps). I tried to run `dh-make-golang estimate` but it fails to do a report, which is not reassuring. It also seems like there is a handful of kubernetes stuff in there which could also be a problem considering the approach *that* package has taken regarding packaging (which is to vendor all that shit in, so we can't reuse their work). It could be possible to strip some of that code out though, maybe we don't need a loki kubernetes operator in Debian for example. But I haven't actually checked why that code is there at all, nor how easy such a change would be. Anyways, here's the dump: $ dh-make-golang make github.com/grafana/loki [...] 2022/03/31 10:14:45 Build-Dependency "sigs.k8s.io/controller-runtime" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "k8s.io/apimachinery" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "github.com/thanos-io/thanos" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "github.com/aws/aws-lambda-go" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "github.com/grafana/dskit" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "github.com/cristalhq/hedgedhttp" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "github.com/cloudflare/cloudflare-go" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "github.com/fluent/fluent-bit-go" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "gopkg.in/Graylog2/go-gelf.v2" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "inet.af/netaddr" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "github.com/weaveworks/common" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "github.com/uber/jaeger-client-go" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "github.com/sony/gobreaker" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "github.com/drone/envsubst" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "github.com/joncrlsn/dque" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "github.com/docker/go-plugins-helpers" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "k8s.io/client-go" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "github.com/ViaQ/logerr" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "github.com/c2h5oh/datasize" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "gopkg.in/fsnotify.v1" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "k8s.io/api" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "github.com/prometheus-operator/prometheus-operator" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "github.com/shurcooL/vfsgen" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "github.com/Workiva/go-datastructures" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "github.com/prometheus/prometheus" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "github.com/fsouza/fake-gcs-server" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "github.com/grafana/regexp" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control 2022/03/31 10:14:45 Build-Dependency "github.com/aws/aws-sdk-go-v2" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control A.
Hi Antoine, I've packaged around 20 dependencies so far. I have yet to submit them (primarily because I find doing this sort of work sucks the energy out of me). I did not quite realize what I was getting myself into :) My TODO list follows. Packages with a # are done, those with a ! have an error, and I forget what - means. loki #github.com/Workiva/go-datastructures #github.com/joncrlsn/dque #github.com/c2h5oh/datasize #github.com/shurcooL/vfsgen github.com/prometheus/prometheus github.com/cortexproject/cortex !github.com/docker/go-plugins-helpers #inet.af/netaddr github.com/thanos-io/thanos #github.com/drone/envsubst gopkg.in/fsnotify.v1 #github.com/sony/gobreaker #github.com/uber/jaeger-client-go #github.com/aws/aws-lambda-go #github.com/fluent/fluent-bit-go #github.com/fsouza/fake-gcs-server #github.com/grafana/dskit #github.com/weaveworks/common cortex #github.com/weaveworks/common github.com/golang-migrate/migrate github.com/thanos-io/thanos github.com/prometheus/prometheus github.com/Masterminds/squirrel #github.com/uber/jaeger-client-go #github.com/sony/gobreaker github.com/grafana/dskit github.com/fsouza/fake-gcs-server netaddr #go4.org/intern intern #go4.org/unsafe/assume-no-moving-gc thanos github.com/lovoo/gcloud-opentracing github.com/fatih/structtag github.com/lightstep/lightstep-tracer-go github.com/cortexproject/cortex #github.com/weaveworks/common github.com/efficientgo/e2e github.com/chromedp/chromedp go.elastic.co/apm github.com/chromedp/cdproto github.com/grafana/dskit gopkg.in/fsnotify.v1 #github.com/uber/jaeger-lib github.com/efficientgo/tools github.com/prometheus/prometheus github.com/baidubce/bce-sdk-go github.com/tencentyun/cos-go-sdk-v5 #github.com/uber/jaeger-client-go jaeger #github.com/uber/jaeger-lib dskit ^go.etcd.io/etcd #github.com/weaveworks/common weaveworks #github.com/sercand/kuberesolver #github.com/felixge/httpsnoop #github.com/weaveworks/promrus #github.com/uber/jaeger-client-go #github.com/uber/jaeger-lib go-kit #github.com/VividCortex/gohistogram #github.com/aws/aws-sdk-go #github.com/influxdata/influxdb #github.com/performancecopilot/speed #github.com/prometheus/client_golang migrate modernc.org/sqlite modernc.org/ql github.com/neo4j/neo4j-go-driver github.com/nakagami/firebirdsql github.com/jackc/pgconn github.com/markbates/pkger github.com/snowflakedb/gosnowflake github.com/ktrysmt/go-bitbucket github.com/dhui/dktest github.com/mutecomm/go-sqlcipher github.com/fsouza/fake-gcs-server github.com/ClickHouse/clickhouse-go github.com/gobuffalo/here github.com/jackc/pgerrcode squirrel: github.com/lann/builder etcd: github.com/form3tech-oss/jwt-go #github.com/grpc-ecosystem/go-grpc-middleware/logging/settable go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc #go.opentelemetry.io/otel -gopkg.in/natefinch/lumberjack.v2 github.com/cockroachdb/datadriven github.com/etcd-io/gofail/runtime I can make an effort to submit the ones I have packaged, but I haven't worked on this for a bit (as other projects took priority)
[...] Impressive! Good job! That must be indeed *so* hard! I think it's worth at least pushing what you have to salsa, so that someone else can carry the torch from where you stopped, if you get tired of this. Who knows, another golang packager might also need the same dep and help you! Really nice, a few comments inline... surely that is packaged somehow. we have the `prometheus` binary package already in Debian, maybe that could be reused? oh wow, they link against cortex! that's massive right there... i think i commented out the code linking to this in some other golang package, and i can't remember which one it is right now. That is probably as unhelpful as it sounds. :p wtf, they link to etcd too.. wth is this thing. that is certainly used by k8s. that one is massive. i started using it for wallabako (#893648) and it brought in a ridiculous number of deps. but it's faster and more portable than the other sqlite lib I was using before, so that would be really nice to have in Debian. ... if a little bonkers (that thing basically transpiles sqlite from C to golang). that's another one I used in wallabako, probably worth packaging in debian in general, just because a bunch of golang packages are likely to use it as well. That's absolutely fine, I think, and it would be really awesome if you could push what you have somewhere already, so that if someone wants to resume the work on loki, they can start from there. And, alternatively, that work would be useful for other packaging efforts, I am sure. Thank you so much! a.
Sorry, # is more like "I packaged it or someone else has" (e.g. no more work necessary) Basically I looked at this and thought "OK that's 20 or so first order deps, I can do this" and then I discovered that several of these had 20 first-order deps of their own. Well, as a first step, I requested access to the golang package group, but it looks like it was never approved. My username is butterball_geometry.
[...] Typical dependency hell. I'm so sorry for your loss of innocence. :p I just approved you.
OK, I uploaded a package to https://salsa.debian.org/go-team/packages/golang-github-c2h5oh-datasize/ Can you have a quick look at it? I used dh-make-golang, but if there's anything I need to tweak I'd like to know before I start uploading. And should I send ITPs for everything, or is it OK to leave the closes as TODO in the changelog?
This looks good to me. Did you ask upstream for tags? Sometimes it actually works! :) Typically I like to fire ITPs for everything, because the roundtrip with NEW can be slow at times. It might also give a heads up for the FTP team of the relationships between the packages, which might not be obvious from the names.
I did not. I'm going to start by focusing on uploading my past work. Sending these out might be the slowest part of all this >.> And what's the proper way to update an existing package? Should I create a PR? Or should I just push directly? (what's the review process like)
no problem. How so? Do you have to copy-paste those in an MTA? :) I found that setting up a local MTA helps tremendously in dealing with those kind of problems, as you can just pipe things around easily... Since this is a 25+ year old project, there's probably a dozen ways to do this. The canonical way is to open a bug report with a patch in the BTS (bugtracker we're talking through here). If that doesn't get a response, in theory, you're allowed to do an NMU. If that doesn't work, you're allowed, again in theory, to salvage the package. And then there's salsa where you can send merge requests. But sometimes those get ignored by graybeards like me who forget it exists or forget to setup notifications, so the BTS is always a safer bet. All those acronyms are expanded somewhere in this article I just wrote about salvaging packages, by pure coincidence (i swear): https://anarc.at/blog/2022-03-31-first-package-salvaged/ Let me know if you have any further question
Sie sind einer der glücklichen Nutznießer der 150 Millionen Dollar schweren Wohltätigkeitsspendeaktion von Elon Musk. Kontaktieren Sie uns (muskfoundation@zohomail.com), um einen Anspruch geltend zu machen.