Prometheus as source for analytics

member · December 1, 2024, 3:03pm

Hello team!

Now I’m trying to connect a new feature, but I ran into a problem. With the Prometheus version 3.0.1 (latest) everything works fine, but with version 2.55.1 the graph is not displayed in the UI.

For the test, I ran docker-compose from the example with image: prom/prometheus:v2.55.1 and got the same effect. Can I somehow fix this myself or will only update to Prometheus 3.0.1?

mark · December 1, 2024, 4:58pm

Hey!

We recently merged a fix that should work with older versions of prometheus. We haven’t yet released it as a new version of Flipt, but will create a new release this coming week.

In the mean time, can you try to use the nightly version of Flipt? I updated the docker-compose file to this:

version: "3"

services:
  prometheus:
    image: prom/prometheus:v2.55.1
    ports:
      - "9090:9090"
    volumes:
      - "./prometheus.yml:/etc/prometheus/prometheus.yml"
    networks:
      - flipt_network

  flipt:
    image: flipt/flipt:nightly
    command: ["./flipt", "--force-migrate"]
    depends_on:
      - prometheus
    ports:
      - "8080:8080"
    networks:
      - flipt_network
    environment:
      - FLIPT_LOG_LEVEL=debug
      - FLIPT_CACHE_ENABLED=true
      - FLIPT_CACHE_BACKEND=memory
      - FLIPT_CACHE_TTL=1m
      - FLIPT_META_TELEMETRY_ENABLED=false
      
  grafana:
    environment:
      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    entrypoint:
      - sh
      - -euc
      - |
        mkdir -p /etc/grafana/provisioning/datasources
        cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
        apiVersion: 1
        datasources:
        - name: Prometheus
          type: prometheus
          access: proxy 
          orgId: 1
          url: http://prometheus:9090
          uid: prometheus
          basicAuth: false
          isDefault: true
          version: 1
          editable: false
        EOF
        /run.sh
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    networks:
      - flipt_network

networks:
  flipt_network:

And it seemed to work using prometheus v2.55.1 and flipt nightly

member · December 1, 2024, 6:31pm

Hey @mark!
Great, I’ll be waiting for the new release.

Checked now with night build:

Version: v1.52.1
Commit: 284e5e6e121e85706126ac07326a628eb8cc5587
Build Date: 2024-12-01T16:53:54Z
Go Version: go1.23.3
OS/Arch: linux/amd64

There is still no analytics in my case (the result is the same as in my screenshot above).

erka · December 2, 2024, 10:39pm

There is no problem with Prometheus v2.55.1. It’s a UI bug. As workaround switch Timezone UTC on/off in Settings. The chart will not be broken after that.

member · December 3, 2024, 10:39am

I tried it and it worked with the https://github.com/flipt-io/flipt/blob/main/examples/analytics/prometheus/docker-compose.yml
image: prom/prometheus:v2.55.1,

but it still doesn’t work for me.

Maybe the problem is on my side, I’ll try to find it

P.S in Grafana everything is ok

andi4000 · December 6, 2024, 4:23pm

Thanks for this! I was scratching my head why is analytics on my v1.52.2 instance not working. Do you have ETA for release?

mark · December 6, 2024, 8:40pm

@andi4000 we just released v1.53.0 that should contain this fix

Please let us know if it resolves the issue!

member · December 7, 2024, 11:01am

Thanks for the update.
For me it still doesn’t work and from this version the graph in Grafana v11.3.1 stopped displaying. In Prometheus itself the numerical metrics are displayed correctly.

member · December 7, 2024, 8:37pm

upd: In Grafana everything is ok, I just checked it again.
It is not displayed only in Flipt analytics.

andi4000 · December 10, 2024, 4:20pm

Thanks for the update! It works on docker compose (examples/analytics/prometheus/docker-compose.yml, edited prometheus to v2.55.1 and flipt to v1.53.1).

But for some reason it’s not yet working on our k8s cluster. I can see data in prometheus, but not at flipt and grafana (with this dashboard)

andi4000 · December 10, 2024, 5:32pm

I noticed this prometheus query here:

github.com

flipt-io/flipt/blob/df4bd3894e214cc4741a6cbd06404c577b75055a/internal/server/analytics/prometheus/client.go#L47


      
          	})
          	if err != nil {
          		return nil, err
          	}
          	promClient := promapi.NewAPI(apiClient)
          	return &client{promClient: promClient, logger: logger}, nil
          }
          
          func (c *client) GetFlagEvaluationsCount(ctx context.Context, req *panalytics.FlagEvaluationsCountRequest) ([]string, []float32, error) {
          	query := fmt.Sprintf(
          		`sum(increase(flipt_evaluations_requests_total{namespace="%s", flag="%s"}[%dm])) or vector(0)`,
          		req.NamespaceKey,
          		req.FlagKey,
          		req.StepMinutes,
          	)
          	r := promapi.Range{
          		Start: req.From.UTC(),
          		End:   req.To.Add(time.Duration(req.StepMinutes) * time.Minute).UTC(),
          		Step:  time.Duration(req.StepMinutes) * time.Minute,
          	}
          	data, warnings, err := c.promClient.QueryRange(ctx, query, r)

But when I check the data points in prometheus, it looks like this:

flipt_evaluations_requests_total{container="flipt", endpoint="http", exported_namespace="team01", flag="foo-my-flag", instance="10.x.x.x:8080", job="flipt", namespace="flipt", otel_scope_name="github.com/flipt-io/flipt", pod="flipt-XXXX-XXXX", service="flipt"}

If I understand this correctly, flipt retrieves zero data because the query is wrong. The namespace key refers to the k8s application namespace, not flipt namespace.

Whereas on local compose setup, the data point looks like this:

flipt_evaluations_requests_total{flag="foo", instance="flipt:8080", job="flipt-metrics", namespace="default", otel_scope_name="github.com/flipt-io/flipt"}

Which makes sense why it works on docker compose, but not in k8s.

Is there something I can do on my end to make this work? Sorry, I’m not very well versed in golang or prometheus setup.

erka · December 10, 2024, 8:56pm

It looks there is no easy way out… Per Prometheus docs

# honor_labels controls how Prometheus handles conflicts between labels that are
# already present in scraped data and labels that Prometheus would attach
# server-side ("job" and "instance" labels, manually configured target
# labels, and labels generated by service discovery implementations).
#
# If honor_labels is set to "true", label conflicts are resolved by keeping label
# values from the scraped data and ignoring the conflicting server-side labels.
#
# If honor_labels is set to "false", label conflicts are resolved by renaming
# conflicting labels in the scraped data to "exported_<original-label>" (for
# example "exported_instance", "exported_job") and then attaching server-side
# labels.
#
# Setting honor_labels to "true" is useful for use cases such as federation and
# scraping the Pushgateway, where all labels specified in the target should be
# preserved.
#
# Note that any globally configured "external_labels" are unaffected by this
# setting. In communication with external systems, they are always applied only
# when a time series does not have a given label yet and are ignored otherwise.
[ honor_labels: <boolean> | default = false ]

What is the best way out for your setup?

andi4000 · December 11, 2024, 10:07am

Thanks for the hint!

I consulted with a colleague and yea that could work. But we’re going to need to create our own ServiceMonitor manifest instead of using the template from flipt’s helm chart, because it’s not exposed there. ~~I’m going to file a PR to expose this in the template~~ → PR link.

Reference for future reader: ServiceMonitor [monitoring.coreos.com/v1] - Monitoring APIs | API reference | OpenShift Container Platform 4.10, look for honorLabels.

But in the long term, isn’t it better for flipt to use other label e.g. flipt_namespace to prevent collision in k8s deployments?

Edit: add PR and docs reference link

andi4000 · December 11, 2024, 2:35pm

Just wanted to confirm that creating own ServiceMonitor with the honorLabels flag works in our k8s setup

Edit: with PR above merged, the solution now is just to override the honorLabels to true flag in flipt’s official helm chart.

member · December 11, 2024, 7:49pm

Thanks for the suggested solution. It worked with this fix.

mark · December 13, 2024, 6:58pm

Hey @andi4000 ! Thanks for the contributions to the helm chart re: ServiceMonitor and honor labels. (And for coming back to update this post!)

I agree that we should probably deprecate namespace and move to more differentiated attributes.

This is currently what we have in code:

// Attributes used in evaluation metrics
AttributeMatch     = attribute.Key("match")
AttributeFlag      = attribute.Key("flag")
AttributeSegment   = attribute.Key("segment")
AttributeReason    = attribute.Key("reason")
AttributeValue     = attribute.Key("value")
AttributeNamespace = attribute.Key("namespace")
AttributeType      = attribute.Key("type")

We should likely add new attributes like:

// Attributes used in evaluation metrics
AttributeMatch     = attribute.Key("match")
AttributeNamespace = attribute.Key("flag_namespace")
AttributeFlag      = attribute.Key("flag_key")
AttributeFlagType  = attribute.Key("flag_type")
AttributeSegment   = attribute.Key("segment_key")
AttributeReason    = attribute.Key("reason")
AttributeValue     = attribute.Key("value")

Topic		Replies	Views
React app fetching failure with default configs Open Source	2	49	October 4, 2024
Using flipt authentication Open Source	1	64	June 22, 2024
Connecting to postgres failing Open Source	5	37	October 21, 2024
Authorization configuration Open Source	26	107	November 15, 2024
Multiple Flipt instances are not working properly Open Source	1	31	November 14, 2024

Prometheus as source for analytics

Related topics