feat: add named component templates for pipeline sources, transforms, and sinks

[sakateka]

Hi! Thanks for the great project - we use it actively. As the number of pipelines grows, managing configs becomes painful: changing a shared default (e.g. a sink) means editing every pipeline.

I explored a templating approach - not sure it’s the best possible design, but I don’t see a cleaner option yet. Would love your feedback.

Proposal: named component templates on the Vector CR

Platform admins define reusable component bodies once on Vector / VectorAggregator via spec.agent.componentTemplates (also available on aggregators through VectorCommon):

• sources, transforms, sinks - maps of template name -> Go text/template body
• rendered output must be valid YAML for a single Vector component

Pipelines reference a template explicitly:

• sourceTemplate / transformTemplate / sinkTemplate: { name, params }
• template params are available as {{ .paramName }} with strict policy missing key => error
• a component is either inline (type: …) or templated - both together is rejected
• the template replaces the whole component (no merge with inline fields)
• for transforms/sinks, inputs from the pipeline override whatever the template rendered

Example - platform defines the sink once (batching, headers, encoding):

spec:
  agent:
    componentTemplates:
      sinks:
        opentelemetry-http: |
          type: opentelemetry
          protocol: http
          uri: "{{ .uri }}"
          method: post
          encoding:
            codec: json
          compression: gzip
          request:
            headers:
              X-Tenant-ID: "{{ .tenant }}"
            timeout_secs: 30
          batch:
            max_events: {{ or (index . "maxEvents") 500 }}
            timeout_secs: 5
          buffer:
            type: disk
            max_size: 268435456
          acknowledgements:
            enabled: true

Each pipeline only passes what actually differs:

sinks:
  app-logs:
    inputs: [parse-logs]
    sinkTemplate:
      name: opentelemetry-http
      params:
        uri: https://otel-prod.example/v1/logs
        tenant: team-payments


  audit-logs:
    inputs: [audit-transform]
    sinkTemplate:
      name: opentelemetry-http
      params:
        uri: https://otel-prod.example/v1/logs
        tenant: team-security
        maxEvents: 100

Change batch size, TLS, headers, or compression once on the Vector CR - all pipelines pick it up on the next reconcile. Pipelines without template refs behave unchanged.

Applied during config build for both agent and aggregator paths.

I haven’t used this code in production or on a test/staging environment yet.

[dkhachyan]

Hi @sakateka!
Thanks a lot for this PR.
One question I had while reading through this: what's the motivation for implementing templating at the controller/CRD level rather than handling this kind of reuse at the deployment layer, e.g. via Helm templating of the VectorPipeline/Vector manifests themselves?

[sakateka]

what's the motivation for implementing templating at the controller/CRD level

Fair question. The motivation originally came from the fact that each of our users has their own chart and has to define a VectorPipeline themselves. Since the VectorPipeline resource is quite large, this led to a lot of duplication across application charts.

However, I now realize that this does not prevent us from solving the problem through templating at the Helm chart level, for example by using a chart dependency that provides the shared VectorPipeline template.

At this point, I think that approach is the right one, so I’m leaning toward dropping this PR.

Thanks for pointing me in that direction!