chore: added experimental OTLP trace exporter support

[abhilash-sivan]

Summary

Introduces the Instana-to-OTLP exporter.

• Adds support for exporting traces via OTLP (4318/v1/traces) with agent.
• Supports OpenTelemetry Semantic Conventions (SemConv) v1.23 to v1.41.1.
• Protocol: http/json

Out of Scope

• Metrics export
• Logs export
• Other protocols like http/protobuf or grpc
• Extended configuration options
• Serverless OTLP endpoint
• Events (exception)

TODO

Core Implementation

• Handle agent port usage (data port vs. connection port) (chore: make agent data port configurable independently from the agent connection port #2559)
• Support multiple span data keys (e.g., span.data.peer, span.data.mongo)
• Finalize overall design
• Implement SemConv lookup handling
• Implement span name mapping
• Complete metadata attribute mapping:
• Direct mappings
• Computed mappings
• Research span status mapping
• Research span name mapping (@abhilash-sivan)
• Implement OpenTelemetry instrumentation-specific handling
• Implement log-specific handling
• Implement resource mapping
• Implement required metrics mappings
• Implement agent connection logic
• Basic Error handling @abhilash-sivan
• Resource-service name mapping
• Dual transformation flow in span data @abhilash-sivan
• Implement configuration option for enabling OTLP export
• agent data port issue
• normalize HTTP client span attributes reafctor: normalized HTTP client span attributes #2617
• span status + http error 400>
• general cleanup coding
• Fix type definitions
• Pipeline green
• remove example apps

Span Attribute Mapping @abhilash-sivan

• HTTP
• Kafka
• aws sqs
• nats
• nats streaming
• bull
• ampq (rabbitmq)
• SNS
• GCPS (Google Cloud Pub/Sub)
• PG (PostgreSQL)
• MYSQL
• MSSQL
• DB2
• MONGO
• REDIS
• COUCHBASE
• ELASTICSEARCH
• DYNAMODB
• MEMCACHED
• PRISMA
• RPC
• GRAPHQL
• GCS (Google Cloud Storage)
• aws S3
• aws KINESIS
• AZSTORAGE (Azure Storage)
• AWS_LAMBDA_INVOKE
• PEER

Testing

Unit Tests

• config module
• resource module
• metrics module
• trace module
• Single-span scenarios
• Multi-span scenarios
• integration tests in collector - agent stub needs support otlp
• SpanBuffer test
• agentconnection
• different semconv version test

End-to-End Testing (@abhilash-sivan)

• SemConv v1.23 validation with the current design
• SemConv v1.41 compatibility validation
• OpenTelemetry wrapper instrumentation testing
• Metadata validation with SemConv v1.23 and span data validation with v1.41
• without metrics - corelation issue

Performance Testing

• Performance testing

Issue Tracking

• UI Issues: https://ibm.box.com/s/0v5wli0pgvrlg8jqpkhi59mb52o28u75
• Results: https://ibm.box.com/s/0v5wli0pgvrlg8jqpkhi59mb52o28u75

Validation Checklist

Before merging, verify the following:

• Traces are successfully exported through the OTLP endpoint (4318/v1/traces).
• The UI correctly identifies and displays applications and attributes.

Future Enhancements/TODO

• Support the agent announcement cycle disablement use case.
• Evaluate exporting logs to the OTLP /v1/logs endpoint.
• Implement metrics mapping.
• Other protocol supports (grpc support, http/protobuf)
• Extended config options like otlp endpoint,protocol etc
• serverless endpoint support
• Events mapping, detailed error + exception + stack trace
• Batching limits? how much spans send in a single otlp request, size restrictions
• Parallel processing of spans in converter.js

References

• https://opentelemetry.io/blog/2023/http-conventions-declared-stable/
• https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/trace/v1/trace.proto

[kirrg001]

test: send traces via otlp port and metrics via standard port - Idenfies 2 diff apps 1. Otel and 2. Node.js

This needs to be communicated in the next WG. Traces & Metrics mapping is required.

[abhilash-sivan]

Not verified rabbitmq per se. It's just to show how we differentiate children under messaging or any other group.

[aryamohanan]

But for RabbitMQ, there’s nothing really common with BASE_MAPPINGS.messaging. I don’t think any of the others share much with it either, apart from Kafka.

[kirrg001]

Why do we need that here? If we have already a custom kafka mapping object?

[abhilash-sivan]

messaging.system is an additional attribute that we append to the span. For Kafka, the system name also happens to be kafka, so it may seem redundant at first glance.

However, additionalAttributes is intended for constant or computed attributes that do not directly map from a single field inside span.data.kafka.

messaging.system falls into that category since it is effectively a constant defined per transformer, rather than something extracted from the incoming span payload itself.

I have changed this to: 'messaging.system': this.systemName

which will fetch the appropriate value in this case

[kirrg001]

Is that debug code?

[aryamohanan]

found a bug introduced by the dual transformation refactoring. When the transformation was moved from addSpan() to transmitSpans(), the normal Collector → Agent flow continued to work, but the serverless/AWS Lambda path use getAndResetSpans() over transmitSpans().As a result, it now receives internal spans instead of transformed backend spans.

Added the transformation here too. Only lambda call this function to get the spans

[aryamohanan]

IMO it's better to extract this refactor, created a PR #2620