Ever since I discovered OpenTelemetry for application monitoring and the flexibility of the OpenTelemetry Collector, I’ve been sharing its wonders with several of my clients – and they’ve had great experiences too.
Having used the Collector for some time, my curiosity about its inner workings grew. That’s when I decided to create my own exporter to get a deeper understanding. To keep the focus on understanding the core mechanics, I crafted a basic exporter that doesn’t process the received metrics, traces, or logs in any way. This project was a perfect chance for me to explore two things: diving into Go, which was new for me, and uncovering more about the OpenTelemetry Collector’s operations, especially how it sends metrics to its configured destinations.
For those keen to see the code, you can find it all here.
Development Environment Setup
Considering my venture into Go development and my growing reluctance to clutter my machine with new tools each time I experiment, I’ve grown fond of devcontainers. This method keeps my machine neat while ensuring I have all the required development tools. If you haven’t explored devcontainers yet, I urge you to do so.
The setup of the environment using
devcontainer.json should look something like this:
I’ve utilized the
postCreateCommand to trigger an external script since we require specific tools beyond Go to craft our collector exporter. The
postCreateCommand.sh file contains the following:
This script ensures the installation of the
delve commands, essential for building the custom collector and aiding in debugging.
I encountered challenges executing these commands directly from
devcontainer.json within the
postCreateCommand. To sidestep excessive troubleshooting, I adopted this approach. A minor inconvenience is that Visual Studio Code, upon detecting changes to
devcontainer.json, suggests a container rebuild and restart. When dependencies are isolated in a separate file, manual container rebuilding becomes mandatory. However, considering these tools are all we need, it’s a manageable trade-off.
forwardPorts, you’ll see ports 4317 (gRPS) and 4318 (http) are specified. As I intent to employ the oltp protocol for telemetry transmission, I added these ports, enabling data flow from our application to the custom collector.
Building a Custom Collector
In order to run our custom exporter, we must build a specialized version of the OpenTelemetry Collector. This enables the testing, debugging, and execution of both the collector and our custom exporter. While the OpenTelemetry documentation provides an excellent guide on this topic, I favor the devcontainer methodology for this project.
OpenTelemetry introduced a tool named
builder to simplify the creation of a custom collector. Assuming you’ve set up the `devcontainer`` as described, this tool should already be installed on your environment.
The builder utility requires a manifest file to direct its build process. I’ve named this file
otelcol-builder.yaml1 and placed it in the root directory of my repository.
Within this configuration, we specify the desired binary name, its description, the output directory, and the collector version. Additionally, we define which exporters and receivers to incorporate into our custom collector. At this stage, we’re integrating the
debugexporter—an OpenTelemetry-provided exporter that relays telemetry to the console—and the
otlpreceiver, which implements the oltp protocol, allowing us to receive telemetry.
Note: Working with a
devcontainerfor this project raised an
output_pathconfiguration issue. To modify it, refer to the
output_pathsection below. 2
Build the custom collector using:
The output should resemble something like this:
As a result, you’ll find an
otelcol-custom binary in the
/tmp/dist directory. Launching this binary activates the custom collector. At this point, the custom collector operates like any other OpenTelemetry Collector and requires a configuration file. I’ve named this configuration
config.yaml and placed it alongside
otelcol-builder.yaml in the repository root. Starting with the debug exporter, it’s easier to validate the custom collector’s operation. A basic
config.yaml looks like this:
To wrap things up, run the following command to view logs, metrics, and traces directly in the console:
The output of that should look something like this:
Creating the exporter
Now that we’ve set up a working custom collector, it’s time to dive into our exporter. I began by creating a new folder named
emptyexporter in my repository’s root. Inside this folder, I created a
go.mod file. This file informs Go that we’re dealing with a module and facilitates the import of other necessary modules. The foundational
go.mod file is:
Typically, the module name mirrors the repository name. While this isn’t a strict requirement, it simplifies module imports in other projects. In this instance, the name isn’t an actual repository name. But since we won’t be publishing this module, that’s fine. However, ensure that the repo name aligns with the folder on your local machine, as Go uses this to locate the module.
require section lists the modules we need. The
go.opentelemetry.io/collector/component module is essential for configuration creation, the
go.opentelemetry.io/collector/exporter module aids in exporter creation, and the
go.opentelemetry.io/collector/pdata module handles the various telemetry categories or signals.
Next, I crafted an
exporter.go file within the
emptyexporter folder. This file houses our exporter code. Given that our exporter won’t be particularly complex, the code remains fairly straightforward:
emptyexporter struct defines our exporter. The functions
pushTraces will be invoked by the collector upon receipt of logs, metrics, and traces, respectively. As we aren’t processing the received telemetry, these functions simply return
NewEmptyexporter serves as a factory function, generating a new instance of the
With the exporter code ready, it’s time to register it with the collector. This requires a
factory.go file in the
emptyexporter directory. This file contains the registration code:
NewFactory is our factory function, crafting a new instance of the
exporter.Factory struct, complete with the
typeStr as the exporter type and the
createDefaultConfig function to produce the exporter’s default configuration. Since our configuration is currently non-existent, it merely returns an empty struct. The
NewFactory function also references the
exporter.WithLogs functions to register the exporter with the collector for traces, metrics, and logs, respectively. Each of these functions, in turn, calls the respective internal factory methods,
The only task left is to change the
otelcol-builder.yaml file to include our newly created
emptyexporter module. Within the exporter section, add:
While a module version is mandatory, any version will suffice since we aren’t publishing this module. The
path denotes the module’s path on your machine. Remember, even if you specify the module’s path, Go will still search for the module using the repository name identical to the module name.
Lastly, we’ll update the
config.yaml file to include the
As you can see we need to add the
emptyexporter exporter to the
exporters section and add it to the relevant signal type pipelines, in my case
Running the custom collector with the updated configuration file should now display the following:
Pay special attention to the
email@example.com/exporter.go:* lines. These lines indicate that the collector has registered our exporter for traces, metrics, and logs. We are now ready to attach a debugger to our custom collector and play around with the received metrics, traces, and logs.
Debugging the Exporter
The primary reason for constructing a custom version of the collector is to facilitate debugging of your exporter. Now that we have a setup ready for debugging, let’s dig into how it can be achieved.
The initial step is to include
debug_compilation: true in the
dist section of the
otelcol-builder.yaml file. This action ensures that the debug symbols are incorporated into the binary during the collector’s build, paving the way for debugging the exporter.
After some exploration, I discovered that the delve tool can be used to debug Go applications. Given the plethora of resources available on using delve for Go debugging, I opted for this method.
You can kickstart the custom collector in debug mode with the following command:
You should see the following output:
Note: An unusual quirk I encountered during my debugging journey was with delve’s termination process. Once delve is active, a simple
ctrl+cdoesn’t suffice to exit. The workaround I employed was to initiate another terminal and run the command
killall dlv. If anyone is aware of a more elegant solution to this, I’m eager to hear it!
launch.json file needs to be created within the
.vscode directory. This file instructs Visual Studio Code on how to connect to
delve and debug the exporter.
To wrap things up, navigate to the
Run and Debug tab in Visual Studio Code. From the dropdown menu, select
Connect to server and hit the play button. Visual Studio Code will now establish a connection with the custom collector. This enables you to set breakpoints within your exporter, specifically on the
pushTraces functions, and observe incoming data in real-time.
By now, you should have a functional custom collector accompanied by a debuggable exporter. While the current setup offers a basic framework for an exporter, it establishes a robust foundation from which you can develop and expand. Building and debugging your own OpenTelemetry collector exporter not only deepens your understanding of the OpenTelemetry ecosystem but also empowers you to tailor monitoring solutions to your unique needs.
For a glimpse into the kind of data your exporter might handle, here’s an example JSON.
otelcol-builder is a acronym for OpenTelemetry Collector Builder ↩︎
While setting up, I found it inconvenient that the
output_pathwas directed to a location within the container. In an attempt to redirect it to a local machine location, I set it as
/workspaces/opentelemetry-embedding-exporter/otelcol-custom. Unfortunately, this adjustment was met with an error upon executing the build command:
1 2 3
Error: failed to compile the OpenTelemetry Collector distribution: exit status 1. Output: error obtaining VCS status: exit status 128 Use -buildvcs=false to disable VCS stamping.
Although I believe there should be a way to make this work, I chose not to go down a troubleshooting rabbit hole. If anyone has insights or solutions regarding this issue, I’m open to suggestions. ↩︎