Apache Beam is a programming model that lets you implement batch and streaming data processing jobs that can run on multiple execution engines, including Dataflow, the execution engine created by Google and available on the Google Cloud Platform.

You have decided to use Dataflow to run your Beam pipelines, great choice! You need to know if your pipeline operates properly, especially if you chose a streaming pipeline. As these jobs run continuously, they should be continuously monitored.

If you feel a bit lost in choosing your monitoring strategy because Dataflow can leverage multiple types of metrics and logs, don’t worry…

Jérémie Gomez

Data Cloud Consultant @ Google

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store