TALEND WEBINAR : March 27th, 2018 | Step-by-Step to Enterprise Data Integration

Is Your Data Integration Platform Container Ready?

Is Your Data Integration Platform Container Ready?

  • Edward Ost
    Over 25 years of experience in application and web development with extensive experience on both Java and .NET platforms. Ed works as the Channels Technical Director at Talend working with technology partners, commercial use customers, and SI partners in the Talend ecosystem. Current focus is on enterprise integration strategies spanning DW, Data Lake, and operational decision support systems using the Talend Unified Platform.
  • July 21, 2017


Docker Containers are widely used and becoming even more prevalent as companies seek to streamline their operations.  Containers help decouple compute resources from applications, increasing the elasticity and hence the efficiency of IT operations.  DataDog reports that Docker usage has grown 40% over the past year, 18.8% of the DataDog sample customers use Docker.  This represents 15% of the servers monitored by DataDog.  Docker adoption within an organization also grows quickly, typically growing to 5x within 9 months.

Clearly, Docker is here to stay and is quickly becoming a pillar of agile DevOps IT operations.  But what of its impact on the applications running on that infrastructure?  Using Docker is one thing, but following true container architecture principles is another.  Many enterprise applications do not provide a simple path to leverage containers.

Historically the application server was king.  You purchased an enterprise class product, and the expectation was the product was secure, managed, scalable, and highly available.  In fact, each such product typically provided its own approach to these concerns, requiring further integration for each aspect.  Even with the integration of security and monitoring, the application servers themselves remained monolithic applications requiring dedicated resources regardless of the actual usage.  This prevents users from realizing the full benefits of containerization.

Modern, lightweight application designs like Talend offer a simple, container ready path.  Talend data integration Jobs are simple Java applications.  They are autonomous, self-contained applications so they can be easily specified in a DockerFile with full confidence that all dependencies have been addressed.

When the Docker image is launched, it can be parameterized so that inputs, configuration, working and staging areas and output folders are all driven with Docker Volumes at execution time.  Such immutable containers also enable greater security.  Entire Docker images can be marked as read-only.  And Docker registries ensure version controlled repeatability and stability for developers and admins alike.  The image can be disposed of immediately upon completion.

The savings in IT operations are potentially significant.  This is reflected in the Data Dog sample.  The median number of containers per physical server increased from 5 to 9 containers.  This shows the value of Docker in providing more granular, targeted allocation of IT resources.  In addition, the median lifetime of Docker containers is also decreasing.  Users of container management tools like Kubernetes have median container lifetimes of less than one day.  So not only does Docker provide more granular use of IT resources, but more dynamic usage as well.

However, realizing these benefits depends on applications being container ready.  In my next article, I’ll walk you through a simple, repeatable way to deploy your Talend Jobs directly to Docker.

Most Downloaded Resources

Browse our most popular resources - You can never just have one.

Join The Conversation


Leave a Reply

Your email address will not be published. Required fields are marked *