When to use Apache Camel?

Apache Camel is one of my favorite open source frameworks in the JVM / Java environment. It enables easy integration of different applications which use several protocols and technologies. This article shows when to use Apache Camel and when to use other alternatives.

The Problem: Enterprise Application Integration (EAI)

Enterprise application integration is necessary in almost every company due to new products and applications. Integrating these applications creates several problems. New paradigms come up every decade, for example client / server communication, Service-oriented Architecture (SOA) or Cloud Computing.

Besides, different interfaces, protocols and technologies emerge. Instead of storing data in files in the past (many years ago), SQL databases are used often today. Sometimes, even NoSQL databases are required in some usecases. Synchronous remote procedure calls or asynchronous messaging is used to communicate via several technologies such as RMI, SOAP Web Services, REST or JMS. A lot of software silos exists. Nevertheless, all applications and products of these decades have to communicate with each other to work together perfectly.

Enterprise Integration Patterns (EIP)

Of course, you could reinvent the wheel for each problem, write some spaghetti code and let the applications work together. Unfortunately, your management will not like the long-term perspective of this solution.

Enterprise Integration Patterns (www.eaipatterns.com) help to fragment problems and use standardized ways to integrate applications. Using these, you always use the same concepts to transform and route messages. Thus, it is a good idea to forget about reinventing the wheel each time you have a problem.

Alternatives for integrating Systems

Three alternatives exist for integrating applications. EIPs can be used in each solution.

Solution 1: Own custom Solution

Implement a individual solution that works for your problem without separating problems into little pieces. This works and is probably the fastest alternative for small use cases. You have to code all by yourself. Maintenance will probably be high if team members change.

Solution 2: Integration Framework

Use a framework which helps to integrate applications in a standardized way using several integration patterns. It reduces efforts a lot. Every developer will easily understand what you did (if he knows the used framework).

Solution 3: Enterprise Service Bus (ESB)

Use an enterprise service bus to integrate your applications. Under the hood, the ESB also uses an integration framework. But there is much more functionality, such as business process management, a registry or business activity monitoring. You can usually configure routing and such stuff within a graphical user interface – you have to decide at your own if that reduces complexity and efforts. Usually, an ESB is a complex product. The learning curve is much higher. But therefore you get a very powerful tool which should offer all your needs.

What is Apache Camel?

Apache Camel is a lightweight integration framework which implements all EIPs. Thus, you can easily integrate different applications using the required patterns. You can use Java, Spring XML, Scala or Groovy. Almost every technology you can imagine is available, for example HTTP, FTP, JMS, EJB, JPA, RMI, JMS, JMX, LDAP, Netty, and many, many more (of course most ESBs also offer support for them). Besides, own custom components can be created very easily.

You can deploy Apache Camel as standalone application, in a web container (e.g. Tomcat or Jetty), in a JEE application Server (e.g. JBoss AS or WebSphere AS), in an OSGi environment or in combination with a Spring container.

If you need more information about Apache Camel, please go to its web site as starting point: http://camel.apache.org. This article is no technical introduction J

When to use Apache Camel?

Apache Camel is awesome if you want to integrate several applications with different protocols and technologies. Why? There is one feature (besides supporting so many technologies and besides supporting different programming languages) which I really appreciate a lot: Every integration uses the same concepts! No matter which protocol you use. No matter which technology you use. No matter which domain specific language (DSL)  you use – it can be Java, Scala, Groovy or Spring XML. You do it the same way. Always! There is a producer, there is a consumer, there are endpoints, there are EIPs, there are custom processors  / beans (e.g. for custom transformation) and there are parameters (e.g. for credentials).

Here is one example which contains all of these concepts using the Java DSL:

from(„activeMQ:orderQueue“)..transaction().log(„processing order“).to(mock:“notYetExistingInterface“)

Now let’s look at another example using the Scala DSL:

„file:incomingOrders?noop=true“ process(new TransformationProcessor) to „jdbc:orderDatastore“

If you are a developer, you should be able to recognize what these routes do, don’t you?

Two other very important features are the support for error-handling (e.g. using a dead letter queue) and automatic testing. You can test EVERYTHING very easily using a Camel-extension of JUnit! And again, you always use the same concepts, no matter which technology you have to support.

Apache Camel is mature and production ready. It offers scalability, transaction support, concurrency and monitoring. Commercial support is available by FuseSource: http://fusesource.com/products/enterprise-camel

When NOT to use Apache Camel?

Well, yes, there exist some use cases where I would not use Apache Camel. I have illustrated this in the following graphic (remember the three alternatives I mentioned above: own custom integration, integration framework, enterprise service bus).

When to use Apache Camel?

If you have to integrate just one or two technologies, e.g. reading a file or sending a JMS message, it is probably much easier and faster to use some well known libraries such as Apache Commons IO or Spring JmsTemplate. But please do always use these helper classes, pure File or JMS integration with try-catch-error is soooo ugly!

Although FuseSource offers commercial support, I would not use Apache Camel for very large integration projects. An ESB is the right tool for this job in most cases. It offers many additional features such as BPM or BAM. Of course, you could also use several single frameworks or products and „create“ your own ESB, but this is a waste of time and money (in my opinion).

Several production-ready ESBs are already available. Usually, open source solutions are more lightweight than commercial products such as WebSphere Message Broker (you probably need a day or two just to install the evaluation version of this product)! Well-known open source ESBs are Apache ServiceMix, Mule ESB and WSO2 ESB. By the way: Did you know that some ESB base on the Apache Camel framework (e.g. Apache Service Mix and the Talend ESB). Thus, if you like Apache Camel, you could also use Apache ServiceMix or the commercial Fuse ESB which is based on ServiceMix.

Conclusion

Apache Camel is an awesome framework to integrate applications with different technologies. The best thing is that you always use the same concepts. Besides, support for many many technologies, good error handling and easy automatic testing make it ready for integration projects.

Because the number of applications and technologies in each company will increase further, Apache Camel has a great future. Today we have application silos, in ten years we will probably have cloud silos which are deployed in Goggle App Engine, CloudFoundry, Amazon EC3, or any other cloud service. So I hope that Apache Camel will not oversleep to be ready for the cloud era, too (e.g. by offering components to connect to cloud frameworks easily). But that’s future… At the moment you really should try this framework out, if you have to integrate applications in the JVM / Java environment.

By the way: I know that I praise Camel in this article, but I am neither a Camel committer nor working for FuseSource. I just really like this framework.

Best regards,

Kai Wähner (Twitter: @KaiWaehner)

Tags: , , , , , , , , ,

24 Responses to “When to use Apache Camel?”

  1. Jeff Phillips Says:

    Nice post. Do you have experience with Spring Integration, so you could contrast it with Apache Camel?

  2. Kai Wähner Says:

    Hey Jeff,

    I do not have experience in a project with Spring Integration. My personal opinion is that Camel has an easier, fluent API (especially the Java / Scala DSL). You can integrate several protocols without coding a lot. Spring Integration could also do the job, but I do not like the Spring programming model – I needs much more (XML-) configuration.

    Thus, if you already use a Spring project which uses several other spring projects, then I would at least evaluate Spring Integration, too. In all other cases, I would choose Camel as integration framework.

    DZone has a great (neutral) comparison of Spring Integration and Camel (unfortunately not using the Java DSL for Camel): http://java.dzone.com/articles/spring-integration-and-apache

    Best regards,
    Kai Wähner (Twitter: @KaiWaehner)

  3. Diwakar Says:

    Hi,
              Good post.
              >> Well-known open source ESBs are Apache ServiceMix, Mule ESB and WSO2 ESB
              When you mention Apache Servicemix ESB are refering to Servicemix 3.x (JBI) or Servicemix 4.x (OSGi)?
              Please let me know your comment.

    With Best Regards,
    Diwakar

  4. Kai Wähner Says:

    Hey Diwakar,
    I (and most other guys) think that the JBI standard is dead already. It always was too complex. Thus, I refer to the current version of ServixMix: 4.x.
    I would not consider any JBI product for a new project. Mule, WSO2 and ServiceMix are all great ESBs because they are up-to-date, lightweight and easy to use.
     
    Best regards,
    Kai

  5. David Foote Says:

    Hi Kai,

    Thank you for your thoughtful treatment of the topic of when to use Camel.  However, I wish to draw your attention to the correct use of the term DSL (Domain-specific Language).  Java, Scala and Groovy are general-purpose languages, the opposite of a DSL.  Examples of DSLs are: HTML, SQL,  Mathematic and YACC.  For reference, please note the Wikipedia article at http://en.wikipedia.org/wiki/Domain-specific_language.

    David Foote 

  6. Oleg Zhurakousky Says:

    Kai

    “You can integrate several protocols without coding a lot.” – When you say without coding a lot do you mean XML, Java, Scala? I am always skeptical when I read such statements. There has got to be a place where you provide a machine instruction, right otherwise how does a runtime knows what to do?

  7. Kai Wähner Says:

    @Oleg:
    What I want to say is: No matter which API you need, you just have to write the route using the same syntaxt and concepts, no matter if you use the Java, XML or Scala DSL.
    Example (Java DSL):
    from(“file:input)”.to(“jms:outputQueue”)
    Now, again the statement: “You can integrate several protocols without coding a lot” => i.e.: You do not have to use the File API, you do not have to use the JMS API, you do not have to code a lot. The Camel components do this for you, their source code contains calls to the File API or JMS API. No matter which other Camel component you use, it’s the same…
    Of course, you still have to write some code: The routes, and you have to do the configuration once (e.g. configuring JMS ressources). Though, it is much less boilerplate code.
    I hope you agree with this explanation :-)
     

  8. Oleg Zhurakousky Says:

    Yes I do agree and wanted to say that the same applies for Spring Integration where pretty sophisticated message flows could be constructed without writing a single line of Java code and only with few configuration elements configured via XML and soon Scala. Let me know and I’ll point you to some samples.
    Also, on the related note its nice how you said “. . .you just have to write the route using. . .” FWIW this is one of the fundamental differences between Camel approach and SI. EIP does not define ‘routes’, instead core EIP patterns is what you use to construct ‘message flows’ which themselves could be used in isolation, extended or included as part of some other flows etc., which means IMHO the idea behind the construct that begins with ‘from’ and ends with ‘to’ contradicts the main idea behind Messaging and EIP in general and that is “physical and logical decoupling of producers and consumers”.
    I can go on, but don’t want to hijack the main idea behind the post ;)
    May be in the future you car write something that compares SI and Camal and than we can duke it out ;)
    Cheers
     

  9. Kai Wähner Says:

    @Oleg:
    I have already written that comparison (and you have already replied): http://www.kai-waehner.de/blog/2012/01/10/spoilt-for-choice-which-integration-framework-to-use-spring-integration-mule-esb-or-apache-camel/
    You can duke it out there :-) Especially, your point about “‘from’ and ‘to’ contradict to the main idea behind messaging and EIP” would be a good argument there!

  10. Hongchao Nie Says:

    Nice post. For the cloud era, I think mule is moving to the correct direction. Mule ESB already provides connectors for salesforce and twitter, etc. Also Mule iON claims to be a iPaaS, though I have not tried this. If the DSL is not of priority, the connection ability is critical to integration architects. Camel has to hurry up.

  11. Kai Wähner Says:

    Hey Hongchao,
    I agree about the components, Mule has several awesome connectors to proprietary products. I have discussed this topic in another blog post: “Comparison of Apache Camel, Mule ESB and Spring Integration”.
    Though, Camel also has connectors for many cloud services such as AWS, GAE, Twitter, and many others (using the jclouds component) already.
    Besides, I think Fuse Fabric is going into the same direction as Mule iON… So I think, has not to hurry up :-)
    Best regards,
    Kai Wähner (Twitter: @KaiWaehner)
     

  12. Murthy Says:

    In this article and other articles of yours, you refer to MuleESB in both Integration Framework context and ESB context. Do you consider MuleESB as more of an Integration Framework or an ESB, because MuleESB itself, based on my understanding, does not have support for BPM/BAM/Registry/Repository which you say are typical features of an ESB.

  13. Kai Wähner Says:

    @Murphy: See discussion in this blog: http://www.kai-waehner.de/blog/2012/01/10/spoilt-for-choice-which-integration-framework-to-use-spring-integration-mule-esb-or-apache-camel/

  14. Soumya Says:

    Nice article – however don’t agree with the concept that ESBs can do BPM. Common mistake – BPM is for “Process orchestration” mainly built on BPEL and more for manual+auto tasks. ESB do orchestration however on “Business Service” level. BPMS can use ESBs for delegating service tasks or ESBs may invoke a BPMS process. To summarise ESBs can do in-memory stateless micro-flows i.e. collection of some services – while BPMS does long runnning process orchestration

  15. Kai Wähner Says:

    Soumya, I agree with you.

    It depends on the definition of the product “ESB”. I explained exactly what you are talking about in a presentation at CamelOne 2012 (Camel + Activiti): http://www.kai-waehner.de/blog/2012/05/17/progress-report-from-camelone-2012-in-boston-apache-camel-activemq-servicemix-cxf/

  16. Anigma Says:

    Hi – This ia very short and nice article. I was wondering if there is any list of features to compare for different products like Camel, Active Matrix (TIBCO) or might be WCF?

  17. Kai Wähner Says:

    Anigma,
    my Slides comparing ESBs vs. Integration Frameworks might help:
    Spoilt for Choice – How to choose the right ESB
    Kai

  18. Sarita Says:

    Can you please help me to decide which ETL tool should I use for my application over spring framework to integrate with HL7 (health domain) format. for now we are considering FTP for HL7 transform. I am currently looking into Apache Camel and spring integration. Is there be any other good option (open source) for this purpose.

  19. Kai Wähner Says:

    Well, I have never used HL7 before.
    Though, besides Apache Camel HL7 component, you can also take a look at Talend Open Studio for Data Integration. It is open source and offers HL7 components including a very easy and powerful GUI designer. Here is the link to Talend’s components: http://www.talendforge.org/components. Search for Healthcare or HL7 on this website.

  20. johndoe Says:

    anyone knows an alternative to apache camel that doesnt require to be connected to the internet?

  21. admin Says:

    Apache Camel does NOT require to be connected to the internet.

    Of course, if you use Maven, then it tries to download packages from internet. But this is an optional feature. You can also just distribute JARs which you need.

    Kai

  22. Andrew Says:

    Nice article.  What factors in integration complexity would make you move up to ESB from Camel? 
     
    In other words; I like the graph.  Its easy to understand.  What are the factors for deciding between one or the other?  Number of Servers? Number of Routes?  Number of different Protocols?  etc.
     
    Thanks,
     
    Andrew

  23. Kai Wähner Says:

    Andrew,
    it is NOT about number of routes, servers, etc.
    Use an ESB if you need good tooling (graphical designer, code generator, monitoring) and commercial support. If you want to write source code by yourself and debug etc. just with an IDE such as Eclipse, then Camel without an ESB is a good choice.
    These are the main differentiators between an integration framework and an ESB whereby an ESB might be on top of of the framework, e.g. Talend ESB and JBoss Fuse both generate Camel code under the hood.
    Kai

  24. kroy Says:

    Help full  thanks
     

Leave a Reply

icon_wink.gif icon_neutral.gif icon_mad.gif icon_twisted.gif icon_smile.gif icon_eek.gif icon_sad.gif icon_rolleyes.gif icon_razz.gif icon_redface.gif icon_surprised.gif icon_mrgreen.gif icon_lol.gif icon_idea.gif icon_biggrin.gif icon_evil.gif icon_cry.gif icon_cool.gif icon_arrow.gif icon_confused.gif icon_question.gif icon_exclaim.gif 

What is 9 + 5 ?
Please leave these two fields as-is:
IMPORTANT! To be able to proceed, you need to solve the following simple math (so we know that you are a human) :-)