Listen to this article on our podcast! Episode 1 of our Cap on Camunda series is now available on Spotify, CapBPM.com, and more! You can also click here to download.
Importance of Microservice Orchestration
Microservice orchestration is something that we all have to do in the enterprise space. Whether you’re using something like Camunda, Kafka, Temporal, or a million other things, you have to have a way for one system to call to another, deal with exceptions and timeouts, escalations retry, throughput, logging and security, multi-tenancy, and more. You have to solve that problem.
Now, are you going to solve that problem with a home brew solution or are you going to use an off the shelf product for that? And this, I think, is sort of a fundamental perspective that we need to be thinking about. There are places where I think a home brew solution makes sense. So if you are doing something super idiosyncratic to your industry and you’re using your own proprietary protocol, I can imagine that an off-the-shelf engine wouldn’t necessarily work for you. But for the most part, the majority of us don’t live in that space.
The majority of us are trying to do fairly conventional things. We want to make a restful call here, make a GRPC call there. We want to talk to the database, get some information, log all of it, encrypt it, and update this other data source. For that purpose, you can either write custom code: If “x” happens, then do “y” unless there’s a timeout, and then escalate and send it to this email, and so on and so forth.
Or you can use a microservice orchestration engine. Now, of the microservice orchestration engines that are out there, you can take something like Kafka, build logic around it and establish your own message protocols and use that as a mechanism to transport messages back and forth.
But you still have to build logic, NEM, and even pseudo grammar around that, right? So when we put messages in the Kafka queue, we do this, we do that, and so on and so forth. Or you can go hardcore. You can write IF statements that are nested. “Hey, if this happens, unless this time’s out, retry three times,” and so on and so forth inside of your code.
Or you can use some kind of a workflow engine. The idea of the workflow engine, or languages that are specific and targeted for solving this particular problem, make a lot of sense in the same way in that using something like SQL makes a lot of sense when you’re dealing with data access. This is because it’s not a generic problem, it’s very specific to a particular type of adventure that we’re on. It makes sense that you would want to be looking at a specialized tool for that – just like the carburetor in your car is a specific tool that does a very specific functionality. You don’t want to have to go out and rebuild a carburetor every time you want to drive to the grocery store. That’s where these engines come in. At the end of the day, you’re better off using a tool than banging two sticks together in order to make fire.
Cap’s Loyalty to Camunda
My loyalty to Camunda is based explicitly and only on its excellence. If it were another company that was excellent in this field, that’s the one I’d be working with, but I’ve worked with IBM, Pega, and Appian. I’ve done a lot of different stuff in this space, and from a pure performance perspective, I don’t know of any engines out there — any service orchestration engines — out there that are better than Camunda. What I like about it is that it’s open source, so if you want to take on the feeding and the maintenance of it, you can do that and you don’t have to work with a vendor.
At the same time, there are SaaS offerings for it offered by Camunda — especially with Camunda 8 — where they say, “Hey, we’ll take on the infrastructure and the security, and you just write your workflows and your processes.” At the end of the day, the majority of the people who work in this space need to solve fairly common problems, but just because they’re common doesn’t mean that they’re easy. Our ability to be able to articulate our problem and our ability to be able to articulate the solution for that problem in a notation specifically designed for that just makes a lot of sense.
I also like the visuals. I like seeing this step go to this step and then go to this step. I like it because I can go to my business partner and go, “Hey, is this right? Am I doing this right?” This is because they know the business side, I know the technical side, and the pictures help us draw that together. I also like the fact that when it is explicitly drawn out, then my technical teams have more clarity in terms of how to manage it, how to change it, what the side effects are, and how to version it. Everything you need is right there. Now, we can focus our energies on how to do an efficient read from a database, what the business logic rules are that we need to implement as opposed to the metadata, and how we’re going to define all of this.
There are tools out there like Temporal, which I think is an exciting tool in the space, but fundamentally, it is a code-based platform as opposed to a visual platform. I fundamentally believe that there’s a finite number of things that you can keep track of in your head when you’re looking at it from a code perspective. For example, most people can bring up an image of the Mona Lisa in their mind, but you’re gonna have a hard time remembering 50 pages of Shakespeare, verbatim. Code and text is harder when it gets complex, where diagrams and images are how our brains are wired. We’re hunter gatherers. We think in terms of visuals and the ability to distinguish differences and colors. So, with all that said, the ability to deal with encroaching complexity is better dealt with with a diagram-based tool than a code-based tool.
Validating Camunda’s Ability to Help
You have to have a high standard. You should always be thinking days and weeks, not weeks and months. You should say, “Hey, I have a test case. I want to make a call to these three restful services, I want to deal with timeouts, escalations, and routing roles, and I want to get this done in two weeks. That is an empirical test that is not subjective. You could actually do that. The trick is going to be making sure that you have somebody to help you.
Whether it’s a partner like Capital BPM or some expert that you know out in the wild, bring in someone who knows how to drive this car so you get the experience of what it’s like if you had this car. You will get there by just figuring out the technology yourself, but you’re not gonna know it on Day 1. But that’s not the important thing. The important thing is to have this vehicle cover the terrain that you need to cover in the time that you need, plus the safety and the security that you need. That is an empirical question, and engineers love empirical questions because we don’t have to argue. We can just see what happens.
I love that approach. I love it philosophically and I love it practically. I would recommend, you know, come up with a use case where you want to do specific tasks and then start the timer. How long is it going to take you to do these things? Is it worth it? It is better to get someone who knows how to do it and see if that works for you. Be empirical.
Overcoming Camunda’s Biggest Weakness
Like all technologies, Camunda has its weaknesses. The main weakness is going to be the fact that it is a different paradigm than what your normal techies are used to. You’re drawing diagrams as opposed to writing code and that is, a fundamental paradigm shift for your traditional programmers. If they can understand, and adapt to the fact that Camunda is the orchestration engine, it acts like a conductor in an orchestra. It will say a little more violin, a little bit more cello, and whatnot. If they understand that, that’s what Camunda does for them.
The things that they’re doing — talking to the database, making a restful call, updating the Kafka queue — those things can still be written in traditional code. You can focus on the excellence of your development team on those atomic tasks. Then let Camunda compose them into a hole. That is the strength and the weakness of it. That is also the place where, from a mindset perspective, where you need the most flexibility. I would say that can definitely be a challenge.
Where Capital BPM Steps In
Typically what we do is we help customers come up with a valid challenge. Within a certain amount of time — both from a development amount of time and an execution amount of time — we help them articulate the problem, and then we help them come up with an objective test that determines whether it’s possible to do this or not. And again, this can be a matter of weeks.
In those weeks, we should be able to build a proof of concept that shows you whether it’s possible to do this or it’s not possible to do. Camunda is not a silver bullet. There are no silver bullets, which is a sad state of affairs, because there are monsters out there. But even as we need to be able to slay these monsters, there are practical things that we can do, and some of these practical things are just a matter of trying it out, seeing if it works, and taking notes on what happened. And that’s where CapBPM can help. We know this technology very well; you know your business domain very well. We can put those together and figure out if this is a useful tool for you or not.