Gregor's Ramblings on Enterprise Integration
Queues are key elements of any asynchronous system because they can invert control flow. Their ability to re-shape traffic enables high-throughput systems that behave gracefully under heavy load. But all that magic doesn't come for free: to function well, queues require flow control
When describing messaging systems, it's natural to focus on data flow. However, control flow is what gives those systems their critical runtime properties like resilience or low latency. It's time to make control flow a first-class consideration.
Event-driven architectures (EDAs) are frequently pitched as loosely coupled, when compared to other forms of integration. Coupling is a multi-faceted and nuanced property, so as architects we should have a closer look.
Coupling is integration's magic word. Despite entire books being written on the subject, it's still widely recited as a "loose coupling is good" mantra. Coupling deserves better, so let's clarify the dimensions and nuances of coupling.
Do serverless solutions lock you in? Let's find out by porting / rebuilding the Serverless Loan Broker on top of Google's Cloud Platform (GCP).
Infrastructure as Code (IaC) brings repeatability into the provisioning and deployment of cloud applications. However, the vocabulary used by most automation tools describes cloud platform resources as compared to the application's intent. As serverless applications take us further away from the infrastructure, we should also find better abstractions to express our automation. Perhaps, Enterprise…
Infrastructure as Code (IaC) has a whole new meaning for serverless applications. Rather than provision resources (the serverless frameworks do that for us), automation determines the system composition and configuration. As expected, quite a few options are available from simple command line to programming frameworks. Time to have a look.
Part 3 of the mini-series on implementing the EIP Loan Broker as a serverless solution with AWS Step Functions uses a Publish-Subscribe Channel and a stand-alone Aggregator to request and process loan quotes.
In part 2 of this blog series I implement the Loan Broker Example using a Recipient List pattern, implemented in DynamoDB, Step Functions, and Lambda.
A lot has happened since we implemented the Loan Broker Example in EIP: we have the cloud, serverless computing, machine learning, service meshes and all sorts of other bells and whistles. Nevertheless, the integration patterns have passed the test of time as we shall see by implementing the original loan broker example with AWS Lambda and AWS Step Functions.
After examining which patterns are embedded in Google Cloud Pub/Sub in an earlier post, I implemented a few common patterns on top of Google Cloud Functions, Google's serverless implementation. It's amazing how easy it has become to get messaging patterns running in the cloud with only a minimum of wrapper code.
Enterprise Integration Patterns are timeless. That's why our book keeps selling well 20 years after its publication in 2003. However, the code examples haven't aged quite as well. I therefore updated several code examples to use modern tech like GoLang, Kafka, RabbitMQ, AWS, Azure Cloud Functions, and Google Cloud Pub/Sub.
About two years ago, I revived the ramblings from a four-year hibernation. Upon resurrection, the ramblings started to take a broader scope, including not only messaging, conversations, and patterns, but also communication and IT architecture topics. The new topics were a natural reflection of my new role as Chief Architect in a massive IT transformation and after a little while I decided that…
OOP is one of the conferences that originated in the "OO" era of the early to mid-nineties. While most of these events have shed the double "O" -- OOPSLA has become SPLASH and JAOO has become GOTO, OOP stayed true to its name and remains one of the premier German developer conferences. This year OOP celebrated its 25th anniversary. Here are my impressions.
The digital world is all about scalability: millions of web sites, billions of hits per month, more data, more tweets, more images uploaded. To make this work, architects have learned a ton about scaling systems. With everything around us scaling to never-before-seen throughput, the limiting element in all this is bound to be us, the human users. And the organizations we work in. One may wonder,…
When you hitchhike through the galaxy, a towel will generally do. To ride the you need to be a bit better equipped. Riding the elevator up and down is exciting and provides tremendous value to the organization. But it is also demanding as you need to adjust to different audiences, backgrounds and dynamics, becoming a technology shapeshifter of sorts. To help with this transformation, I recommended…
As I recently observed, Corporate IT tends to be afraid of code: code is where all the pesky bugs come from, which have to be fixed by quirky, expensive, and unreliable developers or external consultants. This attitude plays to the favor of vendors who peddle products that "only" require configuration, as opposed to coding. Sadly, most configuration is really just programming in a poorly designed,…
Corporate IT lives among zombies: old systems that are half alive and have everyone in fear of going anywhere near them. They are also tough to kill completely. Worse yet, they eat IT staff's brains. It's like Shaun of the Dead minus the funny parts.
I recently wrote about architects and the role they play in large organizations. Still, the question often remains what an architect should be doing besides "riding the elevator". Let's try another analogy: movie characters.
IT loves virtualizing stuff, following the old rule that in computer science every problem can be solved by just one more level of indirection. Cloud computing is based on virtualization of compute resources - you don't need to know which machine your application is actually running on and you can get new ones with the click of a button. Before cloud was a buzzword, though, VMware and others have…
Part of my job is to review system architectures. So I frequently ask teams to show me "their architecture", but almost as frequently I don't consider what I receive an architecture document. The inevitable question "what do you expect?" is also not so easy for me to answer: despite many formal definitions it's not immediately clear what architecture is or whether a document really depicts an…
Architects can often be found commenting or complaining that many things in IT are the same old stuff in new packaging, created by marketing departments who were in need of a new buzzword. For example, aren't microservices really just SOA done right while SOA itself stands for "Same Old Architecture"? And the whole reactive movement seems to have re-discovered callbacks. Maybe there is some truth…
I have not blogged about events in a while, but SATURN 2015 has been an amazing event that's well worth rambling about.
Discussing the Google cloud Pub/Sub system in the last rambling reminded me that the Publish-subscribe Channel pattern makes for a good example of the subtle but important difference between Messaging Patterns and Conversation Patterns. This comparison is timely as I picked up documenting Conversation Patterns again.
Google released the beta version of their publish-subscribe API just a few weeks ago. I show how to build a very simple demo app using the Java API and map the functionality to integration patterns to illustrate the design choices the team made.
As indicated a good while ago I spent some time thinking about patterns that instead of following a message through multiple systems, looks at the message exchange over time between a (mostly) fixed set of systems. I call these message exchanges "conversations". Sadly my writing efforts on Conversation Patterns fell very much asleep around 2008, partly due to my fear that with the rise of REST,…
We were tempted multiple times to extend the EIP icon language, but always felt that simplicity should win over precision. However, seeing the visual vocabulary that my former Google colleague Ivan Gevirtz created, I am convinced that combining the two is useful.
Architects frequently play a critical role as connecting and communicating element between multiple parties. Especially in large organizations such communication is an important factor: too many parties speak different languages to convey different, and often conflicting, objectives. Many layers of management only exacerbate the problem as communicating to corporate leadership resembles the…
Defining what a software or IT architect is or does is no less challenging than defining software architecture itself. The SEI maintains a list of software architecture definitions, yet I have not seen a corresponding list for software architects. One could try to skirt the issue by declaring that software architects are the people who make software architecture. However, I think that this…
One advantage of working for a relatively large organization is that I get to do a little more writing again. Not because I have spare time, but because in times of rapid change, communication across a large audience is critically important. It turns out, short papers are a good way to achieve that and my brief, but accurate technical position / decision papers have become a trademark of my…
At a recent presentation, Martin Fowler introduced himself as a guy "who is good at explaining things". While this certainly has a touch of British Understatement™, it also highlights a critically important but rare skill in IT. As technology invades all parts of personal and professional lives, causing companies to absolutely depend on software and systems, decision makers often can no longer…
Drawing a picture turns out to be a useful system design technique.
I was recently asked to help a team decide whether they should use messaging. Of course, I have not forgotten what I learned during many years in consulting: the consultant always answers "it depends." A good consultant can tell you what the answer depends on. And a truly great consultant can convince you that you are asking the wrong question in the first place.
My New Year's resolution was to write more, so here my thoughts on how to actually make that happen. In a sense it's a plan for myself to be more productive, but hopefully the ideas also work for other folks. Ironically, my work on EIP II has been stalled for a good while, so you're getting advice on how to be a prolific writer from someone who has not written much in 2 years. Read on at your own…
The beginning of a New Year is the time to reflect on the past and make resolutions for the future. It's become my tradition to kick off the year with some reflection on EIP, so here we go.
I joined JavaOne this year as a panelist on Cloud Computing. Here my belated impressions on this year's JavaOne.
By now everyone must have heard about Google Wave, the communication and collaboration platform announced at Google I/O. Google just announced that they are ramping up towards 20,000 developers on the current sandbox and will extend to 100,000 users by end of September. This will give many more people to opportunity to experiment with Wave.
I guess I am the last person to blog about Google I/O. By now, everyone should have read about Wave and the free Android phone (with 30 day SIM!). I am just now getting to write this up on my flight back to Japan, so I’ll give you my personal impressions and viewpoints, if you are still interested.
I have been speaking more frequently about cloud computing in recent days. As SOA is becoming a daily reality, I needed to advance to new, still slightly nebulous topic. What could be a better fit than cloud computing? All jokes aside, cloud computing provides for some interesting discussions and one should safely assume that Google has a word or two to say about it.
Blogging about design patterns seems about as original as blogging about the Java (TM) Programming Language, except for the missing (TM). However, as I just attended a workshop on software service engineering, I realized once again that people from different fields have very different notions about the concept and usage of patterns. Many workshop attendees raised questions about formalizing…
The end of the year is always the time to reflect on the past happenings. My friends in Japan often send a New Year's card with 12 pictures, each showing the significant event during the month. I am not sure something significant related to Enterprise Integration Patterns happened each month, but I think it's still nice to reflect a little bit on the history and current state of the patterns.…
Since Google's Developer Day, I have been promising to speak more regularly about Google products. This is a bit of a balancing act for me, as I want to avoid the slippery slope of becoming a corporate sock puppet. I also don't want to neglect my interest in connected systems, SOA, and asynchronous messaging. With the Internet turning into the largest connected system, the gap between Web-based…
I am just returning from a trip to Montreal and Keystone, CO for the OOPSLA and Colorado Software Summit conferences. I spoke on SOA Patterns, workshopped my conversation pattern paper, and gave six talks on event-driven architectures and building mashups using Yahoo! Pipes and Google Mashup Editor. Phew!
What’s wrong with this picture? I crash a stylish party at a pricey venue (SF Moma). I have no business being there, nor do I have the required invitation. I nonchalantly talk my way in, and hit the crowd with two fellow geeks. Within a few seconds an attractive blonde with a low cut dress approaches us and engages us in a conversation. We grab a few free drinks, indulge on the tirelessly…
I attended the Enterprise Mashup Summit last Friday. It was a small-ish event, with about 50 people attending. We saw about 10 presentations, mostly by vendors plus an open forum. None of the presentations were too sales-ey, which gave the event a nice workshop feel. Given the limited size, it would have been nice to have more time for interaction. Here is a quick rundown of my impressions.
The Internet is cruel. So you finally made the leap from EAI stone age to become a hip mashup developer. Now it turns out you are once again behind the curve, Unless you've written your won Facebook app and scored a million or so installs you are just a worm in the dust of the global data superhighway. Join the club.
As I reported from Mashup Camp, an increasing number of vendors play in the mashup space. I am obviously not the only one who noticed. Dion Hincliffe recently discussed 17 different mashup tools. Maybe I have an MBA hidden inside me (or 10 years of consulting left some marks), but I was temped to conduct my own market segmentation. Gartner would be proud of me. Just be aware that my analysis is…
Mashups pull data from different sources, aggregate and transform the data to be used in different contexts. EAI solutions pull data from different sources, aggregate and transform the data to be used in different contexts. Huh?
I attended the first day of Mashup Camp today. The event took place at the Computer History Museum, which is actually walking distance from my office. Ironically, the only day in the year when something actually happens within walking distance in Silicon Valley it had to rain. Oh well.
Last time I claimed that users like events. This time I want to show how I fulfilled my personal desire for events off the Web. I built two solutions to alert me to new book reviews on Amazon, one using a Python script, the other using Yahoo! Pipes.