From Here to the Asteroid Belt (I)

When I came up with the title line, I had a completely different conclusion in mind. It’s a nice line though. In contrast to the conclusion, it stayed.

Oh and by the way: Spring is finally here:

(spring at the office, so much work lately, so little time to enjoy it)

This is one of those “what’s the right tool for the problem” posts. Most, me being no different, try to use tools they know best for essentially any problem at hand. And this is good instinct. It’s what people have always done and obviously they did something right. Knowing a tool well is of great value and typically supersedes in effectiveness the use of a tool that might be more powerful – if used correctly – but that you are not an expert at.

At scale however, when building something more complex or widely distributed, tool choice becomes decisive and intrinsic qualities such as simplicity, reliability, popularity, platform independence, performance, etc. may outweigh the benefits of tool expertise.

What I want to look at specifically is the applicability of a programming and execution platform for deployment scenarios ranging from an in-house, or SaaS deployments to massively distributed stand-alone applications such as mobile apps or desktop applications.

The latter two form the two endpoints of the custom vs. non-custom development scale and the non-distributed to arbitrarily distributed scale.

The rules are pretty clear:

In-house/SaaS: Complete control. The system is the application is the solution. There is no customization or distribution problem because everything is (essentially) 100% custom and 0% distributed.

Mobile/Desktop: No control over the single instance that is executed somewhere in the wild. Hard to monitor what is going on, minimal to no customization, potentially infinitely many instance in use concurrently.

But what about the places in between. The customized business solutions that drive our economic backbone from production sites to warehouse solutions, from planning to financials, from team productivity to workflow orchestration?


Let’s say you have an application that is part standard solution (to be used as is) but typically requires non-trivial customization, adaptation, extension to be effectively useful.

What are the options?

Option C: Maintain a code line per instance or customer

That is (still?) a popular method – probably because it is simple to start with and it makes sure the original developer is in complete control.

That is also its downside: It does not scale well into any sort of eco-system and licensing model including third-parties. For a buyer it means 100% dependency on a supplier that most likely got paid dearly for a customer specific modification and will asked to be so at any time of further adaptation and extension.

Option P: Build a plugin model on top of a binary platform API

That is the model chosen for browsers and similar applications. It works very well as long as the platform use-case is sufficiently well defined, and the market interesting enough.

It obviously requires to invest significantly into feature rich and stable APIs, as well as into an effective plug-in model, a  development approach for plug-ins, and a distribution channels or bundling/installation model.

In essence you build a little operating system for some specific application case – and that is simply not an easy and cheap task to do right.

Option S: Ship (significant) source code and support extension and customization on site

This approach has several strong advantages: You can supply hot fixes and highly special customization with minimal interference. Customization is technically not limited to particular functions or API. There is no extra cost per installation on the provider side compared to Option C.

It assumes however that the ability to change, version, and deploy is built-in and necessary tools are readily available. As code life-cycle is now managed on-site, some attention need to be paid to handle code life cycle cleanly.

From a consumer’s point of view it reduces dependency and (leaving legal considerations aside) technically enables inclusion of third-party modifications and extensions.

Scenario Determines Tool

In part II we will look at how the three different scenarios above translate into tool approaches. Stay tuned.



Not with the target audience anymore?

Lately when I listen to talks on new and hot stuff for developers, be it in person or on the internet, I have the feeling that its not possibly me who’s being talked to.

It’s not just like having seen the latest web development framework over and over again – it’s that’s it’s only ever Web frameworks – sort of. It’s the unpleasant feeling that there is a hell of a lot of noise about stuff that matters very little.

This post got triggered when my long time colleague Georgi told me about Atomist, the new project by Rod Johnson of Spring Framework fame. Atomist is code generation on steroids and will no doubt get a lot of attention in the future. It allows you create and re-use code transformations to quickly create or alter projectsrather than, say, copy-pasting project skeletons.

There may well be a strong market for tools and practices that focus on rapidly starting a project. And that is definitely good for the show. It is really far away from my daily software design and development experience though, where getting started is the least problem.

Problems we work on are the actual business process implementation, code and module structure, extensibility in projects and for specific technology adaptations, data consistency and data scalability, point-in-time recovery, and stuff like “what does it mean to roll this out into production systems that are constantly under load?”.

Not saying that there is no value in frameworks that can do some initial stuff, or even consistently alter some number of projects later (can it?– Any better than a normal refactoring?) – but over the lifetime of a project or even product this seems to add comparatively little value.

So is this because it is just so much simpler to build and marketeer a new Web framework than anything else? Is there really so much more demand? Or is this simply a case of Streetlight effect?

I guess most of the harder problems that show up in the back-ends of growing and heavily used applications cannot be addressed by technology per se – but instead can only addressed by solution patterns (see e.g. Martin Fowler’s Patterns of Enterprise Architecture) to be adhered to. The back-end is where long-lasting value is created though. Far away from the front end. So it should be worthwhile to do something about it.

There is one example of an extremely successful technology that has a solid foundation, some very successful implementations, an impressively long history, and has become the true spine of business computing: The relational database.

Any technology that would standardize and solve problems like application life cycle management, work load management – just to name two – on the level that the relational database model and SQL have standardized data operations should have a golden future.


  1. Atomist
  2. Rod Johnson on Atomist
  3. Streetlight Effect
  4. Martin Fowler’s Patterns of Enterprise Architecture

Some more…

* it may as well be micro-services (a technical approach that aims to become the “Web” of the backend). But then look at stuff like this

IT Projects vs. Product Projects

A while back when I was discussing a project with a potential client, I was amazed at how little willingness to invest into analysis and design there was. Instead of trying to understand the underlying problem space and projecting what would have to happen in near and mid-term future, the client wanted an immediate solution recipe – something that simply would fix it for now.

What had happened?

I acted like a product developer – the client acted like an IT organization

This made me wonder about the characteristic differences between developing a product and solving problems in an IT organization.

A Question of Attitude

Here’s a little – incomplete –  table of attitudes that I find characterize the two mindsets:

IT Organization Product Organization
Let’s not talk about too much – make decisions! Let’s think it through once more.
The next goal counts. No need to solve problems we do not experience today. We want to solve the “whole” problem – once and forever.
Maintenance and future development is a topic for the next bugdet round. Let’s try to build something that has defined and limited maintenance needs and can be developed further with little modification.
If it works for hundreds, it will certainly work perfectly well for billions. Prepare for scalability challenges early.
We have an ESB? Great let’s use that! We do not integrate around something else. We are a “middle” to integrate with.
There is that SAP /ORCL consultant who claims to know how to do it? Let him pay to solve it! We need to have the core know-how that helps us plan for the future.

I have seen these in action more than once. Both points of view are valid and justified: Either you care about keeping something up and running within budget or you care about addressing a problem space for as long and as effectively as possibly. Competing goals.

It gets a little problematic though when applying the one mindset onto the other’s setting. Or, say, if you think you are solving an IT problem but are actually having a product development problem at hand.

For example, a growing IT organization may discover that some initially simple job of maintaining software client installations on workstations reaches a level that procedures to follow have effectively turned into a product – a solution to the problem domain – without anybody noticing nor paying due attention.

The sad truth is that you cannot build a product without some foresight and long lasting mission philosophy. Without growing and cultivating an ever refined “design story” any product development effort will ends up as the stereotypical big ball of mud.

In the case of the potential client of ours, I am not sure how things worked out. Given their attitude I guess they simply ploughed on making only the most obvious decisions – and probably did not get too far.


As an IT organization make sure not to miss when a problem space starts asking for a product development approach – when it will pay off to dedicate resources and planning to beat the day-to-day plumbing effort by securing key field expertise and maintaining a solution with foresight.

Local vs. Distributed Complexity

As a student or programming enthusiast, you will spend considerable time getting your head around data structures and algorithms. It is those elementary concepts that make up the essential tool set to make a dumb machine perform something useful and enjoyable.

When going professional, i. e. when building software to be used by others, typically developers end up either building enabling functionality, e. g. low level frameworks and libraries (infrastructure) or applications or parts thereof, e. g. user interfaces, jobs (solutions).

There is a cultural divide between infrastructure developers and solution developers. The former have a tendency to believe the latter do somehow intellectually inferior work, while the latter believe the former have no clue about real life.

While it is definitely beneficial to develop skills in API design and system level programming, without the experience of developing and delivering an end-to-end solution however, this is like knowing the finest details on kitchen equipment without ever cooking for friends.

The Difference

A typical characteristic of an infrastructure library is a rather well-defined problem scope that is known to imply some level of non-trivial complexity in its implementation (otherwise it would be pointless):


Local complexity is expected and accepted.


In contrast, solution development is driven by business flows, end-user requirements, and other requirements that are typically far from stable until done and much less over time. Complete solutions typically consists of many spread out – if not distributed – implementation pieces – so that local complexity is simply not affordable.


Distributed complexity is expected, local complexity is not acceptable.


The natural learning order is from left to right:



Unfortunately many career and whole companies do not get past the infrastructure/solution line. This produces deciders that have very little idea about “the real” and tend to view it as a simplified extrapolation of their previous experience. Eventually we see astronaut architectures full of disrespect for the problem space, absurd assumptions on how markets adapt, and eventually how much time and reality exposure solutions require to become solid problem solvers.


Not much to say but…

Working on two super interesting posts on z2 v2.3 Maven repository support and smart property pre-processing (check out the roadmap) as well as, and on the other end of the scale,  on how to make secondary indexes for HBase applications.

Anyway, didn’t make it in time and there are seasonal priorities after all.

Hope you have a good start into 2014!


Upcoming Talks

speaker-02There are two talks on Z2 upcoming in the near term: In only two weeks from now I will be talking about “Stop wasting your time with Java build tools” on May 11th at Codemotion 2013 in Berlin. Shouldn’t be too hard to guess what that talk will be about.

ETschatten_0Secondly, we are very happy to be present at the Entwicklertag 2013 on June 5th in Karlsruhe where I will be talking about “Buildfrei skalieren für Big Data mit Z2” which is all about how Big Data application scale-out benefits from the “self-distributing” approach of Z2. There will be plenty of time to meet and talk.

Update (2013-05-16): Here is the codemotion slide dec.

Checking out Netweaver Cloud

Just recently I have been taking a look at SAP’s Netweaver Cloud offering.

SAP is not exactly known as a great technology vendor. Neither as great vendor of technology nor as a vendor of great technology. As I have worked for SAP and I have been using SAP software and as we (ZFabrik) thought more than once about creating software in the SAP space, I was curious as to how “meaningful” NW Cloud looks from my perspective, that is, from the outside of SAP and to somebody not working in an SAP account.

Getting started is really easy. Once you have an SDN account, you enter the SAP HANA Cloud Cockpit (don’t be fooled by that. It’s all called HANA regardless of whether you get in touch with anything HANA or whether you would even have any clue what HANA could possibly be. Don’t worry. Ignore it. It’s simply the latest and they couldn’t help it).

Technically, what you get in the trial is a Suse Linux VM that runs some LJS server, which presents itself very much like an Apache Tomcat Web Application Server of some 7.x version (7.0.35 in my case). Note also the PS below. In my case the OS had 2 GB of memory, the JVM had a 1GB heap with 256M perm space (which I believe cannot be changed).

In order to deploy some simple Web app, you only need to install some rather harmless looking Eclipse plugins (of course – you get some more than you would ask for, but at least you do not get something like Netweaver Developer Studio) and the NW Cloud SDK. After installation and getting over with some config as explained in the guide, you can control the VM and deployment of your apps from there.

All in all: It’s a glorified Tomcat on in a hosted Linux VM. All data administration, VM configuration, etc… it all looks rather hard coded. Add that you can only deploy a web applications and it all does not look too impressive.

So what’s the business model? Why would somebody like me be tempted to consider NW Cloud for anything? Where is the market?

It’s all along the same lines why RIM (the Blackberry maker) was so successful in the beginning: Corporate IT departments would not be willing to accept the (perceived) risk of allowing access to MS Exchange from outside of the company network. RIM took that responsibility from them. Now SAP departments are not any more adventurous. Providing access to internal SAP systems? No way! NW Cloud comes with a Connectivity Service that promises to do exactly that: Provide access to internal SAP systems (via some VPN channel or so) to applications running on NW Cloud that itself are directly accessible from the internet. A simplified development and production setup hence looks like this:

nwcloudThat does actually make sense and may give rise to some interesting opportunities for software development for SAP accounts.

Happy Easter!

Ps.:In fact, it seems that the Web Container instance used in NW Cloud is actually based on Eclipse Virgo (formerly known as Spring dm Server). So it is actually not Apache Tomcat. Similar. But not quite the same in terms of naming integration and other low-level aspects.

Z2 V2.1 is coming…

While I wanted to write about something completely different – much more technical and possibly quite boring to most – finishing up version 2.1 of Z2 keeps us so busy that I thought it may be much nicer to simply brag a bit about what is really cool in v2.1.

1. We do finally have a decent project management tool in place

We switched from Trac to Redmine and finally have issues, repositories, samples, and Wiki in one place and all linked up with each other. It’s here: Redmine at

The Wiki has turned out nice. Look here: Z2-Environment Wiki. Still some work in progress content of course.

2. We moved completely in with Git now

It’s not like we have become true Giterons now. Far from it. Subversion is supported as it was before.

Git does however fit nicely with Z2 – much more so than we thought a while back. Also, Git does help with cross-repository operations, where Subversion is rather weak.

In contrast to Subversion, with Git there is a natural tendency to have more and smaller projects. That is reflected in the repositories that make Z2 and its add-ons.

And we are know hosting all repositories ourself. Via Redmine they are all nicely integrated with the Web UI and we have all permission management abstracted out from the OS layer.

3. Add-ons

Previously we had two distributions: z2@base and z2@spring and planned for z2@hadoop. From v2.1 on there is only z2-base and add-ons and samples. I.e. there is a Spring add-on that has all the Spring modules (and a Wiki page and samples) as there is a Hadoop add-on (and a Wiki page and samples) and hopefully more.

To make use of an add-on, all you need to do is to declare another repository with Z2. Samples do so and you hardly notice. Want to try one? Takes no more than 5 minutes:

With the simple ability to create your own repositories (here Git is nice), scenario setup is simple and still cleanly separated and in relation with the original repositories:

This is really great! Previously our approach was more like: Take a copy of the z2@base repository, modify the environment, and add your repository on top.

With the modular repository approach and the overriding of the environment (see How to create your own system) and by setting up via “git pull” this has become much simpler and repeatable – and upgradable!

4. Z2 Zero Downtime Upgrades

Now to a true software feature: Zero Downtime Upgrades (implemented via the Gateway project actually). Updating stateful Web applications typically implies downtime. Most do not have serializable session state, and even if they have, you would still have a short downtime when updating. With Z2 Zero Downtime Upgrades we use the built-in capability of worker process management to implement no-downtime upgrades, by putting old worker processes in retirement until all sessions bound to them have terminated and the gateway terminates them.

Old sessions can complete with the old code, while new sessions will make use of the new code.

5. Samples

Via the Wiki and the linked repositories we have a very convenient way for us to roll-out samples and for you to try and understand them. There is samples for Spring, transaction manager integration, hadoop (in progress), and we are working on more.

6. Lots of improvements and updates

Finally we are on Jetty 8. Eclipsoid integrates with the source code resolution during debugging, etc., etc.