Scripting along…

This post is inspired by a discussion with a development lead of a company nearby and by the blog post “What about complexity in software“.

The specific discussion was about the idea of using a scripting language (think PHP, Python, Ruby, Groovy et al) for developing business applications and integration features.

A perfectly reasonable approach. Everything inside me rejected it.

But:

There is more context to the discussion of course – which is why the formulation above is slightly unfair to the reader. But before going into any detail, let’s disect the situation a little bit.

For simplicity let’s assume you already have some software asset that you need to develop further in some specified but rather broad direction (i.e. expectations unclear as so often – but there is a distinct feeling of necessity).

Now, every time you make a decision about adding some tool (or technology, or team split, or…) into the big picture  you do not only make a step in some direction but you do also add complexity and limit further possibilities. While you can achieve more short term, it will typically be hard to revert that other decision or to make a competing, challenging, similar decision later on.

How hard changing a decision is, depends on how easily the change can be implemented. When changes can be implemented more or less atomically and others involved are not confronted with the unexpected, we tend to call that a refactoring**. All other cases are revolutions, causing significant pain before showing tangible improvements. If the pain and risk factor is large and your organization is sufficiently political, decisions will probably not be taken at all (or late, or for political reasons) and you should generally consider yourself screwed (bye bye progress!).

That said you should naturally strive to delay the unavoidable and:

  1. Make sure you really need to pick a tool at all.
  2. Make sure to pick a tool that carries as far as possible – albeit possibly not the most sexy one short term (so that you can live with it longer – even if you do not love it).
  3. Make sure to pick a tool that can scale with your problem space.

Back to the Point

Enough of that and back to the beginning of the post and the specific question of whether to use a scripting language for application development is a smart thing to do.

Scripting languages (at least the ones named above) are tuned for productivity. The general idea is provide for productivity by requiring little declarative “overhead”, provide simple and versatile data structures, high-level rich operations, and be relaxed about type safety and sometimes syntax.

Scripting languages are great for “wide but thin”: Shallow layering of code, mostly local data structure, many variations of the same pattern. Heavy re-use of non-trivial data structures and interfaces, deeply designed call hierarchies, performance-critical code are not what you should bring to the party.

Considering bullet point two above, if you know you are in for “wide but thin” and you can handle other known implications (e.g. deployment and distribution), go for the lightness and productivity of your favorite scripting tool. If you don’t know what you are in for, pick a tool that carries further – i.e. Java (it was about time I’d say that!), or C#, or C/C++ even (depending on your problem domain again – of course) – although that may be perceived as harder.

**) The ability to refactor instead of “throw away and redo” or “live with the pain eternally” cannot be praised enough.

Advertisements

System Centric Development

In my previous life at SAP, I was first part of the team designing and developing the runtime of the Enterprise Portal product and later I was part of the Java EE Engine (a.k.a. SAP Netweaver/Java) team.

Both systems were initially really simple. Simple in the sense that a runtime developer could typically work on the whole system at once – as far as he or she was concerned at least. It would all fit into one development workspace. If you wanted to test or debug something: No problem. Just check out the whole thing, build it on your machine, and run it.

Life was good.

That changed. Bug reports were coming in for some already (internally) released version. How to find out what was really running on the other side? How to reproduce the bug in your environment? Still you could stretch things somehow. People might allow you to deploy patches and try again. Still somehow manageable. We got used to it.

But then things got more complex. And all the sudden you found yourself not anymore at the more comfortable lower layers but rather sandwiched between some framework you used (that library over here) and users of your code (this complex HR application over there). Add the versioning problem and the mess is complete.

At some point in time setting up a complete environment required some non-trivial installation and tweaking that could easily stop you for half a day. Potentially leading to the devastating outcome that the weekly build of service pack x actually didn’t really have the fix you expected because change list 52304 did not make it into that week’s build. Needless to say that developers rather stayed longer with what they had. Longer than healthy: Integration problems eradicated all deadline fantasies out there.

Was that just lack of discipline? Not really… it was a mix of non-human-manageable complexity and the superhuman forensic instinct required to identify exactly that source code revision made your environment and whose fault it was (in a system of hundreds of components that were last updated at various points in time in various source code repositories with a deep dependency graph). I guess there is a way to set things up so that everything is easily trackable in such complex environments. I doubt that anybody has managed. Not over any significant time. Not with more than a handful of interdependent components that are developed at the same time.

Life was not good anymore.

One day I was working on a Web Service interop and performance test. One side implemented in an ABAP system, the other side implemented on Netweaver/Java.

Something was broken on the ABAP side. Couldn’t get it to work. Called a developer: „Hey, ABC is not working in BYX, any idea why?“, „Hmm.. let’s see… (click, click,…) ah yes, there is that bug XYZ. It’s a trivial fix. Ok. fixed. Can you try again?“. Wow… works!

Something was broken on the Java side. Couldn’t get it to work. Called a developer. „Hey, DEF is not working on my installation. Any idea why?“. „What version do you have?“. „Should be DWI CW34“, „Let’s see… (wait)… I think that bug TFB was fixed in CW42“. „Can you send me a patch?“. „Hmmm…(wait)… you should deploy the latest SDAs (deployables) from here,… but let’s see… they require this and that.. you could try?!“. Well… ok,.. maybe… maybe better not.

That is what System Centric is all about:

Know what is in the system. Be able to modify anything in the system down to the single line level. Allow system-local modification. Be self-contained: Always keep the overall consistency by definition and without dependency on external infrastructure.

In a system centric approach the difference between the source and configuration store vs. the executing runtime is the same as the difference between a blueprint of a house and an actual house. The runtime is nothing more than an actual instance of the system exactly as described and configured in the system’s repository. When looking at the repository you would say “this is the system”, just as when looking at the blueprint of a house you would say “this is the house”. We like to say The source is the system.

Contrast that with a server that is build of a mix of binary components pulled from many different places cobbled together with a number of deployed applications from even more indefinite sources. Phew…

Want more? Go here: http://www.z2-environment.eu/v20doc.

Set them free: Using Lombok in z2

As you have undoubtedly learned by now, z2 compiles on its own, on-demand. It uses the Eclipse Compiler for Java (ECJ) for that. It used to use the build it Javac from the JDK more on that in some other post.

Lombok is a rather cool (although maybe slightly disturbing) approach to add getters and setters and more to pretty plain Java objects w/o typing them. A minor feature in a way – and maybe not really important – but definitely a noise remover. And a cool example to show off how well z2 holds things together even when enhancing the compilation step.

Using Lombok, you can use POJOs w/o the need to declare setters and getters before you actually have a good reason to do so:

package test;

import lombok.Getter;
import lombok.Setter;

public class MyBean {

    @Getter
    @Setter
    private String text;
}

Lombok, under the hood makes sure that methods setText, getText are available.

Compiling Java source code to Java byte code is  just one of the things that need to be happening before code can be executed – in some cases. To give extensions a chance to be able to enhance what happens during compilation, z2 offers a compiler API.

Both use-cases apply:

Modifying source code before compilation: In some cases it is most handy to prepare source code before compilation already – like a preprocessor. This is what we use for Lombok below. If you know Lombok, that may come as a surprise. Read on.

Modifying byte code after compilation: Other tools understand byte code already and better than source code. So you can do that as well. This method was used in the z2@Spring distribution supporting weaving of the AspectJ aspect of the spring framework. No idea what that means? Read on here: Spring features in Z2, Spring AspectJ.

But back to Lombok: Here is a short descriptions how it’s done. More links below.

The compiler API is very simple. There is only one interface: ICompiler, that has one method:

boolean compile(String compName, File[] src, File dest, ClassLoader cl);

Whenever called, the job of a compiler is to transform source files in the folders passed in src or compilation results of other compiler invocations in the folder passed as dest as desired.

The set of compilers to be used is a Java component specific configuration. I.e. on a Java component descriptor (the z.properties file) you specify the set of compilers to apply as well as their invocation order. E.g. like this:

java.compile.order=lombok,java

The default setting is “java.compile.order=java”.

Now to the construction of the actual extension:

A compiler extensions is declared as a z2 component (of course). In its component descriptor it declares its type, its unique id as a compiler, as well as its implementation class:

com.zfabrik.component.type=com.zfabrik.compiler
compiler.id=lombok
component.className=com.zfabrik.impl.LombokCompiler

The implementation class uses the Delombok source converter tool as follows:

  1. Create a temp folder at <component root>/gen/lombok
  2. Compute a class path based on the class path class loader (see above). Delombok requires a class path.
  3. Process all source files in the src folders using Delombok
  4. Copy the processed source files back to where they were found originally

View the source code at: LombokCompiler.java (use z2_base/z2_base)

If you have a running z2 setup (see Check it out), you can play around with this by simply checking out the projects at

http://www.z2-environment.net/svn/z2_base/trunk/offline/lombok

(again: use z2_base/z2_base when asked)

into your eclipse workspace and sync’ing your runtime. The simple web app at /lombok demos a simple Lombok processed Bean in a JSP.

There is one pre-requisite however: You need to have the tools.jar from your JDK on your classpath. The simplest way to do so is to copy tools.jar from JAVA_HOME/lib to JAVA_HOME/jre/lib/ext.

Why is this cool?

  1. We reduced the application of lombok to your Java code to a mere one-liner
  2. From the outside, e.g. when using the Eclipsoid plugin to fullfill project dependencies, it’s as if the Java Beans were indeed normal Java Beans in the first Place.
  3. On the downside you need a JDK rather just a JRE and need to provide the tools.jar (that contains the Javac that Delombok requires).

Links:

Welcome

My name is Henning Blohm. I am co-found and co-owner of a small software startup, ZFabrik Software, located near Heidelberg in Germany.

Previously I spent a significant part of my life at SAP AG, mostly working on Java topics. SAP does not do as much Java as some people may believe – I learned a lot by working in an environment where Java was more in a defending position rather than the champion. By far, most of the code SAP customers get to enjoy is written in ABAP, SAP’s proprietary programming language. Arguably an old-fashioned, hardly beautiful language. But don’t be fooled: It’s not about the language. It’s all about the software engineering approaches behind it.

A common tune of this blog will be just that: Things that suck in “Java-style” software engineering vs. other approaches.

One tangible result of my learnings at SAP is the z2-Environment: A “System Centric” approach to Java development (w/o breaking Java’s neck). I am sure that will play a big role in this blog.

Some other technologies that we use heavily in projects will certainly play a role. Most notably Hadoop/HBase and the Vaadin UI toolkit.

It was about time. Wanted to get started on this a long time back: A blog about all things “z”.