It took me a while to work it out, I will be the first to admit that.
Looking back as short a time as 3 years, I still believed that the most important part of being a developer was knowing the language. Sure, the language changed over time, C, then C++ (I like so many of those around me prided myself on an insane amount of knowledge of the functioning of vtables, everything about the STL, a dozen ways to implement a linked list (and the most efficient, at least until a better way came along). I actually started to feel the threads of the complex mass that I was trying to hold in my head beginning to unravel when Java came along and gave me new focus (of interest, I tried helping a friend with C++ fairly recently, and while I could probably patch up or maintain an application in it, I found much of the more esoteric knowledge I had is already gone).
Java, and even more so python, another language I learned at around the same time, showed me that I could write code much quicker without needing to know vtables inside out, understand the rules of name mangling, andso forth. The problem was, I didn't really listen. I started to learn Java inside out. The language was relatively simple, but look at all those libraries and frameworks to learn! Gravy! Here was a challenge worthy of my efforts.
So, in essence, I swapped intricate knowledge of an overly complex language for intricate knowledge of an overly complex platform.
Now, personally I am more fan of than detractor from the J2EE platform (or Java Enterprise Edition as it is now called). I will be the first to say that many of the technologies, EJBs in particular, deal with a pretty scary problem domain filled with threading, multi-user, scaling and network issues. It is an attempt to get a handle on all of that, and a decent attempt too.
Outside of software development, I found myself similarly obsessed with complexity and control. For example, I had a home network setup with a Web server, my own mail server, print server, and specially configured Linux-based firewall, all of which I thought I needed because I wanted things just so, and all of which extracted an enormous toll in time and effort.
What Fixed Me?
Simple: So Much To Do, So Little Time
As time went on, results became more of a focus than knowing a language and platform inside out.
It sounds like heresy, but it is something that a lot of developers have lost sight of. Language facts and examples can be gleaned from a book, or google. There is no need to try and hold them all in your head anymore.
On top of my work, extra responsibilities started drawing on my time. It was all stuff I wanted to do, but it's amazing how being behind all the time will focus you on what really matters.
Meanwhile, every time there was a network glitch, or something changed in my DSL provider's configuration, I could end up spending an evening trying to fix the problem. Meanwhile we were usually not receiving e-mail, the Web sites were down, and the pressure was on.
I would like to say I had an epiphany—that would certainly be the cool thing to say—but the truth was that this constant grind slowly wore me down. It was a gradual realization, not a sudden one, although I can remember the final point that something clicked in my head (I promise this wordy explanation has a point and it is worth the journey):
Like many people, I had received an invitation to gmail, and had started an account. Also like many people, I had created the account, messed with it for a few hours, thought "that's cool," and promptly forgotten about it.
Then one evening, my DSL provider changed its policy on port 25 traffic and closed the port (SMTP for those in the know) and suddenly I could not get e-mail. I ranted down the phone at the DSL provider and asked why it was necessary, I tried to find ways around the blockage, and eventually I implemented a fetchmail cron job to go and grab my mail off of their server, which, as it turned out, was flaky at best.
Then, I started thinking about the reason I actually ran my own mail server. The main reason was that during the course of a day, I end up using as many as seven different computers, and in order to not have to deal with mail more than once on each, or worse—not having access to mail that I dealt with on another machine—I set up an IMAP server that I controlled, so that I had centralized e-mail. Then, I remembered gmail again. I thought, it will never do everything my IMAP server can do, and I was right. It doesn't, but here is the crucial point, it did what I needed.
So, I dusted off the gmail account, redirected my domain mail to it, and never looked back. As time went on, I discovered all manner of features in gmail that make life much easier—and this is also an important point—things that I did not know to look for in my e-mail previously.
I went on, happy at not having to maintain my IMAP server or postfix any longer, converted my wife over to gmail as well, and showed her the stuff it could do, which she loved. Then, I thought nothing more of it for a while.
However, as time went on, I suddenly realized how much time I was saving now, and I started to look at how I could apply the lessons I learned in other parts of my life, both professional and otherwise.
OK, I really don't want to be trite. I don't expect these lessons will work for everyone or that you even want to hear them, but this was my experience and I wanted to share it anyway:
- It saves a lot of time and effort if you can get someone else to do the work of tasks which, for want of a better term, are not "core" to your interests. For example, I had no interest in running a mail server at home; I merely wanted the benefits of doing so. That Google was offering to do the hard work for me, and for free even, made it a no brainer.
- 90% of the features for only doing 10% of the effort is a good trade. There were a couple of minor things I lost when I let my IMAP server go, but it was a good trade. On the other hand, there were things I did need from my e-mail, and I did not want to lose sight of those necessary features.
- If you are willing to take the plunge to try a new tool or service, you may find that it brings its own benefits over what you know already.
- Getting back to "core" interests, letting gmail provide my mail service gave me more time to concentrate on the things I really wanted to. I have no problem getting into deep complexity if the situation warrants it, but you only have so much time and can't spend it everywhere.
- The cost of switching (and there is always a cost) is often minor compared with the gains that result.
Applying the Lessons
So what does this mean to development? Well, not much, yet, but the relevance is coming.
I reasoned that if I could simplify my development world, use as much in the way of services and tools to cut out the work that I didn't need to do, I could concentrate on the tasks that I needed to solve because other people have not already.
For example, NewEnergy Associates, the company I work for, has some unique "crown jewels" like the formula service engine. This is like the Excel formula engine on steroids. It models the passage of time, acquisitions and mergers, and so forth. This is clearly custom work and value add in the industry—it is core to our business.
To display the data, we use a cell grid—much like an Excel spreadsheet. We also have to feed data and formulas into the grid, which is held in a database. Writing data display grids and persistence engines to solve these two is clearly not core. That much is clear and I doubt whether many companies would take it upon themselves to write these components when off-the-shelf ones can be bought (or open source ones used)?
Grabbing a few components to do the parts that are obviously plug-and-play is not the full story. I wonder how many of you have a policy of always looking for pre-written components before embarking on designing and coding your own. Beyond that, how many times have you had to write your own because of just a couple of things that the pre-written ones couldn't do?
Now we get into the meat. Could those pre-written bought (or open sourced) components be used if you accepted a few limitations? Perhaps the persistence library can't store one of your more esoteric data types. Can you get by with dropping that data type and making it use more common data storage types? Think about whether it is worth the tradeoff.
Likewise, can you make do without a couple of features from the display grid? If there is a pre-written display grid that is easy to use, but doesn't allow pasting a range of cells into it, can you get by without the paste functionality, or come up with an inferior but workable alternative, or are you going to sit down and code your own display grid instead?
Tools can be a similar source of inefficiency. Many companies have a "recommended" tool base, and this is generally a good thing because it is a set of tools that you can train everyone for and have a standards skillsets. The problems I see with this approach generally are:
- To minimize the number of tools selected and offer the broadest possible development options from the least number of tools, very flexible but complex tools tend to be selected. In other words, do-it-all tools are selected over simple, focused ones.
- The expense of selecting a fixed set of tools can be high, so once they are set they tend not to change for a while. Meanwhile, new versions or better tools are coming out that could greatly simplify and speed development, but are being ignored because they are not approved.
Taking things a little further, and this is a decision you will have to make, maybe it is better to select simpler, targeted tools that suit projects well and can be picked up quickly, rather than one Swiss Army knife of a tool that takes weeks to learn properly and even then slows development by being over-complicated. A decent developer will be able to pick up a simple new IDE very quickly in my experience. IDEs look similar at the most basic levels (project navigator, editor window, compile, debug, breakpoints, and so forth all work very similarly among IDEs). Using an IDE that suits the project and minimizes distractions immediately makes for simple and faster development. By dropping some features you don't really need and switching tools, you can make life much easier for yourself and possibly find some new must-have features that you didn't even know you were missing.
So What's to be Done?
Based on my experience, some of the longest to write, hardest to maintain and most complex code has resulted from those projects that were rushed. It is an irony that in the development world, more haste really does mean less speed.
When a project is rushed, the temptation is to get coding. There have been many diatribes about this subject and I am sure you are already familiar with the idea that good process will save you time and produce simpler and more manageable code.
Suffice it to say that I have personal experience that will back it up. Rushed projects that lacked the proper process have resulted in complex code bases with subsystems that were difficult to test in isolation, components that were written when they could have been bought or open-source equivalents used, and complexity that arose from trying to close the last 5% gap any way that would work, however ugly.
In general, I consider the following points to be important for simplifying development long before you get to the coding phase:
- Understand the problem domain fully
- Look for outlying requirements that appear to complicate matters or prevent bought or freely available components from being used, and see whether they can be dropped or modified
- Identify those parts of the task that are core to the value you or your company is trying to add, and try and ensure that those are the parts you will spend your time on
- See what third-party components can be obtained. (Remember to include the reputation of the source as a factor in these decisions. I will more readily use a released project from Apache than a small project from a group I have never heard of—unless that project looks to be of very high quality and has passed a stringent examination.)
- For large systems, understand the architecture and responsibilities of the subsystems and figure out the APIs needed between those systems. Then, you can test those APIs thoroughly, isolating any bugs rather than trying to track them down through an entire stack. (This point is critical—poor architecture always causes extra complexity in my experience.)
Is Java the Problem?
There is a movement gaining popularity right now that Java itself is the problem, and that the only solution is to replace it with something simpler and cleaner.
I am not a complete subscriber to this viewpoint. There are many holes in it, but I understand that there is also some truth to the claims.
From the one side of things, Java was designed to be a simple language with a consistent set of rules, and the language remains largely unchanged. At its core, the language is still simple. That is a great place to start from.
Java 5 added some new features, such as the new for loop (which I believe simplifies the language further), autoboxing (which does likewise to help break down some of the confusion between primitives and Object-derived types) and some other features such as generics that are the cause of some debate as to whether they simplify or complicate the language.
It's important to remember that having features in a language does not mean that you have to use them. I land on the side of generics being a benefit, the extra type-safety, and the fact that you don't have to remember to cast everything coming out of a collection are reason enough for me, but if you don't like them, don't use them.
So, at its core Java is still a simple language. Maybe it's a little wordier than some of the slicker scripting languages out there (one of my favorites is Python, and I have known and used it slightly longer than I have used Java), but it's simple enough for anyone serious about learning it to grasp quickly.
No, Java's problem is Java Enterprise Edition (what used to be called J2EE). In fact, even more than that, it's EJBs (most people outside of the Java world still believe EJBs and Java Enterprise Edition are the same thing). Certainly any whitepaper I have ever seen railing against of the complexity of using Java Enterprise Edition bases its case on that.
Beyond that, it's not just EJBs that are complex, but the full on EJB experience. Multiple classes, strict inheritance rules, and XML-based deployment descriptors edited by hand. Yes, that is overly complex (for most things, there are occasions you may be glad that you have the option to interact at that level).
But, let's examine the alternatives to Java, and where they might be useful.
Is Ruby on Rails the answer? How about Groovy? Should we just abandon Java as overcomplicated and past it? That certainly seems to be a popular viewpoint right now, but that answer is going to have to come from you (and likely your company). Despite being a big fan of dynamic languages, I see them as a complement rather than a full alternative.
For example, I have just finished a project that uses Groovy as a fast way of constructing translators (import/export code if you prefer) between file formats and a database. The built-in syntactic and semantic support for XML and SQL was extremely useful and both simplified and sped development. For the rest of the system, however, we used Java, and drew on IDEs to speed and simplify the development there.
There is a lot of interest in Ruby on Rails for rapidly building Web front end/database back end applications right now, and it does make for a very compelling demo. One thing I have noticed, however, is that the comparison is always made to a more traditional approach of Java development, maybe using Entity EJBs, JDO or Hibernate for the persistence layer, and Struts, Tapestry, or similar for the Web front end.
What I have yet to see was a comparison of Ruby on Rails building the same app as a Java stack using the sort of high-level IDEs that are now available for building such apps in Java. To me, a face-off between Ruby on Rails and Java Studio Creator would be far more interesting. Most of the applications I have seen produced in Ruby on Rails demonstrations would only take minutes to construct in Java Studio Creator also, and they both occupy almost exactly the same problem domain.
On top of this, though, with the Java option I have more mature tools for construction and debugging, wysiwyg editing for the Web pages, and many more advantages from a more mature and feature rich tools platform. Those kind of things also speed and simplify development.
I regularly draw on Python, Jython, Groovy, Ruby, and Perl both for personal and work usage, but still Java remains the primary platform for both personal and work usage.
The point I am trying to make is that if you are finding Java development too complex (or any other platform for that matter), the complexity may not lie in the language, but rather in your approach or choice of tools. Examine those before throwing out the baby with the bathwater.
Simplify Your Tools
As you can tell from the above, I think that tools play an important part in whether development is simple or complicated.
Higher level tools can help isolate you from some of the complexity. Regular readers of Developer.com will see that in previous articles, I, and other authors, have demonstrated that Java EE doesn't mean EJBs, and if it does, that EJBs can be made easier.
My favorite approach to doing so is the aggressive use of high-level IDEs that help you manage the complexity. The best of these do not run you through a Q&A wizard and spit out boilerplate code; rather, they integrate those Java EE standards needed into the environment in a way that lets you work with the standards without getting lost in complexity.
A great example of this is Sun's Java Studio Creator. I have been involved from a very early time with this IDE; I was interested when the IDE was still known as Project Rave. The reason I like it so much is that it does what few Java tools are brave enough to do. In other words, it doesn't try and do everything.
Java Studio Creator concentrates on one very focused task: simply and rapidly creating Web-based applications using data stored in a database. This will sound very familiar to anyone who has looked at the problem domain Ruby on Rails is aiming for.
As a bonus, Java Studio Creator can also consume (in other words, be a client for) Web Services and EJBs. It does so in a similarly simple, straightforward, and fast way. It cannot create EJBs or Web Services (at least it does not offer you any assistance at all in doing so), but it makes client apps for these systems very easy to construct.
I am delighted to see that Sun, Oracle, and others are starting to do the same with many of their development tools at present. Oracle's recent versions of JDeveloper try and limit the complexity by trying to tailor the IDE to the type of project you are developing, and not overwhelming you with choices. Sun's NetBeans 5.0 is concentrating on simplifying EJB and Java EE development through the use of high level integration removing you from having to write or edit any of the XML configuration files. On top of that, their new Matisse GUI builder is exactly the way things should be going (finally, a really easy and productive Swing GUI builder). I am not a GUI guy, so the easier and quicker I can throw a GUI together, the quicker I can concentrate on the real development work.
As an aside, just recently I had the benefit of helping a good friend get a new start using Java after many years of C/C++. It had been a long time since I had looked at the Java world through a beginner's eyes, and watching his stumbling points was a rare opportunity to make some observations about the problems of getting started.
He started using Eclipse, a natural enough choice because it is the most widely used IDE for Java, and immediately got lost in its myriad of options from the start. Having started with the Eclipse IDE from the viewpoint of already knowing Java well, I knew that it was a powerful IDE that could do everything I wanted. The problem was that it never occurred to me how alien or overwhelming it might be for someone new to the whole field.
Another thing that bit him early was the lack of built-in support for GUI building and easy EJB construction. Although EJBs in particular are an advanced subject, it was clear that he would need to understand them before building the large system his company was planning. Selecting and downloading plugins for EJBs and GUI development was extra time and complexity.
Be an Engineer, not a Reference Book
I once heard a story that Albert Einstein refused to learn his own telephone number, saying that he would not clutter up his brain with information that could be retrieved from a book.
I don't know whether the story is true, and I suspect that it isn't, but it is still a great story with a good moral. If you have a limited amount of time, it is better to spend it on understanding the problem than understanding the tools and language inside out.
I don't mean to put anyone off of learning about a language or tool. If that floats your boat, then obviously knowing both the problem domain and the language and tools thoroughly well make you a more effective engineer. But if time forces you to choose, choose understanding the problem domain thoroughly, and pick the rest up from books as you go.
Worried About Outsourcing?
By sticking to the core tasks, those that bring direct value to you or your company, those that you were hired to do, what is there to be outsourced? Persistence layers and hundreds of JSP pages are ripe for being outsourced, but if you are able to construct these things yourself very quickly and simply, then get back to concentrating on the real value add, what's to be outsourced?
Experience the Speed
I have no real interest in GUI construction, Object Relational mapping, and other commonplace tasks. I am very interested in tools that let me do them and get them out of the way as quickly and as simply as possible, because that lets me get back to the real work.
Choose Your Battles
Of course, you can't simplify everything to the point of absurdity. There are some things that are not already done, things that add real value. These are the ones that you should be concentrating on. If you want to drop out of complexity altogether, software development probably isn't the right field for you. Choose your battles, concentrate on those problems you can really bring value to, and for the rest, re-use whatever you can.
This was a difficult article to write. There is so much more I would like to have said, so many areas where I would have liked to have backed up what I said more, but the article was already long.
Suffice to say that the material presented here represents my best knowledge about simplifying the development process at present. It may change in the future (I hope it does; such things should always be evolving as you learn more).
I believe that complexity is often a problem of our own manufacture. It is easy to blame the language, the requirements, or any one of a number of other sources, but the truth is that it is usually process and approach that cause complexity.
A language does not automatically make development complex. Anything can be coded simply in any language (even Perl).
Libraries likewise do not automatically make things complex. In fact, this is an easy one to dispel. If a library is causing your complexity, don't use it.
Stick to what is core. The more you can hand off outlying tasks to third parties, open source or commercial components, to concentrate on the core of your task, the faster and simpler your development will be (and the less likely to be outsourced).
If a few requirements are causing much more complexity, go back and re-examine whether they are really necessary. Think of creative ways to work around them to try and allow you to use ready components instead of having to write your own.
Try and be an engineer instead of a reference book. You can buy reference books and keep them on hand, ready for use. It is far better to have a good understanding of the system you are constructing than an intricate knowledge of the language and tools you are constructing it with, especially when the language and tools come with manuals, and usually the system does not.
Keep looking at new tools and options. It still appears that frameworks such as JSF and JDBC Rowsets, and tools like Java Studio Creator are relatively unknown for reducing complexity.
And finally—never run a mail server at home when gmail will do everything you need and more.
A future article will cover more details about the process that we voluntarily implemented at my company in an effort to manage the increasing complexity.
About the Author
Dick Wall is a Lead Systems Engineer for NewEnergy Associates, A Siemens Company based in Atlanta, GA that provides energy IT and consulting solutions for decision support and energy operations. He can be reached for comment on this and other matters at email@example.com. He also co-hosts the Java Posse with Tor Norbye and Carl Quinn, a podcast devoted to Java News and Interviews, which can be found at http://javaposse.com.