JLOFT: The Project Continues

It has been a month since my last blog entry because I could not recall precisely where the blog was! :-) So, I have now happily reconnected with the location and will dutifully bookmark it so as not to lose it once again.

A lot has happened in a months time and I want to communicate to everyone what this process is like from a new persons point-of-view:

First -- SCRUM (not Eiffel): The SCRUM method is something I highly recommend. However, bear in mind there are many Agile methods for you to choose from, ranging from Extreme Programming (XP) to the overly structured and documentation burdened. SCRUM hangs out somewhere in the middle and (so far) is working very well for us. The primary benefits are: Focus and Flexibility. Each Sprint (iteration) allows the team to concentrate their collective focus on a finite list of things to produce (not just to do). The question getting asked most often as a result is, "Is this what we really need to be spending our time on?" -or- "How is what we're doing now get us to the goal?" The focus of each team and team member is very tightly honed in on the tasks at hand. Additionally, there is a wonderful flexibility in the process as well.

Flexibility in each Sprint iteration or cycle is the freedom to say, "Are we in too deep on this?" In every software construction project comes the Unknown-unknown's -- it is the list of tasks that must be done to finish the objective goal that can only be known by the process of discovery. This process of discovery is very raw, real and even palpable! My own awareness of previously unknown tasks as they percolate to the surface of our awareness is quite real and almost fun to experience. Thus, as an Unknown-unknown bubbles up into plain view, we have the flexibility to ask, "Do we need to continue with this, or do we take this off our plate and put it back on the to the backlog for later?"

The answer to the questions are held consistently within the time-box surrounding the Sprint. The raw fact of a team agreement to a specific period of time in which to accomplish a list of tasks helps keeps all team members accountable to each other and constantly double and triple checking each other to keep each other on the path towards the goal. The nearly free side-effect to this reality is how feature creep is consistently gauged and watch-dogged as the Sprint iteration is underway. I cannot recall a single project I have been on where the goodness of such focus was felt by the entire team. Working the SCRUM method is truly a remarkable experience and I highly recommend any software development project adopt some form of agile methodology.

Finally -- Daily Stand-up Meetings: These meetings are another excellent tool in the SCRUM method. So far, the entire team has done a terrific job of keeping the meetings short and sweet. We only ever allow about 20 minutes (max, and usually 10 to 15) for the meeting. With three teams (e.g. two Eiffel and one SQL Server DBA), this provides about 5 to 6 minutes of talk time for one team member to answer 6 very basic questions. The questions are really simple, such as: What did you do yesterday? What do you plan for today? and Do you have anything blocking your way to completing your tasks? Simple questions asked first thing in the morning not only communicate to the entire group what is happening, but the mentally prepare each team to re-groove themselves from where they left off the day before to pick up the task and continue driving forward.

The real power of tools like SCRUM are brought into clarity when they are juxtaposed against the normal style found in many engineering shops all over the world -- that is -- silo-programmers operating in a monolithic, self-paced, day-to-day world, where the job of designing, coding, testing and delivering is a project-long blur. Consider -- for the SCRUM team, the end of each day and the end of each Sprint cycle is a list of: "We FINISHED ______!" The psychological and emotional lift for each individual and for the team as a whole is without equal. It makes all the hard stuff involved with software engineering come back to what it was when I was learning and each new thing learned-and-working brought excitement about what was next. I have personally worked in shops where we all lived in cubes and work was "tossed over the wall" (literally) from one person to the next. I will not trade back!

Second -- Pair Programming: The companion to SCRUM is pair programming. This is another highly recommended tool for any development team. Our daily experience is that of high quality code, good record keeping, excellent design and professional interaction that leads to smarter and more competent team members faster. Our experience of this has several remarkable aspects to it.

Except for our two experienced Eiffel Engineers, the entire Jinny Corp. team are newbies. Even my brief experience with Eiffel 10 years ago hardly qualifies me as an Eiffel engineer by any stretch of the imagination. Working alongside two seasoned engineers proves that to me each day. Nevertheless -- one of the goals of working as pairs (or more in our case so far) is the transfer of knowledge. This is happening very quickly for us. What was very difficult in the opening weeks of training is now fairly simple. I can liken it to graduating from elementary school and junior high into high school. Somewhere here, we're going to make that transition into feeling like full-fledged undergrads! The main thing to get from this is how the knowledge transfer rate is on, steady and flowing well.

There are a whole host of good points about Pair Programming to be pointed out. Catching simple typos is not as trivial as it seems. I often ask myself how much time has been saved by having a second (or third) pair of eyes watching as code is being typed. Having another brain involved in design is equally compelling. One of our engineers pointed out how there is a false-security that Eiffel engineers can be lulled into: I used to call it Design-by-Compiler. He pointed out how some programmers think that it is enough that the code will compile and the compiler is happy. Our other engineer then pointed out the new and next step: Design-by-Auto-test -- that is -- The compiler is happy and my new test suite is also happy, thus -- my code must be alright. In our case, we're being lifted above this false sense of security and given good tools through watching-learning-imitation-and-asking while we are side-by-side.

EIFFEL STUFF

Oh yeah -- and we're doing all of this in Eiffel! So, where have we been in the last four weeks?

Stub Generation -- Not everything goes as planned!

We tried and experiment and it ultimately didn't work out. What was the experiment and what was it about?

Eiffel has what is a new feature to me: Auto-Test. This is a great tool. I like it a lot. It will be some time before I am able to exercise the tool to the extent it can be utilized. Nevertheless -- Auto-Test can only work on effected classes. Many of the foundational classes for JLOFT are deferred and not directly testable by Auto-Test. In order to facilitate this testing, one of our engineers suggested building an Eiffel program to examine each class in the system universe and auto-generate an effected STUB class for each one deferred. We called it "StubGen."

The stub generation program ultimately failed due to the complexities of inheritance. While the concept is valid and perhaps doable with enough time and resources, it was not within our grasp with our limited budget and resources. There are several lessons I take away from this process:

1. Eiffel Studio is a collection of the finest software tools I have ever encountered. Still, it is a work-in-progress. There was a time when Auto-Test didn't exist and now -- it does. Similarly: There are tools yet to come. Perhaps you and I will help inspire and perhaps even build some of them. Who knows. The basic lesson for me is simply this: An idea of a tool or the notion of a need for a tool doesn't necessarily translate to the reality of "We can easily make this!"

2. Project leads and managers need to be mindful of "pie in the sky" and use not simply good or sound judgment to reveal the "not doable" but tools like SCRUM. Honestly, the SCRUM Sprint cycle helped us to slam on the brakes before more time and effort was spent on what turned out to be a nearly fruitless venture. For the record -- there is no one person or persons to "blame". We all thought about the idea and agreed that it was worth the try. Nevertheless -- if it were not for the good practice of having a planned and restrained design/build cycle (e.g. Sprint), there is a temptation that lingers in programmers heads that the goal is just out of reach -- perhaps only hours away. Even our engineer who first suggested the tool made this type of statement. Ultimately, the dream of imminent completion was tempered by the reality of, "We gave ourselves X amount of time. Can we realistically finish on-time?"

3. Some things matter as much as the code: After 20 years as a software engineer, I have learned that writing software and building systems is a human endeavor. It is emotional (both up and down). There are victories and defeats. Such things are deeply involved when writing code. Stated another way -- when you look at code, you'd do well to realize the humanity living behind each block, module, class and keyword. All it takes is having one project or code task fail to get the reality of this fully in your face. An immediate goodness that arose from #2 above is how it was easier to swallow this small failure than it would be to have it living in a never-ending-goal-just-beyond-our-reach-we're-almost-there-CRASH scenario and then trying to save-face and pick up the pieces after!

End-to-End -- Start Thin, Get Fat

One of the immediate triumphs was based on a suggestion from our two engineers. The suggestion was simply this: Let's build something thin from end to end. Let's make a small example system that deals with one or two tables in the database, pulls the data, processes it through the business tier and places it in the GUI or presentation tier for the end user to see. We simply called it: End-to-End.

End-to-End worked beautifully. It ferreted and culled out errors. It exposed and allowed discovery of previously Unknown-unknown's. It provided the basis for making design choices easily made, easily coded, easily tested and quickly attained. The net result were immediate extensions of base classes from the architecture. Another fine result were also attained, like Coding Standards and Naming Conventions. Moreover, there was the learning factor for a team of new Eiffel developers. It gave us a place to stretch our legs, learn and grow without fear of causing too much damage if we got off the rails (and some of us did from time to time). Even so, the small, thin slice of the system universe also allowed our experienced engineers the chance to offer guidance, learn our business logic and grow in their way of understanding what the business needs and wants as well. This isn't just a one-way street from them to us.

The net result of End-to-End was "End-to-End-2" -or- "Son of End-to-End!" We figured, what worked well the first time might be good for a second round. Actually, this was the brain-child of our IT Director, David Wallace. He felt that what we gained could be further enhanced by taking that work and extending it, enlarging it and expanding it. This we did and more.

Son-of-End-to-End is what provided a platform upon which to build the tier proxy classes (see below). It also has provided a place to expand the RM2OOM tool that ships with Eiffel. Here is one tool that is going to get an excellent make-over to allow it to produce really well crafted and even complex data access templates for ODBC. Not only will it be able to do SELECT and UPDATE commands, but have two-way interaction with SQL Stored Procedures, complete with argument passing and so on. We see these as low-hanging fruit and perhaps there is more Unknown Unknown's (fruit-wise) that we'll discover as we move along.

The net result of all this is simply proving the concepts on a thin slice of code and then scaling them up as a template for the remainder of the system. Some of our next goals for E2E-3 (End-to-End 3) are items such as: User Security, Publish-Subscribe modeling and even testing deployment over a wide are network. On that last item: Jinny is located in five (5) locations around the country. The goal is to have each local system operate both locally and on and enterprise level. Data will be shared (depending on security access). Thus, that last item about deployment is important. We want to know how deployment on a large scale will impact our design. Having an End-to-End thin system to deploy and test will teach us many lessons.

Eiffel Store

Eiffel Store was another surprise for us. Using the ODBC client to access the SQL Server database had some Unknown-unknown's of its own. For instance: We discovered that not all ODBC drivers work the same. There are errors in older drivers when it comes to Unicode. Working with the folks at ISE, we were finally able to determine the issues and get them handled. Moreover, we discovered design choices we would like to have to make working with data more convenient (e.g. UPDATE commands that get built for you based on the settings made in those template classes produced by RM2OOM).

I won't say much more about Eiffel Store in this blog because there wasn't really that much (in the end of it). Nevertheless, just know that the code is solid and good, but there is room for improvement. One of our goals in this is to exercise the Eiffel libraries very well and help solidify them even more.

MIXED_NUMBER and MIXED_QUANTITY

The ISE team provided Manu Stapf to us in the opening month of the project as our system architect. In this process, Manu was quick to note a new generic class that knows how to work with Mixed Numbers. The need for this class was spawned from our inventory need to model packages of items. For instance: We might have a product that is a package of 36 hair clips. A client buys a package and then returns several clips as damaged items. There are many products that fit this mold (e.g. packages of bottles, mittens, scarves and so on). The need to model X number of packages with Y number of loose pieces (e.g. 3 3/36 -- 3 pkgs 3 each of 36) is essential to the ability of our inventory system to properly model not only customer returns, but vendor returns and even the resale of returned items that remain salable to our customers.

In the end, MIXED_NUMBER is not what we needed. It turns out that the math needed by the inventory system is not the math used by the typical mixed number in mathematics. Thus, the MIXED_QUANTITY class was born and modeled from the MIXED_NUMBER. What has turned out to be the end result is a MIXED_NUMBER class that might find its way into the ISE Eiffel library and a well-designed, well-thought-out and well-tested MIXED_QUANTITY class for our inventory system.

N-Tier Design and Proxies

The last thing to mention on this VERY lengthy blog entry is the fine work of the PROXY classes. The PROXY classes were born out of a system need.

Our old system is single threaded. Opening the software and performing actions like the saving of an order are very labor intensive processes. They probably didn't have to be that way -- but -- they are. Nevertheless, it started a discussion of: "The main complaint from end users is -- IT'S SLOW! So, speed is one of our top priorities."

A lot of the speed issue was solved by choosing a more industrial strength database system in SQL Server. The previous software was written in Microsoft Visual FoxPro. While VFP is a very fine database system in its own right, it is older technology, single threaded and works best for a local company database system and not one spread over the entire country. Thus, moving to SQL Server was the first leg-up in this journey.

The second step is in the software. This is where multi-threading and Eiffel come in. We asked to have some measure of multi-threading in our new system to give us design flexibility to handle any problems we might encounter with slowness. Nowhere else in the old system was this felt more than in the GUI and at start-up time. What we asked for was a GUI that would start a long-running task in its own thread and then come back to the user to get more to do while the first task was underway. To this end, our engineers have hammered out a system of PROXY classes.

The PROXY classes essentially allow GUI objects to toss EXECUTION_REQUEST objects to a proxy of a business class instead of talking to the business class directly. The communication is a two-way street -- that is -- the business tier can send up requests to the GUI, getting responses back if needed. The system works on a series of queues with callback agents. A new addition to the proxies is signally code allowing threads to tell each other when they are complete, facilitating coordination between the tiers and threads.

One final thing to note about the proxies is how we've managed to use Design by Contract to allow class features on the thread boundaries to know if a the calling thread is on the right side of the boundary. Thus, a GUI thread cannot spin-up a business tier thread or the other way around. For us new guys, I think this will prove to be a real benefit to help us keep our threads on the right side of the road!