I’ll be there!

Building software modules within a team requires communicating, at a minimum,  method or class signatures and a general idea of the functionality.  This may include a detailed design document clearly specifying method signature and a pseudo-code implementation or method stubs and comments about what the method should do right in the source file.  One thing that should never be used for communicating a signature and functionality is dummy code.

Dummy code looks like real code.  It is real code but it rarely works.  Even if it compiles it doesn’t produce the correct result when run.  Nor are there any tests that prove it (in)correct.  Dummy code exists only for the sake of its author’s gratification.

Rather than talk about the desired functionality through some more productive means the dummy code author thinks he is helpfully providing a guesstimate of correct implementation.  What he has actually done is provide a jumbled mess of non-working code cleverly disguised as a working part of the project when he checks it in to the SCM.  The dummy code author thinks someone else will come along and provide the correct implementation “later”.

What he doesn’t realize is that his dummy code is almost indistinguishable from code that should be part of the project.  The next person to try to reuse this code is in for hours of WTF moments trying to figure out WTF the code is supposed to do and why it’s got code branching five levels deep.

At my first post-college job I had a wonderful mentor.  Thys (pronounced Tays, like “taste” without the second ‘t’) and I were talking one day when he asked me a question.  “Anthony, if you were given a task of writing software that would only be run once would you do it quick-and-dirty or with a well-measured, engineered approach?”

How bad could “quick-and-diry” be if the software would run only once?  I thought about it for a moment before answering.  Quick and dirty.

Thys said he would never write quick and dirty software.  Doing so figuratively creates a  monkey on the developerment team’s back.  The software that only needs to run once inevitably needs to run a second time and a third time and eventually run regularly.  Thys helped me realize that software evolves and escapes.  Useful software will run indefinitely.  Any time and effort saved on the initial write will be lost 10 times (I made that number up.  It is actually higher.  Any want to provide real data or examples?) over on the maintenance, bug fixes and rewrites.

Writing dummy code is writing software quick and dirty without any idea of the implementation details.  Bad code and bad algorithms.  The worst of both worlds.  It’s been a long time since Thys and I had that conversation but the years have proven him right again and again.  Any code worth checking in to SCM is worth writing well.  That includes unit tests.

Dummy code has no place in SCM.  If you’re not going to correctly implement a method you stub out then don’t provide an implementation at all.  Leave some comments around the method stub if you have to.  Open a ticket in the issue tracking system to let someone know the method needs an implementation and describe the input and output.  Let the implementer determine which algorithms and data structures to use.

Dummy code is a hazard to software projects.  It violates the principle of least astonishment.  Next time you think about writing some remember that the next person that uses the code will waste time figuring out why it doesn’t work as expected.

If you check code in to your SCM where other people can pull it, take a look at what you’re checking in.  Does it work?  Does it work well?  Are there tests that prove it?  Helping someone by coding less is sometimes the best help of all.

I don’t make any money running the group.  In fact it costs me money.  I do a good amount of planning, coordinating, buying, booking and raffle prize finding for the group.  I love seeing 30+ people show up for our after work meetings.   It’s a lot of work but I do it because I am passionate about emerging trends in application scalability.  It’s a labor of love that I find professionally and personally rewarding.  Like all good geeks I maintain a healthy hobby writing software outside business hours.

I’m interested in building bigger, better software.  Batch applications, database interactions, web applications, CPU intensive algorithims should be as effective as possible.  Effective does not necessarily mean efficient.  Effectiveness is based on price, time-to-market, performance, maintenance cost, etc.  Analyzing these facets is important. The tools used to arrive at the solution are important, too.

BostonSUG has wonderful speakers who generously donate their time to speak at our meetings.  These people come in from all over the country.  Just a few examples, we’ve hosted Orion Letizi and Nikita Ivanov from San Francisco, Billy Newport from Minnesota and Mike Culver from Washington.  We’ve had numerous local speakers who we appreciate just as much as those that fly to see us.  Patrick Peralta, Tom O’Hare and Rakesh Chaudhary had large audiences.  Each of these speakers is an expert in a different facet of building scalable applications.  They are industry leaders in compute clouds, data grids, object caching, streaming media and more.

One thing we haven’t touched yet is building a web site that performs well.  The person that replied to the meeting reminder said:

save your time:
only put strings in session; and only put one item per-user in the session; unless they are doing some heavy form processing.
use native clustering.
put professional load balancer in from.
max pipe.

Thanks.  That is a helpful reply.  It immediately invalidates the hours of insightful discussion we’ve had at past BostonSUG meetings that have spent less than five minutes total dedicated to the problem to which he so generously offers his solution</sarcasm>.

This is a picture of what is described in the reply post.  As an industry we have known this works for a long time.  That’s why we don’t talk about the architecture depicted by this diagram at BostonSUG.  Sure, we’ll talk about HOW an app server achieve scalability.  We’ll talk about the implementation, not just about using JBoss or Mongrel.  We got a good look inside GridGain.  We had a guided tour around Terracotta.  BostonSUG evokes deeper discussion than “small sessions, load balancer, database cluster, done!”.  You should do that anyway.

Past meetings aside we have a lot to talk about this year.  Rails 3, Java 7, CloudFront, Hibernate Shards, Google AppEngine, tons of topics that push the industry foward.  If you’re interested in evolving and not building the same old software then come to BostonSUG.  If you’re interested in building what we know works (and have known since pre-2000) then it’s not the meeting for you.

I didn’t intend for this to turn into a meeting plug but I might as well include the link to the meeting announcement.  Wednesday January 28, 2009 @ 6 p.m.  IBM Innovation Center Waltham, MA 02451

I was told, quite authoritatively, a better way of integrating with an application is to give the other application developers your messaging system and have them integrate with that.  Not just the message broker but the entire end-to-end message system, producers and consumers included.  I panicked when I heard this because I knew this guy was serious.  He wanted to build an application like this?!

Deep Integration

The messaging platform needs some kind of web server to handle the RESTful calls coming from the client application.  It needs a message broker, ActiveMQ, RabbitMQ, WebSphereMQ, take your pick.  It needs some way of consuming messages produced by the web/application server.  It needs at least one dedicated server to run it.

Hardening an appliance for use on a customer site is significantly more time consuming and expensive than running that platform in-house.  The appliance has to “just work” for the customer.  The customer also has access to the provider database.  Notice how the message consumer communicates with that provider database through both corporate firewalls.  The provider database is exposed and the customer has a box with the credentials on their premises.

The push for this “deep integration” with the customer was so that their developers wouldn’t have to learn how to make RESTful API calls.  Instead they would learn how to administer the provider platform (keeping the database credentials safe, of course), learn the Ruby/Java/C#/C++ API to put messages on the queue and learn the RESTful API anyway because that’s the easiest way to interact with the message platform.

I can’t help but wonder why the customer needs to be bothered with the provider message platform.  The provider isn’t in the business of providing a message platform.  The provider only needs to provide the services the customer is buying.  Pushing some messaging platform on them doesn’t help them get their job done.  It’s a waste of time, effort and money.  Giving them the messaging platform and saying, “here, you administer it” shows laziness and ignorance in the provider.  What could possibly be gained from making the customer use the messaging platform?  It makes the provider’s tech guys feel good because they worked on what seems like a big, important project.

RESTful Integration

Most of the time this works just fine.  Keep your message platforms to yourself.

This is the first time we’ve done the round-table style meeting and I’m excited to see how it goes.  Guests are encouraged to come prepared with questions, answers and opinions on application scalability tools, strategies and designs.  Hot topics will include platform and software stack, cloud computing and resources, vendor tools and support and CDNs. Those are my guesses about hot topics doesn’t mean the meeting is limited to those topics.

Guests, be the regular or first-timers, will drive the direction of the discussion.  We’ll talk about solving technical problems based on past experience or serve as an advisory panel on where and when to use a particular tool.

Full meeting details are at the BostonSUG web site.  We ask that you sign up for the meeting at the meeting registration page so we have an idea of how much food to buy.  There will be snacks and bottled water at this meeting.

Hope to see everyone on the 28th at 6 p.m.!

I used jhat to find out that each bookmark object uses 1.5 kilobytes of memory. It doesn’t take much to see that a LOT of bookmarks will fit into 8 gigs of memory on a medium sized server. It would take hundreds of thousands of users to generate enough bookmark data to fill that memory. At this point it’s clear the application server box can handle the data load (that isn’t to say that it can handle the network load, however).

I spent some time coming up with a nice data model and ORM design so accessing the data once it’s in memory is lightning fast, maybe even faster. But what about when it’s on a hard disk, deep in the database? Dragging my 700 bookmarks out of the database when I only want to view the 25 most frequently visited is a bit overkill, especially when you consider dragging everyone else’s bookmarks out at the same time. Even though this is still a small-time application I decided to be nice to the database anyway and implement some kind of paging solution. Spring Web MVC doesn’t offer much that I’m aware of in regard to pagination help so it looks like I’m on my own.

Pagination is something that must take place in the model with the help of the view. Right away we’ve got two things to deal with, though it’s best to deal with them in isolation. The first thing I did was get the model right. I’m using Hibernate for ORM and the Criteria API makes it easy to set offsets and limits. That is, I can ask for the 50 next items (limit) starting at row 20 (offset).

The Criteria API is great at getting your data into a workable format without HQL or SQL (not that either of them is a bad thing. I enjoy working with them). Let’s take a look at the method that gets some new bookmarks.

public List<Bookmark> getNewestBookmarks(int offset, int numBookmarks) { DetachedCriteria c = DetachedCriteria.forClass(Bookmark.class); c.addOrder(Order.desc("createdDate")); return getHibernateTemplate().findByCriteria(c, offset, numBookmarks); }

So this method gets the model for us, but where do the offset and limit come from? From the controller, of course. This method is called from the controller like so:

public ModelAndView newest(HttpServletRequest request, HttpServletResponse response) { int numberOfBookmarks = requestUtils.getSafeValueFromRequest(request, "numberOfBookmarks", 250, defaultNumberOfNewest); int offset = requestUtils.getSafeValueFromRequest(request, "offset", 300, 0); List bookmarks = bookmarkService.getNewestBookmarks(offset, numberOfBookmarks); Map model = requestUtils.generateBookmarkModel(offset, numberOfBookmarks, 300, bookmarks); model.put("bookmarkListName", "Newest Bookmarks"); return new ModelAndView("index", model); }

After sanity checking the offset and limit received from the request they are passed on to the getNewestBookmarks method to create and execute a somewhat unsightly query. At this point, rather than gathering up all 700 of my bookmarks only 25 or so are returned. What if I one day want to view the top 50 most visited bookmarks though? Or what if I want to scroll through them one page at a time? Passing in 0, 25 for the offset and limit of every method call won’t cut it. The view needs to tell us how many bookmarks to retrieve and where to start the search.

Let’s say I’m viewing the top 25 bookmarks in my collection and I want to view 26-50. Somehow the view has to tell the controller to tell the model about the desired offset and limit. This can be done by passing them in as request parameters. In the code sample above, the HttpServletRequest object passed in to the controller contains values for offset and limit (numberOfBookmark) variables. The JSP code below shows how to pass these parameters when a user clicks a link. (Actually I’m having a hard time getting the JSP to show up right when the code is rendered as HTML. Just click the link and view the entire file instead.)

JSP Pagination Operations

This page uses the JSTL core tag library to build the links that display the “previous/next/more/less” options. Right after the body tag there is code to create four URLs, one for each action. Each one starts off with a name and a value. c:url var=”viewMoreUrl” value=”" creates a page-scoped variable called viewMoreUrl. This variable will later be used to display the link that allows me to display five more bookmarks when I view the list. The value is left blank which tells the tag to use the current base URL in creating the new one. If I’m on a page that is located at /bookmarks/pages/viewUserBookmarks.html the new URL will start off as /bookmarks/pages/viewUserBookmarks.html.

Adding parameters to a URL is easy. Using the c:param tag we get c:param name=”numberOfBookmarks” value=”${numberOfBookmarks + 5}”. This tag appends a variable called numberOfBookmarks to the base URL described in the previous paragraph. The value is a JSP expression that uses the numberOfBookmarks value passed in the model and increases it by five to pass back in the request. Because we want the link only to display five more bookmarks and not change the offset that value will remain the value passed out in the model. That gives us a URL that looks like /bookmarks/pages/viewUserBookmarks.html?numberOfBookmarks=30&offset=0.

When this URL is accessed the request values for offset and numberOfBookmarks are exposed in the HttpServletRequest object for use by the controller. When the getNewestBookmarks method is called the offset and limit are 0 and 30 instead of 0 and 25.

Obtaining a new offset value is done like getting a new limit value. We start off with the same base URL and leave the numberOfBookmarks value unchanged. We need to change the offset value though. c:param name=”offset” value=”${offset + numberOfBookmarks}” creates the offset value of the URL. If we are viewing the newest bookmarks (offset 0) and we are viewing 30 bookmarks when we click the link we should see bookmarks 31-60.

Here is an interesting thing to consider: what if a user goes to the second page using the default values. The user will be viewing bookmarks 26-50. The user then clicks the “view more” link three times and is now viewing 40 bookmarks. The “previous” will now request that the offset be -15. Unfortunately we can’t have negative bookmarks. In this case the controller has some sanity checking built in to it. The input parameter values must always be validated before they are used. The controller uses a utility method to get a safe value for the offset and limit and ensures that neither can be a negative value. This safe value is then passed back to the view once the model has been built. Even though the offset in the request says -15 the controller will only go as low as 0. More interesting work takes place in the generateBookmarksModel method, which I will talk about more next week.

What did I learn from making this effort? Well, giving 8GB of RAM to an application server is a little overkill for a domain model like this. The pagination that I came up with is still rudimentary, but a little more work could add more features, namely displaying page number links and the ability to change the number of bookmarks added at a time.

What is gained from this effort? Being nice to the database opens up a few doors. It dramatically improves scalability because it allows these requests to be stateless. If an app server queried the database for all of my bookmarks then my session would have to be directed to that app server every time I made a request in order to not incur the cost of hitting the database again by another app server. If round-robin load balancing is used the Hibernate session on each app server would contain a copy of my data and that’s a waste of resources, most notably it’s a hit to the database that didn’t have to happen.

Reducing the load on the database by getting only what we need and stateless sessions gives me the option of scaling the database with less difficulty. A mostly-read application can get by with fewer hits to the write master and do a majority of their work from read-only slaves. Spreading complex queries over multiple read slaves is much preferable to every app server hitting one database for every query. By having stateless requests I can have more app servers (using less memory) hitting more database servers without maxing out the CPU utilization due to the cost of complex queries.

What should I do with the unused memory in the app server boxes? Rather than give it to one app server instance for use as a cache it should be given to the pool of application servers for use as a cache. One way to do that is with memcached. I’ll talk about that more in the future too.

As always please leave comments. I enjoy getting feedback on what I’ve written.

Here are some of the topics I want to talk about:

• Data growth and access – What kind of DBMS topologies allow maximum scalability without breaking the budget? What if the budget wasn’t a problem? How do you migrate your data model from hundreds of users to millions of users? Should you partition user data across shards or keep it in a central database? How do you profile data access paterns? Is your application mostly-read or mostly-write? When should you use an LDAP directory for data storage? When should you use MySQL and when should you use Oracle?
• Application server scaling – app server clustering, web server integration, load balancing, application session management, data caching
• Web Tier – load balancing reverse proxies, data caching, working with HTTP, considerations for exposing functionality via REST
• Language conisderations and platform choices – Dynamic languages vs. Static languages, Linux vs. Windows, Solaris vs. Linux, RedHat vs. Oracle, IBM vs. Oracle – How do you make these choices? What evaluation critera are most important? Which ones are misleading?
• Framework scalability – How can you scale if your framework can’t? Does Rails really scale better than Spring Web MVC? Where do PHP frameworks fit in? What are the alternatives? This is where we will investigate what you gain and lose by binding yourself to a particular framework, how to keep the coupling to a minimum and how to use your framework as a solid foundation instead viewing it as a cage.
• Emerging technology – Should you become familiar with Map Reduce and Hadoop? What kind of impact do object databases have on your application? Should you buy your own Sun or Dell boxes or use Amazon’s EC2 and S3?
• Application architecture – How do you write an application that scales? What does your application look like as it grows from servicing hundreds to millions of users? How does it handle session management? How does it access datastores? Are there any design patterns that are helpful? What are the anti-patterns to be aware of?

Like I said, I haven’t found a group dedicated to discussing these topics. If you know of one in the Boston area please let me know. Assuming there isn’t one I am willing to start one. I’d like to start off small and meet at coffee shops around Burlington or Lowell. I have no delusions that this is going to start off or even become as big as NEJUG is now. If it starts off as a few people getting together to talk about scalability trends, cool caching solutions and specific products then I’d call it a good start.

The meeting location is still TBD and will be based on how many people are interested in attending. It will probably be somewhere in Burlington, MA. There is no planned presentation at this time. Instead we will have a meet and greet and then discuss scalability trends and news, what approaches to scalability are commonly used now and what are the plans for the future.

If you are interested in coming or in finding out more information please email me at <my first name>@<my domain name>.<my tld> or leave a comment below. Please make sure to include your email address when you fill out the form so that I can get in touch with you – your email address will not be displayed on my site.