Web Development Techniques Focused on Scalability & Growth

All websites are built to scale out in some level. When developing a new social network or webapp it’s all too common that developers will jump on the scalability issue hoping to clear out the problem before it even begins. However the honest truth is that very few websites will grow to the scale of Facebook or Twitter where you need multiple servers.

In this article I want to discuss scaling your website based on more of an average audience range. I’m thinking in terms of blogs, corporate websites, e-commerce, or even social networks with a smaller market scale. Because in truth how you scale your code and website configuration will determine the extent of problems you’ll run into later on down the road.

Mange Organized Code Bases

Possibly the first and best suggestion I can offer is to make sure your files are organized and easy to read through. Not just in code but also in filenames and folder structures. When building a website from the ground up you are often responsible for creating the whole server structure.

The best method is to follow MVC hierarchy where you separate content, views, and backend code into alternate directories. Understand there is no ultimate right and wrong way of doing things. But it’s a nice idea to have your backend logic separated from your frontend code views.

If the website is built with no real backend then you shouldn’t run into this problem! Most pages would be created using HTML or a single PHP file which is routed to different views. Either way in these situations you have a much smaller website to scale, and thus you have a lot more options to choose from.

Optimize for Network Connections

The web server hosting your website is constantly pulling in and giving out network data. HTTP requests are common in larger server clusters especially where multiple websites are hosting on the same IP address. You want to keep your HTML grabbing as many files locally as possible. This will cut down on time requesting external resources from Google or Yahoo! or Github or wherever.

But even aside from external resources you should take a deeper look at how your own design is structured. Use a single image sprite sheet for CSS backgrounds instead of separate images so that you are only calling one HTTP request. This trick will also work great for related icons in your layout. Image compression is usually not going to make a difference but if you feature hundreds or thousands of different photos it may have an effect over time.

What you really want to compress are your HTML resources and response data. Resources could refer to external JS or CSS files which you can minify and strip out extra whitespace. Additionally response data is the typical HTML/JSON/XML you retrieve from an Ajax request. This request generates a new HTTP request method sent without reloading the entire page.

These are not always as taxing on your server compared to loading a full webpage or gallery of images. But jQuery has popularized Ajax effects and now they are extremely commonplace. Be sure you optimize the return data so it contains exactly what you need to work with – no more, no less.

Contextual Browser Caching

Forcing users to reload your page and pull from the database every single time can be a hassle. WordPress and Joomla! and other popular Content Management Systems have many plugins to include for caching effects. You’ll save loads of CPU resources by channeling database queries to a local cache storage file.

You can implement a custom setup with your own webapp on pages which are most popular with your visitors. Think of websites like Digg and Reddit which have received a tremendous amount of traffic to viral news stories. These are of course some very extreme examples, but scale the ideas down to your own website infrastructure.

You won’t need a caching mechanism on every single page because it would ultimately be a waste of server resources. For large media files such as videos it would be a good idea to keep the streamed data in cache memory where possible. Then if somebody wants to re-watch the video or stream audio later in the day it won’t require an active connection.

Ignore Scaling and Simply Maintain

One extremely simplified way to look at the scalability issue is by not even considering the idea of “scaling” horizontally or vertically. When you hit traffic numbers that are crashing your server and you need more than a dedicated machine, even then you should figure out a multi-server setup or virtualized machines in the cloud. High-class scaling as with Twitter or Instagram does not need to happen with the majority of websites.

But every single website should follow guidelines of maintaining the status quo. Give your server the best environment to serve website files without getting too clogged up on HTTP requests. Maintaining frontend HTML/CSS is a lot easier if you know how to build proper structures right from the beginning. Chances are you won’t need to edit these files very much unless you’re fixing bugs or redesigning a new layout.

Backend files will take a bit more finesse and research to understand common practices. Naming conventions are big with stuff like classes, functions, includes, and 3rd party scripts. And honestly it may be worthwhile to learn a micro-framework such as Silex or Symfony. Both libraries are great examples for newer developers to learn very good programming habits and best practices.

But your methods of maintainability will change depending on the current project at hand. You will need much less time to build a simple small business website compared to an eCommerce shop. Gauge each situation for its strengths & weaknesses before planning a development strategy. You may be pleasantly surprised by how impactful a little preemptive brainstorming can be to affect your entire workload.

Final Thoughts

Not all of these tips will be useful for every project but it’s worthwhile to spend a bit of time contemplating these ideas on your own. Consider what fallacies you may run into on your frontend or backend systems. There isn’t often a way to prepare for issues in advance other than understanding what may go wrong and solutions to possibly fix the problem.

If you end up working with alternative open source CMS solutions then scaling will certainly be much easier. Products like WordPress and Drupal have a community of developers who are very intelligent about the inner workings of the codebase. Use this knowledge to help solve problems and don’t be afraid of getting in touch with other developers to answer more specific questions.