Designing Integrated High Quality Linux Applications

Imagine you'll install that expansive product your company bought from ACME, and realized you'll have to do the following:

Sounds familiar? Who never faced this sad situation, that inducts the user to make mistakes? If your products' installation process sound like Uncompress-Copy-Configure-ConfigureMore-Use, like this one, you have a problem, and the user won't like it.

Users like to feel that your Product integrates well with the OS. You should not demand that the OS adapt himself to your Product (changing environment variables, etc). It must let the user Install-and-Use.

The Install-And-Use glory is easily achieved using a 3 ingredients receipt:

We'll discuss here what are these ingredients and how to implement them.

Let's see how universal is this concept analyzing some types of softwares:

Pay attention that the Software on its Own contains all your product business logic, which could be useless if you hadn't a Configuration to define how to work with a data bundle, provided by the user. So, Configurations are what connects your product to the user.

We can use a metaphor about a Sculptor (business logic), that needs Bronze (content) and a Theme or Inspiration (configuration) from a Mecenas (user), to produce a beautiful work (content). He make annotations in his Journal (logs) about his day-by-day activities, to report to his Mecenas (user).

OK, so let's be more practical. The fact is, if we correctly use the universal parts concept, we greatly improve the quality of our Product. We'll do that simply separating, encapsulating, each one of these parts in different system directories (having only different files for each part is not sufficient). There is a standard called FHS that defines the Linux directories for each part, and we'll discuss it later in Section 4, “Linux Directory Hierarchy: Oriented to the Software Parts”.

By now let's see the value of this separation to the user:

Let's make some exercise with separation using as example a system called MySoftware, in which the business logic is in Example 1, “A Shell program referring an external configuration file” and the configuration is in Example 2, “File containing only the configurations for MySoftware”.

Example 1. A Shell program referring an external configuration file

#!/bin/sh

#############################################################################
##
## /usr/bin/MySoftware
##
## Business logic of MyProgram system.
## Do not change nothing in this file. All configuration can be
## made on /etc/MySoftware.conf
##
## We'll not support any modifications made here.
##

# Default configuration file
CONF=/etc/MySoftware.conf	

# Minimal content directories
MIN_CONTENT_PATH=/var/www:/var/MySoftware/www	

if [ -r "$CONF"]; then
        . "$CONF"	
fi

# All the content I'll serve are the "minimal" plus the ones provided
# by the user in the configuration file $CONF
CONTENT_PATH=$MIN_CONTENT_PATH:$CONF_CONTENT_PATH	

.
.
.

	Definition of the configuration file name.
	Definition of some static parameters.
	The configuration is readed from an external file, if exists.
	After reading the configuration file, all content directories -- user's + product's -- goes together in the $CONTENT_PATH, that will be used from now on.

Example 2. File containing only the configurations for MySoftware

#############################################################################
##
## /etc/MySoftware.conf
##
## Configuration parameters for MySoftware.
## Change as much as you want.
##

# Content directory.
# A ':' separated list of directories for your content.
# The directories /var/www and /var/MySofware are already there, so
# include here your special directories, if any.
CONF_CONTENT_PATH=/var/NewInstance:/var/NewInstance2	

# Your e-mail address, for notifications.
EMAIL=john@mycompany.com	

# Logs directory
LOG_DIR=/var/log/myInstance	

These are user defined parameters.

When I was a system administrator for IBM e-business Hosting Services, I was fascinated by Apache's flexibility letting us do things like this:

bash# /usr/sbin/httpd &
bash# /usr/sbin/httpd -f /etc/httpd/dom1.com.br.conf &
bash# /usr/sbin/httpd -f /etc/httpd/dom2.com.br.conf &
bash# /usr/sbin/httpd -f /etc/httpd/dom3.com.br.conf &

If we don't pass any parameter (like the first example), Apache loads its default, hardcoded configuration file from /etc/httpd/conf/httpd.conf. We built other configs, one for each customer, with a completely different structure, IP address, loaded modules, content directory, passwords, domains, log strategy etc.

This same concept is used by a text editor of a multiuser desktop (like Linux). When the code is loaded, it looks for a configuration file on the user's $HOME, and depending who invoked him (user A or B), it will appear differently because each user has its own personal configuration.

The obvious conclusion is that the Software's body (business logic) is pure e completely oriented by his manipulator's spirit (configuration). But the competitive advantage lays on how easy we switch from one spirit to another, like in Apache's example. It is very healthy to promote it to your user. You'll be letting him create intimacy, reliability, confort with your Product.

We used this approach with many different Softwares in that e-business Hosting time, and it was extremely usefull for maintenance etc. In a version migration we had total control over where were each of its parts, and upgraded and downgraded Software with no waste of time, with obvious success.

But there were some Products that refused to work this way. They had so many hardcoded parameters, that we couldn't see what divided the body from their spirit (or other parts). These Softwares were marked as bad guys and discarded/replaced as soon as possible.

We concluded that the good guys Softwares were intuitively blessed by their developer's four parts vision. And they made our life easyer. In fact, in that time we formulated this theory, that continues to prove itself.

Do you want to deploy bad guy or good guy Software?

So let's summarize what the FHS has to say about Linux directories:

You may think is a bad idea to break your Software (as a whole) in many pieces, instead of keeping it all under a self-contained directory. But a package system (RPM) has a database that manages it all for you in a very professional way, taking care of configuration files, directories etc. And if you spread your Software using the FHS, beyond the user friendliness, you'll bring an intuitive way to the sysadmin configure it, and work better with performance and security.

Now that we know where each part of our software must be installed, lets review the Universal Parts Table applied to the FHS.

If you are a systems administrator, this section is not for you. This is a subject for developers and packagers, to make sysadmin's life easyer.

The /opt and /usr/local directories are used by sysadmins to manualy non-packaged files (without RPM) of a software, precisely to not loose control over those files. Notice how separated this folder are from the rest of the system.

A manual installation process (without RPM, or based on simple file copy) is documented in forgoten document inside a drawer (if it was documented), and inside the head of who made installation. If he moves to another job, that installations becomes obscure to the rest of the team, and is a time bomb.

With RPM is different. RPM (or any other package system) is an installation "process" by itself. It is self-documented in his database and pre and post-install actions, which permits total control. Turns installations independent from who did it, turning installtions in a business process.

Installations based on coping files into /opt or /usr/local are far from providing the organization, system visibility and control that RPM provides. I can say /opt and /usr/local would be obsoleted when all softwares become RPMized.

It is very important to Linux evolution and popularization (especially in the desktop battlefield), that developers stop using this hell directories, and start using the FHS. After reading this section, if you still think this folders are good business, please drop me an e-mail.

Products that are entirely installed under one directory, use the self-contained approach, that has several problems:

Many developers believe that the "self-contained" approach let them work with several versions of the same product, for testing purposes, or whatever. Yes, agree, with this or any good reason in the planet. But remember that a High Quality Software (or Commercial Grade Software) objective is to be practical for the final user, and not to be easy to their developers and testers. Invite yourself to visit an unexperienced user (but potential customer) and watch him installing your product.

Developer, don't be afraid of spreading your files according to FHS because RPM will keep an eye on them.

If you have a business reason to let the user work with several versions of your Product simultaneously (or any other reason), make a relocatable package, which is described in the Maximum RPM book. Be also aware about the implications of using this feature, described in the same book.

Red Hat and derivated distributions allways use the directory standard, instead of /opt or /usr/local. Read what Red Hat says about this subject, and think about it.

	Note
	The Makefiles of an OpenSource Software that is portable to other UNICES must have the standard installation in /usr/local for compatibility reasons. But must also give the option, and induct the packager, to create the package using FHS specifications.

I would like to close this subject inviting the reader a se abstratir and think about any Software can be treated as an extension to the lower level Software. In the same way a third party plugin is an extension to your Software, your Software is also an extension to the OS (lower level). This is where all the Integration (from the title of this document) magic lives. So we can apply all the ease-of-use concepts we discussed before to the plugin architecture design of your Software.

You should give user the option to install only the part of your Software he wants. Imagine your Software has a client part and a server part, and both use files and libraries in common. You should break them in 3 RPMs. For instance, lets say the name of your product is MyDB, so you'll provide the packages:

and last 2 packages depends on the first. If the user is installing a client profile, he will use:

If he is installing a server profile:

This approach will help the user save disk space, and be aware of how your Software is organized.

Better then a simple graphical interface is a consistent integrated desktop. So developer, please do not reinvent the wheel using proprietary libraries. Today's Linux desktop is full-featured, complete APIs that makes your life easier.

The desktops today in Linuxland are KDE and GNOME. Try to allways use one of them, or both.

KDE is the most outstanding, offering a true consistent desktop, flexible, with an extremely elegant architecture, using components (like Microsoft's COM and COM+), intercommunication, performance etc. It is constantly evolving, and is developed in C++. Its applications have an familiar integrated look-and-feel, is light and mature. People say that KDE 3 is shiny diamond, ready to be used, and is my first suggestion to you.

GNOME also brings the integrated desktop proposal, but it is far from the maturity and ease-of-use of KDE. From the other side, is very well supported by the comunity, and good improvements are appearing.

Motif isn't an integrated desktop. It is a widgets library (button, scrollbar etc), plus a window-manager. It was born commercial, is mature and popular in commercial applications. But is considered obsolete in front of KDE and GNOME, that integrates the desktop. Motif source code was opened by the OpenGroup and because that was renamed to OpenMotif.

Java is being used more and more for graphical interfaces, specially in server Software, where the graphics are only helpers to configuration and administration.

Nowadays every desktop has a browser, and if your Product is a server application, the Web Interface is the right choice, because it lets a user administer it from anywhere. But keep in mind the security and organization of your CGIs, because they use to be front doors for crackers. Web interface (CGI) is completely different programming paradigm. Try to understand it conceptually first, starting from "how a web-server works", "what is a URL", etc, to get on this without compromising your Product's security.

Specially for a commercial Product, your Software must provide a graphical installer. Believe me, they are impressive in a demonstration, and CIOs love them.

More then just installation, a wizard helps in the initial configuration of your Product, collects info like activation key etc, and shows the developer license.

A wizard should not do more than this:

So the wizard hides the RPM installation and writes initial personalization. RPM is still responsable for putting all your software files in the correct places. This role should never be of your installer. Think that an experienced user (there are a lot of them in the Linux world) should be able to reproduce your Product installation without the graphical help, using only RPM commands. In fact, in big data centers, where people make mass installations, a graphical installer only disturbs.

RPM provides tools that help your graphical installer interact with them, like installation percentage viewer. Documentation for use are allways in the RPM manual (man rpm) and in the Maximum RPM book.

Lets review what happens when we boot Linux:

The runlevels mechanism lets Linux initialize itself in different ways. And also lets us change from one profile (runlevel) to another without rebooting.

The default runlevel is defined in /etc/inittab with a line like this:

id:3:initdefault:

Runlevels are numbers from 0 to 6 and each one of them is used following this standard:

You can switch from one runlevel to another using the telinit command. And you can see the current runlevel and the last one with the runlevel command. See bellow how we switched from runlevel 3 to 5.

bash# runlevel
N 3
bash# telinit 5
bash# runlevel
3 5
bash# 

Subsystems examples are a web-server, data base server, OS network layer etc. We'll not consider a user oriented application (like a text editor) as a subsystem.

Linux provides an elegant and modular way to organize the subsystems initialization. An important fact to think is about subsystems interdependencies. For instance, it makes no sense to start a web-server before basic networking subsystem is active.

Subsystems are organized under the /etc/init.d and /etc/rc.d/rcN.d directories:

So to make your Software start automatically in the boot process, it must be a subsystem, and we'll see how to do it in the following section.

Your Software's files will spread across the filesystems, but you'll want to provide a simple and consistent interface to let the user at least start and stop it. Subsystems architecture promotes this ease-of-use, also providing a way (non obrigatoria) to be automatically started on system initialization. You just have to create your /etc/init.d script following a standard to make it functional.

Example 6. Skeleton of a Subsystem control program in /etc/init.d

#!/bin/sh
#
# /etc/init.d/mysystem
# Subsystem file for "MySystem" server
#
# chkconfig: 2345 95 05	
# description: MySystem server daemon
#
# processname: MySystem
# config: /etc/MySystem/mySystem.conf
# config: /etc/sysconfig/mySystem
# pidfile: /var/run/MySystem.pid

# source function library
. /etc/rc.d/init.d/functions

# pull in sysconfig settings
[ -f /etc/sysconfig/mySystem ] && . /etc/sysconfig/mySystem	

RETVAL=0
prog="MySystem"
.
.	
.

start() {	
	echo -n $"Starting $prog:"
	.
	.	
	.
	RETVAL=$?
	[ "$RETVAL" = 0 ] && touch /var/lock/subsys/$prog
	echo
}

stop() {	
	echo -n $"Stopping $prog:"
	.
	.	
	.
	killproc $prog -TERM
	RETVAL=$?
	[ "$RETVAL" = 0 ] && rm -f /var/lock/subsys/$prog
	echo
}

reload() {	
	echo -n $"Reloading $prog:"
	killproc $prog -HUP
	RETVAL=$?
	echo
}

case "$1" in	
	start)
		start
		;;
	stop)
		stop
		;;
	restart)
		stop
		start
		;;
	reload)
		reload
		;;
	condrestart)
		if [ -f /var/lock/subsys/$prog ] ; then
			stop
			# avoid race
			sleep 3
			start
		fi
		;;
	status)
		status $prog
		RETVAL=$?
		;;
	*)	
		echo $"Usage: $0 {start|stop|restart|reload|condrestart|status}"
		RETVAL=1
esac
exit $RETVAL

	Although these are comments, they are used by chkconfig command and must be present. This particular line defines that on runlevels 2,3,4 and 5, this subsystem will be activated with priority 95 (one of the lasts), and deactivated with priority 05 (one of the firsts).
	Besides your Software's own configuration, this script can also have a configuration file. The standard place for it is under /etc/sysconfig directory, and in our case we call it mySystem. This code line reads this configuration file.
	Your script can have many functions, but it is obrigatorios the implementation of start and stop methods, because they are responsible for (de)activation of your Subsystem on boot. Other methods can be called from the command line, and you can define as much as you want.
	After defining the script actions, the command line is analyzed and the requested method (action) is called.
	If this script is executed without any parameter, it will return a help message like this: bash# /etc/init.d/mysystem Usage: mysystem {start|stop|restart|reload|condrestart|status}
	Here you put your Software's specific command.

The mysystem subsystem methods you implemented will be called by users with the service command like this example:

bash# service mysystem start
Starting MySystem:			[ OK ]
bash# service mysystem status
Subsysten MySystem is active with pid 1234
bash# service mysystem reload
Reloading MySystem:			[ OK ]
bash# service mysystem stop
Stopping MySystem:			[ OK ]
bash# 

You don't have to worry about managing the symbolic links in /etc/rc.d/rcN.d. The chkconfig command makes it for you, based on the control comments defined in the beginning of your script.

bash# chkconfig --add mysystem
bash# chkconfig --del mysystem

Read the chkconfig manual page to see what more it can do for you.

When you'll create the RPM, put your Subsystem script in /etc/init.d and do not include any /etc/rc.d/rcN.d link, because it is a user decision to make your subsystem automatic or not. If you include them and the user makes any change, the RPM file inventory will become inconsistent.

The symbolic links must be created and removed dynamically by the post-installation and pre-uninstallation process of your package, using the chkconfig command. This approach guarantees 100% package and filesystem consistency.