Category Archives: Uncategorized

The next bit

I did not gain very much, however, by my inspection. Our visitor bore every mark of being an average commonplace British tradesman, obese, pompous, and slow. He wore rather baggy grey shepherd’s check trousers, a not over-clean black frock-coat, unbuttoned in the front, and a drab waistcoat with a heavy brassy Albert chain, and a square pierced bit of metal dangling down as an ornament. A frayed top-hat and a faded brown overcoat with a wrinkled velvet collar lay upon a chair beside him. Altogether, look as I would, there was nothing remarkable about the man save his blazing red head, and the expression of extreme chagrin and discontent upon his features.

Sherlock Holmes’ quick eye took in my occupation, and he shook his head with a smile as he noticed my questioning glances. “Beyond the obvious facts that he has at some time done manual labour, that he takes snuff, that he is a Freemason, that he has been in China, and that he has done a considerable amount of writing lately, I can deduce nothing else.”

Mr. Jabez Wilson started up in his chair, with his forefinger upon the paper, but his eyes upon my companion.

“How, in the name of good-fortune, did you know all that, Mr. Holmes?” he asked. “How did you know, for example, that I did manual labour. It’s as true as gospel, for I began as a ship’s carpenter.”

“Your hands, my dear sir. Your right hand is quite a size larger than your left. You have worked with it, and the muscles are more developed.”

“Well, the snuff, then, and the Freemasonry?”

“I won’t insult your intelligence by telling you how I read that, especially as, rather against the strict rules of your order, you use an arc-and-compass breastpin.”

“Ah, of course, I forgot that. But the writing?”

“What else can be indicated by that right cuff so very shiny for five inches, and the left one with the smooth patch near the elbow where you rest it upon the desk?”

“Well, but China?”

“The fish that you have tattooed immediately above your right wrist could only have been done in China. I have made a small study of tattoo marks and have even contributed to the literature of the subject. That trick of staining the fishes’ scales of a delicate pink is quite peculiar to China. When, in addition, I see a Chinese coin hanging from your watch-chain, the matter becomes even more simple.”

Mr. Jabez Wilson laughed heavily. “Well, I never!” said he. “I thought at first that you had done something clever, but I see that there was nothing in it after all.”

“I begin to think, Watson,” said Holmes, “that I make a mistake in explaining. ‘Omne ignotum pro magnifico,’ you know, and my poor little reputation, such as it is, will suffer shipwreck if I am so candid. Can you not find the advertisement, Mr. Wilson?”

“Yes, I have got it now,” he answered with his thick red finger planted halfway down the column. “Here it is. This is what began it all. You just read it for yourself, sir.”

I took the paper from him and read as follows:

“TO THE RED-HEADED LEAGUE: On account of the bequest of the late Ezekiah Hopkins, of Lebanon, Pennsylvania, U. S. A., there is now another vacancy open which entitles a member of the League to a salary of £4 a week for purely nominal services. All red-headed men who are sound in body and mind and above the age of twenty-one years, are eligible. Apply in person on Monday, at eleven o’clock, to Duncan Ross, at the offices of the League, 7 Pope’s Court, Fleet Street.”

“What on earth does this mean?” I ejaculated after I had twice read over the extraordinary announcement.

Holmes chuckled and wriggled in his chair, as was his habit when in high spirits. “It is a little off the beaten track, isn’t it?” said he. “And now, Mr. Wilson, off you go at scratch and tell us all about yourself, your household, and the effect which this advertisement had upon your fortunes. You will first make a note, Doctor, of the paper and the date.”

“It is The Morning Chronicle of April 27, 1890. Just two months ago.”

 

External links

Rewards program

http://ekoutanov.blog.com/

http://www.blogster.com/loyaltyprogram/tips-to-making-a-rewards-program-stick

http://ekoutanov.livejournal.com/

http://ekoutanov.xanga.com/762787158/how-to-make-your-loyalty-program-count/

https://www.myspace.com/585349715/blog/545721359

The next bit

I did not gain very much, however, by my inspection. Our visitor bore every mark of being an average commonplace British tradesman, obese, pompous, and slow. He wore rather baggy grey shepherd’s check trousers, a not over-clean black frock-coat, unbuttoned in the front, and a drab waistcoat with a heavy brassy Albert chain, and a square pierced bit of metal dangling down as an ornament. A frayed top-hat and a faded brown overcoat with a wrinkled velvet collar lay upon a chair beside him. Altogether, look as I would, there was nothing remarkable about the man save his blazing red head, and the expression of extreme chagrin and discontent upon his features.

Sherlock Holmes’ quick eye took in my occupation, and he shook his head with a smile as he noticed my questioning glances. “Beyond the obvious facts that he has at some time done manual labour, that he takes snuff, that he is a Freemason, that he has been in China, and that he has done a considerable amount of writing lately, I can deduce nothing else.”

Mr. Jabez Wilson started up in his chair, with his forefinger upon the paper, but his eyes upon my companion.

“How, in the name of good-fortune, did you know all that, Mr. Holmes?” he asked. “How did you know, for example, that I did manual labour. It’s as true as gospel, for I began as a ship’s carpenter.”

“Your hands, my dear sir. Your right hand is quite a size larger than your left. You have worked with it, and the muscles are more developed.”

“Well, the snuff, then, and the Freemasonry?”

“I won’t insult your intelligence by telling you how I read that, especially as, rather against the strict rules of your order, you use an arc-and-compass breastpin.”

“Ah, of course, I forgot that. But the writing?”

“What else can be indicated by that right cuff so very shiny for five inches, and the left one with the smooth patch near the elbow where you rest it upon the desk?”

“Well, but China?”

“The fish that you have tattooed immediately above your right wrist could only have been done in China. I have made a small study of tattoo marks and have even contributed to the literature of the subject. That trick of staining the fishes’ scales of a delicate pink is quite peculiar to China. When, in addition, I see a Chinese coin hanging from your watch-chain, the matter becomes even more simple.”

Mr. Jabez Wilson laughed heavily. “Well, I never!” said he. “I thought at first that you had done something clever, but I see that there was nothing in it after all.”

“I begin to think, Watson,” said Holmes, “that I make a mistake in explaining. ‘Omne ignotum pro magnifico,’ you know, and my poor little reputation, such as it is, will suffer shipwreck if I am so candid. Can you not find the advertisement, Mr. Wilson?”

“Yes, I have got it now,” he answered with his thick red finger planted halfway down the column. “Here it is. This is what began it all. You just read it for yourself, sir.”

I took the paper from him and read as follows:

“TO THE RED-HEADED LEAGUE: On account of the bequest of the late Ezekiah Hopkins, of Lebanon, Pennsylvania, U. S. A., there is now another vacancy open which entitles a member of the League to a salary of £4 a week for purely nominal services. All red-headed men who are sound in body and mind and above the age of twenty-one years, are eligible. Apply in person on Monday, at eleven o’clock, to Duncan Ross, at the offices of the League, 7 Pope’s Court, Fleet Street.”

“What on earth does this mean?” I ejaculated after I had twice read over the extraordinary announcement.

Holmes chuckled and wriggled in his chair, as was his habit when in high spirits. “It is a little off the beaten track, isn’t it?” said he. “And now, Mr. Wilson, off you go at scratch and tell us all about yourself, your household, and the effect which this advertisement had upon your fortunes. You will first make a note, Doctor, of the paper and the date.”

“It is The Morning Chronicle of April 27, 1890. Just two months ago.”

The beginning

Ihad called upon my friend, Mr. Sherlock Holmes, one day in the autumn of last year and found him in deep conversation with a very stout, florid-faced, elderly gentleman with fiery red hair. With an apology for my intrusion, I was about to withdraw when Holmes pulled me abruptly into the room and closed the door behind me.

“You could not possibly have come at a better time, my dear Watson,” he said cordially.

“I was afraid that you were engaged.”

“So I am. Very much so.”

“Then I can wait in the next room.”

“Not at all. This gentleman, Mr. Wilson, has been my partner and helper in many of my most successful cases, and I have no doubt that he will be of the utmost use to me in yours also.”

The stout gentleman half rose from his chair and gave a bob of greeting, with a quick little questioning glance from his small fat-encircled eyes.

“Try the settee,” said Holmes, relapsing into his armchair and putting his fingertips together, as was his custom when in judicial moods. “I know, my dear Watson, that you share my love of all that is bizarre and outside the conventions and humdrum routine of everyday life. You have shown your relish for it by the enthusiasm which has prompted you to chronicle, and, if you will excuse my saying so, somewhat to embellish so many of my own little adventures.”

“Your cases have indeed been of the greatest interest to me,” I observed.

“You will remember that I remarked the other day, just before we went into the very simple problem presented by Miss Mary Sutherland, that for strange effects and extraordinary combinations we must go to life itself, which is always far more daring than any effort of the imagination.”

“A proposition which I took the liberty of doubting.”

“You did, Doctor, but none the less you must come round to my view, for otherwise I shall keep on piling fact upon fact on you until your reason breaks down under them and acknowledges me to be right. Now, Mr. Jabez Wilson here has been good enough to call upon me this morning, and to begin a narrative which promises to be one of the most singular which I have listened to for some time. You have heard me remark that the strangest and most unique things are very often connected not with the larger but with the smaller crimes, and occasionally, indeed, where there is room for doubt whether any positive crime has been committed. As far as I have heard, it is impossible for me to say whether the present case is an instance of crime or not, but the course of events is certainly among the most singular that I have ever listened to. Perhaps, Mr. Wilson, you would have the great kindness to recommence your narrative. I ask you not merely because my friend Dr. Watson has not heard the opening part but also because the peculiar nature of the story makes me anxious to have every possible detail from your lips. As a rule, when I have heard some slight indication of the course of events, I am able to guide myself by the thousands of other similar cases which occur to my memory. In the present instance I am forced to admit that the facts are, to the best of my belief, unique.”

The portly client puffed out his chest with an appearance of some little pride and pulled a dirty and wrinkled newspaper from the inside pocket of his greatcoat. As he glanced down the advertisement column, with his head thrust forward and the paper flattened out upon his knee, I took a good look at the man and endeavoured, after the fashion of my companion, to read the indications which might be presented by his dress or appearance.

 

External links

Rewards program

http://ekoutanov.blog.com/

http://www.blogster.com/loyaltyprogram/tips-to-making-a-rewards-program-stick

http://ekoutanov.livejournal.com/

http://ekoutanov.xanga.com/762787158/how-to-make-your-loyalty-program-count/

https://www.myspace.com/585349715/blog/545721359

Tips to making your loyalty program work

Wouldn’t it be nice to have simple rule in building a rewards program? Sure! But, every business is different. Size, culture, industry, budget, location, and so on, are some of the main,primary,key factors that separate businesses. For instance, gas receipts for Woolworths customers might be more beneficial than getting frequent flyer points on their own. This is because we need to fill our cars up every week. As it follows, we are pressured to shop at the supermarket regularly to get gas receipts. Now then, there happen to be several things you ought to think about when designing and implementing your loyalty program:

  • Your target market
  • The loyalty program’s objectives
  • Make it attractive
  • Keep it easy to understand
  • Easy to join
  • Promote it

To start with, it is crucial that you understand your target market or the customer demographic who purchase from your business regularly. This is vital in all businesses. This allows you to enumerate over the incentives that you can offer and your clientele are likely to appreciate.

Secondly, we need to define the desired outcomes for the loyalty program. How much will you reward to take on how many customers and to increase what percentage of sales? An prime example could be to increase profits by 10% every year after having the loyalty scheme. You can aim for a couple of objectives to begin with, after which you can add or improve them later. Having well-defined outcomes will enable you to appraise the effectiveness of the rewards program and make alterations to improve it. Staying focused on what you are aiming for also makes it easy to design a scheme that is effective.Your shop will reap a lot from a carefully designed loyalty scheme. It encourages sales and allows you to analyse your customers’ behaviour.

So have a go. Let them have what they want. Things are easy when most people come to you for muffins – you can just give them a free one after a few purchases. But promoting planting seeds to all Bunning’s loyal customers will not cut it since hardware stores sell such a broad product range and people are after different things each time. This is why big businesses like Bunnings give out gift cards so customers can choose what to redeem their hard-earned points on.Your rewards scheme must be straight-forward for shoppers to comprehent. If they don’t get it, they will not buy in. You could have one reward level applied to all customers or feature a number of reward levels for different customer demographics with different spending amount. For instance, you might give more rewards to customers who purchase over $2000 at your store in 12 months. The point note is to keep the structure of the rewards program simple to understand for your shoppers.

Punch cards are one of the easiest ways for customers to get on board. The real drawback is you can’t really monitor and analyse your customers’ buying behaviour and evaluate the efficacy of the scheme. Should you require a form for people to register, keep it simple, to the point, and include guides and information about the rewards offering. MazeCard makes it so easy for customers to join, online or using their smartphone.

Finally, the loyalty program cannot yield any real benefits if people don’t know about it. Your business needs to promote it – much like any other products or services your customers use and love. Ensure information pertinent to your rewards scheme is up-to-date, easily obtainable and displayed in the shop and on the web. Have a chat with your customers. Ask if they already are or want to join the program. Ensure your customers are fully aware of the benefits so they will be sure to hop on board.

There are many CRM businesses that will be happy to help you design a rewards program for your business. One emerging new trend is the use of online loyalty programs that don’t require punchcards and track your customers spending online. If you’re an Australian resident, you could turn to MazeCard which is both secure and very convenient. Otherwise, programs such as Punchd (which are less secure) could be the way to go if you’re in the states.

I hope this article provides you a few things to think of before building one.

Listed on: link directory

External links

Rewards program

http://ekoutanov.blog.com/

http://www.blogster.com/loyaltyprogram/tips-to-making-a-rewards-program-stick

http://ekoutanov.livejournal.com/

http://ekoutanov.xanga.com/762787158/how-to-make-your-loyalty-program-count/

https://www.myspace.com/585349715/blog/545721359

A budget HA disk stack

A budget HA disk stack

Highly available disk stacks are nothing new. At the time of writing, Dell will happily sell you a no-single-point-of-failure MD3000 (SAS) or an MD3000i (iSCSI) array with a pair of 146GB 15K RPM SAS drives for about $4,500. Not bad, eh? Still, if you’re on a first name basis with Linux and have a couple of machines to spare, you can set up a shared-nothing disk cluster for next to nothing.

Just how might that be? The good folks at LINBIT have kindly offered their Distributed Replicated Block Device (DRBD) under the GPL license. DRBD is an online disk clustering suite that, as stated in their own words, can be seen as a “network-based raid1″.

About DRBD

DRBD works by injecting a thin layer in between the file system (and the buffer cache) and the disk driver. The DRBD kernel module intercepts all requests from the file system and splits them down two paths – one to the actual disk and another to a mirrored disk on a peer node. Should the former fail, the file system can be mounted on the opposing node and the data will be available for use.

DRBD works on two nodes at a time – one is given the role of the primary node, the other – a secondary role. Reads and writes can only occur on the primary node. The secondary node must not mount the file system, not even in read-only mode. This last point requires some clarification. While it’s true to say that the secondary node sees all updates on the primary node, it can’t expose these updates to the file system, as DRBD is completely file system agnostic. That is, DRBD has no explicit knowledge of the file system and, as such, has no way of communicating the changes upstream to the file system driver. The two-at-a-time rule does not actually limit DRBD from operating on more than two nodes. DRBD supports further “stacking”, where a higher level DRBD module appearing as a block device to the operating system forks to a pair of lower-level block devices which themselves are DRBD modules (and so on).

Replication takes place using one of three protocols:

Protocol A queues the data written on the primary node to the secondary node, but doesn’t wait for the secondary node to confirm the receipt of the data before acknowledging to its own host that the data has been safely committed. Those familiar with NFS will draw relationships to “asynchronous replication” for this is, indeed, the case. Being asynchronous, it is the fastest of all replication protocols but suffers from one major drawback – the failure of the primary device does not guarantee that all of the data is available on the secondary device. However, the data on the secondary device is always consistent, that is, it accurately represents the data stored on the primary device at the time of the last synchronisation.

Protocol B awaits the response from the secondary host prior to acknowledging the successful commit of the data to its own host. However, the secondary host is not required to immediately persist the replicated changes to stable storage – it may do so some time after confirming the receipt of the changes from the primary host. This ensures that, in the event of a failure, the secondary node is not only consistent, but completely up-to-date with respect to the primary node’s data. In the authors’ own words, this protocol can be seen as “semi-synchronous” replication. This protocol is somewhat slower than protocol A, as it exercises the network on each and every write operation.

Protocol C not only awaits the response from the secondary host, but also mandates that the secondary host secures the updates to stable storage prior to responding to the primary. Because of the added disk I/O overhead, Protocol C is considerably slower than protocol B. Drawing back to our NFS example, this protocol equates to fully synchronous replication.

The protocols above represent varying levels of assurance with respect to the integrity of the data replication process and trade speed for security. Protocol A is the fastest of all, but is not particularly safe. Protocol C offers the most resiliency to failure, but incurs the most amount of latency. LINBID claim that most customers should be using protocol C. This is debatable – protocol B is just as safe, while incurring far less overheads. Protocol B only comes unstuck if both nodes were to black out or power cycle at exactly the same time. This scenario should be guarded against using a UPS and/or redundant power lines. If redundant power is not available, protocol C is, indeed, the most appropriate.

Setting up DRBD

Obtaining DRBD

DRBD has been incorporated into the Linux kernel since 2.6.33. If you’ve been blessed with an older kernel but are a paying customer of LINBIT, you might be provided with a pre-built package to match your distribution. But since this is an “on a budget” thing, you’ll just have to download a tarball distribution from the DRBD website (or get a recent kernel). The following instructions apply to DRBD version 8.3.8.1.

Building DRBD

You’re probably familiar with the famous Linux trio: configure-make-install. This one’s no different, although you do have specify an extra switch or two to get the build going.
$ ./configure --with-km --sysconfdir /etc $ make # make install

NB: In every relevant place, the DRBD documentation states that configuration files will be searched in the sequence of /etc/drbd-83.conf, /etc/drbd-08.conf, followed by /etc/drbd.conf. However, the header file (user/config.h) generated by running the configure script points to the /usr/local/etc directory instead, contradicting all documentation, including the man pages. The --sysconfig switch overwrites this behaviour. Furthermore, according to the source code of version 8.3.8.1 (user/drbdadm_main.c), there is an extra configuration file drbd-82.conf that is searched after drbd-83.conf, which has been omitted from the documentation. Our recommendation to the folks at LINBIT would be either switch the default configuration directory to /etc, or to update the documentation to indicate otherwise.

Verifying the build

After building, load the module to verify that it was built correctly:
# modprobe drbd

If modprobe fails to load the module, it may be due to the fact that DRBD has placed the module in the wrong directory – one that is not commensurate with your kernel release (this wouldn’t be the first time DRBD got confused). You can try to search for the module, as so:
# find /lib/modules -name drbd.ko

If you find the module, copy it to your /lib/modules/`uname -r`/kernel/drivers/block directory. Having done that, register the module:
# depmod -a

Alternatively, enter the drbd subdirectory of the DRBD source tree and run the following (this time forcing the kernel revision):
$ make clean $ make KDIR=/lib/modules/`uname -r`/build # make install

Then try running modprobe drbd again.

Configuring DRBD

The layout of the DRBD disk cluster must be described in a single configuration file located at /etc/drbd.conf. In our example, replication will take place over two virtual machines, interconnected by a single private link. The machines are named ‘spark’ and ‘flare’. Both hosts will be commingling on the /dev/sda3 block device. The corresponding configuration file is depicted below:

global {
usage-count yes;
}
common {
protocol C;
}
resource r0 {
device /dev/drbd1;
disk /dev/sda3;
meta-disk internal;
on spark {
address 192.168.100.10:7789;
}
on flare {
address 192.168.100.20:7789;
}
}

It’s obvious from the configuration that protocol C is being employed. The resource section lists the details of a single resource named r0. (DRBD may have multiple resources configured and operational.) The two on sections represent the configurations specific to the nodes ‘spark’ and ‘flare’. The device, disk and meta-disk entries are common to both nodes. However, if any of these items were to differ between the two nodes, you would be expected to move them down into the on sections. The address entries will invariably differ between the two nodes. I feel compelled to say that the two addresses must be cross-routable, and appropriate arrangements must be made to allow DRDB traffic to traverse any firewalls on the ports nominated in the address entry.

Configuring the metadata

DRBD requires a dedicated storage area on each node for keeping metadata – information about the current state of synchronisation between the DRBD nodes.

Metadata can be external, in which case you must dedicate an area on the disk outside of the partition you wish to replicate. External metadata can offer the greatest performance since you can employ a second disk on each node to parallelise I/O operations.

Metadata can also be internal, that is, inlined with the partition being replicated. This mode offers a worse I/O performance compared to external metadata. It is somewhat simpler, however, and does have the advantage of coupling metadata closer to the real data – in case you have to physically relocate the disk. Internal metadata is placed at the end of the partition or Logical Volume (LV) occupying the target file system. To prevent the metadata from overwriting the end of the file system, the latter must first be shrunk to make room for the metadata.

In our example we’ll be using internal metadata. In either case, metadata takes up some space on the device; the space varies depending on the size of the replicated file system. Before determining the size of the metadata, we must accurately gauge the size of the file system to be replicated. When we talk about sizes, we refer to the raw size of the file system, i.e. the amount of space it takes up on the disk – not the amount of usable space the file system presents to the applications. The best way to determine the size of the file system is to look at the size of the underlying partition or LV, since file systems tend to occupy the entire partition/LV. We’ll use the parted utility in our example of replicating /dev/sda3 – a 4GB Ext3 partition.

# parted /dev/sda3 unit s print
Model: Unknown (unknown) Disk /dev/sda3: 8193150s Sector size (logical/physical): 512B/512B Partition Table: loop Number Start End Size File system Flags 1 0s 8193149s 8193150s ext3

Determine the size of the metadata:

given by: ceiling(Size/218) x 8 + 72 = 328 (where the ceiling function rounds the input up to the nearest integer)

NB: The observant among you will notice that the actual requirement of the internal metadata block size will be smaller than the stated figure, because by shrinking the file system we’re decreasing the demand for metadata. Still, the difference in size will be negligible, and it’s simplest to compute the metadata block size from the pre-shrunk size.

Check file system for errors (Ext2/Ext3 file systems):
# e2fsck -f /dev/sda3

Calculate the new size of the FS, allowing for DRBD metadata:
given by: Size – 328 = 8192822

Resize the file system:
# resize2fs /dev/sda3 8192822s

Finally, create the metadata block:
# drbdadm create-md r0

Loading DRBD on startup

In most cases it’s desirable to load the DRBD kernel module and activate DRBD replication on start-up. DRBD is distributed with a daemon for just this purpose. (Replace {DRBD_DIR} with the directory where DRBD was unpacked to.)
# cp {DRBD_DIR}/scripts/drbd /etc/rc.d/init.d # chkconfig --add drbd

Activating DRBD

Start the daemon:
# service drbd start

Observe the status of the disks:
$ cat /proc/drbd
version: 8.3.8.1 (api:88/proto:86-94) GIT-hash: 0d8589fcc32c874df57c930ca1691399b55ec893 build by emil@flare, 2010-08-04 20:45:00 1: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r---- ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:4096408

The Inconsistent/Inconsistent disk state is expected at this point. This simply means that the disks have never been synchronised.

Initial synchronisation

The next step is the initial synchronisation, and involves the complete overwrite of the data on one peer’s disk, sourced from the disk of another peer. You must select which of the peers contains the correct data, and issue the following command on that peer:
# drbdadm -- --overwrite-data-of-peer primary r0

Now, on either of the peer nodes, do:
$ watch "cat /proc/drbd"

You will see a progress bar, similar to the one below: version: 8.3.8.1 (api:88/proto:86-94) GIT-hash: 0d8589fcc32c874df57c930ca1691399b55ec893 build by emil@flare, 2010-08-04 20:45:00 1: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r---- ns:0 nr:24064 dw:24064 dr:0 al:0 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:4072344 [>....................] sync'ed: 0.7% (4072344/4096408)K finish: 2:49:40 speed: 324 (320) K/sec

Depending on the size of your file system, and the speed of the network, this operation may take some time to complete. Using a pair of virtual machines and a virtual internal network, a 4GB Ext3 file system took about 3.5 hours to synchronise. That said, you should be able to start using the primary disk as soon as it’s up, without waiting for the synchronisation process to complete. However, refrain from performing any mission-critical operations on the primary file system until the intial synchronisation completes (even if using protocol C).

Mounting the file system

Next, we can mount the disk on the primary node. But first, we must ensure that one node is selected as the primary node. On the primary node, issue the following:
# drbdadm primary r0

Observe the output of cat /proc/drbd, having made a node primary:
version: 8.3.8.1 (api:88/proto:86-94) GIT-hash: 0d8589fcc32c874df57c930ca1691399b55ec893 build by emil@spark, 2010-08-06 08:01:01 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r---- ns:32768 nr:0 dw:0 dr:32984 al:0 bm:4 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
The output of cat /proc/drbd on the secondary node should be very similar, only the Primary/Secondary roles will appear reversed.

The order of our HA disk stack (lowest level first) is as follows:

Physical disk partition, LVM (if applicable), DRBD, file system

When mounting the disk, we refer to the special DRBD block device, rather than the actual device (e.g. /dev/sda3). Like real partitions, DRBD devices are suffixed with a 1-based index. For convenience, it’s worth appending the following entry to the end of the /etc/fstab file:
/dev/drbd1 /mnt/drbd1 ext3 noauto 0 0

The noauto option in the ‘mount options’ column tells the operating system to refrain from mounting the device at startup. Otherwise, one of the nodes would invariably fail trying to mount the file system, as only one node can have the file system mounted at any given time.

Now mount the block device:
# mount /dev/drbd1
NB: Because of the entry in /etc/fstab we don’t have to specify a mountpoint to the mount command.

So there you have it: a highly available, no-single-point-of-failure disk stack for the price of a pair of Linux boxes. And all in the time it took you to drink 17 cups of coffee.

Further reading

Gridlock

DRBD fully integrates with Gridlock – the world’s best high availability cluster. Whether you’re after a high performance, highly available shared-nothing architecture, or off-site replication and disaster recovery, Gridlock is up to the challenge.

The problem with using Linux-based (or an OS-specific) clustering software is that you’ll always be tied to the operating system.

Gridlock, on the other hand, works at the application level and isn’t coupled to the operating system. I think this is the way forward, particularly seeing that many organisations are running a mixed bag of Windows and Linux servers – being able to cluster Windows and Linux machines together can be a real advantage. It also makes installation and configuration easier, since you’re don’t have separate instructions for a dozen different operating systems and hardware configurations.

The other neat thing about Gridlock is that it doesn’t use quorum and doesn’t rely on NIC bonding/teaming to achieve multipath configurations – instead it combines redundant networks at the application level, which means it works on any network card and doesn’t require specialised switchgear.

Split brain

When running in an active-standby configuration, only one DRBD node can be made primary at any given time. Two (or more) disks coexisting in the primary state can result in the branching of the data sets. Stated otherwise, one node could have changes not visible to its peer, and vice versa. This condition is known as a split brain. When the drbd daemon is started, it will check for a split brain condition, and abort synchronisation while appending an error message to /var/log/messages.

The first step in recovering from a split brain condition is to identify the changes made to both nodes following the split brain event. If both nodes have important information that needs to be merged, it’s best to back up one of the nodes (call it node A, or the trailing node) and re-sync data from the other node (node B, or the leading node). When the re-sync is complete, both nodes will contain the data set of node B, with the latter being the primary node. Following that, demote node B to secondary status, and promote node A to primary status. Hand-merge the changes from the backup data set on node A – these changes will propagate to node B.

On the trailing node, backup the data and issue the following commands:
# drbdadm secondary r0 # drbdadm -- --discard-my-data connect r0

On the leading node, do:
# drbdadm primary r0 # drbdadm connect r0

Observe /proc/drbd – it should now show the nodes synchronising.

Having synchronised the nodes, reverse the roles and manually merge the changes on the new primary node.

Startup barrier

Be default, when a DRBD node starts up, it waits for its peer node to start. This prevents a scenario where the cluster is only booted using one node, and mission-critical data is written without being replicated onto a peer’s disk. The default timeout is ‘unlimited’, that is, a node will wait indefinitely for its peer to come up before proceeding with its own boot sequence. Despite this, DRBD will present you with an option to skip the wait. To control the timeouts, add a startup section inside the resource section, as shown below:

startup {
wfc-timeout 10;
degr-wfc-timeout 10;
outdated-wfc-timeout 10;
}

In this example, we have explicitly specified the timeout to be 10 seconds. So the node will allow some time for its peer to come up, but the absence of the peer won’t prevent the node from booting.

Synchronisation options

DRBD’s synchronisation mechanism is optimised for slow computers with slow network connections by default. This is just too bad, as the out-of-the-box configuration requires quite a bit of tinkering to get going on even the most basic hardware. The default synchronisation rate is capped at around 250 KB/s, which is roughly 2% of a 100 MBps LAN. While the presence of a throttling feature is good, its default settings are too conservative. Furthermore, DRBD by default will transmit all blocks that it thinks may be out of sync. Compare this with rolling checksums and compression used by tools such as rsync. While compression is not yet an option, it is possible to tell DRBD to compare the digests of each block with the primary’s copy, and only transfer the block if the digests differ. Bear in mind though – the use of a checksum will trade CPU cycles for bandwidth. A more free-flowing throttle cap and the use of MD5 checksums for a faster resync can be specified by adding a syncer section to the common section, as shown below:

common {
...
syncer {
rate 5M;
csums-alg md5;
}
...
}

In the example above, the sync rate has been capped to 5 MB/s, which is around 50% of the capacity of a 100Base-T Ethernet fabric, taking into account TCP/IP framing overheads. This configuration uses the MD5 algorithm to compute digests over the replicated blocks, which must be supported by your kernel (most will). The two settings are completely independent: one can specify a new throttle without setting a checksum algorithm, and vice versa.