Rolled out my first load-balanced service today and OpenBSD just makes the whole thing so much simpler. I wanted to spread the load of sending/receiving email between to Mail Servers (MX) primarily so if either machine fails, the service is not disrupted and I have time to ‘fix’ replace the broken machine.

Requirement

Due to compliance requirements to ‘eliminate’ Single Points of Failure I’m required to put up warm backups or services for most of our company servers.

Having a ‘warm’ backup server (that sits around powered on, doing nothing but waiting to be pushed into production) is such a waste of resources, so we wanted to put anything that’s a backup into ‘live’ systems.

There are many advantages to having a live failover instead of a warm backup, and suffice it to say OpenBSD gives us different ‘simple’ to configure options. Two solutions released ‘out-of-the-box’ with the base OS are:

• carp, and
• relayd

CARP

We use CARP on our firewalls, which essentially means that you have two machines set up to handle the work of a single machine. In a firewall situation, CARP provides instant failover from one host to the other in the event one of the machines fail.

For example, machine 1 as MASTER handles all traffic but also pushes needed information to machine 2 so that if machine 1 blows up, the backup machine #2 can take over the work without any users noticing the change.

CARP allows multiple servers to share the same ‘face’/IP so external hosts see only one machine although 2 or more machines may be behind the CARP configuration.

Major/Minor requirement: All hosts support CARP.

RELAYD

relayd takes advantage of OpenBSD’s firewall facilities so the firewall can act as a gateway between the ‘world’ and your disparate servers.

For example: use relayd infront of 10 web servers, so users always see the same IP.

Nice things about relayd.

1. Target Servers do not have to be OpenBSD boxes, and don’t even have to be running exactly the same thing.
   1. One of our future goals is to provide seamless load balancing for a few Windows Hosted servers.
2. Low overhead
3. Relayd monitors the target servers to make sure they are up before forwarding connections to them.
4. Relayd configuration rules are nice and simple, with simple default examples.

Read It, Learn It, Live It, Love It.

Can’t leave things alone, and have to piece together a little disinformation of my own.

US needs 'digital warfare force'

The US has set up specialised detachments dealing with IT problems

The head of America's National Security Agency says that America needs to build a digital warfare force for the future, according to reports.

Lt Gen Keith Alexander, who also heads the Pentagon's new Cyber Command, outlined his views in a report for the House Armed Services subcommittee.

In it, he stated that the US needed to reorganise its offensive and defensive cyber operations.

So, the land of the brave and the dead buffaloes, that have openly broken all forms of international law through kidnapping individuals, revoking life, liberty and the pursuit of anything to various groups and individuals in pursuit of “the American Way” is going to expect you and me to believe that all those spy satellites and telecommunication eavesdropping services do not already put them well ahead of every body else on invading not only their own Citizen’s privacy but everyone elses?

Please, …

The worrying problem is the apathy for the real loss of your privacy.

People didn’t move to encrypting their email when they all knew that the US was eaves dropping, now we people’s whole lives on the Internet being assessed and reviewed by the US machine. They’ve been tapping Australian international phone traffic since Woomera, and who knows whether the Australian Government is turning a co-operative blind eye for spying on Australian citizens internal communications.

I wonder what will finally take us over the edge for end-to-end encrypted communications (e.g. email, phone, web browsing, et. al.)

Encrypting your email is so easy these days, but it’s really hard to communicate in an encrypted manner because people find it too ‘difficult’ to use the additional tools to provide this encryption.

Avoided it for a couple of hours, but after looking it up it wasn’t that hard after all.

Summary:

I needed to connect to a client’s broadband modem to do some maintenance. Unfortunately we’ve set the client up such that administrating the modem is only possible ‘from inside’ the client’s side of the cable modem.

The 2nd problem is that the modem is administered through a web interface so the question is, how can I securely get Internet Explorer to connect through a machine on the inside back to this modem ?

In fact, only one machine on the network can access the modem.

I was side-tracked with another problem using tunnels, but the solution for this particular scenario was relatively simple.

ssh –L local-port:modem-ip:modem-port internal-host

local-port is the port on my local machine that I will point the browser to (for example: http://localhost:local-port)

modem-ip is the IP address for the modem, from the internal-host. For example, a non-routeable/private IP address such as 172.16.11.1.

modem-port. The port on the modem where the web interface is listening. For example 80 or 443

internal-host is the Host inside the network to which I can jump to from the outside (usually a machine with a public-ip)

ssh –L 4321:172.16.11.1:80 host.example.org

I can access the modem by starting up Internet Explorer and using the address http://localhost:4321

ssh –L 4322:172.16.11.1:443 host.example.org

I can now access the SSL secured interface by using the address https://localhost:4322

Using the above scenario you can supposedly daisy-chain (connect from one server to the next) by having multiple terminals making one link to the next.

There’s also some ssh fu where you can chain from one machine to the next to the next on a single command-line, but we’ll leave that for another day.

Woo hoo, built my first box in aeons.

Been playing with various bits and pieces at work trying to piece together at least another functional box. Sometime later we decided that we needed a new box and we would look at reusing as many components from the trash pile I was playing with.

Unfortunately, bits and pieces of the trash pile was working, but together there was no ensemble. We decided to get new bits for the parts that looked like were dead and yesterday was my turn to put the bits together (and pray I don’t fry anything.)

I think the last time I actually had to put a box together from scratch was back in 1998? As I recall we had a bum machine at QSC and had to get the motherboard from Australia(?) Ever since then I’ve basically had some under my wings that I told to read the Taiwanese documentation and cable the box together. Of course that was an experience in itself in finding ports not working because they just weren’t wired up.

Anyhow, a relative newbie and not wanting to ever open this box again I made sure every lose wire got plugged into something even if there was no likelihood that it would ever get used. Double checked the bits I couldn’t figure out with our resident hardware dude, crossed my fingers and pushed 240v into the machine.

Poof, no-sound, nothing! Woo hooo, go software dude. In the distant pass, when computers don’t power up, and you’re somewhat certain that the power supply works fine, pull the PCI boards out and see what happens. So, pulled out a few boards and voila machine sings beautifully.

That wasn’t too bad, now was it?

Had my first session of validating firewall rules on Monday and Tuesday, wohooo that’s an experience. My previous installations were of small systems, so I have previous experience in ‘drafting’ the firewall rules, putting it in and letting it go live. Testing and validating the firewall essentially meant sitting there in front of the firewall server and watching traffic, tweaking issues as they became known.

Firewalls are the quality of the walls between buildings. The higher grade your firewall, the higher probability your building isn’t going to burn down, should the building next door go up in flames.

The quality of the construction material of your firewall is just part of the toolkit for minimising danger to your building, you also need to ensure that there’s no open passage for the fire to enter your building while avoiding your firewall barrier. One building that went up in flames had a decent firewall, but they had large ventilation shafts between the building and the next building, leading directly to highly combustible material. Fire from the adjoining building spread into our building through the ventilation shafts and the building came down, while the firewall held firm.

The burnt building looked like the aftermath of a bombing, the inside collapsed in soot while the firewall stood alone.

Lesson 1: Physical firewalls have the same limitations as their electronic / communications firewall counter-parts. They are only as good as the material their built with, and the ventilation shafts between your side of the firewall and the next.

Unless you want to burn your firewall to test it, the general idea is to test the materials and the process of producing your firewall.

With our computer firewall firewall, we have existing best practise procedures for designing and building the firewall, and we’re now in the stage of testing the “ventilation” shafts built into our firewalls to validate whether the rules we’ve set up for what to allow in and out through the ventilation shafts behave as we expect.

I haven’t heard of any automated tools for doing the testing, so if you’ve heard of one please do tell us.

At the moment the process of testing the open ventilation shafts (in computer speak “open ports”) is to set up a simulated network on either side of our firewall and generate network traffic trying to get through the firewall in both directions. Unfortunately, the generated cannot be purely random, each “open port” or “potentially open port” has to have a specific test.

Unless you have the money, you can’t really duplicate your live network in this test environment, so you end up spending a lot of time doing the network configuration dance, continuously readjusting your various test machines to simulate other machines and providing different services as well as simulating trying to get through the firewall to the other side.

Lesson 2: You really want a set of command-line tools for doing this. Windows greater user-feedback (GUI?) is nice, but it can really use up your time when things don’t work as expected (and how often is that the truth in a test environment.)

This is when it’s good to have several machines on an independent set of networks (i.e. at minimum you’re testing the firewall with two networks) but just as importantly several monitors, keyboards, and a cool smooth swivel chair to spin around in.

Don’t bother doing this using terminal/ssh connections, that is just a recipe for frustration and avoiding configuration options you need to consider (because often enough changes you need to do will throw you out of your terminal/ssh session)

Lesson 3: Physical hardware is way cooler than the virtual world on its own.

Most of what we tested only needed testing a direct connection to the server, but our last test before quitting for the day last night was to test whether a connection from a connection would go through on a virtual connection (VPN.) Woo hoo, that wasn’t easy, but it wasn’t as hard as initially expected (since we’d done similar stuff previously.)

If you’ve got almost the cash, where you can’t afford a full simulated network, but can afford a good size beefy duo of machines for either side of the simulated network, then you would probably go with using a network of virtual machines on either side of your firewall. Now, that would be way cool, but I don’t think my laptop is beefy enough (yet)

Oh yeah, my preferred firewall ? OpenBSD with PF, of course. For user VPNs, I’m doing pretty good with installing OpenVPN.

Arrggghh, ya gotta hate those install moments that just fails because a 'package' doesn't exist, but the 'package' isn't really a package, and there's not much documentation on the web to help us out (i.e. minimise thinking)

OpenBSD 4.3 has the fontconfig libraries as part of the xbase, so you have to install it as part of the full install, or after installing your box to extract the files.

I'm kind of promoting that people don't use passwords for their connections, but to use keys instead ?

Environment:

Windows XP Desktop want to connect securely -to-
Unix Server running OpenSSH (e.g. Linux, BSD)

Why?

The primary rationale for promoting the use of keys amongst friends is the susceptibility of people to create less than 12 character passwords and easily fall into the habit of reusing or choosing simple passwords, which invariably increases the possibility that a hacker can automate an attack to get into your system.

By using keys, which are significantly more difficult to whack, you not only offer a higher level of security, but you can now use seriously difficult to crack passwords.

Thunderbird 2.0.0.X

Problem:

Printing email messages results in getting half-a-page of mail header information, before the actual message content. This is ugly as well as wasting paper and ink.

Summary:

For the past couple of months I've been having this problem with Thunderbird 2.0.0.X (5-pre at the moment) whereby printing mail messages means that I always get a print of mail envelope headers which can be very long (nearly half-a-page for some messages.) I couldn't find anything in the print-options to turn the thing off and have been looking at different options for the past month.

Today, I finally hit upon: mail.show_headers default integer 2

[enigmail: userprefs]

Replacement of Mozilla's show all headers (because the original value is overriden)
user_pref("extensions.enigmail.show_headers",1);
JS: Both mail.show_headers and extensions.enigmail.show_headers control the viewing of the headers (normal=1 / all=2).
As Enigmail needs to see all headers, it sets mail.show_headers to 2 and stores the desired view in extensions.enigmail.show_headers.
The default is derived from the setting of mail.show_headers.

Of course, once you know where the 'problem' is, it becomes easier to find the 'solution.'

Unfortunately, the printing process doesn't have a separate setting (to allow you to differentiate what you get on screen as opposed to what you get out the printer.) The solution to my printing problem is:

Set mail.show_headers to "1" (without the quotes)

But what happens to my enigmail now?

Will soon be in the market for getting a new PC, largely because this thing I'm running has hit its last legs and consistently freezes when I'm working with new image files from my 10MB digital camera.

I was going to take a look at getting the new gig from a local vendor (i.e. sorry DELL and others) but reading stories such as Jeff and Scott putting together their new machine just makes you wonder whether it isn't time to splash it on a custom home building kit.

Building a PC part 1

Over the next few days, I'll be building Scott Hanselman's computer. My goal today is more modest: build a minimal system that boots.

I'd like to dispel the myth that building computers is risky, or in any way difficult or complicated. If you can put together a LEGO kit, you can put together a PC from parts. It's dead easy, like snapping together so many LEGO bricks. Well, mostly. Have you seen how complicated some of those LEGO kits are?

Granted, building computers isn't for everybody. There are plenty of other things you might want to do with your time, like, say, spending time with your children, or finding a cure for cancer. That's why people buy pre-assembled computers from Dell. But if you need fine-grained control over exactly what's inside your PC, if you desire a deeper understanding of how the hardware fits together and works, then building a PC is a fun project to take on. You can easily match or beat Dell's prices in most cases, while building a superior rig -- and you can learn something along the way, too.

Here's the complete set of parts we ordered, per the component list.

All you need is a few basic tools to build this PC. I typically use needle-nose pliers, wire cutters, and a small phillips screwdriver.

Note to self:

a) Do not hit the DROP database in phpmyadmin

b) Do not hit the DROP database in phpmyadmin

c) Do not hit the DROP database in phpmyadmin

For some clever and stupid reason, I slacked off and hit the DROP button on phpmyadmin when I was intending to clean up some unused tables on Nomoa's database.

Guess what happened ? Two weeks of posts disappeared before you could say Kalamazoo or Niuatoputapu depending on how far away you wanted to go.

The database backup was two weeks old, so we got most of the data back, and I had some of the posts backedup on the clients I use for updating the website, but essentially. If you want to be an administrator, don't do it on a slow link from the middle of Tonga and do not, repeat do not, hit DROP.

Another adventure into the wild world of computer software, which resulted in our Gallery2 just failing altogether which seems to have been a combination of upgrading to Xaraya 1.1.3 as well as running the SVN Gallery2 2.3svn builds.

The 1st major disaster was Gallery2 2.3svn just failing to login, and there's a lot of new magic with the passwords, but there were also problems with just getting the database files working correctly.

Following no original plan, but with two thoughts in mind, I disastered that I was going to make a clean install. The two problems with the previous installation was:

• A lot of bogus users had creeped into the Xaraya installations, and by association into the Gallery2 installation.
• A lot of fluff in the database remained from the Gallery1 update to update to updates.

After quite a bit of hocus pocus, and a great deal of time trying to avoid a full new installation with the current svn code, we now have 2.2 Branch svn being used and since there's a way to switch to 2.3 when it is stable we'll go that route from now on.

Why was I on svn anyway? Because of those security faults that can wipe out your server. Now that I've learned how to use branches and switch between branches using the svn code repositories, I can be secured as soon as the code is updated instead of having to wait for a binary release etc. etc. etc.

Next problem was my Gallery Remote failing again, but fortunately we knew about that problem from previous reinstalls so we just had to find it on the web as shown below.

Gallery2 and Gallery Remote Issues

I was trying to get Gallery remote to work… but kept getting an error saying that it couldn’t find gallery_remote2.php.  After searching the Gallery forums for a few I foudn a few things, and it fixed the problem.

You need to find the  “GalleryRemote.properties”, mine was located at “C:Documents and SettingsShelby.GalleryRemoteGalleryRemote.properties”.  Open that file and add “forceGalleryVersion.n=2″ to the top line and then save it,

Next create a file named “gallery_remote2.php”, in that file add this following information.

< ?
header(”HTTP/1.0 404 Not Found”);
exit;
?>

Save that file and upload it to your gallery2/ directory and then you should be all set to use the gallery remote.

 Also @ the codex

There seems to be some interesting groupware products out there that are trying to remove Microsoft's Exchange server from the King of the Hill position it is on right now. Unfortunately,  most of the supposedly open source solutions are really closed source solutions with little teasers that are open source.

They are sort of like, we'll let you have the free cd player, radio, but you have to pay for the car. Like, give me the car without the cd player and I can put in my own!!!

Fortunately, there are a few truly open groupware products out there, and I've just come across a few that might be interesting to investigate further:

There are a number of different considerations for submerging your group into a Groupware solution, one being the maintainability of the system and the application of existing knowledge. Some of the solutions, especially the half-open source solutions, attempt to bring together a best of breed solution. In that way, you get a great, well tested base of components (mail server, calendar server, firewall, etc) that is integrated by the groupware team.

The other path, is to create everything yourself and hopefully have a better integration story (such as is with Microsoft Exchange.)

This is a common enough problem that when I came across the related discussion on the misc.openbsd mailing list I just had to summarise.

http://comments.gmane.org/gmane.os.openbsd.misc/125685

Problem:

In a given directory we have a number of photos that we would like to rename to something else. In our instance, the directory contains image files of the same naming convention, for example:

file0.jpg
file1.jpg
file2.jpg
...
...
filen.jpg

Will Backman has a great podcast on how you can better secure your communications between yourself and your servers from remote unsecured spaces through the use of One Time Passwords (passphrases) on FreeBSD, NetBSD, and OpenBSD.

The Joy of S/Key

One Time Passwords (OTP) are certainly nothing new. In fact, they have been in use for over ten years. The idea is essentially very simple: every time you login to a system, you use a different password. If someone were to eavesdrop on the connection, the password they captured would be useless to them.

In 1994, Neil Haller of Bellcore announced the “S/KEY One Time Password System” at the Symposium on Network and Distributed System Security. It described a practical way to implement OTP that was both secure and simple. Over the years it has matured into strong, practical system that is now described by RFC2289.

The initial summary of Will's podcast is

bsdtalk117 - One Time Passwords

Important when you don't trust the computer you are using, such as a library computer or internet kiosk.

Available by default in Free/Net/Open BSD.

FreeBSD uses OPIE, Net/Open use S/Key.

One time passwords are based on your pass phrase, a non-repeating sequence number, and a seed.

Initial setup should be done directly on the server.

"skeyinit" for Net/Open, "opiepasswd -c" for FreeBSD.

Now you can safely (?) login to your machine from insecure locations.

Again from The Joy of S/Key

It is true that SSH arguably does a better job of protecting passwords from eavesdroppers. In fact SSH provides for more than that, and it also protects all content from eavesdroppers. However there is one very common form of attack to which SSH is not immune: keylogging. Keyloggers record the keys you hit, and they don't care whether you're using an SSH client or telnet. They have to be installed on the machine you are using, either in software or hardware. However, now that we live in the age of Microsoft and Cybercafes, using a trojanised machine is all too easy to do. What most people don't realise is that SSH, or at least OpenSSH, is already S/KEY aware. So why not use it ?

So, please download and listen to the podcast

Cyber Punk Cafe: You silly rabbit Linux is for kids

Computers are relatively inexpensive these days however the software that runs on the computer is still at a fairly high cost if you are looking at purchasing several titles. It seems ridiculous to go out and buy 1000's of dollars worth of software that your children can play educational games and do their school work. If you own an older computer (Pentium 2 or newer) and have a few hours free one weekend you can build your children a great computer so that they can play educational games and do their school work. Now you can finally have your computer back.

How you ask?

You can make your kids a usable computer by installing a free open source operating system  and some great 100% free open source applications. So first we need a operating system,  My choice of operating system for this particular task would be Ubuntu Linux primarily because it is as easy to use as Microsoft windows,  and is a one disc ISO image that you can download at  http://www.ubuntu.com/products/GetUbuntu/download?action=show&redirect=download. All the instructions you need to burn and boot the disk  are located either on or are linked from that page and are very easy to read and follow.

If you have  an extra windows or mac license and are inclined to use Windows or mac os some of these applications will run on windows and mac. There are many great applications for  children. weather they are younger  or  older   i will start with a list of applications aimed at the younger kids and i will list the application for older kids in part 2 of this article which will be posted soon

Visit bTonga

2007.05.11 - I've finally compeleted the download of the 4.1 i386 release of OpenBSD and am way excited about trying the system as the desktop client for my sister-in-law as a major requirement for her (as if she knows?) is a functional Office Productivity Suite (aka as MS Office replacement --> OpenOffice.)

The other major requirements work just fine, already work well but have gathered further enhancements. Desktop Publishing/Scribus, Email/Evolution-or-Thunderbird, Internet Browser/Thunderbird.

For the kids, two great additions are childsplay.sf.net and gcompris, two 'educational' programs for OSS operating systems.

Visit bTonga

For those who thought there doesn't exist a code of ethics for System Administrators? I'm always having to explain a number of rudimentary 'ethical standards' for new system administrators so it was time to look up the web to see what the 'professionals' have to say.

Well there are a number of groups, System Administrators' with their published codes of ethic, so I thought it would be good to have one hanging around, care of: http://lopsa.org/CodeOfEthics 

Find more at:

• www.sage.org/ethics/ethics_horiz.pdf
• FEANI's Code of Conduct for Professional Engineers
• BCS Code of Conduct and Code of Good Practice
• SAGE Code of Ethics
• SANS IT Code of Ethics

Visit bTonga

I remember circa 2002/2003 Pulu and I experimented and put together a system whereby you can have all print jobs in a networked environment go to a PDF file (for archival reasons) before going out on the printers.

CUPS-PDF

This software is designed to produce PDF files in a heterogeneous network by providing a PDF printer on the central fileserver. It is available under the GPL and is packaged for many different distributions or can be built directly out of the source files.

Apparently someone else thought of a better automated solution and created a program for it @ CUPS-PDF.

At the time, we thought that it was cool and infinitely most practical for archives that all networked print jobs should be archived as proper/certifiable copies of print documents sent out from an organisation. Now, with larger / cheaper disk space it should be seriously considered?

Visit bTonga

One of the cousins came through with his humoungous laptop and it seems a stream of consciousness is going through his University since they've got a group of students installing Ubuntu on their laptops.

So, I've downloaded the CDR and installed in under Virtual PC 2007 to see how it goes (not) which is an adventure in its own right.

Nonetheless, the post is really to point out Ubuntu's torrent site where you can get your torrents of the release builds.

I'm using torrents at the moment because the connection I'm using is quite flakey and I'm finding torrents more resistant than FTP fro large file downloads (apart from the fact that I'm downing the ISO's instead of separate files.)