The Challenge – Step 2: Deploy a Spacewalk server on CentOS 6

Spacewalk is an open source (GPLv2) Linux systems management solution. It is the upstream community project for Red Hat Satellite. Its capabilities include:

  • Inventory your systems (hardware and software information)
  • Install and update software on your systems
  • Collect and distribute your custom software packages into manageable groups
  • Provision (kickstart) your systems
  • Manage and deploy configuration files to your systems
  • Provision virtual guests
  • Start/stop/configure virtual guests
  • Distribute content across multiple geographical sites in an efficient manner

Sounds pretty sweet to me, so let’s build out own Spacewalk server. In this guide, I used the following variables that you’ll want to modify to reflect your own environment:

  • hostname: spacewalk01.homelab.virtnet
  • IP address (via DHCP): 10.0.0.174
  • I use PuTTY for SSH access however you can use whichever SSH client you prefer.
  • I also use vi, which is notorious for not being welcoming to beginners. Feel free to install and use whichever text editor you prefer.

Here we go…

Installing CentOS 6

New VM in Hyper-V: connect to an external virtual switch, set type to Generation 1, and configure disk space per this article (I aimed high with 750 GB):

Partition your disk to suit your needs. Important note regarding partitioning: This blog article assumes some defaults, based on those defaults you should be aware that there are two locations which need sufficient disk space, you may want to keep this in mind while partitioning:

/var/satellite (5GB per distro)
/u01/app/oracle/oradata/XE (1GB per distro)

Installing CentOS is outside the scope of this guide. Here the tl;dr on the CentOS 6.8 install that I used: boot from DVD1 ISO, totally vanilla install, use whole disk when prompted, enable network during install, set hostname with FQDN (this article uses spacewalk01.homelab.virtnet), opt for ‘Minimal’ role as opposed to Basic Server, Virtual Host, etc. to save disk space.

When CentOS boots, get IP address:

[root@spacewalk01 ~]# ifconfig eth0 | grep "inet addr:"

Thanks to Hyper-V, you have two options here: you can continue to connect to your server from within Hyper-V, or you can SSH into your server. SSH allows for much easier copy/paste, scrolling, etc. If you need a refresher on how to SSH to your CentOS server, here’s a quick guide. Using whatever method you wish, login to your CentOS Spacewalk server and get to the command prompt:

[root@spacewalk01 ~]#

For the rest of this article, we’ll forgo the [username@host path]# and instead just display the # prompt to save space.

First, update everything:

# yum update -y

Note your hostname:

# hostname

Note your IP address:

# ifconfig eth0 | grep "inet addr:"

Add your IP address and hostname (FQDN and hostname, as shown below) to /etc/hosts (remember, yours will likely be different than the ones shown below):

# echo "10.0.0.174 spacewalk01.homelab.virtnet spacewalk01" >> /etc/hosts

Confirm that your server’s hostname can be resolved:

# ping spacewalk01 -c1
# ping spacewalk01.homelab.virtnet -c1

Installing Spacewalk

Add the Spacewalk 2.6 repo:

# rpm -Uvh http://yum.spacewalkproject.org/2.6/RHEL/6/x86_64/spacewalk-repo-2.6-0.el6.noarch.rpm

Add the JPackage 5.0 repo manually because the official repo is a bit messy from what I’ve read:

# cat > /etc/yum.repos.d/jpackage-generic.repo << EOF
[jpackage-generic]
name=JPackage generic
mirrorlist=http://www.jpackage.org/mirrorlist.php?dist=generic&type=free&release=5.0
enabled=1
gpgcheck=1
gpgkey=http://www.jpackage.org/jpackage.asc
EOF

Add the Extra Packages for Enterprise Linux (EPEL) for CentOS 6 repo for things like OpenJava:

# rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm

Backup current iptables settings:

# iptables-save > /etc/sysconfig/iptables.pre_spacewalk

Verify backup:

# cat /etc/sysconfig/iptables.pre_spacewalk

Open firewall ports:

# vi /etc/sysconfig/iptables

Locate this line in the config:

-A INPUT -j REJECT --reject-with icmp-host-prohibited

Immediately before that line, add the following:

## Spacewalk firewall rule additions
# PXE for network boot
-A INPUT -m state --state NEW -m tcp -p tcp --dport 69 -j ACCEPT
# HTTP server interface
-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
# HTTPS secure SSL server interface
-A INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
# Spacewalk monitoring service
-A INPUT -m state --state NEW -m tcp -p tcp --dport 4545 -j ACCEPT
# Push connectivity (for pushing actions to clients + responses)
-A INPUT -m state --state NEW -m tcp -p tcp --dport 5222 -j ACCEPT
# Push connectivity for Spacewalk Proxies
-A INPUT -m state --state NEW -m tcp -p tcp --dport 5269 -j ACCEPT
# Next rule drops all incoming connections

Restart firewall:

# /etc/init.d/iptables restart

Install Spacewalk & PostgreSQL:

# yum -y install spacewalk-setup-postgresql spacewalk-postgresql

Configure Spacewalk:

# spacewalk-setup

Connect to spacewalk via the server’s IP address from another computer, accepting the warning that the SSL cert is not trusted:

https://10.0.0.174

Congrats, you’ve got Spacewalk up and running. Next up: let’s make it do things.

Setting up Spacewalk: Channels

If you are already familiar with the concept of channels, feel free to skip ahead.

Earlier in this guide, you ran the yum command which connected to the system’s configured repo’s (aka repositories) and downloaded the latest version of the system’s installed software. New code was put into place, security patches were installed, and overall, the system health increased. For home or lab use, this is totally normal and should cause zero problems.

However, in the corporate world, it’s simply not so simple. Commit this to memory: stability. This is your #1, end all, top most priority in corporate IT. Fast? Backed up? Scaled out? Sure, but none of that matters if the whole thing crashes constantly. In the enterprise world, it has to stay up and be available. Bringing down yourcompany.com because you changed something in production without knowing the consequence? We call those. career alerting decisions, or resume generators. Avoid them if at all possible.

Enter Spacewalk. One of the many things that Spacewalk does is add a layer of management between your environment and the world’s software repos known as a channel. Spacewalk can connect to a given repo (or multiple repos) and download a complete set of software that is appropriate for your environment, storing it in a channel. Even software that you have yet to install (examples include vi, nano, bind, dhcp, and literally anything else you can install on CentOS) can be downloaded by Spacewalk and in effect ‘frozen’ in a channel so that your environment is guaranteed to only have access to a known, tested, stable, deployed version of a given piece of software.

This allows you, the system admin, to choose both when and which software packages are made available to install. Perhaps PHP was just upgraded and is being pushed into the public repos… but you know that your company’s Big Application requires a legacy version of PHP. By the way, get used to legacy software because it is everywhere in corporate IT. With Spacewalk, you can ensure that the systems that need that legacy version are not allowed to upgrade to the newer release by not allowing that new release into a given channel.

The opposite is also true: you can choose when all of your systems must update to a given release of software or install a certain program. Wouldn’t it be nice to click a few buttons in one place and have Really Cool Software v1.28.006 installed across all of your environment? Spacewalk does exactly that – all members of a given channel can be told to install or update a certain package from that channel.

Finally, Spacewalk allows you to keep multiple channels in different states. One way to make use of this is to create DEV, TEST, and PROD environments, each with their own channels. At first, all the channels would be in sync because all environments would start from the same known state. Then, as new software is released or requested, it could be added to the DEV channel, pushed to those servers, and poked and prodded to see what breaks. Once the bugs are squished, the approved package could be added to the TEST channel, and run against production code, but in a non-production environment. After a burn in period passes and nothing breaks, the PROD channel could be updated and the software pushed out into the live production environment without fear of failure. Sure, things go wrong from time to time, but with this sort of tiered deployment strategy, your far more likely to have no issues in production because you caught all the bugs in DEV and worked out the kinks in TEST.

So, let’s setup our CentOS 6 default channels.

I like to setup one parent channel for each OS version that I’ll be working with. This parent channel won’t have any repos associated with it (more on that in a bit). Instead, it’s sole purpose is to group together all the other channels, known as child channels, that relate to CentOS 6. A client can use all child channels associated with the parent channel it’s registered to (more on that later too).

Spacewalk > Channels > Manage Software Channels > Create Channel
Spacewalk > Channels > Manage Software Channels > Create Channel

 Keeping it simple, let’s name the parent channel centos6:

creating parent channel
creating parent channel

Scroll down and click Create Channel. Once that is done, we’ll need to create an activation key. This is used later when clients subscribe to the parent channel. The process to make the key is simple:

Systems > Activation Keys > Create Key
Systems > Activation Keys > Create Key

You only need to make a single change on the Activation Key Details form:

select centos6
select centos6

Scroll down and click Create Activation Key to create the key and associate it with your parent channel.

Now we’ll make the child channels that go under the parent channel. For this example, we’ll be making 6 child channels:

  • CentOS 6 OS – child channel name will be centos6_os
  • CentOS 6 update – child channel name will be centos6_update
  • CentOS 6 extras – child channel name will be centos6_extras
  • CentOS 6 EPEL – child channel name will be centos6_epel
  • PostgreSQL 8.4 – child channel name will be postgresql
  • Spacewalk 2.6 – child channel name will be spacewalk

The first 3 cover CentOS 6 itself. PostgreSQL requires some software packages that are contained in EPEL, which is described as:

Extra Packages for Enterprise Linux (or EPEL) is a Fedora Special Interest Group that creates, maintains, and manages a high quality set of additional packages for Enterprise Linux, including, but not limited to, Red Hat Enterprise Linux (RHEL), CentOS and Scientific Linux (SL), Oracle Linux (OL).

EPEL packages are usually based on their Fedora counterparts and will never conflict with or replace packages in the base Enterprise Linux distributions. EPEL uses much of the same infrastructure as Fedora, including buildsystem, bugzilla instance, updates manager, mirror manager and more.

Finally, the last two child channels are included because we used them earlier when installing Spacewalk. Using the same steps we followed when making the parent channel, we’ll create the child channels – only this time we’ll make sure to select centos6 under Parent Channel. Here is what the first channel, centos6_os looks like when it’s being setup:

creating child channels
creating child channels

Scroll down and click Create Channel. Follow this procedure to make these 5 remaining child channels, all of which use centos6 as the Parent Channel:

  • centos6_update
  • centos6_extras
  • centos6_epel
  • postgresql
  • spacewalk

Once done, you should see all 6 child channels nested under the centos6 parent channel:

Channels > Manage Software Channels
Channels > Manage Software Channels

Now that we have the child channels setup, we need to setup repos for each one and assign them accordingly.

SETTING UP SPACEWALK: Repositories

Simply stated, a software repository is a storage location from which software packages may be retrieved and installed on a computer. There is a little more to it than that – repos can be used intelligently to figure out which software is newer than what you have installed, which software isn’t installed, which software requires additional software to operate, etc. You can tie repos and channels together in different ways, either many repos to a single channel, or many channels to a single repo, or anything in between. For this article, we’ll keep it simple with a 1:1 pairing of repo to channel.

Let’s setup the 3 CentOS repos first. To make sure that your system is able to update quickly and easily, you’ll want to use a mirror that is located close to you geographically. Use this list of mirrors for CentOS to find a mirror closer to you. Once you find a local mirror, simply modify the URLs below from:

http://centos.s.uw.edu/centos/6/os/x86_64/

To this:

http://<your_mirror_url>/centos/6/os/x86_64/

Note that the latter part of the URL stays the same – that is the path to the specific repo that we want to use, only the first part of the URL needs to be altered.

First, let’s create a repo in Spacewalk:

Channels > Manage Repositories > Create Repository
Channels > Manage Repositories > Create Repository

From there, the data entry is pretty easy:

create repository
create repository

Scroll down and click Create Repository. Follow this procedure to make the rest of the CentOS 6-specific repos:

  • centos6_update – http://centos.s.uw.edu/centos/6/updates/x86_64/
  • centos6_extras – http://centos.s.uw.edu/centos/6/extras/x86_64/

For EPEL, use their mirror list to find a location near you:

  • centos6_epel – http://<your_mirror_url>/epel/6/x86_64/

I was unable to find a mirror for PostgreSQL, but that’s not an issue because this is a single software package so we won’t be moving a lot of data. PostgreSQL’s repo is located here:

  • postgresql – https://yum.postgresql.org/8.4/redhat/rhel-6-x86_64/

The same is true for Spacewalk:

  • spacewalk – http://yum.spacewalkproject.org/2.6/RHEL/6/x86_64/

Once all of the repos are setup, your Spacewalk instance should show this:

Channels > Manage Repositories
Channels > Manage Repositories

Next, we need to tie each repo to its respective child channel. First, select your first child channel. Here I am starting with centos6_os:

Channels > Manage Software Channels
Channels > Manage Software Channels

From there, you assign the repo that will be tied to that channel:

select the repo that matches your child channel
select the repo that matches your child channel

Scroll down and click Update Repositories. Follow this process for all of the child channels, making sure to match them 1:1 with their repo. When everything is matched up, your Spacewalk instance should look like this:

Channels > Manage Repositories
Channels > Manage Repositories

Note that every repo is matched to exactly 1 channel in our example. As mentioned before, you don’t have to do it this way, but for our example, we are doing it to keep it easier to visualize what is happening in the background.

Speaking of the background, we’re about to get down and dirty with Spacewalk and the command line. This is not a required step, but I find that it helps illustrate what exactly Spacewalk is doing with everything we just setup. If you wish to skip the command line portion and work with the GUI, feel free to skip ahead.

Have Spacewalk show us which channels are configured:

# spacewalk-repo-sync --list

Which produces output that should look like this:

14:01:44 ======================================
14:01:44 | Channel Label | Repository |
14:01:44 ======================================
14:01:44 postgresql | https://yum.postgresql.org/8.4/redhat/rhel-6-x86_64/
14:01:44 centos6_epel | http://mirror.sfo12.us.leaseweb.net/epel/6/x86_64/
14:01:44 centos6_update | http://centos.s.uw.edu/centos/6/updates/x86_64/
14:01:44 spacewalk | http://yum.spacewalkproject.org/2.6/RHEL/6/x86_64/
14:01:44 centos6_os | http://centos.s.uw.edu/centos/6/os/x86_64/
14:01:44 centos6_extras | http://centos.s.uw.edu/centos/6/extras/x86_64/
14:01:44 centos6 | No repository set

Here you can clearly see all of the channels and the associated repos. Note that the parent channel, centos6, doesn’t have a repo associated with it.

Our next step is to tell Spacewalk to connect to the internet and download a complete copy of each of these repos, storing them locally on the Spacewalk server. This allows Spacewalk to draw upon it’s locally stored repos when managing your infrastructure rather than the public repos that might wreak havoc with your environment (recall part above about DEV/TEST/PROD). Before you move forward, make sure that your CentOS VM has at least 30GB of free space, and that the hard drive hosting the virtual disk file has at least 30GB of free space! If you followed the suggestions at the top of this article, you should have well beyond that in free space, however here is a guide that shows you how to increase the disk size should you need more space.

Starting with one of the smaller repos, postgresql, here is how you download the entirely of the repo into Spacewalk via the command line:

# spacewalk-repo-sync -c postgresql

Running this command will query the public repo, get a count of the software packages hosted there, and start downloading them. At the time this article was written, there were 155 packages in the PostgreSQL x86_64 repo. Thanks to the low number of packages – and trust me, 155 is a low number! – the sync should happen pretty quickly depending on your connection speed.
Once the sync is complete, go back to the Spacewalk GUI and you’ll see the 155 packages that have been synced under the postgresql repo. Start by selecting the postgresql channel:

Channels > Manage Software Channels
Channels > Manage Software Channels

Then switch to the Packages tab and click List / Remove Packages:

list packages
list packages

Here you’ll find a list of every package that Spacewalk downloaded:

lots of info in available about each package
lots of info in available about each package

As indicated by the arrow, each package is a link that provides lots of info about that package: what it does, when it was made, what software it is dependent on, and more. Dig in at your leisure, but that info is beyond the scope of this article, so we’ll press on.

All of the packages shown are now contained in your local postgresql repo, which is attached to the child channel postgresql. Later, when we connect servers to Spacewalk, they can subscribe to the parent channel centos6 and by association, they will gain access to the child channel postgresql, which is what you are viewing in Spacewalk.

You’ll want to go ahead and sync all of the local repos with their public counterparts. You can continue doing this by the command line (my preferred method) or you can use the Spacewalk GUI. For the GUI, first select the appropriate channel:

Channels > Manage Software Channels
Channels > Manage Software Channels

Then switch to the Repositories tab and click Sync:

Repositories > Sync
Repositories > Sync

The button in the bottom right triggers the same process that we ran previously via the command line, only you get less feedback on progress.

Using whatever method you wish, sync all of the repos that you’ve configured in Spacewalk. This will take about 45 mins even with a fast internet connection, longer if your connection is slow. As of the date this article was written, here are the package counts you’ll see when you are done:

that is a lot of packages
that is a lot of packages

Congrats on getting this far – we’re almost done! Next up, eratta.

SETTING UP SPACEWALK: ERATTA

When one of the software packages that make up CentOS is updated, either for security, bug fix, or enhancement purposes, an eratta is published that explains the type of update (security, bug fix, or enhancement), what is being updated, and why that matters. To be a good system admin, you need access to the errata for your repos. Keeping up to date on what security problems are know to exists with packages with your repos is step one to keep your environment safe and secure.

Up to this point, everything we’ve done has been very, ‘out of the box’ as far as Spacewalk is concerned. Unfortunately, the CentOS community and/or the errata community just don’t see eye to eye so the errata for CentOS exists… but not in a very easy to read format. Here, see for yourself. There is another source of similar data from the OVAL project. Here, I’ll let them explain:

The Open Vulnerability and Assessment Language (OVAL) project, maintained by The MITRE Corporation, is an international, information security effort that promotes open and publicly available security content, and seeks to standardize the transfer of this information across the entire spectrum of security tools and services. Refer to http://oval.mitre.org/ for further information.

All you need to do is parse all of the data from the CentOS errata and OVAL… sort it by package… assign it to each package individually… and enter it all into Spacewalk. Yeah… I’m not going to do it that way.

Geeks to the rescue! Steve Meier has written a script which automates all of this (and then some). I used two sites, Steve’s and another, to get the script setup and working with Spacewalk in my environment. 100% of the credit for the rest of this article goes to them. Here’s what you’ll need to do to get it working in your environment.

First, we’ll need wget, which we can install via yum:

# yum -y install wget

Second, let’s get the script itself onto your Spacewalk server, save it in a convenient directory, and make it executable:

# mkdir /opt/tools/
# cd /opt/tools/
# wget https://github.com/stevemeier/cefs/raw/master/errata-import.pl
# chmod +x errata-import.pl

The script needs a few Perl packages installed:

# yum -y install perl-Frontier-RPC perl-Text-Unidecode perl-XML-Simple

Next, we need to create a user within Spacewalk that has the right to create errata for your various channels. This is easily accomplished in the GUI. First, let’s make a new user:

Users > Create User
Users > Create User

The Create User form needs some basic data. For this article, we’ll use the username su-errata. The rest of the data is pretty much up to you.

creating a new user
creating a new user

Once the user has been created, we need to assign a role that will allow the user to create and modify errata:

select your new user to modify it's roles
select your new user to modify it’s roles

Add the role Channel Administrator to your new user:

adding Channel Administrator role
adding Channel Administrator role

Scroll down and click Update to save the changes. Once that’s done, we’ll need to run a cron job on the Spacewalk server that will do a few things for us on a daily basis:

  • update all of our child channels*
  • download the latest errata XML file
  • download the latest OVAL XML file
  • run some quick checks on those files to make sure they downloaded correctly (fancy!)
  • parse both of these files, assigning errata as needed to our repos

* – Note: If you do not want your channels kept in sync with the public channels (remember DEV/TEST/PROD from earlier) the script will need to be modified to not run the spacewalk-repo-sync commands. Just comment those lines out with hashes (#).

Here is how to set that up on your server. First, open an empty file in vi:

# vi /etc/cron.daily/spacewalk_sync.cron

Paste all of this into the blank file, making sure to enter the password you specified for the su-errata user earlier:

#!/bin/sh
MAILTO=root

# try to create the lock and check the outcome
LOCKFILE=/var/run/spacewalk_sync.lock
#lockfile -r 0 ${LOCKFILE} 1>/dev/null 2>&1
#status=$?
#if [ ${status} -ne 0 ] ;then
if [ -e "$LOCKFILE" ]; then
 echo "Another instance already running. Aborting."
 exit 1
else
 touch "$LOCKFILE"
fi
trap "rm ${LOCKFILE}" EXIT

#sync channels and publish updates
/usr/bin/spacewalk-repo-sync -c centos6_os >/dev/null
/usr/bin/spacewalk-repo-sync -c centos6_extras >/dev/null
/usr/bin/spacewalk-repo-sync -c centos6_update >/dev/null
/usr/bin/spacewalk-repo-sync -c centos6_epel >/dev/null
/usr/bin/spacewalk-repo-sync -c postgresql >/dev/null
/usr/bin/spacewalk-repo-sync -c spacewalk >/dev/null

#get errata file and checksums
cd /tmp
wget -N http://cefs.steve-meier.de/errata.latest.xml 1>/dev/null 2>&1
wget -N http://cefs.steve-meier.de/errata.latest.md5 1>/dev/null 2>&1
wget -N http://www.redhat.com/security/data/oval/com.redhat.rhsa-all.xml.bz2 1>/dev/null 2>&1
bunzip2 -f /tmp/com.redhat.rhsa-all.xml.bz2

#verify integrity
grep "errata.latest.xml$" errata.latest.md5 > myerrata.md5
md5sum -c myerrata.md5 1>/dev/null 2>&1
if [ "$?" == 0 ]; then
 #ok - import errata
 SPACEWALK_PASS=CHANGE_ME SPACEWALK_USER=su-errata /opt/tools/errata-import.pl --server localhost --errata errata.latest.xml --include-channels=centos6_os,centos6_extras,centos6_update,centos6_epel,postgresql,spacewalk --rhsa-oval=/tmp/com.redhat.rhsa-all.xml --publish 1>/dev/null
 if [ "$?" != 0 ]; then
 echo "It seems like there was a problem while publishing the most recent errata..."
 exit 1
 fi
 rm /tmp/myerrata.md5
else
 #errata information possibly invalid
 echo "ERROR: md5 checksum mismatch, check download!"
 exit 1
fi

Save the file, and make it executable:

# chmod +x /etc/cron.daily/spacewalk_sync.cron

You can run the cron job yourself manually, but be warned it can take 10 mins or more to complete, even when there aren’t any updates to process. There is also no visual indication that anything is happening (note all the > /dev/null statements in the script) so be patient. If you wish to run the cron job manually, all you need to do is call it:

# /etc/cron.daily/spacewalk_sync.cron

If there are no issues it will return to the hash prompt after several minutes.

To see the fruits of your labor, check out the Spacewalk GUI:

Channels > All Channels
Channels > All Channels

Congrats, your Spacewalk instance is ready to manage some servers!

 

Leave a Reply

Your email address will not be published. Required fields are marked *