The purpose of this guide is to help an IT administrator install and configure the STARRS network asset tracking application for a basic site.
Audience
The intended audience of this guide is IT administrators with Linux experience (specifically Red Hat) and a willingness to use open-source software.
Commitment
The install process should take around an hour provided appropriate resources.
Description
Definition
STARRS is a self-service network resource tracking and registration web application used to monitor resource utilization (such as IP addresses, DNS records, computers, etc). For more details, view the project description here.
Physical
STARRS requires a database host and web server to operate. It is recommended to use a single virtual appliance for all STARRS functionality.
Process
The virtual appliance is provisioned (out of scope of this guide)
Core software is installed
Dependent software packages are downloaded and installed
STARRS is acquired, configured, and installed
The web server is configured to allow access
Installation
Virtual Appliance
STARRS was tested only on RHEL-based Linux distributions. Anything RHEL6.0+ is compatible. There are no major requirements for the virtual appliance and a minimal software install will suffice. In this example we will be using a Scientific Linux 6.4 virtual machine.
Connectivity
Ensure that you are able to log into your remote system as any administrative user (in this example, root) and have internet access.
12
[root@starrs-test ~]# cat /etc/redhat-release
Scientific Linux release 6.4 (Carbon)
System Security
Firewall
Firewalls can get in the way of allowing web access to the server. Only perform these steps if you have a system firewall installed and intend on using it. In this example we will use the RHEL default iptables.
Add a rule to the system firewall to allow HTTP and HTTPS traffic to the server.
Save the firewall configuration (no restart required)
1
service iptables save
NOTE: If you have IPv6 enabled on your system, make sure to apply firewall rules to the IPv6 firewall as well.
SELinux
Any RHEL administrator has dealt with SELinux at some point in their career. There are system-wide settings that allow/deny actions by programs running on the server. Disabling SELinux is not a solution.
STARRS heavily depends on the PostgreSQL database engine for operation. PgSQL must be at version 9.0 or higher, which is NOT available from the standard RHELish repositories. PgSQL will also require the PL/Perl and PL/Python support packages (also NOT located in the repos). You will need to add new software repositories to your appliance.
Utilities
These programs will be needed at some point or another in the installation process.
Install the following packages through yum.
1
yum install cpan wget git make perl-YAML -y
We will also need the development tools group of packages installed.
1
yum groupinstall "Development tools" -y
PostgreSQL Database Engine
On your own computer, open up yum.postgresql.org and click on the latest available PostgreSQL Release link. In this case we will be using 9.2.
Locate the approprate repository link for your operating system (Fedora, CentOS, SL, etc) and architecture (i386, x86_64, etc). In this case we will be using Scientific Linux 6 - i386.
Download the package from the link you located onto the virtual appliance into any convenient directory.
Install the PostgreSQL repository package file that was downloaded with yum. Answer yes if asked for verification.
1
yum install pgdg-sl92-9.2-8.noarch.rpm
You need to ensure that the base PostgreSQL packages are hidden from future package searches and updated. Add an exclude line to your base OS repository file located in /etc/yum.repos.d/. This will prevent any PgSQL packages from being used out of the base packages.
1234567
# This example uses the sl.repo file. This will depend on your variant of OS (CentOS-base.repo for CentOS).
[sl]
name=Scientific Linux $releasever - $basearch
...
exclude=postgresql*
[sl-security]
Install the required PgSQL packages using the Yum package manager.
[root@starrs-test ~]# service postgresql-9.2 initdb
Initializing database: [ OK ]
[root@starrs-test ~]#
Make PgSQL start on system boot
1
chkconfig postgresql-9.2 on
Start the PgSQL service
123
[root@starrs-test ~]# service postgresql-9.2 start
Starting postgresql-9.2 service: [ OK ]
[root@starrs-test ~]#
su to the Postgres account and assign a password to the postgres user account using the query below. Exit back to root when done. This password should be kept secure!
123456789101112
[root@starrs-test ~]# su postgres -
bash-4.1$ psql
could not change directory to "/root"
psql (9.2.4)
Type "help" for help.
postgres=# ALTER USER postgres WITH PASSWORD 'supersecurepasswordhere';
ALTER ROLE
postgres=# \q
bash-4.1$ exit
exit
[root@starrs-test ~]#
The STARRS installer requires passwordless access to the postgres account. This is achievable by creating a .pgpass file in the root home directory.
You can replace the passwords with whatever you want. Note the passwords for later.
This file should be readable only by the user that created it. Change permissions accordingly.
1
chmod 600 .pgpass
You now need to allow users to login to the database server from the server itself. Open the /var/lib/pgsql/9.2/data/pg_hba.conf and change all methods to md5 like so:
12345678
# TYPE DATABASE USER ADDRESS METHOD
# "local" is for Unix domain socket connections only
local all all md5
# IPv4 local connections:
host all all 127.0.0.1/32 md5
# IPv6 local connections:
host all all ::1/128 md5
This will enable login from Localhost only. If you need remote access, only allow the specific IP addresses or subnets that you need. Security is good.
Reload the postgres service to bring in the changes
1
service postgresql-9.2 reload
Create admin and client accounts for STARRS.
12345678910
[root@starrs-test ~]# psql -h localhost -U postgres
psql (9.2.4)
Type "help" for help.
postgres=# create user starrs_admin with password 'adminpass';
CREATE ROLE
postgres=# create user starrs_client with password 'clientpass';
CREATE ROLE
postgres=# \q
[root@starrs-test ~]#
Verify that you can log into the database server without being prompted for a password.
1234567
[root@starrs-test ~]# psql -h localhost -U postgres
psql (9.2.4)
Type "help" for help.
postgres=# \q
[root@starrs-test ~]#
If you get prompted for a password, STOP! You need to have this working in order to proceed. Make sure you typed everything in the file correctly and its permissions are set.
Dependencies
STARRS has many other software dependencies in order to function. These are mostly Perl modules that extend the capabilities of the language. These modules are (CPAN and package are provided however not all packages may be available):
Net::IP (perl-Net-IP)
Net::LDAP (perl-LDAP)
Net::DNS (perl-Net-DNS)
Net::SNMP (perl-Net-SNMP)
Net::SMTP (perl-Mail-Sender)
Crypt::DES (perl-Crypt-DES)
VMware::vCloud
Data::Validate::Domain
NOTE: The first time you run CPAN you will be asked some basic setup questions. Answering the defaults are fine for most installations.
Install each of these modules. Some of them are available as packages in yum.
STARRS comes in two parts: The backend (database) and the Web interface. Each one is stored in it’s own repository on Github. You will need to download both in order to use the application. Right now we will focus on the backend.
You will need a directory to store the downloaded repos in. I recommend using /opt. Clone (download) the current versions of the repositories using Git into that directory.
12345
[root@starrs-test ~]# cd /opt/
[root@starrs-test opt]# git clone https://github.com/cohoe/starrs -q
[root@starrs-test opt]# ls
starrs
[root@starrs-test opt]#
Open the installer file at /opt/starrs/Setup/Installer.pl. You will need to edit the values in the Settings section of the file to match your specific installation. Example:
dbsuperuser is the root postgres account (usually just ‘postgres’).
dbadminuser is the STARRS admin user you created above.
dbclientuser is the STARRS client user you created above.
sample will populate sample data into the database. Set to anything other than undef to enable sample content.
Run the install script
1
perl /opt/starrs/Setup/Installer.pl
A lot of text will flash across the screen. This is expected. If you see errors, then something is wrong and you should revisit the setup instructions.
You can verify that STARRS is functioning by running some simple queries.
1234567891011121314151617181920
[root@starrs-test starrs]# psql -h localhost -U postgres starrs
psql (9.2.4)
Type "help" for help.
starrs=# SELECT api.initialize('root');
NOTICE: table "user_privileges" does not exist, skipping
CONTEXT: SQL statement "DROP TABLE IF EXISTS "user_privileges""
PL/pgSQL function api.initialize(text) line 19 at SQL statement
initialize
------------------
Greetings admin!
(1 row)
starrs=# SELECT * FROM api.get_systems(NULL);
system_name | owner | group | comment | date_created | date_modified | type | os_name | last_modifier | platform_name | asset | datacenter | location
-------------+-------+-------+---------+--------------+---------------+------+---------+---------------+---------------+-------+------------+----------
(0 rows)
starrs=# \q
[root@starrs-test starrs]#
If you see “Greetings admin!” and you can perform the queries without error, then your backend is all set up and is ready for the web interface.
Apache2/httpd
The STARRS web interface requires a web server to be installed. Only Apache has been tested. If you have a new system then you might not have Apache (or in RHELish, httpd) installed.
If you do not have Apache/httpd installed, install it.
1
yum install httpd php php-pgsql -y
Navigate to the /etc/httpd/conf.d directory.
Remove the welcome.conf that exists there.
12
cd /etc/httpd/conf.d/
rm -rf welcome.conf
Enable httpd to start on boot
1
chkconfig httpd on
Start the httpd service. (Warning messages are fine, as long as the service starts you should be fine)
12345
[root@starrs-test ~]# service httpd start
Starting httpd: httpd: apr_sockaddr_info_get() failed for starrs-test.grantcohoe.com
httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
[ OK ]
[root@starrs-test ~]#
PHP Configuration
A minor change to the system PHP configuration allows the use of short tags for cleaner code. This feature must be enabled in the /etc/php.ini file.
In the php.ini file, set short_open_tag = on
Web Interface
You will be cloning the starrs-web into the Apache web root directory to simply deployment.
Change directory to /var/www/html/ and clone the repository. Note the . at the end of the clone command.
Copy the application/config/impulse.php.example to application/config/impulse.php
For this web environment, the defaults in this file are fine. In this file you can change which environment variable to get the current user from.
Apache needs to be given some special instructions to serve up the application correctly. Create a file at /etc/httpd/conf.d/starrs.conf with the following contents:
Restart Apache to apply the changes. (A reload is not sufficient enough)
1
service httpd restart
Testing
At this point you should have a fully functioning STARRS installation. Navigate to your server in a web browser and you should be prompted for login credentials. As we established in the authentication database file, the username is root and the password is admin. If you get the STARRS main page, then success! Otherwise start looking through log files to figure out what is wrong.
Detailed troubleshooting is out of the scope of this guide. System Administrator cleverness is a rare skill, but is the most useful thing when trying to figure out what happened. Shoot me an email if you really feel something is wrong.
Data backups are something every single person should do. This presentation is with specific regard to non-technical users who want a little bit of information and some suggestions on what to do.
This presentation was never given explicitly, but it was used for a Public Speaking class.
This presentation discusses the impact of disk alignment with specific regard to virtual environments. Performance can be greatly hurt by misaligned partitions both on the VM and host and all layers in between.
We wanted to deploy our own wireless network across the floor. I decided to make the project happen.
Requirements
Spectrum Use
We can only use three channels in the 5 GHz spectrum. RIT has a massive unified wireless deployment across all of campus. Policy states that third-parties (students, staff, etc) cannot operate wireless access points within range of the RIT network to prevent signal overlap and interference. This is not an issue in the 5 GHz spectrum, where there are many more non-overlaping channels. This limits potential clients to only those with dual-band wireless radios.
Authentication
No client-side software can be required. This is a requirement set by us to allow ease of access by users. Most operating systems support PEAP/MsCHAPv2 (via RADIUS) out of the box and require no extra configuration to make them work. Unfortunately this requires a bit more system infrastructure to make it work. Our authentication backend (MIT Kerberos) does not support this type of user authentication so we need to do some hax to make it.
Speed
We need to be faster than RIT. There is absolutely no reason to use CSH wireless over RITs unless we can be faster. By using Channel Bonding in two key areas, we can achieve this requirement and double the speed of RIT wireless. We also need to be able to do fast-reassociation between access points so that users can walk up and down floor and not lose connectivity.
Setup
Access Points
First we had to figure out where and how to deploy our range of APs. Since we have relatively few resources we used a combination of what we had lying around, purchased, and donated harware.
3x Cisco 1230
1x Cisco 1142
1x Cisco 1131
1x Cisco 1252
Looking at the map of the floor, I wanted to place the channel-bonded APs in the areas with the largest concentration of users (the Lounge and the User Center). The others would be dispersed across the other public rooms on floor.
[[ IMAGE HERE ]]
Wireless Domain Services
Cisco WDS is essentially a poor-mans controller. WDS allows you to do authentication once across your wireless domain and do relatively seamless handoff between access points. I had an extra 1230 laying around without antennas so I parked it in my server room and configured it to act as the WDS master. When a client attempts to authenticate to an AP, the auth data is sent to the WDS server where it is then processed and a response sent to the AP to let them in or not. If the client roams to another AP then the WDS server promises the new AP that the client is OK and skips the authentication phase.
This is the only device that will ever talk to the RADIUS server, so all of the configuration for that is only needed once.
NOTE: In this example the WDS server is at IP address 192.168.0.250 and the RADIUS server is at 192.168.0.100.
1234567891011121314151617181920212223242526272829
aaa new-model
!
!
aaa group server radius rad_local
server 192.168.0.250 auth-port 1812 acct-port 1813
!
aaa group server radius rad_eap
server 192.168.0.100 auth-port 1812 acct-port 1813
!
aaa authentication login eap_local group rad_local
aaa authentication login eap_methods group rad_eap
!
radius-server local
no authentication mac
nas 192.168.0.250 key 7 SUPERSECRETKEYHERE
user wds-authman nthash 7 AUTHMANPASS
!
radius-server host 192.168.0.250 auth-port 1812 acct-port 1813 key 7 SUPERSECRETRADIUSKEYHERE
radius-server host 192.168.0.100 auth-port 1812 acct-port 1813 key 7 SUPERSECRETRADIUSOTHERKEYHERE
!
!
wlccp ap username wds-authman password 7 AUTHMANPASS
wlccp authentication-server infrastructure eap_local
wlccp authentication-server client eap eap_methods
ssid prettyflyforawifi
wlccp authentication-server client leap eap_local
ssid prettyflyforawifi
wlccp wds mode wds-only
wlccp wds priority 255 interface BVI1
The access points need to use the AP username defined above to talk to the WDS server. RADIUS configuration here should be optional.
1234567891011121314151617
aaa new-model
!
aaa group server radius rad_eap
server 192.168.0.100 auth-port 1812 acct-port 1813
!
aaa authentication login default local
aaa authentication login eap_methods group rad_eap
dot11 ssid prettyflyforawifi
authentication open eap eap_methods
authentication network-eap eap_methods
authentication key-management wpa
guest-mode
!
radius-server host 192.168.0.100 auth-port 1812 acct-port 1813 key 7 SUPERSECRETRADIUSKEY
!
wlccp ap username wds-authman password 7 AUTHMANPASS
wlccp ap wds ip address 192.168.0.250
You are a nerd. You like doing nerd-y things. You run an entirely *nix-based network. But like all nerds, you have that one pesky Windows machine that you want to have available for your users. Or you just want the same SSO password from your Kerberos KDC. Regardless, you want Windows to talk to MIT Kerberos. Believe it or not, this is SUPPORTED!
Process Overview
We start with the user entering their credentials. These are entered in the standard Windows logon screen (which I dearly miss from Windows 2003). From there, the client is configured to have it’s default domain (realm in this case) to be the MIT Kerberos realm that your machine is a member of. This would be the equivalent of an Active Directory domain.
From here, your workstation acts just like any other Kerberized host. It uses it’s host principal (derived from what it thinks its hostname is) and configured password to authenticate to the KDC. Once it has verified it’s identity, it then goes to authenticate you. And if your password is correct, Windows lets you in!
The question is: who does it let you in as? A Kerberos principal is nothing more than a name. It is not an account, or any object that actually contains information for the system. You must map Kerberos principals to accounts. These accounts are created in Windows as either local users or via Active Directory (a whole other can of worms). Usually, you will want to create a one-to-one mapping between principal and account (grant@GRANTCOHOE.COM principal = Windows account “grant”).
KDC Setup
On your KDC, open up kadmin with your administrative principal and create a host principal for the new machine. It needs to have a password that you know, so use your favorite password generating source.
12345678910
miranda ~ # kadmin -p cohoe/admin
Authenticating as principal cohoe/admin with password.
Password for cohoe/admin@GRANTCOHOE.COM:
kadmin: ank host/caprica.grantcohoe.com@GRANTCOHOE.COM
WARNING: no policy specified for host/caprica.grantcohoe.com@GRANTCOHOE.COM; defaulting to no policy
Enter password for principal "host/caprica.grantcohoe.com@GRANTCOHOE.COM":
Re-enter password for principal "host/caprica.grantcohoe.com@GRANTCOHOE.COM":
Principal "host/caprica.grantcohoe.com@GRANTCOHOE.COM" created.
kadmin: quit
miranda ~ #
That’s it for the KDC. On to the Windows machine!
Windows 7 Client
Windows 7 (as well as Server 2008 I believe) include the basic utilities required to support MIT Kerberos. This was available for XP and Server 2003 as “Microsoft Support Tools”. The utility we will be using is called ksetup. It allows for configuration of foreign-realm authentication systems.
So to set your machine to authenticate to your already existing MIT Kerberos KDC, open up a command prompt and do the following:
After that, you need to set a Group Policy setting to automatically pick the right domain for you to login to. Open up the Group Policy Editor (gpedit.msc) and navigate to Local Computer Policy->Computer Configuration->Administrative Templates->System->Login->Assign a default domain for logon. Set this to your realm (GRANTCOHOE.COM in my case).
Do a nice reboot of your system, and you should be ready to go!
Active Directory
Active Directory can be configured to trust a foreign Kerberos realm. It will NOT synchronize information with anything, but if you just need a bunch of users to log into something and no data with it, the process is not too terrible. Rather than duplicate the information, you can find the guide that I used here: Microsoft trusting MIT Kerberos
Lets say you are in a domain, and you wish to access several different web services. How often do you find you have to enter the same username and password over and over again to get to each website? Wouldn’t it be nice if you could just enter it once and automagically have access to all the web services you need? Well guess what? You can!
It all works off of MIT Kerberos. Kerberos is a mechanism that centralizes your authentication to one service and allows for Single Sign-On. The concept of SSO is fairly simple: Enter your credentials once and that’s it. It save you from typing them over and over again and protects against certain attacks.
WebAuth was developed at Stanford University and brings “kerberization” to web services. All you need is some modules, a KDC, and a webserver.
In this guide, we will be utilizing the Kerberos realm/domain name of EXAMPLE.COM. We will also be using two servers:
gmcsrvx2.example.com: The Webauth WebKDC
gmcsrvx3.example.com: A secondary web server
webauth.example.com: A DNS CNAME to gmcsrvx2.example.com
Note that you will need a pre-existing Kerberos KDC in your network.
There are three components of the WebAuth system. WebAuth typically refers to the component that provides authorized access to certain content. The WebKDC is the process that handles communication with the Kerberos KDC and distributes tickets out to the client machines. The WebLogin pages are the login/logoff/password change pages that are presented to the user to enter their credentials. Certain binary packages provided by Stanford do not contain components of the WebAuth system. This is why we are going to build it from source.
Server Setup
Packages
Each machine you plan to install WebAuth on needs the following packages:
httpd-devel
krb5-devel
curl-devel
mod_ssl
mod_fcgid
perl-FCGI
perl-Template-Toolkit
perl-Crypt-SSLeay
cpan
Enterprise Linux does not include several required Perl modules for this to work. You are going to need to install them via CPAN. If you have never run CPAN before, do it once just to get the preliminary setup working.
CGI::Application::Plugin::TT
CGI::Application::Plugin::AutoRunmode
CGI::Application::Plugin::Forward
CGI::Application::Plugin::Redirect
You also need Remctl, an interface to Kerberos. For EL6 we are using remctl 2.11 available from the Fedora 15 repositories.
Open up ports 80 (HTTP) and 443 (HTTPS) since we will be doing web stuff.
NTP
Since you will be doing Kerberos stuff, you need a synchronized clock. If you do not know how to do this, see my guide on Setting up NTP.
Kerberos
Your machine needs to be a Kerberos client (meaning valid DNS, krb5.conf, and host prinicpal in its /etc/krb5.keytab). We will be using three keytabs for this application:
/etc/krb5.keytab
host/gmcsrvx2.example.com@EXAMPLE.COM
/etc/webauth/webauth.keytab
webauth/gmcsrvx2.example.com@EXAMPLE.COM
/etc/webkdc/webkdc.keytab
service/webkdc@EXAMPLE.COM
NOTE: If you wrote these in your home directory and cp’d them into their respective paths, check your SELinux contexts.
SSL Certificates
You need an SSL certificate matching the hostname of each server (gmcsrvx2.example.com, gmcsrvx3.example.com, webauth.example.com) avaiable for your Apache configuration.
IMPORTANT:You also need to ensure that your CA is trusted by Curl (the system). If in doubt, cat your CA cert into /etc/pki/tls/certs/ca-bundle.crt. You will get really odd errors if you do not do this.
Shared Libraries
We will be installing WebAuth into /usr/local and this need its libraries to be linked in the system.
We will need to run ldconfig to load in the WebAuth libraries later on.
Compile and Install WebAuth
The latest version at the time of this writing is webauth-4.1.1 and is avaiable from Stanford University. Grab the source tarball since the packages do not include features that you will need in a brand new installation.
123456789101112
[root@gmcsrvx2 ~]# tar xfz webauth-4.1.1.tar.gz
[root@gmcsrvx2 ~]# cd webauth-4.1.1
[root@gmcsrvx2 webauth-4.1.1]# ./configure --enable-webkdc
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
....
....
config.status: executing depfiles commands
config.status: executing libtool commands
config.status: executing include/webauth/defines.h commands
[root@gmcsrvx2 webauth-4.1.1]#
Assuming everything is all good, go ahead and build it.
Now ensure that the new libraries get loaded by running ldconfig. This looks at the files in that ld.so.conf.d directory and adds them to the library path. MAKE SURE YOU DO THIS!!!
The Apache modules for WebAuth have been compiled into /usr/local/libexec/apache2/modules:
12
[root@gmcsrvx2 ~]# ls /usr/local/libexec/apache2/modules/
mod_webauth.la mod_webauthldap.la mod_webauthldap.so mod_webauth.so mod_webkdc.la mod_webkdc.so
mod_webauth Configuration
Make sure you have the proper keytab set up at /etc/webauth/webauth.keytab. This file can technically be located anywhere as long as you configure the module accordingly.
Next you need to create a local state directory for WebAuth. Usually this is /var/lib/webauth. The path can be changed as long as you set it in the following configuration file.
The Apache module needs a configuration file to be included in the Apache server configuration. On Enterprise Linux, this can be located in /etc/httpd/conf.d/mod_webauth.conf:
12345678910111213141516171819202122232425
# Load the module that you compiled.
LoadModule webauth_module /usr/local/libexec/apache2/modules/mod_webauth.so
# Some fancy WebAuth stuff
WebAuthKeyRingAutoUpdate on
WebAuthKeyringKeyLifetime 30d
# The path to some critical files. These should be secured.
WebAuthKeyring /var/lib/webauth/keyring
WebAuthServiceTokenCache /var/lib/webauth/service_token_cache
WebAuthCredCacheDir /var/lib/webauth/cred_cache
# The path to the keytab that webauth will use to authenticate with your KDC
WebAuthKeytab /etc/webauth/webauth.keytab
# The URL to point to if you need to login. MAKE SURE THIS IS CORRECT!
WebAuthLoginURL "https://webauth.example.com/login"
WebAuthWebKdcURL "https://webauth.example.com/webkdc-service/"
# The Kerberos principal to use when authenticating with the keytab above
WebAuthWebKdcPrincipal service/webkdc
# SSL is a good thing. Plaintext passwords are bad. Secure this server.
WebAuthSSLRedirect On
WebAuthWebKdcSSLCertFile /etc/pki/example.com/example-ca.crt
Again, adjust paths to suit your tastes. The WebAuthWebKdcSSLCertFile should be your public CA certificate that you probably cat’d earlier.
mod_webkdc Configuration
Make sure you have the proper keytab set up at /etc/webkdc/webkdc.keytab. This file can technically be located anywhere as long as you configure the module accordingly.
Next you need to create a local state directory for the WebKDC. Usually this is /var/lib/webkdc. The path can be changed as long as you set it in the configuration files.
The WebKDC utilizes an ACL to allow only certain hosts to get tickets. This goes in /etc/webkdc/token.acl:
123
# These lines allow all principals that under the webauth service to generate a token
krb5:webauth/*@EXAMPLE.COM id
krb5:webauth/gmcsrvx2.example.com@EXAMPLE.COM cred krb5 krbtgt/EXAMPLE.COM@EXAMPLE.COM
For the WebLogin pages, another configuration file is used to specify the information for it (in /etc/webkdc/webkdc.conf). This is seperate from the Apache module configuration loaded by the webserver.
12345
# The KEYRING_PATH should match what you put in your httpd config$KEYRING_PATH="/var/lib/webkdc/keyring";$URL="https://webauth.example.com/webkdc-service/";# You can make custom skins for the weblogin page. Change the path here$TEMPLATE_PATH="./generic/templates";
NOTE: You CANNOT change the location of this file or the WebKDC module will freak out.
Note that certain directives must match the Apache configuration. We will make that now in /etc/httpd/conf.d/mod_webkdc.conf:
123456789101112131415
# Load the module that you compiled.
LoadModule webkdc_module /usr/local/libexec/apache2/modules/mod_webkdc.so
# Some fancy WebKdc stuff
WebKdcServiceTokenLifetime 30d
# The path to some critical files. These should be secured.
WebKdcKeyring /var/lib/webkdc/keyring
# The path to the keytab and access control list that the webkdc will use to authenticate with your KDC
WebKdcKeytab /etc/webkdc/webkdc.keytab
WebKdcTokenAcl /etc/webkdc/token.acl
# Debugging information is wonderful. Turn this off when you get everything working.
WebKdcDebug On
Ensure that your paths match what you set up earlier.
mod_webauthldap Configuration
This step should only be done if you have an LDAP server operating in your network and can accept SASL binds. You can make certain LDAP attributes available in the web environment as server variables for your web applications. This is configured in /etc/httpd/conf.d/mod_webauthldap.conf:
12345678910111213141516171819
# Load the module that you compiled
LoadModule webauthldap_module /usr/local/libexec/apache2/modules/mod_webauthldap.so
# Webauth Keytab & credential cache file
WebAuthLdapKeytab /etc/webauth/webauth.keytab
WebAuthLdapTktCache /var/lib/webauth/krb5cc_ldap
# LDAP Host Information
WebAuthLdapHost ldap.example.com
WebAuthLdapBase ou=users,dc=example,dc=com
WebAuthLdapAuthorizationAttribute privilegeAttribute
WebAuthLdapDebug on
<Location />
WebAuthLdapAttribute givenName
WebAuthLdapAttribute sn
WebAuthLdapAttribute cn
WebAuthLdapAttribute mail
</Location>
Misc Permissions
Most of the files that you created do not have the appropriate permissions to be read/written by the webserver. Lets fix that.
I created a VHost for both “webauth.example.com” and “gmcsrvx2.example.com”, with the latter requiring user authentication to view. I name these after the hostname they are serving in /etc/httpd/conf.d/webauth.conf, Just throw a simple Hello World into the DocumentRoot that you configure for your testing host. Note that you must have NameVirtualHost-ing setup for both ports 80 and 443.
<VirtualHost *:80>
ServerName webauth.example.com
ServerAlias webauth
# Send them to somewhere useful if they request the root of this VHost
RedirectMatch permanent ^/$ https://gmcsrvx2.example.com/
# Send non-HTTPS traffic to HTTPS since we are dealing with passwords
RedirectMatch permanent ^/(.+)$ https://webauth.example.com/$1
</VirtualHost>
<VirtualHost *:443>
# Name to respond to
ServerName webauth.example.com
ServerAlias webauth
# Root directory
DocumentRoot /usr/local/share/weblogin
# SSL
SSLEngine On
SSLCertificateFile /etc/pki/example.com/webauth/host-cert.pem
SSLCertificateKeyFile /etc/pki/example.com/webauth/host-key.pem
SSLCACertificateFile /etc/pki/example.com/example-ca.crt
# Web Login directory needs some special love
<Directory "/usr/local/share/weblogin">
AllowOverride none
Options ExecCGI
AddHandler fcgid-script .fcgi
Order allow,deny
Allow from all
</Directory>
# This allows you to not need to put the file extension on the scripts
ScriptAlias /login "/usr/local/share/weblogin/login.fcgi"
ScriptAlias /logout "/usr/local/share/weblogin/logout.fcgi"
ScriptAlias /pwchange "/usr/local/share/weblogin/pwchange.fcgi"
# More special options to make things load right based on your template
Alias /images "/usr/local/share/weblogin/generic/images"
Alias /help.html "/usr/local/share/weblogin/generic/help.heml"
Alias /style.css "/usr/local/share/weblogin/generic/style.css"
# This is the actual web KDC
<Location /webkdc-service>
SetHandler webkdc
</Location>
</VirtualHost>
And the host file at /etc/httpd/conf.d/gmcsrvx2.conf:
12345678910111213141516171819202122
<VirtualHost *:80>
ServerName gmcsrvx2.example.com
ServerAlias gmcsrvx2
DocumentRoot /var/www/html
RedirectMatch permanent ^/(.+)$ https://gmcsrvx2.example.com/$1
</VirtualHost>
<VirtualHost *:443>
ServerName gmcsrvx2.example.com
ServerAlias gmcsrvx2
DocumentRoot /var/www/gmcsrvx2
SSLEngine On
SSLCertificateFile /etc/pki/example.com/gmcsrvx2/host-cert.pem
SSLCertificateKeyFile /etc/pki/example.com/gmcsrvx2/host-key.pem
SSLCACertificateFile /etc/pki/example.com/example-ca.crt
# Require a webauth valid user to access this directory
<Directory "/var/www/gmcsrvx2">
AuthType WebAuth
Require valid-user
</Directory>
</VirtualHost>
Once you are all set, start Apache. Then navigate your web browser to the host (http://gmcsrvx2.example.com). This should first redirect you to HTTPS, then bounce you to the WebLogin pages. After authenticating, you should be able to access the server.
Conclusion
This is by no means a simple thing to get set up. An intimate knowlege how how Kerberos, Apache, and LDAP work is crucial to debugging issues relating to WebAuth. All of my sample configuration files can be found in the Archive.
First let’s review how your traffic gets to it’s destination. You open up your favorite web browser and punch in “www.google.com”. Since your computer works with IP addresses, and not names (like people), you need to do a process of resolving the name. The Domain Name System (DNS) is the service that does this. Your ISP runs DNS servers, that are typically given to you in your DHCP lease. Your computer sends a query to this DNS server asking “what is the IP address of www.google.com”. Since your ISP does not have control over the “google.com” domain, the request is forwarded to another server. Eventually someone says “Hey! www.google.com is at 72.14.204.104”. This response is sent back through the servers to your computer.
After resolving the name, your computer now creates a data packet that will be sent to the computer at 72.14.204.104. The packet gets there by looking at your hosts route table and will hit your default gateway.
The Technology
IP over DNS is a method of encapsulating IP packets inside a DNS query. This essentially creates a VPN between a server and a client. In my case, I setup and configured the Iodine DNS server on my server located at RIT that I would use to route my traffic to the internet. Inside the tunnel exists the subnet of 192.168.0.0/27, with the server being at 192.168.0.1. My client connected and received IP address 192.168.0.2.
The Problem
Certain WLAN installations will include a guest SSID that non-controlled clients can associate to (In this scenario, I will use ExampleGuest, keeping the actual one I used anonymous). Often your traffic will get routed to a captive portal first, requiring you to enter credentials supplied by your organization. Until you do so, none of your traffic will reach it’s destination outside the LAN… UNLESS you discover a vulnerability, such as the one we are going to explore.
When I walked into the Example Corp site, I flipped open my trusty laptop and got to work. After associating with the ExampleGuest SSID, I was given the IP address 10.24.50.22/24 with gateway 10.24.50.1. I opened my web browser in attempt to get to “www.google.com”, and was immediately directed to the captive portal asking me to log in. Obviously I do not have credentials to this network, so I am at a dead end at this time.
Next I whipped out a terminal session, and did a lookup on “www.google.com”. Lo and behold the name was resolved to it’s external IP address. This means that DNS traffic was being allowed past the captive portal and out onto the network. If “www.google.com” resolved to something like “1.1.1.1” or an address in the 10.24.0.0/16 space, I know that the captive portal is grabbing all of my traffic. This would be the end of this experiment. However as I saw, this was not the case. External DNS queries were still being answered. Time to do some hacks.
The Exploit
Step 1) I need my Iodine client to be able to “query” my remote name server. I added a static route to my laptop that forced traffic to my server through the gateway given to me by the DHCP server. (ip route add 129.21.50.104 via 10.24.50.1)
Step 2) I need to establish my IPoDNS tunnel. (iodine -P mypasswordwenthere tunnel.grantcohoe.com) A successful connection gave me the IP address 192.168.0.2. I was then able to ping 192.168.0.1, which is the inside IP address of the tunnel).
Step 3) I need to change my default gateway to the server address inside the tunnel (ip route add default via 192.168.0.1).
And with that, all of my traffic is going to be encapsulated over the IP over DNS tunnel and sent to my server as DNS queries, this giving me unrestricted internet access bypassing the captive portal.
The Defense
This entire experiment would grind to a halt if DNS queries were not being handled externally. The network administrator should enable features necessary to either drop DNS requests or answer them with the captive portal IP address.
The Conclusion
End result: Unrestricted internet access over a wireless network that I have no credentials to.
Difficulty of setting this up: HIGH
Speed of tunneled internet: SLOW
Worth it for practical use: NOT AT ALL
Worth it for education: A LOT
This sort of trick can work on airplane and hotel wireless systems as well. Most sites do not think to have their captive portals capture DNS traffic in addition to regular IP traffic. As we can see here, it can be used against them.
I have a shiny new VM server sitting in my dorm room. I have access to two networks, one operated by my dorm organization and the other provided to me by RIT. Both get me to the internet, but do so through different paths/SLAs. I want my server to be accessible from both networks. I also want to be able to attach VM hosts to NAT’d networks behind each respective network. This gives me a total of four possible VM networks (primary-external, primary-internal, secondary-external, secondary-internal). I have a Hurricane Electric IPv6 tunnel endpoint configured on the server, and want IPv6 connectivity available on ALL networks regardless of being external or internal. And to complicate matters, my external IP addresses are given to me via DHCP so I cannot set anything statically. Make it so.
Dependencies
IPTables
EBTables
Kernel 2.6 or newer
IPCalc
You also need access to two networks.
Script Setup
First, like any good shell script, we should define our command paths:
Now we need to load in the relevant subnet layouts. In an ideal world this would be set statically. However due to the nature of the environment I am in, both of my networks serve me my address via DHCP and is subject to change. This is very dirty and not very efficient, but it works:
12345678910111213141516171819202122
# Here we will get the subnet information for the primary network
PRIMARY_EXTERNAL_NETWORK=`$IPCALC -n $(ip -4 addr show dev $PRIMARY_EXTERNAL_INTERFACE | grep "inet" | cut -d ' ' -f 6) | cut -d = -f 2`
PRIMARY_EXTERNAL_PREFIX=`$IPCALC -p $(ip -4 addr show dev $PRIMARY_EXTERNAL_INTERFACE | grep "inet" | cut -d ' ' -f 6) | cut -d = -f 2`
PRIMARY_EXTERNAL_NETWORK="$PRIMARY_EXTERNAL_NETWORK/$PRIMARY_EXTERNAL_PREFIX"
# Then the NAT'd network behind the primary
PRIMARY_INTERNAL_NETWORK=`ip -4 addr show dev $PRIMARY_INTERNAL_INTERFACE | grep "inet" | cut -d ' ' -f 6 | sed "s/[0-9]\+\//0\//"`
# Now the subnet information for the secondary network
SECONDARY_EXTERNAL_NETWORK=`$IPCALC -n $(ip -4 addr show dev $SECONDARY_EXTERNAL_INTERFACE | grep "inet" | cut -d ' ' -f 6) | cut -d = -f 2`
SECONDARY_EXTERNAL_PREFIX=`$IPCALC -p $(ip -4 addr show dev $SECONDARY_EXTERNAL_INTERFACE | grep "inet" | cut -d ' ' -f 6) | cut -d = -f 2`
SECONDARY_EXTERNAL_NETWORK="$SECONDARY_EXTERNAL_NETWORK/$SECONDARY_EXTERNAL_PREFIX"
# And it's NAT'd network.
SECONDARY_INTERNAL_NETWORK=`ip -4 addr show dev $SECONDARY_INTERNAL_INTERFACE | grep "inet" | cut -d ' ' -f 6 | sed "s/[0-9]\+\//0\//"`
# This is where we load in the IP addresses of the interfaces
PRIMARY_EXTERNAL_IP=`ip -4 addr show dev $PRIMARY_EXTERNAL_INTERFACE | grep inet | cut -d ' ' -f 6 | sed "s/\/[0-9]\+$//"`
PRIMARY_INTERNAL_IP=`ip -4 addr show dev $PRIMARY_INTERNAL_INTERFACE | grep inet | cut -d ' ' -f 6 | sed "s/\/[0-9]\+$//"`
SECONDARY_EXTERNAL_IP=`ip -4 addr show dev $SECONDARY_EXTERNAL_INTERFACE | grep inet | cut -d ' ' -f 6 | sed "s/\/[0-9]\+$//"`
SECONDARY_INTERNAL_IP=`ip -4 addr show dev $SECONDARY_INTERNAL_INTERFACE | grep inet | cut -d ' ' -f 6 | sed "s/\/[0-9]\+$//"`
# We get the gateways by pinging once out of the appropriate interface to the multicast address of All Routers on the network.
PRIMARY_GATEWAY_IP=`ping -I $PRIMARY_EXTERNAL_IP 224.0.0.2 -c 1 | grep "icmp_seq" | cut -d : -f 1 | awk '{print $4}'`
SECONDARY_GATEWAY_IP=`ping -I $SECONDARY_EXTERNAL_IP 224.0.0.2 -c 1 | grep "icmp_seq" | cut -d : -f 1 | awk '{print $4}'`
System Configuration
In order to get most of these features working, we need to enable IP forwarding in the kernel.
123
echo "Enabling IP forwarding..."
sysctl -w net.ipv4.ip_forward=1
sysctl -w net.ipv6.conf.all.forwarding=1
Routes
The routes are the most critical part of making this whole system work. We use two cool features of the linux networking stack, Route Tables and Route Rules. Route Tables are similar to your VRF tables in Cisco-land. Basically you maintain several different routing tables on a single system in addition to the main one. In order to specify what table a packet should use, you configure Route Rules. A rule basically says “Packets that match this rule should use table A”. In this system I use rules to force a packet to use a route table based on its source address.
# Kernel IP Routes
echo "Setting system default gateways"
ip route del default
ip route add default via $PRIMARY_GATEWAY_IP
ip -6 route add ::/0 dev $IPV6_TUNNEL_INTERFACE
echo "Adding default routes for primary external network..."
ip route flush table $PRIMARY_EXTERNAL_INTERFACE-routes
ip route add $PRIMARY_EXTERNAL_NETWORK dev $PRIMARY_EXTERNAL_INTERFACE src $PRIMARY_EXTERNAL_IP table $PRIMARY_EXTERNAL_INTERFACE-routes
ip route add default via $PRIMARY_GATEWAY_IP table $PRIMARY_EXTERNAL_INTERFACE-routes
echo "Adding default routes for secondary external network..."
ip route flush table $SECONDARY_EXTERNAL_INTERFACE-routes
ip route add $SECONDARY_EXTERNAL_NETWORK dev $SECONDARY_EXTERNAL_INTERFACE src $SECONDARY_EXTERNAL_IP table $SECONDARY_EXTERNAL_INTERFACE-routes
ip route add default via $SECONDARY_GATEWAY_IP table $SECONDARY_EXTERNAL_INTERFACE-routes
echo "Creating routes for primary internal network..."
ip route flush table $PRIMARY_INTERNAL_INTERFACE-routes
ip route add $PRIMARY_INTERNAL_NETWORK dev $PRIMARY_INTERNAL_INTERFACE table $PRIMARY_INTERNAL_INTERFACE-routes
ip route add default via $PRIMARY_GATEWAY_IP table $PRIMARY_INTERNAL_INTERFACE-routes
echo "Creating routes for secondary internal network..."
ip route flush table $SECONDARY_INTERNAL_INTERFACE-routes
ip route add $SECONDARY_INTERNAL_NETWORK dev $SECONDARY_INTERNAL_INTERFACE table $SECONDARY_INTERNAL_INTERFACE-routes
ip route add default via $SECONDARY_GATEWAY_IP table $SECONDARY_INTERNAL_INTERFACE-routes
echo "Creating route rules for primary external network..."
ip rule del from $PRIMARY_EXTERNAL_IP
ip rule add from $PRIMARY_EXTERNAL_IP table $PRIMARY_EXTERNAL_INTERFACE-routes
echo "Creating route rules for secondary external network..."
ip rule del from $SECONDARY_EXTERNAL_IP
ip rule add from $SECONDARY_EXTERNAL_IP table $SECONDARY_EXTERNAL_INTERFACE-routes
echo "Creating route rules for primary internal network..."
ip rule del from $PRIMARY_INTERNAL_NETWORK
ip rule add from $PRIMARY_INTERNAL_NETWORK lookup $PRIMARY_INTERNAL_INTERFACE-routes
echo "Creating route rules for secondary internal network..."
ip rule del from $SECONDARY_INTERNAL_NETWORK
ip rule add from $SECONDARY_INTERNAL_NETWORK lookup $SECONDARY_INTERNAL_INTERFACE-routes
Firewall
My script also maintains the host firewall rules, so we’ll enable those:
echo "Reseting firewall rules..."
$SERVICE iptables restart
$SERVICE ip6tables restart
echo "Setting up IPv4 firewall rules..."
$IPTABLES -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
$IPTABLES -A INPUT -p ipv6 -j ACCEPT
$IPTABLES -A INPUT -p icmp -j ACCEPT
$IPTABLES -A INPUT -i lo -j ACCEPT
$IPTABLES -A INPUT -s mason.csh.rit.edu -p tcp --dport 5666 -j ACCEPT
$IPTABLES -A INPUT -p tcp --dport 53 -j ACCEPT
$IPTABLES -A INPUT -p udp --dport 53 -j ACCEPT
$IPTABLES -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
$IPTABLES -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
$IPTABLES -A INPUT -j REJECT --reject-with icmp-host-prohibited
echo "Setting up IPv6 firewall rules..."
$IP6TABLES -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
$IP6TABLES -A INPUT -p ipv6-icmp -j ACCEPT
$IP6TABLES -A INPUT -i lo -j ACCEPT
$IP6TABLES -A INPUT -p tcp --dport 53 -j ACCEPT
$IP6TABLES -A INPUT -p udp --dport 53 -j ACCEPT
$IP6TABLES -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
$IP6TABLES -A INPUT -j REJECT --reject-with icmp6-adm-prohibited
echo "Setting up firewall rules for primary internal network..."
$IPTABLES -A FORWARD -s $PRIMARY_INTERNAL_NETWORK -j ACCEPT
$IPTABLES -t nat -A POSTROUTING -s $PRIMARY_INTERNAL_NETWORK -j SNAT --to-source $PRIMARY_EXTERNAL_IP
echo "Setting up firewall rules for secondary internal network..."
$IPTABLES -A FORWARD -s $SECONDARY_INTERNAL_NETWORK -j ACCEPT
$IPTABLES -t nat -A POSTROUTING -s $SECONDARY_INTERNAL_NETWORK -j SNAT --to-source $SECONDARY_EXTERNAL_IP
echo "Setting up default firewall actions..."
$IPTABLES -A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
$IPTABLES -A FORWARD -j REJECT --reject-with icmp-port-unreachable
Router Advertisements
One of the features of my networking setup is that all networks, no matter internal or external have IPv6 connectivity. This is achieved by sending Router Advertisements out all interfaces. This is all well and good for the VMs that I am hosting, but these advertisements travel out of the physical interfaces to other hosts on the network! This is not good in that I am allowing other users to use my IPv6 tunnel interface. This puzzled me for a long time until I discovered a handy little program called “ebtables”. Ebtables is basically iptables for layer 2. As such, I was able to filter all router advertisement broadcasts out of the physical interfaces.
123456
echo "Blocking IPv6 router advertisement to the world..."
$SERVICE radvd stop
$SERVICE ebtables restart
$EBTABLES -A OUTPUT -d 33:33:0:0:0:1 -o eth1 -j DROP
$EBTABLES -A OUTPUT -d 33:33:0:0:0:1 -o eth0 -j DROP
$SERVICE radvd start
End Result
You now have a dual-homed server with two external networks, two internal networks, and IPv6 connectivity to all. Both external networks can receive their configuration via DHCP and will dynamically adjust their routing accordingly.
If I am the requester or the requestee, it means that I have had a conversation with you (Not chat/texted. Actual person-to-person) on at least two separate occasions. (Occasionally this may be reduced to one depending on circumstances) Do not get this confused with what I consider a “dialog”. A dialog is nothing more than “Hi! How are you today? What’s up?”. A conversation entails we discussed and compared our thoughts on at least one topic at length >5min. Also If you are from my present, it means that I consider there a statistically significant chance that we will see each other in person again in the future. If you are from my past, then we talked at least twice on several (>2) occasions. If you are from my future, well that feature hasn’t been implemented yet and will probably cause me to segfault.
The logic behind these guidelines is relatively simple. There is no substitute for real life interaction. The idea of “meeting someone over Facebook” doesn’t sit well with me. Whenever I see a person friend-request someone they don’t really know or have had any degree of conversation with, it makes me question their ability to do so face-to-face. It also helps avoid the potential “stalker” effect that can come about easily with social media.
Take for example Jack and Jill. Jack is an avid fan of a school sports team. Jill is his favorite player on the sports team. Jack and Jill do not know each other, and have had nothing more than a dialog (as defined earlier) between them at various meet-n-greet events. Jill gets a friend-request from Jack several days later. Jill not wanting to potentially upset Jack accepts the friend-request. She now encounters frequent chat messages from Jack that lead a quick and dull conversation. Various other actions get Jack labeled as a “stalker” in Jills mind. This is not where Jack wanted to end up as he is now considered “the bad guy”. While his motives (assume everything here to be at face-value, no hidden agendas) are honorable, they have led to an awkward situation for both of them.
By my guidelines, Jill should never have accepted the friend-request. Similarly Jack should have never sent one. Since neither of them have had any level of in-person conversations, they have no common ground on which to establish a friendship.
I will occasionally receive friend-requests from people from High School. These I find myself subjecting to a higher degree of scrutiny since I do not associate with this crowd a whole lot anymore. If I did not particularly enjoy your presence back then, I probably have no reason to change that.
My organization stores user information in an OpenLDAP directory server. This includes contact information, shell preferences, iButton IDs, etc. The directory server does not store the users password and relies on a Kerberos KDC to provide authentication via SASL. This is a widely supported configuration and is popular because you can achieve SSO (Single Sign-On) and keep passwords out of your directory. Our shell access machines are configured to do LDAP authentication of users which basically instructs PAM to do a bind to the LDAP server using the credentials provided by the user. If the bind is successful, the user logs in. If not, well sucks to be them. Before this project we had a single server handling the directory running CentOS 5.8/OpenLDAP 2.3. If this server went poof, then all userdata is lost and no one could log into any other servers (or websites if configured with Stanford Webauth. See the guide for more on this). Obviously this is a problem that needed correcting.
Technology
LDAP Synchronization
OpenLDAP 2.4 introduced a method of replication called “mirrormode”. This allows you to use their already existing syncrepl protocol to synchronize data between multiple servers. Previously (OpenLDAP <= 2.3) only allowed you to do a master-slave style of replication where the slaves are read-only and would be unwilling to perform changes. Obviously this has some limitations in that if your master (or “provider” in OpenLDAP-ish) were to die, then your users cannot do any changes until you get it back online. Using MirrorMode, you can create an “N-way Multi-Master” topology that allows all servers to be read/write and instantly replicate their changes to the others.
High Availability
To enable users to have LDAP access even if one of the servers dies, we need a way to keep the packets flowing and automatically failover to the secondary servers. Linux has a package called “Heartbeat” that allows this sort of functionality. If you are familiar with Cisco HSRP or the open-source VRRP, it works the same way. Server A is assigned an IP address (lets say 10.0.0.1) and Server B is assigned a different address (lets say 10.0.0.2). Heartbeat is configured to provide a third IP address (10.0.0.3) that will always be available between the two. On the primary server, an ethernet alias is created with the virtualized IP address. Periodic heartbeats (keep-alive packets) are sent out to the other servers to indicate “I’m still here!”. Should these messages start disappearing (like when the server dies), the secondary will notice the lack of updates and automatically create a similar alias interface on itself and assign that virtualized IP. This allows you to give your clients one host, but have it seemlessly float between several real ones.
SASL Authentication
The easy way of configuring MirrorMode requires you to store your replication DN’s credentials in plaintext in the config file. Obviously this is not very secure since you are storing passwords in plaintext. As such we can use the hosts Kerberos principal to bind to the LDAP server as the replication DN and perform all of the tasks we need to do. This is much better than plaintext!
SSL
Since we like being secure, we should really be using LDAPS (LDAP + SSL) to get our data. We will be setting this up too using our PKI. We will save this for the end since we want to ensure that our core functionality actually works first.
New Servers
I spun up two new machines, lets call them “warlock.example.com” and “ldap2.example.com”. Each of them runs my favorite Scientific Linux 6.2 but with OpenLDAP 2.4. You cannot do MirrorMode between 2.3 and 2.4.
Configuration
Packages
First we need to install the required packages for all of this nonsense to work.
openldap (LDAP libraries)
openldap-servers (Server)
openldap-clients (Client utilities)
cyrus-sasl (SASL daemon)
cyrus-sasl-ldap (LDAP authentication for SASL)
cyrus-sasl-gssapi (Kerberos authentication for SASL)
krb5-workstation (Kerberos client utilities)
heartbeat (Failover)
If you are coming off of a fresh minimal install, you might want to install openssh-clients and vim as well. Some packages may not be included in your base distribution and may need other repositories (EPEL, etc)
Kerberos Keytabs
Each host needs its own set of host/ and ldap/ principals as well as a shared one for the virtualized address. In the end, you need a keytab with the following principals:
host/ldap1.example.com@EXAMPLE.COM
ldap/ldap1.example.com@EXAMPLE.COM
host/ldap.example.com@EXAMPLE.COM
ldap/ldap.example.com@EXAMPLE.COM
WARNING: Each time you write a -randkey’d principal to a keytab, it’s KVNO (Key Version Number) is increased, thus invalidating all previous written principals. You need to merge the shared principals into each hosts keytab. See my guide on Kerberos Utilities for information on doing this.
Put this file at /etc/krb5.keytab and set an ACL on it such that the LDAP user can read it.
If you see something like “kerberos error 13” or “get_sec_context: 13” it means that someone cannot read the keytab, usually the LDAP server. Fix it.
NTP
Kerberos and the OpenLDAP synchronization require synchronized clocks. If you aren’t already set up for this, follow my guide for setting up NTP on a system before continuing.
SASL
You need saslauthd to be running on both LDAP servers. If you do not already have this set up, follow my guide for SASL Authentication before continuing.
Logging
We will be using Syslog and Logrotate to manage the logfiles for the LDAP daemon (slapd). By default it will spit out to local4. This is configurable depending on your system, but for me I am leaving it as the default. Add an entry to your rsyslog config file (usually /etc/rsyslog.conf) for slapd
1
local4.* /var/log/slapd.log
Now unless we tell logrotate to rotate this file, it will get infinitely large and cause you lots of headaches. I created a config file to automatically rotate this log according the the system defaults. This was done at /etc/logrotate.d/slapd:
123
/var/log/slapd.log {
missingok
}
Then restart rsyslog. Logrotate runs on a cron job and is not required to be restarted.
Data Directory
Since I am grabbing the data from my old LDAP server, I will not be setting up a new data directory. On the old server and on the new master, the data directory is /var/lib/ldap/. I simply scp’d all of the contents from the old server over to this directory. If you do this, I recommend stopping the old server for a moment to ensure that there are no changes occurring while you work. After scp-ing everything, make sure to chown everything to the LDAP user. I also recommend running slapindex to ensure that all data gets reindexed.
Client Library Configuration
Now to make it convenient to test your new configuration, edit your /etc/openldap/ldap.conf to change the URI and add the certificate settings.
1234
BASE dc=example,dc=com
URI ldaps://ldap.example.com
TLS_CACERT /etc/pki/tls/example-ca.crt
TLS_REQCERT allow
When configuring this file on the servers, TLS_REQCERT must be set to ALLOW since we are doing Syncrepl over LDAPS. Obviously since we are using a shared certificate for ldap.example.com, it will not match the hostname of the server and will fail. On all of your clients, they should certainly “demand” the certificate. But in this instance that prevents us from accomplishing Syncrepl over LDAPS.
Certificates for LDAPS
Since people want to be secure, OpenLDAP has the ability to do sessions inside of TLS tunnels. This works the same way HTTPS traffic does. To do this you need to have the ability to generate SSL certificates based on a given CA. This procedure varies from organization to organization. Regardless of your method, the hostname you chose is critical as this will be the name that is verified. In this setup, we are creating a host called “ldap.example.com” that all LDAP clients will be configured to use. As such the same SSL certificate for both hosts will be generated for “ldap.example.com” and placed on each server.
I placed the public certificate at /etc/pki/tls/certs/slapd.pem, the secret key in /etc/pki/tls/private/slapd.pem, and my CA certificate at /etc/pki/tls/example-ca.crt. After obtaining my files, I verify them to make sure that they will actually work:
If you are using a previous server configuration, just scp the entire /etc/openldap code over to the new servers. Make sure you blow away any files or directories that may have been added for you before you copy. If you are not, you might need to do a bit of setup. OpenLDAP 2.4 uses a new method of configuration called “cn=config”, which stores the configuration data in the LDAP database. However since this is not fully supported by most clients, I am still using the config file. For setting up a fresh install, see one of my other articles on this. (Still in progress at the time of this writing)
The following directives will need to be placed in your slapd.conf for the SSL certificates:
Depending on your system, you may need to configure the daemon to run on ldaps://. On Red-Hat based systems this is in /etc/sysconfig/ldap.
You need to add two things to your configuration for Syncrepl to function. First to your slapd.conf:
123456789101112131415
# Syncrepl ServerID
serverID 001
# Syncrepl configuration for mirroring instant replication between another
# server. The binddn should be the host/ principal of this server
# stored in the Kerberos keytab
syncrepl rid=001
provider=ldaps://ldap2.example.com
type=refreshAndPersist
retry="5 5 300 +"
searchbase="dc=example,dc=com"
attrs="*,+"
bindmethod=sasl
binddn="cn=ldap1,ou=hosts,dc=example,dc=com"
mirrormode TRUE
The serverID value must uniquely identify the server. Make sure you change it when inserting the configuration onto the secondary server. Likewise change the Syncrepl RID as well for the same reason.
Secondly you need need to allow the replication binddn full read access to all objects. This should go in your ACL file (which could be your slapd.conf, but shouldnt be).
123
access to *
by dn.exact="cn=ldap1,ou=hosts,dc=example,dc=com" read
by dn.exact="cn=ldap2,ou=hosts,dc=example,dc=com" read
WARNING: If you do not have an existing ACL setup, doing just these entries will prevent anything else from doing anything with your server. These directives are meant to be ADDED to an existing ACL.
You should be all set and ready to start the server.
123
[root@ldap1 ~]# service slapd start
Starting slapd: [ OK ]
[root@ldap1 ~]#
Make sure that when you do your SASL bind, the server reports your DN correctly. To verify this:
That DN is the one that should be in your ACL and have read access to everything.
Since the LDAP user will always need this principal, I recommend adding a cronjob to keep the ticket alive. I wrote a script that will renew your ticket and fix an SELinux permissioning problem as well. You can get it here. Just throw it into your /usr/local/sbin/ directory. There are better ways of doing this, but this one works just as well.
1
0 */2 * * * /usr/local/sbin/slaprenew
Firewall
Dont forget to open up ports 389 and 636 for LDAP and LDAPSSL!
Searching
After configuring both the master and the secondary servers, we now need to get the data initially replicated across your secondaries. Assuming you did like me and copied the data directory from an old server, your master is the only one that has real data. Start the service on this machine. If all went according to plan, you should be able to search for an object to verify that it works.
If you get errors, increase the debug level (-d 1) and work out any issues.
Replication
After doing a kinit as the LDAP user described above (host/ldap2.example.com), start the LDAP server on the secondary. If you tail the slapd log file, you should start seeing replication events occurring really fast. This is the server loading its data from the provider as specified in slapd.conf. Once it is done, try searching the server in a similar fashion as above.
Heartbeat provides failover for configured resources between several Linux systems. In this case we are going to provide high-availability of an alias IP address (10.0.0.3) which we will distribute to clients as the LDAP server (ldap.example.com). Heartbeat configuration is very simple and only requires three files:
/etc/ha.d/haresources contains the resources that we will be failing over. It should be the same on both servers.
The reason the name of the host above is not “ldap.example.com” is because the entry must be a node listed in the configuration file (below).
/etc/ha.d/authkeys contains a shared secret used to secure the heartbeat messages. This is to ensure that someone doesnt just throw up a server and claim to be a part of your cluster.
12
auth 1
1 md5 mysupersecretpassword
Make sure to chmod it to something not world-readable (600 is recommend).
Finally, /etc/ha.d/ha.cf is the Heartbeat configuration file. It should contain entries for each server in your cluster, timers, and log files.
After all this is set, start the heartbeat service. After a bit you should see another ethernet interface show up on your primary. To test failover, unplug one of the machines and see what happens!
DNS Hack
A side affect of having the LDAP server respond on the shared IP address is that it gets confused about its hostname. As such you need to edit your /etc/hosts file to point the ldap.example.com name to each of the actual host IP addresses. In this example, ldap1 is 10.0.0.1 and ldap2 is 10.0.0.2. You should modify your hosts file to include:
Likewise if you log into a client machine (in my case, I made this the KDC) you should be able to search at only the ldap.example.com host. The others will return errors.
The reason is that the client has a different view of what the server’s hostname is. This difference causes SASL to freak out and not allow you to bind.
slapd[9888]: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
slapd[9888]: text=SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
This means your DNS is not hacked or functional. Fix it according to the instructions
GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache permissions incorrect)
You need to change the SELinux context of your KRB5CC file (default is /tmp/krb5cc_$UIDOFLDAPUSER on Scientific Linux) to something the slapd process can read. Since every time you re-initialize the ticket you risk defaulting the permissions, I recommend using my script in your cronjob from above. If someone finds a better way to do this, please let me know!
If after enabling SSL on your client you receive (using -d 1):
1
TLS: can't connect: TLS error -5938:Encountered end of file.
Check your certificate files on the SERVER. Odds are you have an incorrect path.
Summary
You now have two independent LDAP servers running OpenLDAP 2.4 and synchronizing data between them. They are intended to listen on a Heartbeat-ed service IP address that will always be available even if one of the servers dies. You can also do SASL binds to each server to avoid having passwords stored in your configuration files.
If after reading this you feel there is something that can be improved or isnt clear, feel free to contact me! Also you can grab sample configuration files that I used at http://archive.grantcohoe.com/projects/ldap.
