Distributed Service Management

Distributed Service Management (or DSM for short) is a way to start/stop/restart services across multiple servers.

Changelog

Download

dsm-1.1.zip

About

Distributed Service Management (or DSM for short) is a way to start/stop/restart services across multiple servers.

The guts of DSM work by using ssh to execute remote commands on other servers. The actual commands executed are stored in a simple config file dsm.cfg

DSM can start/stop any type of service. It can do anything you normally do on the command line; just faster. It can restart init style services, daemontools services, or it can run processes and scripts directly.

Initial configuration

Initial configuration involves listing the services you want to mange in dsm.cfg and exchanging ssh keys so that you can execute the restarts on other servers

example dsm.cfg

[hosts]
[[app01]]
enabled = apache,memcached
[[app02]]
enabled = apache,memcached

[services]
[[apache]]
runas = root
start = /etc/init.d/httpd restart
stop  = /etc/init.d/httpd stop
restart = /etc/init.d/httpd graceful

[[memcached]]
start = svc -u /service/memcached
stop = svc -d /service/memcached

bash alias

I find it easy to setup a bash alias for dms called "menu". This can be put into your ~/.bashrc file. This is assumed for all the examples listed here.

alias menu='python /path/to/dsm.py'

Example Usage

This is an example of how you could restart the services (apache and memcached) on both servers (app01 and app02). Each of the following blocks are equivalent 

menu stop apache,memcached on app01,app02
menu start apache,memcached on app01,app02

menu restart apache,memcached on app01,app02

menu restart apache,memcached on all

menu restart all on all

I lied a little; given the config file above restart executes a different command for apache then start/stop does.

Config File Details

the start/stop commands get executed on remote hosts in the following way

If you have /etc/init.d/httpd start as your start command and you execute menu start apache from app01 from user1's shell the following command gets executed

su - root -c '/etc/init.d/httpd start

if you instead run menu start apache on app02 from app01 (to start it on the other server) the following command will get executed

ssh -n root@app02 /etc/init.d/httpd start

if you wanted to restart memcached on app02 from app01 and ran the following command menu restart memcached on app02 the following statement would get executed

ssh -n user1@app02 svc -d /service/memcached \&\& svc -u /service/memcached

notice that it ran the command as 'user1'. This is the default user as specified in dsm.py, and can be overridden on a per command basis in the config file (as is done for the apache example)

more about command line options

Command line options can be complicated or simple. you can string commands together, or run them individualy. You can also extract and modify state information directly from the command line which makes dsm very extenable, and easy to integrate into other scripts.

To restart multiple services string them together with commas (no spaces)

menu restart apache,memcached

To execute the command on multiple servers string them together with commas (no spaces)

menu restart all on app01,app02

About the expandable 'all' keyword. 'All' is a context sensitive expansion. It will expand to all the enabled services or hosts. So if i only have apache enabled on app01, and memcached enabled on app02. menu restart all on all will restart apache on app01, and memcached on app02. Thats the same as the following longer command; but in the heat of the moment you don't have to look up what is where

menu restart all on all
menu restart apache on app01 restart memcached on app02

you can exclude services; good when the list of servers or services grows

menu restart all except memcached on app01
menu restart all on all except app01

A few words about state

The first section of the config file (hosts) keeps track of which hosts you want to run commands from, and which services are supposed to be running where

You can extract state via the command line with several commands

menu isenabled apache on app01
menu isdisabled memcached on app02

you can also modify it

menu enable apache on app01
menu disable memeached on app02

you can also see a human readable list of enabled services

menu list apache
menu list app01
menu list services
menu list hosts

optional host name

Whenever the host name is omitted from a command line it is assumed to be the localhost (as identified via a system 'host' command)

Security

managing services across multiple servers in an easy way can be tricky to do securely. The examples above use ssh to connect to root to restart apache. You probably don't want to do that, even if you are on a closed network.

You will be better off connecting as a non-privileged user and using sudo to execute the commands. The memcached example uses daeamontools to restart memcached from a nonprivelidged 'user1' account.

In the end only you can decide what meets your security requirements; this is just a tool to make some things easier.