Installing graphite on Windows Server 2003

* This is still work in progress.

Cloud computing is the in thing. It is now possible for anyone to manage, control and run software on multiple machines. Horizontal scaling is increasingly becoming the most used used, over-used and abused word in the industry. Every startup is doing something that involves horizontal scaling and is on the cloud.

There have been some amazing successes and companies are becoming more and more open. There has been a lot of activity towards open source software. And there has been a new generation of softwares and tools being open sourced on git.

One such contribution is statsd from etsy. Statsd is easily combined with graphite to monitor several servers on a cluster. This all works, and works well on unix/linux and mac based machine. Open source does not really help on windows. Making any of these software work on windows is still a nightmare. Getting statsd to work on a ubuntu is as simple as following well defined set of instructions. All one needs to do is install graphite and then follow this statsd blog. These same steps become zillion more steps on windows.

This blog is an attempt to get graphite and statsd working on a windows 2003 server. We start from very basic, by installing python first and then installing graphite and statsd respectively.

Installations:

  • install python
  • install git
  • install cygwin
  • install node.js
  • clone statsd from its github page.

I installed nodejs in its default directory on windows and it went into c:/program files. To run node, it needs to be in your PATH. Now we can test statsd and nodejs both with below command:

node stats.js yourConfileFileForStatsd.js

Most likely, one would get an error relating to problems with watchFile function.In node 0.6+*watchFile* has been replaced with watch. To get rid of this error replace all occurrences of watchFile with watch.

Once done, we can see that node is able to run stats.js properly.

Now let's install graphite following instructions here. One of the very first errors we would get is related to txamqp dependency while installing carbon:

error: Unable to find vcvarsall.bat

I followed several articles on the internet and finally came to this stackoverflow link. This link suggests to install mingw32 to get rid of the error.

This error makes way for a new one:

gcc: error: unrecognized command line option '-mno-cygwin'

Another round of investigation leads to this stackoverflow question. (stackoverflow is the best thing that has happend to mankind since programming started). That answer gets us past that error.

But this is windows!, we are not yet done. We are still trying to install carbon!. Another error now (not really a window related though):

path '/opt/graphite/lib' cannot be absolute!

Fix for this one is to modify carbon source (downloaded with pip for python). Change setup.cfg file and change prefix variable to the correct one pointing to the local installation folder.

With the last carbon error past us, the installation should have been a success. As per the graphite installation instructions now download checkdependencies.py and run. Install any missing requirements. In my case i was missing cairo.

Install cairo from http://www.lfd.uci.edu/~gohlke/pythonlibs/#pycairo

Now install graphite-web for python. Since graphite-web is a web application for python, we would need a server to run it own, which is apache. To host python web application on apache there are two options:

  1. mod_python
  2. mod_wsgi

Which one to choose? Stackoverflow to the rescue and we come to know that mod_python is not the way to go forward.

So we pick mod_wsgi and configure it as described here. Create a .wsgi script and create a virtual host in apache httpd.conf file. Check and see if we get a running wsgi script.

Now we need to create a *graphite database* as per the original graphite installation instructions. And the errors do not seem to go away. On running

sudo python manage.py syncdb

we get

error on syncdb about timezone.

Install pytz for python with

pip install pytz

timezone errors are now gone and we get below error:

settings.DATABASE is not properly configured.

This error is due to wrong settings for dianjo in settings.py file. local_settings.py is not picked up for some reason. The error was due to a wrong graphite db path. Point that path to a local path and the error goes.

Finally, we are at a point where we can start carbon.

carbon-cache.py start

and…. another error!

no module pwd

pwd is unix module which is not available on windows.

We would need pywin32 modules from here to get rid of this error. And now We can replace call to pwd within graphite/lib/carbon/util.py with a function described here at stackoverflow.

And guess what….another error now in twisted module:

no module syslog

And this post continues….