Python XML comparator

Although I am not a big fan of xml formatting myself, but dealing with it is inevitable given how widely it is used. So when searching for a solution to compare two generic XML files, I came across several solution, but none who would completely satisfy a comparison of two generic XML trees in python.  Plus, the code shared was incomplete and without any particular instructions of how to complete it.

So I grabbed different solutions and came up with a solution to achieve the desired result. Therefore, the result was the following python class.

__author__ = 'syawar'

import xml.etree.ElementTree as ET
import logging

class xmlTree():
    """
    .. module:: xmltree
        :platform: Ubuntu, Linux
        :synopsis: convert the xml files passed to an iterable object

    ..
    this module handles conversion of an xml file to an iterable object
    """

    def __init__(self):
        self.logger = logging.getLogger(__name__)
        self.hdlr = logging.FileHandler('/var/log/xml-minipulations.log')
        self.formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
        self.hdlr.setFormatter(self.formatter)
        self.logger.addHandler(self.hdlr)
        self.logger.setLevel(logging.DEBUG)

    @staticmethod
    def convert_file_to_tree( filename):
        """

        :param filename:
        :return:
         the xml file in tree format
        """
        return ET.parse(filename)

    def xml_compare(self, x1, x2, excludes=[]):
        """
        Compares two xml etrees
        :param x1: the first tree
        :param x2: the second tree
        :param excludes: list of string of attributes to exclude from comparison
        :return:
            True if both files match
        """
        if x1.tag != x2.tag:
            self.logger.debug('Tags do not match: %s and %s' % (x1.tag, x2.tag))
            return False
        for name, value in x1.attrib.items():
            if not name in excludes:
                if x2.attrib.get(name) != value:
                    self.logger.debug('Attributes do not match: %s=%r, %s=%r'
                                 % (name, value, name, x2.attrib.get(name)))
                    return False
        for name in x2.attrib.keys():
            if not name in excludes:
                if name not in x1.attrib:
                    self.logger.debug('x2 has an attribute x1 is missing: %s'
                                 % name)
                    return False
        if not self.text_compare(x1.text, x2.text):
            self.logger.debug('text: %r != %r' % (x1.text, x2.text))
            return False
        if not self.text_compare(x1.tail, x2.tail):
            self.logger.debug('tail: %r != %r' % (x1.tail, x2.tail))
            return False
        cl1 = x1.getchildren()
        cl2 = x2.getchildren()
        if len(cl1) != len(cl2):
            self.logger.debug('children length differs, %i != %i'
                         % (len(cl1), len(cl2)))
            return False
        i = 0
        for c1, c2 in zip(cl1, cl2):
            i += 1
            if not self.xml_compare(c1, c2):
                self.logger.debug('children %i do not match: %s'
                             % (i, c1.tag))
                return False
        return True

    def text_compare(self, t1, t2):
        """
        Compare two text strings
        :param t1: text one
        :param t2: text two
        :return:
            True if a match
        """
        if not t1 and not t2:
            return True
        if t1 == '*' or t2 == '*':
            return True
        return (t1 or '').strip() == (t2 or '').strip()

Usage:

The usage is simple. Create object, either convert xml files to xml.etree.ElementTree objects by using convert_file_to_tree or just pass xml.etree.ElementTree to the xml_compare, which will return true if the files are the same and false if files are different.

The excludes list can be passed in case there is a property of a node that you do not want to compare. A prime example is the last-modified  field that most xml files used posses. the function call will then have excludes=[“last-modified”].

 

And that is all folks. Hope this helps!

UCARP-Multiple VIPs onto One interface

For a while now, my colleague and I had been trying to map two VIPs to the same physical interface. We tried various recipes that we found on-line but could not come up with a working solution to the problem and experimented on our own. The major problem was the second VIP would not get adopted at all, or if it would, it would show up on a virtual interface of the previously declared ucarp, which of-course was incorrect.

After days of experiments and failure, we were able to come up with a clean solution to our problem, and since there were no other recipes specifically tailored for this particular issue, I decided to share it with you guys.

NOTE: The following configuration has been developed and tested on debian squeeze.

Assuming that the single interface is eth0, the two VIPS between the two machines are 192.168.0.1 and 192.168.0.2.

The files that were changed/added were:

  • /etc/network/interfaces
  • /usr/share/ucarp/vip-up-1.sh
  • /usr/share/ucarp/vip-up-2.sh
  • /usr/share/ucarp/vip-down-1.sh
  • /usr/share/ucarp/vip-down-2.sh

Here are the configuration files listed above: (The scripts were duplicated on both machines with the IPs changed to what was suitable)

/etc/network/interfaces

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.0.3
netmask 255.255.255.0
gateway 192.168.101.1
up ucarp -i eth0 -s 192.168.0.3 -v 1 -p word -a 192.168.0.1 -u /usr/share/ucarp/vip-up-1 -d /usr/share/ucarp/vip-down-1 -P -B
up ucarp -i eth0 -s 192.168.0.3 -v 2 -p word -a 192.168.0.2 -u /usr/share/ucarp/vip-up-2 -d /usr/share/ucarp/vip-down-2 -P -B
down pkill ucarp

iface eth0:ucarp:0 inet static
address 192.168.0.1
netmask 255.255.255.0

iface eth0:ucarp:1 inet static
address 192.168.0.2
netmask 255.255.255.0

/usr/share/ucarp/vip-up-1.sh

#!/bin/sh

/sbin/ifup $1:ucarp:0

/usr/share/ucarp/vip-down-1.sh

#!/bin/sh

/sbin/ifdown $1:ucarp:0

/usr/share/ucarp/vip-up-2.sh

#!/bin/sh

/sbin/ifup $1:ucarp:1

/usr/share/ucarp/vip-down-2.sh

#!/bin/sh

/sbin/ifdown $1:ucarp:1

The vip-up/Down scripts are duplicated over both machines, however, the interfaces file will have to be altered to suit the other machine, which is rather simple.

I hope this helps!!!

OpenNMS Chrome Extension Official Launch!!!

Finally, the chrome extension that Dominick have been developing for OpenNMS, is available on the Chrome Web store .

The objective behind the development of this application was to allow another method of notifications for OpenNMS and so after introducing it at the the OpenNMS Dev-Jam last month, it is finally available on the chrome store.

You can have a look at it on its official website and add it to your google chrome browser. We have already started planning the next release, which should be sometime the beginning of next year, to include more features and provide better functionality.

The extension is completely opensource and you can get the code at https://bitbucket.org/ssaqibyawargmailcom/opennms-chrome-extension.

Have Fun!!!

OpenNMS Dev-Jam 2013

I spent this week with at the OpenNMS Dev-Jam in Minneapolis. It was an amazing experience and even though I was the least adept with the software at this conference, all the participants were more than happy to steer me in the right direction .

Dominick (friend) and I, developed a Chrome extension and show cased it at the conference, and I was amazed at the amazing response from all of them. Although I plan to submit the extension to the chrome app store, you can get the current beta from here.

It was a great experience, gained some valuable knowledge and made some good friends. I think anyone who wishes to develop seriously in OpenNMS, need to attend these conferences to get an idea of where to begin and how to go about what needs to be developed.

Cheers to the OpenNMS group for organizing this amazing conference!!!

Net-SNMP extension using Perl on debian

I was recently tasked with retrieving some data from a server, to be monitored and graphed withing OpenNMS. Easy enough I though and created some scripts to gather the data and extended Net-SNMP like I did in the previous post, adding it to the OpenNMS data-collection and got into defining the graphs. But then I realized that the graphs were in-correct because the data I was receiving from the extension was String while I needed a COUNTER.

After some research I came across these docs which explained my problem is detail and the solution as well. Extending SNMP through perl. Although the implementation is easy enough, some small configurations and dependencies caused me some hassle, so I thought I would lay the whole plan out.

Dependencies before beginning:

  1. Perl
  2. snmp
  3. snmpd
  4. libnet-snmp-perl
  5. libsnmp-perl

The Perl Script:

A general perl script is below, but you may extend it as you see fit:

#!/usr/bin/perl

use NetSNMP::agent (':all');
use NetSNMP::ASN qw(ASN_OCTET_STR ASN_INTEGER);

sub hello_handler {
  my ($handler, $registration_info, $request_info, $requests) = @_;
  my $request;
  my $string_value = "hello world";
  my $integer_value = "8675309";

  for($request = $requests; $request; $request = $request->next()) {
    my $oid = $request->getOID();
    if ($request_info->getMode() == MODE_GET) {
      if ($oid == new NetSNMP::OID(".1.3.6.1.4.1.8072.9999.9999.1.0")) {
        $request->setValue(ASN_OCTET_STR, $string_value);
      }
      elsif ($oid == new NetSNMP::OID(".1.3.6.1.4.1.8072.9999.9999.1.1")) {
        $request->setValue(ASN_INTEGER, $integer_value);
      }
    } elsif ($request_info->getMode() == MODE_GETNEXT) {
      if ($oid == new NetSNMP::OID(".1.3.6.1.4.1.8072.9999.9999.1.0")) {
        $request->setOID(".1.3.6.1.4.1.8072.9999.9999.1.1");
        $request->setValue(ASN_INTEGER, $integer_value);
      }
      elsif ($oid < new NetSNMP::OID(".1.3.6.1.4.1.8072.9999.9999.1.0")) {         
        $request->setOID(".1.3.6.1.4.1.8072.9999.9999.1.0");
        $request->setValue(ASN_OCTET_STR, $string_value);
      }
    }
  }
}

my $agent = new NetSNMP::agent();
$agent->register("hello_world", ".1.3.6.1.4.1.8072.9999.9999",
                 \&hello_handler);

The above script, however, only uses the STRING and INTEGER library. The COUNTER (ASN_COUNTER) and GAUGE (ASN_GAUGE) libraries can also be used.

 Extending Net-SNMP:

Assuming the above perl script is located at  /usr/src/snmp-extend/extension.pl , edit you snmpd.conf file (commonly located at /etc/snmp/snmpd.conf) to include the following:

disablePerl false
perl do "/usr/src/snmp-extend/extension.pl"

Now restart snmpd and your snmpwalk on the mib or the machine will show the above mibs present in the format you designated.

Note:

Unlike the normal extension, where changing the scripts would change the snmp and you would have to do nothing else, any changes to the perl script will only be applied when the snmpd is restarted.

 

Hope this helps!!!