Discussion:
Python design patterns for hierarchical structure of objects
(too old to reply)
Martin Visser
2010-02-16 02:32:18 UTC
Permalink
Hi,

I'm starting to build a Python program to help better describe systems I
work with. While I am making progress, I am interesting in whether anyone is
aware of some existing projects or even libraries/snippets that might help.

Essentially I need to be able to establish a structure of components
(multi-level members), record connections between components, and record
attributes. It needs to be flexible enough (allowing it to determine valid
attributes and connections), depending on the position of the component in
the structure. While not particularly hard, and I am making progress, being
a relative Python noob, I was hoping for to look at ways other than what I
am writing for it to be done.

I will also need to parse proprietary configuration files to build the
structure, but that is whole other can of worms I guess. (And eventually I
want to output the "structure" in both a normalized text, and graphical
format(eg Graphviz). Eventually I would like to build to something that can
be interacted with directly (probably via Web2.0ish interface).

I have built something similar to this in Perl, however it was very
procedural, and hence turned out very locked into to a fixed set of input
files, and hard to maintain. (I started out learning FORTRAN, so it is hard
to change old habits).

(Googling doesn't seem to find what I want all that easy - I probably need
the right Computer Science keywords to get what I want).


Regards, Martin

***@gmail.com
Matthew Hannigan
2010-02-16 03:28:37 UTC
Permalink
Augeas comies to mind.

http://augeas.net/
"Augeas - a configuration API
Augeas is a configuration editing tool. It parses
configuration files in their native formats and transforms
them into a tree. Configuration changes are made by
manipulating this tree and saving it back into native
config files."


You may also want to look at Puppet if you're not aware if it.

http://reductivelabs.com/products/puppet/

The front page is a bit sales talky so I won't quote it here.
It's also in ruby which is slightly unfortunate.
Post by Martin Visser
Hi,
I'm starting to build a Python program to help better describe systems I
work with. While I am making progress, I am interesting in whether anyone is
aware of some existing projects or even libraries/snippets that might help.
Essentially I need to be able to establish a structure of components
(multi-level members), record connections between components, and record
attributes. It needs to be flexible enough (allowing it to determine valid
attributes and connections), depending on the position of the component in
the structure. While not particularly hard, and I am making progress, being
a relative Python noob, I was hoping for to look at ways other than what I
am writing for it to be done.
I will also need to parse proprietary configuration files to build the
structure, but that is whole other can of worms I guess. (And eventually I
want to output the "structure" in both a normalized text, and graphical
format(eg Graphviz). Eventually I would like to build to something that can
be interacted with directly (probably via Web2.0ish interface).
I have built something similar to this in Perl, however it was very
procedural, and hence turned out very locked into to a fixed set of input
files, and hard to maintain. (I started out learning FORTRAN, so it is hard
to change old habits).
(Googling doesn't seem to find what I want all that easy - I probably need
the right Computer Science keywords to get what I want).
Regards, Martin
_______________________________________________
coders mailing list
http://lists.slug.org.au/listinfo/coders
Martin Visser
2010-02-16 04:09:58 UTC
Permalink
Matthew,

Thanks. I was aware of Puppet, but not Augeas.

Unfortunately, neither seem to be written in Python, and I really want this
to be a bit of coding exercise as much as getting the job done. I really
want to code this - but hopefully using some design concepts from others.
(And I like Python, as it seems to strike the right level in terms of
maturity, utility and snazziness (tm) (

I guess I was a little obscure in my original posts, but the configuration
files I specifically want to analyse will be those of network devices (eg
routers, firewall, load-balancers) that usually have many hundreds of lines
that go to make up the configuration of one entity. (Of course there is no
reason that I won't want to be able coalesce the configuration of multiple
heterogeneous devices to see/verify inconsistencies between them).

I suppose at this time my program will bemore of a "looker" than a "doer"
which is what I think Augeas or Puppet aim for.

Regards, Martin
Post by Matthew Hannigan
Augeas comies to mind.
http://augeas.net/
"Augeas - a configuration API
Augeas is a configuration editing tool. It parses
configuration files in their native formats and transforms
them into a tree. Configuration changes are made by
manipulating this tree and saving it back into native
config files."
You may also want to look at Puppet if you're not aware if it.
http://reductivelabs.com/products/puppet/
The front page is a bit sales talky so I won't quote it here.
It's also in ruby which is slightly unfortunate.
Post by Martin Visser
Hi,
I'm starting to build a Python program to help better describe systems I
work with. While I am making progress, I am interesting in whether anyone
is
Post by Martin Visser
aware of some existing projects or even libraries/snippets that might
help.
Post by Martin Visser
Essentially I need to be able to establish a structure of components
(multi-level members), record connections between components, and record
attributes. It needs to be flexible enough (allowing it to determine
valid
Post by Martin Visser
attributes and connections), depending on the position of the component
in
Post by Martin Visser
the structure. While not particularly hard, and I am making progress,
being
Post by Martin Visser
a relative Python noob, I was hoping for to look at ways other than what
I
Post by Martin Visser
am writing for it to be done.
I will also need to parse proprietary configuration files to build the
structure, but that is whole other can of worms I guess. (And eventually
I
Post by Martin Visser
want to output the "structure" in both a normalized text, and graphical
format(eg Graphviz). Eventually I would like to build to something that
can
Post by Martin Visser
be interacted with directly (probably via Web2.0ish interface).
I have built something similar to this in Perl, however it was very
procedural, and hence turned out very locked into to a fixed set of input
files, and hard to maintain. (I started out learning FORTRAN, so it is
hard
Post by Martin Visser
to change old habits).
(Googling doesn't seem to find what I want all that easy - I probably
need
Post by Martin Visser
the right Computer Science keywords to get what I want).
Regards, Martin
_______________________________________________
coders mailing list
http://lists.slug.org.au/listinfo/coders
Matthew Hannigan
2010-02-16 04:51:08 UTC
Permalink
Post by Martin Visser
Matthew,
Thanks. I was aware of Puppet, but not Augeas.
Unfortunately, neither seem to be written in Python, and I really want this
Augeas has bindings in python, ruby, ocaml, perl, haskell, java and php,
according to the download page. I've no idea on how mature / stable they are.
Post by Martin Visser
I suppose at this time my program will bemore of a "looker" than a "doer"
which is what I think Augeas or Puppet aim for.
Perhaps 'facter' (part of puppet) is what you're after.
Lindsay Holmwood
2010-02-16 04:21:15 UTC
Permalink
Post by Matthew Hannigan
Augeas comies to mind.
http://augeas.net/
   "Augeas - a configuration API
   Augeas is a configuration editing tool. It parses
   configuration files in their native formats and transforms
   them into a tree. Configuration changes are made by
   manipulating this tree and saving it back into native
   config files."
You may also want to look at Puppet if you're not aware if it.
   http://reductivelabs.com/products/puppet/
The front page is a bit sales talky so I won't quote it here.
It's also in ruby which is slightly unfortunate.
Speaking of other things that are fortunately written in Ruby, you
might be interested in TreeTop - it helps you quickly write parsers
for all sorts of interesting things:

http://treetop.rubyforge.org/

Cucumber's Gherkin specification parser is written in it, and there's
another neat project called Vimmish that translate Vim keystrokes into
human readable text:

http://github.com/dira/vimmish

Specifically for your use case Martin, you might want to check out
Pars, which has a bunch of TreeTop parsers for various Unix
configuration files:

http://github.com/atmos/pars

Hope that helps!
Lindsay
Post by Matthew Hannigan
Post by Martin Visser
Hi,
I'm starting to build a Python program to help better describe systems I
work with. While I am making progress, I am interesting in whether anyone is
aware of some existing projects or even libraries/snippets that might help.
Essentially I need to be able to establish a structure of components
(multi-level members), record connections between components, and record
attributes. It needs to be flexible enough  (allowing it to determine valid
attributes and connections), depending on the position of the component in
the structure. While not particularly hard, and I am making progress, being
a relative Python noob, I was hoping for to look at ways other than what I
am writing for it to be done.
I will also need to parse proprietary configuration files to build the
structure, but that is whole other can of worms I guess. (And eventually I
want to output the "structure" in both a normalized text, and graphical
format(eg Graphviz). Eventually I would like to build to something that can
be interacted with directly (probably via Web2.0ish interface).
I have built something similar to this in Perl, however it was very
procedural, and hence turned out very locked into to a fixed set of input
files, and hard to maintain. (I started out learning FORTRAN, so it is hard
to change old habits).
(Googling doesn't seem to find what I want all that easy - I probably need
the right Computer Science keywords to get what I want).
Regards, Martin
_______________________________________________
coders mailing list
http://lists.slug.org.au/listinfo/coders
_______________________________________________
coders mailing list
http://lists.slug.org.au/listinfo/coders
--
w: http://holmwood.id.au/~lindsay/
t: @auxesis
Martin Visser
2010-02-17 00:49:39 UTC
Permalink
Post by Lindsay Holmwood
Speaking of other things that are fortunately written in Ruby, you
might be interested in TreeTop - it helps you quickly write parsers
http://treetop.rubyforge.org/
Cucumber's Gherkin specification parser is written in it, and there's
another neat project called Vimmish that translate Vim keystrokes into
http://github.com/dira/vimmish
Specifically for your use case Martin, you might want to check out
Pars, which has a bunch of TreeTop parsers for various Unix
http://github.com/atmos/pars
Hope that helps!
Lindsay
Thanks Lindsay.

Parsing Expression Grammer used in TreeTop looks interesting. It seems that
there is PyPEG as well that would be analogous.

It seems that a lot of parsers either show examples of parsing programming
languages, or very basic .INI style configuration files.

(A long time ago I had to maintain some lex and yacc that was used parsing
some configuration files, so I have some experience as to how to how this
stuff works)

What I want to parse and produce an intelligent (meaning sensible and
useful) hierarchy for is config files like say from a Nortel Switched
Firewall snippet (network config, not firewall rules):-

/cfg/net/if 102/.
addr 10.10.97.36
mask 255.255.255.240
broad 0.0.0.0
vlan 4
ena n
/cfg/net/if 102/port/.
add 3
add 4
/cfg/net/if 102/vrrp/.
vrid 44
ip1 10.10.97.37
ip2 10.10.97.38
/cfg/net/if 124/.
addr 1.2.3.4
mask 255.255.255.192
broad 0.0.0.0
vlan 24
ena n

or say a Nortel Load Balancer:-

/c/slb/real 31
ena
ipver v4
rip 10.10.22.31
name "zzzzvweb1"
/c/slb/real 32
ena
ipver v4
rip 10.10.22.32
name "zzzzzweb2"
/c/slb/group 20
ipver v4
metric hash
add 31
add 32
name "zzzzz"
/c/slb/virt 20
ena
ipver v4
vip 1.2.234.132
dname "zzzz.yyyy.com.au"
/c/slb/virt 20/service http
group 20


or totally different looking machine like a Juniper SRX firewall:-

security {
nat {
source {
pool scic-internet {
address {
1.2.3.4/32;
}
}
rule-set inside-NAT {
from zone inside;
to zone internet;
rule inside-NAT-rule {
match {
source-address [ 10.10.0.0/16 33.33.0.0/16 ];
}
then {
source-nat {
pool {
scic-internet;
}
}
}
}
}
}
destination {
pool mail-pool {
address 10.10..2.4/32;
}
pool vpn-pool {
address 10.10.2.19/32;
}
rule-set internet-NAT {
from zone internet;
rule mail {
match {
destination-address 1.2.3.5/32;
destination-port 25;
}
then {
destination-nat pool mail-pool;
}
}
rule vpn {
match {
destination-address 1.2.3.4/32;
destination-port 443;
}
then {
destination-nat pool vpn-pool;
}
}
}
}
}

Regards, Martin

***@gmail.com
Loading...