Monday, May 06, 2013

puppet-cleaner 0.3.0 released

This release adds a new option for ensuring that resource titles are quoted.

By the way, two transformations are done by default and are not optional: the use of ${} for variable interpolation in strings and the replacement of double with single quotes when possible.

The optional transformations are:
  • alignment of fat arrows (=>)
  • removal of unneeded quotes around boolean literals
  • reordering of ensure attribute to the first position
  • declaration of symbolic links by using ensure => link and target attributes
  • conversion of /* */ style comments to #
  • representation of file modes as a 4 digits string
  • use of quotes around resource titles
  • indentation by n spaces
  • removal of unneeded quotes around variables
  • removal of trailing white space
Installation: sudo gem install puppet-cleaner
Website: https://github.com/santana/puppet-cleaner

Monday, April 29, 2013

puppet 0.2.0 released

It happened that puppet 3 didn't introduced anything that could disturb the monkey patching of the lexer :) just some weird change in the require order of the parser.

puppet 0.2.0 is now available, tested successfully with puppet 3.

Friday, April 26, 2013

puppet 0.1.1 released

puppet-cleaner is a set of tools that helps you keep your manifests compliant with the style guide.

Website:
  https://github.com/santana/puppet-cleaner/

How to install:
  sudo gem install puppet-cleaner

This is a maintenance release that fixes a run-time error. Thanks to github user blindsey to bring it to my attention.

Tuesday, April 16, 2013

puppet-cleaner 0.1.0 released


I'm currently a puppet master for thousands of servers, for which thousands of lines of puppet DSL code have been written, and keep being written every day.

The Challenge


One of the immediate goals was to make those thousands of lines code to comply with the current Puppetlabs' style guide.

For this purpose I wrote a library for applying arbitrary transformations to a puppet DSL code source.

It was very interesting because I learned a lot about how puppet is built while looking for an economic way to do these transformations.

Object orientation and code reuse

Honestly, I considered a combination of sed, awk and perl the first time I tackled this challenge. puppet-lint for example uses regular expressions to tokenize the code and then warns you about violations to the style guide.

I didn't want to reinvent the wheel so I thought of tapping on puppet's lexical analyser to do the job for me.

It wasn't as easy to extend it as I expected it to be for an object oriented piece of software. My approach required some changes to the analyser in order to reuse it that were "impossible" due to some internal design decisions or shortcuts.

I ended up exploiting Ruby's dynamic typing by monkey patching Puppet::Parser::Lexer and finally get the BLANK and RETURN tokens I needed.

The design

The next challenge was to design the library, but hopefully it just presented to me: a production line (stream of tokens) where specialized workers (transformation algorithms) were waiting for the token they know how to work on and transform the set of tokens that matches the pattern they were taught to recognize.

Testing


The final challenge was to test the library output for correctness. The usefulness of puppet-cleaner depended on this. What use is a code cleaner if it modifies the behaviour of the original code?

Then I dug up a serialization method from puppet's source code that would help me prove that two different text files were actually the same puppet DSL code, and put it in puppet-diff and puppet-inspect.

The result


It was fun indeed. Lots of new stuff.

The goal was achieved: thousands of puppet DSL code were made to comply in a matter of minutes (I still had to be very cautious.)

Today I'm uploading the product of this effort to github for anyone to use. It's faster that puppet-lint although  not as complete, but it actually fixes your code, instead of just warning you about it.

Finally, here's an input and output example.

input:


/*
  multiline comment
  trailing white space here ->     
*/

class someclass($version = "5", $platform = "rhel6")
{
if ! ($version in ["4", "5"] or !($platform in ["rhel5", "rhel6"]))  {
     fail("Version $version on $platform is not supported")
}
       else {
    notice("Defining class for version \"$version\" on '$platform'")
  package {
    [
          "package-1",
"package-2",
  ]:
  }

     sysctl2::conf {"${something}":
  settings => [
     "set key1 'value1'",
     "set key2 'value2'",
 ]
  }
   file {
        "/file/a": # test
    source  =>    'puppet:///someclass/file/a',
       owner                  =>   'root',
     ensure               =>   ensure_a,
 group=>         "${group}";
        "/file/b":
    source  =>    'puppet:///someclass/file/b',
 mode=>755,
     ensure               =>   ensure_b,
       owner                  =>   'root',
 group=>         "${group}";
  }

   file {
        "/file/c":
    source  =>    'puppet:///someclass/file/c',
       owner                  =>   'root',
     ensure               =>   ensure_c,
 group=>         "${group}";
        "/file/d":
    source  =>    'puppet:///someclass/file/d',
 mode=>"755",
       owner                  =>   'root',
 group=>         "${group}"
  }
   file {
        "/file/e":
    source  =>    'puppet:///someclass/file/c',
       owner                  =>   'root',
      alias  => ["$hostname"],
 group=>         "${group}",
     ensure               =>   'absent';
        "/file/f":
     ensure               =>   'ensure_f';
  }

  user {
    "jdoe":
   managed_home => "true",
  }
}
}


output: 


#
#   multiline comment
#   trailing white space here ->


class someclass($version = '5', $platform = 'rhel6')
{
  if ! ($version in ['4', '5'] or !($platform in ['rhel5', 'rhel6']))  {
    fail("Version ${version} on ${platform} is not supported")
  }
  else {
    notice("Defining class for version \"${version}\" on '${platform}'")
    package {
      [
        'package-1',
        'package-2',
      ]:
    }

    sysctl2::conf {$something:
      settings => [
        'set key1 \'value1\'',
        'set key2 \'value2\'',
      ]
    }
    file {
      '/file/a':
        ensure => link, # test
        source => 'puppet:///someclass/file/a',
        owner  => 'root',
        target => ensure_a,
        group  => $group;
      '/file/b':
        ensure => link,
        source => 'puppet:///someclass/file/b',
        mode   => '0755',
        target => ensure_b,
        owner  => 'root',
        group  => $group;
    }

    file {
      '/file/c':
        ensure => link,
        source => 'puppet:///someclass/file/c',
        owner  => 'root',
        target => ensure_c,
        group  => $group;
      '/file/d':
        source => 'puppet:///someclass/file/d',
        mode   => '0755',
        owner  => 'root',
        group  => $group
    }
    file {
      '/file/e':
        ensure => 'absent',
        source => 'puppet:///someclass/file/c',
        owner  => 'root',
        alias  => [$hostname],
        group  => $group;
      '/file/f':
        ensure => link,
        target => 'ensure_f';
    }

    user {
      'jdoe':
        managed_home => true,
    }
  }
}

Sunday, April 07, 2013

Stating the obvious

My comments on this post: http://venturebeat.com/2013/04/06/developer-first-security/

Three are the sources for vulnerabilities:

• badly designed protocols
• badly written code
• badly administered systems

Too much focus has been given to the third point: security polocies (PCI DSS), security appliances (firewalls, IDS...), security software layers (SELinux).

However, must of the current security issues have their origin in the other two points.

Security is nothing you can buy or rent. It's as simple as the process of doing our jobs right.

That means quality, not as an added thing, but as a way of living.

This has been known to the OpenBSD Team since 1996, which has been pursuing correctness since then.

This must be obvious in 2013, don't you think?

Tuesday, April 17, 2012

Ruby/Informix 0.8.0 released

The point of this release is to make it compatible with Ruby 1.9.3. It's no longer compatible with Ruby 1.8 which is about to lose support.

Other changes include:

  • the tests now load the extension just built, instead of the one installed
  • adds a gemspec file, that replaces the old Rakefile, to build the gem
  • Rakefile is now used for running the tests, passing the name of the database to test against as a task argument
  • when installing the gem a warning is printed if INFORMIXDIR is not set and exits, instead of trying to guess and failing miserably
You'll need to have the test-unit gem installed in order to run the tests.

Wednesday, May 21, 2008

binpatch 1.1.0 released

binpatch is a ports-like framework for building binary patches for OpenBSD.

Thanks Mike Erdely (merdelly@) for contributing to this release.

Visit http://openbsdbinpatch.sourceforge.net to learn more about binpatch.