RESTapi – A teach yourself lab!

Here is a lab I put together for a recent event, it was well received and I think it is very valuable to the ManageIQ and CloudForms user bases.

In the lab you will GET and POST RESTapi actions. Its a basic look at some common use cases such as,

  • Query a VM’s hardware inventory.
  • Query the VMS for any who are tagged with “Function/Database”
  • Show the tags on a specific VM
  • Create new Category and Tag in the appliance
  • Assign a Category/Tag to a VM
  • Provision a VM
  • Query the status of the Provision request.

The lab expects you to have an appliance provisioned and connected to a provider (the lab uses AMAZON EC2) Therefore when querying attributes of a vm/instance the return list is specific to the provider. CPU’s are not available on AWS EC2, whereas availability zones are.

Note : Cut and Pasting the code from the slides can introduce ASCII translation errors depending on platform. Take care to ensure quotes are “.

I will do another post in the future to cover off “PUT, PATCH and DELETE” actions.

The slides are here in PDF format – https://www.dropbox.com/s/14vwhd1fcb97prg/RESTapi.pdf?dl=0

 

Slide02 Slide03 Slide04 Slide05 Slide06 Slide07 Slide08 Slide09 Slide10 Slide11 Slide12 Slide13 Slide14 Slide15 Slide16 Slide17 Slide18 Slide19 Slide20 Slide21 Slide22 Slide23 Slide24

 

 

 

CLOUDFORMS 5.3.0 GEM List

Here is the list of GEMs and their info that are included in the CLOUDFORMS 5.3.0 appliance. I thought it would be useful to post, I need this recently for some work I am doing. It was really easy as most things in CLOUDFORMS usually are I simply wrote some ruby to utilise an existing gem called GEMS, that pulls this data from rubygems.org. I wrapped the code in some file open and close, e.g. I dumped the list of gems in the appliance using “gem list > gems.txt” then had my little nugget of code read that in. Here it is.

require 'rubygems'
require 'gems'

outFile = File.open("/Users/johnhardy/Desktop/Gems-CF-5.3.0.html","w")
outFile.write "<html>"
outFile.write "<body>"
outFile.write "<table>"

gemList = File.open("/Users/johnhardy/Desktop/Gems.txt","r") do |infile|
  while (line = infile.gets)
    gemItem = line.split
    gemInfo =  Gems.info "#{gemItem[0]}"
    puts gemInfo['name'] + "," + gemInfo['info']
    outFile.write "<tr>"
    outFile.write "<td>"
    outFile.write gemInfo['name']
    outFile.write "</td>"
    outFile.write "<td>"
    outFile.write gemInfo['info']
    outFile.write "</td>"
    outFile.write "</tr>"
    end
end

outFile.write "</table>"
outFile.write "</body>"
outFile.write "</html>"
outFile.close

 

actionmailer Email on Rails. Compose, deliver, receive, and test emails using the familiar controller/view pattern. First-class support for multipart email and attachments.
actionpack Web apps on Rails. Simple, battle-tested conventions for building and testing MVC web applications. Works with any Rack-compatible server.
actionwebservice Adds WSDL/SOAP and XML-RPC web service support to Action Pack
active_hash Includes the ability to specify data using hashes, yml files or JSON files
activemodel A toolkit for building modeling frameworks like Active Record. Rich support for attributes, callbacks, validations, serialization, internationalization, and testing.
activerecord Databases on Rails. Build a persistent domain model by mapping database tables to Ruby classes. Strong conventions for associations, validations, aggregations, migrations, and testing come baked-in.
activeresource REST on Rails. Wrap your RESTful web app with Ruby classes and work with them like Active Record models.
activesupport A toolkit of support libraries and Ruby core extensions extracted from the Rails framework. Rich support for multibyte strings, internationalization, time zones, and testing.
acts_as_list This “acts_as” extension provides the capabilities for sorting and reordering a number of objects in a list. The class that has this specified needs to have a “position” column defined as an integer on the mapped database table.
acts_as_tree A gem that adds simple support for organizing ActiveRecord models into parent–children relationships.
addressable Addressable is a replacement for the URI implementation that is part of
Ruby’s standard library. It more closely conforms to the relevant RFCs and
adds support for IRIs and URI templates.
akami Building Web Service Security
american_date American style month/day/year date parsing for ruby 1.9
amq-protocol amq-protocol is an AMQP 0.9.1 serialization library for Ruby. It is not an
AMQP client: amq-protocol only handles serialization and deserialization.
If you want to write your own AMQP client, this gem can help you with that.
ancestry Ancestry allows the records of a ActiveRecord model to be organized in a tree
structure, using a single, intuitively formatted database column. It exposes
all the standard tree structure relations (ancestors, parent, root, children,
siblings, descendants) and all of them can be fetched in a single sql query.
Additional features are named_scopes, integrity checking, integrity restoration,
arrangement of (sub)tree into hashes and different strategies for dealing with
orphaned records.
arel Arel Really Exasperates Logicians

Arel is a SQL AST manager for Ruby. It

1. Simplifies the generation of complex SQL queries
2. Adapts to various RDBMSes

It is intended to be a framework framework; that is, you can build your own ORM
with it, focusing on innovative object and collection modeling as opposed to
database compatibility and query generation.

arrayfields string/symbol keyword access to arrays
awesome_print Great Ruby dubugging companion: pretty print Ruby objects to visualize their structure. Supports custom object formatting via plugins
awesome_spawn AwesomeSpawn is a module that provides some useful features over Ruby’s Kernel.spawn.
aws-sdk Version 1 of the AWS SDK for Ruby. Available as both `aws-sdk` and `aws-sdk-v1`.
Use `aws-sdk-v1` if you want to load v1 and v2 of the Ruby SDK in the same
application.
bcrypt-ruby bcrypt() is a sophisticated and secure hash algorithm designed by The OpenBSD project
for hashing passwords. The bcrypt Ruby gem provides a simple wrapper for safely handling
passwords.
bigdecimal This library provides arbitrary-precision decimal floating-point number class.
binary_struct BinaryStruct is a class for dealing with binary structured data. It simplifies
expressing what the binary structure looks like, with the ability to name the
parts. Given this definition, it is easy to encode/decode the binary structure
from/to a Hash.
builder Builder provides a number of builder objects that make creating structured data
simple to do. Currently the following builder objects are supported:

* XML Markup
* XML Events

bundler Bundler manages an application’s dependencies through its entire life, across many machines, systematically and repeatably
bundler_ext Simple library leveraging the Bundler Gemfile DSL to load gems already on the system and managed by the systems package manager (like yum/apt)
bunny Easy to use, feature complete Ruby client for RabbitMQ 3.3 and later versions.
capybara Capybara is an integration testing tool for rack based web applications. It simulates how a user would interact with a website
childprocess This gem aims at being a simple and reliable solution for controlling external programs running in the background on any Ruby / OS combination.
color Color is a Ruby library to provide basic RGB, CMYK, HSL, and other colourspace
manipulation support to applications that require it. It also provides 152
named RGB colours (184 with spelling variations) that are commonly supported in
HTML, SVG, and X11 applications. A technique for generating monochromatic
contrasting palettes is also included.

The Color library performs purely mathematical manipulation of the colours
based on colour theory without reference to colour profiles (such as sRGB or
Adobe RGB). For most purposes, when working with RGB and HSL colour spaces,
this won’t matter. Absolute colour spaces (like CIE L*a*b* and XYZ) and cannot
be reliably converted to relative colour spaces (like RGB) without colour
profiles.

Color version 1.7.1 adds Color::RGB::RebeccaPurple for the colour #663399 in
honour of Rebecca Meyer, the daughter of Eric Meyer, who passed away on the 7th
of June, 2014. Her favourite colour was purple.
{rebeccapurple}[http://meyerweb.com/eric/thoughts/2014/06/19/rebeccapurple/]

Barring bugs introduced in this release, this is the last version of color that
supports Ruby 1.8, so make sure that your gem specification is set properly (to
<tt>~> 1.6</tt>) if that matters for your application.

colored >> puts “this is red”.red

>> puts “this is red with a blue background (read: ugly)”.red_on_blue

>> puts “this is red with an underline”.red.underline

>> puts “this is really bold and really blue”.bold.blue

>> logger.debug “hey this is broken!”.red_on_yellow # in rails

>> puts Color.red “This is red” # but this part is mostly untested

daemons Daemons provides an easy way to wrap existing ruby scripts (for example a self-written server) to be run as a daemon and to be controlled by simple start/stop/restart commands. You can also call blocks as daemons and control them from the parent or just daemonize the current process. Besides this basic functionality, daemons offers many advanced features like exception backtracing and logging (in case your ruby script crashes) and monitoring and automatic restarting of your processes if they crash.
dalli High performance memcached client for Ruby
default_value_for The default_value_for plugin allows one to define default values for ActiveRecord models in a declarative manner
diff-lcs Diff::LCS computes the difference between two Enumerable sequences using the
McIlroy-Hunt longest common subsequence (LCS) algorithm. It includes utilities
to create a simple HTML diff output format and a standard diff-like tool.

This is release 1.2.4, fixing a bug introduced after diff-lcs 1.1.3 that did
not properly prune common sequences at the beginning of a comparison set.
Thanks to Paul Kunysch for fixing this issue.

Coincident with the release of diff-lcs 1.2.3, we reported an issue with
Rubinius in 1.9 mode
({rubinius/rubinius#2268}[https://github.com/rubinius/rubinius/issues/2268]).
We are happy to report that this issue has been resolved.

elif A port of File::ReadBackwards, the Perl module by Uri Guttman, for reading a file in reverse, line by line. This can often be helpful for things like log files, where the interesting information is usually at the end.
erubis Erubis is an implementation of eRuby and has the following features:

* Very fast, almost three times faster than ERB and about 10% faster than eruby.
* Multi-language support (Ruby/PHP/C/Java/Scheme/Perl/Javascript)
* Auto escaping support
* Auto trimming spaces around ”
* Embedded pattern changeable (default ”)
* Enable to handle Processing Instructions (PI) as embedded pattern (ex. ”)
* Context object available and easy to combine eRuby template with YAML datafile
* Print statement available
* Easy to extend and customize in subclass
* Ruby on Rails support

eventmachine EventMachine implements a fast, single-threaded engine for arbitrary network
communications. It’s extremely easy to use in Ruby. EventMachine wraps all
interactions with IP sockets, allowing programs to concentrate on the
implementation of network protocols. It can be used to create both network
servers and clients. To create a server or client, a Ruby program only needs
to specify the IP address and port, and provide a Module that implements the
communications protocol. Implementations of several standard network protocols
are provided with the package, primarily to serve as examples. The real goal
of EventMachine is to enable programs to easily interface with other programs
using TCP/IP, especially if custom protocols are required.
excon EXtended http(s) CONnections
execjs ExecJS lets you run JavaScript code from Ruby.
ezcrypto Makes it easier and safer to write crypto code.
facade The facade library allows you to mixin singleton methods from classes
or modules as instance methods of the extending class.
factory_girl factory_girl provides a framework and DSL for defining and
using factories – less error-prone, more explicit, and
all-around easier to work with than fixtures.
fattr a “fatter attr” for ruby
ffi Ruby FFI library
fog The Ruby cloud services library. Supports all major cloud providers including AWS, Rackspace, Linode, Blue Box, StormOnDemand, and many others. Full support for most AWS services including EC2, S3, CloudWatch, SimpleDB, ELB, and RDS.
formatador STDOUT text formatting
gssapi A FFI wrapper around the system GSSAPI library. Please make sure and read the
Yard docs or standard GSSAPI documentation if you have any questions.

There is also a class called GSSAPI::Simple that wraps many of the common features
used for GSSAPI.

gyoku Gyoku translates Ruby Hashes to XML
haml Haml (HTML Abstraction Markup Language) is a layer on top of HTML or XML that’s
designed to express the structure of documents in a non-repetitive, elegant, and
easy way by using indentation rather than closing tags and allowing Ruby to be
embedded with ease. It was originally envisioned as a plugin for Ruby on Rails,
but it can function as a stand-alone templating engine.
haml-rails Haml-rails provides Haml generators for Rails 3. It also enables Haml as the templating engine for you, so you don’t have to screw around in your own application.rb when your Gemfile already clearly indicated what templating engine you have installed. Hurrah.
handsoap Handsoap is a library for creating SOAP clients in Ruby
highline A high-level IO library that provides validation, type conversion, and more for
command-line interfaces. HighLine also includes a complete menu system that can
crank out anything from simple list selection to complete shells with just
minutes of work.
hike A Ruby library for finding files in a set of paths.
hirb Hirb provides a mini view framework for console applications and uses it to improve ripl(irb)’s default inspect output. Given an object or array of objects, hirb renders a view based on the object’s class and/or ancestry. Hirb offers reusable views in the form of helper classes. The two main helpers, Hirb::Helpers::Table and Hirb::Helpers::Tree, provide several options for generating ascii tables and trees. Using Hirb::Helpers::AutoTable, hirb has useful default views for at least ten popular database gems i.e. Rails’ ActiveRecord::Base. Other than views, hirb offers a smart pager and a console menu. The smart pager only pages when the output exceeds the current screen size. The menu is used in conjunction with tables to offer two dimensional menus.
hoe Hoe is a rake/rubygems helper for project Rakefiles. It helps you
manage, maintain, and release your project and includes a dynamic
plug-in system allowing for easy extensibility. Hoe ships with
plug-ins for all your usual project tasks including rdoc generation,
testing, packaging, deployment, and announcement..

See class rdoc for help. Hint: `ri Hoe` or any of the plugins listed
below.

For extra goodness, see: http://docs.seattlerb.org/hoe/Hoe.pdf

httparty Makes http fun! Also, makes consuming restful web services dead easy.
httpclient gives something like the functionality of libwww-perl (LWP) in Ruby
httpi Common interface for Ruby’s HTTP libraries
i18n New wave Internationalization support for Ruby.
inifile Although made popular by Windows, INI files can be used on any system thanks
to their flexibility. They allow a program to store configuration data, which
can then be easily parsed and changed. Two notable systems that use the INI
format are Samba and Trac.

More information about INI files can be found on the [Wikipedia Page](http://en.wikipedia.org/wiki/INI_file).

### Properties

The basic element contained in an INI file is the property. Every property has
a name and a value, delimited by an equals sign *=*. The name appears to the
left of the equals sign and the value to the right.

name=value

### Sections

Section declarations start with *[* and end with *]* as in `[section1]` and
`[section2]` shown in the example below. The section declaration marks the
beginning of a section. All properties after the section declaration will be
associated with that section.

### Comments

All lines beginning with a semicolon *;* or a number sign *#* are considered
to be comments. Comment lines are ignored when parsing INI files.

### Example File Format

A typical INI file might look like this:

[section1]
; some comment on section1
var1 = foo
var2 = doodle
var3 = multiline values \
are also possible

[section2]
# another comment
var1 = baz
var2 = shoodle

io-console add console capabilities to IO instances.
io-extra Adds the IO.closefrom, IO.fdwalk, IO.pread, IO.pread_ptr, IO.pwrite, and
IO.writev singleton methods as well as the IO#directio, IO#directio? and
IO#ttyname instance methods (for those platforms that support them).
jbuilder Create JSON structures via a Builder-style DSL
journey Journey is a router. It routes requests.
json This is a JSON implementation as a Ruby extension in C.
json_pure This is a JSON implementation in pure Ruby.
linux_admin LinuxAdmin is a module to simplify management of linux systems.
It should be a single place to manage various system level configurations,
registration, updates, etc.
little-plugger LittlePlugger is a module that provides Gem based plugin management.
By extending your own class or module with LittlePlugger you can easily
manage the loading and initializing of plugins provided by other gems.
log4r See also: http://logging.apache.org/log4j
logging Logging is a flexible logging library for use in Ruby programs based on the
design of Java’s log4j library. It features a hierarchical logging system,
custom level names, multiple output destinations per log event, custom
formatting, and more.
mail A really Ruby Mail handler.
main a class factory and dsl for generating command line programs real quick
map the awesome ruby container you’ve always wanted: a string/symbol indifferent ordered hash that works in all rubies
mime-types The mime-types library provides a library and registry for information about
MIME content type definitions. It can be used to determine defined filename
extensions for MIME types, or to use filename extensions to look up the likely
MIME type definitions.

MIME content types are used in MIME-compliant communications, as in e-mail or
HTTP traffic, to indicate the type of content which is transmitted. The
mime-types library provides the ability for detailed information about MIME
entities (provided as an enumerable collection of MIME::Type objects) to be
determined and used programmatically. There are many types defined by RFCs and
vendors, so the list is long but by definition incomplete; don’t hesitate to to
add additional type definitions (see Contributing.rdoc). The primary sources
for MIME type definitions found in mime-types is the IANA collection of
registrations (see below for the link), RFCs, and W3C recommendations.

This is release 2.4.3, restoring full compatibility with Ruby 1.9.2 (which will
be dropped in mime-types 3.0). It also includes the performance improvements
from mime-types 2.4.2 (since yanked because of the broken Ruby 1.9.2 support)
and the 2.4.1 fix of a bug in observed use of the mime-types library where
extensions were not previously sorted, such that

MIME::Types.of(‘image.jpg’).first.extensions.first

returned a value of +jpeg+ in mime-types 1, but +jpe+ in mime-types 2. This was
introduced because extensions were sorted during assignment
(MIME::Type#extensions=). This behaviour has been reverted to protect clients
that work as noted above. The preferred way to express this is the new method:

MIME::Type.of(‘image.jpg’).first.preferred_extension

Łukasz Śliwa created the
{friendly_mime}[https://github.com/lukaszsliwa/friendly_mime] gem, which offers
friendly descriptive names for MIME types. This functionality and
English-language data has been added to mime-types as MIME::Type#friendly. To
make it easy for internationalization, MIME::Type#i18n_key has been added,
which will return a key suitable for use with the
{I18n}[https://github.com/svenfuchs/i18n] library.

As a reminder, mime-types 2.x is no longer compatible with Ruby 1.8 and
mime-types 1.x is only being maintained for security issues. No new MIME types
or features will be added.

mime-types (previously called MIME::Types for Ruby) was originally based on
MIME::Types for Perl by Mark Overmeer, copyright 2001 – 2009. It is built to
conform to the MIME types of RFCs 2045 and 2231. It tracks the {IANA Media
Types registry}[https://www.iana.org/assignments/media-types/media-types.xhtml]
with some types added by the users of mime-types.

minitest minitest provides a complete suite of testing facilities supporting
TDD, BDD, mocking, and benchmarking.

“I had a class with Jim Weirich on testing last week and we were
allowed to choose our testing frameworks. Kirk Haines and I were
paired up and we cracked open the code for a few test
frameworks…

I MUST say that minitest is *very* readable / understandable
compared to the ‘other two’ options we looked at. Nicely done and
thank you for helping us keep our mental sanity.”

— Wayne E. Seguin

minitest/unit is a small and incredibly fast unit testing framework.
It provides a rich set of assertions to make your tests clean and
readable.

minitest/spec is a functionally complete spec engine. It hooks onto
minitest/unit and seamlessly bridges test assertions over to spec
expectations.

minitest/benchmark is an awesome way to assert the performance of your
algorithms in a repeatable manner. Now you can assert that your newb
co-worker doesn’t replace your linear algorithm with an exponential
one!

minitest/mock by Steven Baker, is a beautifully tiny mock (and stub)
object framework.

minitest/pride shows pride in testing and adds coloring to your test
output. I guess it is an example of how to write IO pipes too. :P

minitest/unit is meant to have a clean implementation for language
implementors that need a minimal set of methods to bootstrap a working
test suite. For example, there is no magic involved for test-case
discovery.

“Again, I can’t praise enough the idea of a testing/specing
framework that I can actually read in full in one sitting!”

— Piotr Szotkowski

Comparing to rspec:

rspec is a testing DSL. minitest is ruby.

— Adam Hawkins, “Bow Before MiniTest”

minitest doesn’t reinvent anything that ruby already provides, like:
classes, modules, inheritance, methods. This means you only have to
learn ruby to use minitest and all of your regular OO practices like
extract-method refactorings still apply.

more_core_extensions MoreCoreExtensions are a set of core extensions beyond those provided by ActiveSupport.
multi_json A common interface to multiple JSON libraries, including Oj, Yajl, the JSON gem (with C-extensions), the pure-Ruby JSON gem, NSJSONSerialization, gson.rb, JrJackson, and OkJson.
multi_xml Provides swappable XML backends utilizing LibXML, Nokogiri, Ox, or REXML.
net-http-persistent Manages persistent connections using Net::HTTP plus a speed fix for Ruby 1.8.
It’s thread-safe too!

Using persistent HTTP connections can dramatically increase the speed of HTTP.
Creating a new HTTP connection for every request involves an extra TCP
round-trip and causes TCP congestion avoidance negotiation to start over.

Net::HTTP supports persistent connections with some API methods but does not
handle reconnection gracefully. Net::HTTP::Persistent supports reconnection
and retry according to RFC 2616.

net-ldap Net::LDAP for Ruby (also called net-ldap) implements client access for the
Lightweight Directory Access Protocol (LDAP), an IETF standard protocol for
accessing distributed directory services. Net::LDAP is written completely in
Ruby with no external dependencies. It supports most LDAP client features and a
subset of server features as well.

Net::LDAP has been tested against modern popular LDAP servers including
OpenLDAP and Active Directory. The current release is mostly compliant with
earlier versions of the IETF LDAP RFCs (2251-2256, 2829-2830, 3377, and 3771).
Our roadmap for Net::LDAP 1.0 is to gain full <em>client</em> compliance with
the most recent LDAP RFCs (4510-4519, plutions of 4520-4532).

net-ping The net-ping library provides a ping interface for Ruby. It includes
separate TCP, HTTP, LDAP, ICMP, UDP, WMI (for Windows) and external ping
classes.
net-scp A pure Ruby implementation of the SCP client protocol
net-sftp A pure Ruby implementation of the SFTP client protocol
net-ssh Net::SSH: a pure-Ruby implementation of the SSH2 client protocol. It allows you to write programs that invoke and interact with processes on remote servers, via SSH2.
netrc This library can read and update netrc files, preserving formatting including comments and whitespace.
nokogiri Nokogiri (鋸) is an HTML, XML, SAX, and Reader parser. Among Nokogiri’s
many features is the ability to search documents via XPath or CSS3 selectors.

XML is like violence – if it doesn’t solve your problems, you are not using
enough of it.

nori XML to Hash translator
open4 open child process with handles on pid, stdin, stdout, and stderr: manage child processes and their io handles easily.
outfielding-jqplot-rails jqplot packaged for the Rails 3.1+ asset pipeline
ovirt_metrics OvirtMetrics is an ActiveRecord-based gem for reading the oVirt History database.
parallel Run any kind of code in parallel processes
pdf-writer This library provides the ability to create PDF documents using only native Ruby libraries. There are several demo programs available in the demo/ directory. The canonical documentation for PDF::Writer is "manual.pdf", which can be generated using bin/techbook (just "techbook" for RubyGem users) and the manual file "manual.pwd".
pg Pg is the Ruby interface to the {PostgreSQL RDBMS}[http://www.postgresql.org/].

It works with {PostgreSQL 8.4 and later}[http://www.postgresql.org/support/versioning/].

A small example usage:

#!/usr/bin/env ruby

require ‘pg’

# Output a table of current connections to the DB
conn = PG.connect( dbname: ‘sales’ )
conn.exec( “SELECT * FROM pg_stat_activity” ) do |result|
puts ” PID | User | Query”
result.each do |row|
puts ” %7d | %-16s | %s ” %
row.values_at(‘procpid’, ‘usename’, ‘current_query’)
end
end

Platform Hopefully robust platform sensing
polyglot The Polyglot library allows a Ruby module to register a loader
for the file type associated with a filename extension, and it
augments ‘require’ to find and load matching files.
princely A wrapper for the PrinceXML PDF generation library.
prototype-rails Prototype, Scriptaculous, and RJS for Ruby on Rails
qpid_messaging Qpid is an enterprise messaging framework.
rack Rack provides a minimal, modular and adaptable interface for developing
web applications in Ruby. By wrapping HTTP requests and responses in
the simplest way possible, it unifies and distills the API for web
servers, web frameworks, and software in between (the so-called
middleware) into a single method call.

Also see http://rack.github.com/.

rack-cache Rack::Cache is suitable as a quick drop-in component to enable HTTP caching for Rack-based applications that produce freshness (Expires, Cache-Control) and/or validation (Last-Modified, ETag) information.
rack-ssl Rack middleware to force SSL/TLS.
rack-test Rack::Test is a small, simple testing API for Rack apps. It can be used on its
own or as a reusable starting point for Web frameworks and testing libraries
to build on. Most of its initial functionality is an extraction of Merb 1.0’s
request helpers feature.
rails Ruby on Rails is a full-stack web framework optimized for programmer happiness and sustainable productivity. It encourages beautiful code by favoring convention over configuration.
railties Rails internals: application bootup, plugins, generators, and rake tasks.
rake Rake is a Make-like program implemented in Ruby. Tasks and dependencies are
specified in standard Ruby syntax.

Rake has the following features:

* Rakefiles (rake’s version of Makefiles) are completely defined in
standard Ruby syntax. No XML files to edit. No quirky Makefile
syntax to worry about (is that a tab or a space?)

* Users can specify tasks with prerequisites.

* Rake supports rule patterns to synthesize implicit tasks.

* Flexible FileLists that act like arrays but know about manipulating
file names and paths.

* A library of prepackaged tasks to make building rakefiles easier. For example,
tasks for building tarballs and publishing to FTP or SSH sites. (Formerly
tasks for building RDoc and Gems were included in rake but they’re now
available in RDoc and RubyGems respectively.)

* Supports parallel execution of tasks.

rdoc RDoc produces HTML and command-line documentation for Ruby projects. RDoc
includes the +rdoc+ and +ri+ tools for generating and displaying documentation
from the command-line.
rest-client A simple HTTP and REST client for Ruby, inspired by the Sinatra microframework style of specifying actions: get, put, post, delete.
rspec BDD for Ruby
rspec-core BDD for Ruby. RSpec runner and example groups.
rspec-expectations rspec-expectations provides a simple, readable API to express expected outcomes of a code example.
rspec-fire More resilient test doubles for RSpec.
rspec-mocks RSpec’s ‘test double’ framework, with support for stubbing and mocking
rspec-rails RSpec for Rails
ruby-graphviz Ruby/Graphviz provides an interface to layout and generate images of directed graphs in a variety of formats (PostScript, PNG, etc.) using GraphViz.
ruby-hmac This module provides common interface to HMAC functionality. HMAC is a kind of “Message Authentication Code” (MAC) algorithm whose standard is documented in RFC2104. Namely, a MAC provides a way to check the integrity of information transmitted over or stored in an unreliable medium, based on a secret key.

Originally written by Daiki Ueno. Converted to a RubyGem by Geoffrey Grosenbach

ruby-plsql ruby-plsql gem provides simple Ruby API for calling Oracle PL/SQL procedures.
It could be used both for accessing Oracle PL/SQL API procedures in legacy applications
as well as it could be used to create PL/SQL unit tests using Ruby testing libraries.
ruby-progressbar Ruby/ProgressBar is an extremely flexible text progress bar library for Ruby.
The output can be customized with a flexible formatting system including:
percentage, bars of various formats, elapsed time and estimated time remaining.
rubyforge A script which automates a limited set of rubyforge operations.

* Run ‘rubyforge help’ for complete usage.
* Setup: For first time users AND upgrades to 0.4.0:
* rubyforge setup (deletes your username and password, so run sparingly!)
* edit ~/.rubyforge/user-config.yml
* rubyforge config
* For all rubyforge upgrades, run ‘rubyforge config’ to ensure you have latest.

rubyntlm Ruby/NTLM provides message creator and parser for the NTLM authentication.
rubyrep This rubygem does not have a description or summary.
rubywbem This is a short description
rubyzip rubyzip is a ruby module for reading and writing zip files
rufus-lru LruHash class, a Hash with a max size, controlled by a LRU mechanism
rufus-scheduler job scheduler for Ruby (at, cron, in and every jobs).
ruport Ruby Reports is a software library that aims to make the task of reporting
less tedious and painful. It provides tools for data acquisition,
database interaction, formatting, and parsing/munging.
sass Sass makes CSS fun again. Sass is an extension of CSS3, adding
nested rules, variables, mixins, selector inheritance, and more.
It’s translated to well-formatted, standard CSS using the
command line tool or a web-framework plugin.
savon Heavy metal SOAP client
secure_headers Add easily configured browser headers to responses.
selenium-webdriver WebDriver is a tool for writing automated tests of websites. It aims to mimic the behaviour of a real user, and as such interacts with the HTML of the application.
shoulda-matchers Making tests easy on the fingers and eyes
simple-rss A simple, flexible, extensible, and liberal RSS and Atom reader for Ruby. It is designed to be backwards compatible with the standard RSS parser, but will never do RSS generation.
snmp A Ruby implementation of SNMP (the Simple Network Management Protocol).
soap4r An implementation of SOAP 1.1 for Ruby.
sprockets Sprockets is a Rack-based asset packaging system that concatenates and serves JavaScript, CoffeeScript, CSS, LESS, Sass, and SCSS.
state_machine Adds support for creating state machines for attributes on any Ruby class
syntax Syntax is Ruby library for performing simple syntax highlighting.
thin A thin and fast web server
thor Thor is a toolkit for building powerful command-line interfaces.
tilt Generic interface to multiple Ruby template engines
timecop A gem providing “time travel” and “time freezing” capabilities, making it dead simple to test time-dependent code. It provides a unified method to mock Time.now, Date.today, and DateTime.now in a single call.
transaction-simple Transaction::Simple provides a generic way to add active transaction support to
objects. The transaction methods added by this module will work with most
objects, excluding those that cannot be Marshal-ed (bindings, procedure
objects, IO instances, or singleton objects).

The transactions supported by Transaction::Simple are not associated with any
sort of data store. They are “live” transactions occurring in memory on the
object itself. This is to allow “test” changes to be made to an object before
making the changes permanent.

Transaction::Simple can handle an “infinite” number of transaction levels
(limited only by memory). If I open two transactions, commit the second, but
abort the first, the object will revert to the original version.

Transaction::Simple supports “named” transactions, so that multiple levels of
transactions can be committed, aborted, or rewound by referring to the
appropriate name of the transaction. Names may be any object except nil.

Transaction groups are also supported. A transaction group is an object wrapper
that manages a group of objects as if they were a single object for the purpose
of transaction management. All transactions for this group of objects should be
performed against the transaction group object, not against individual objects
in the group.

Version 1.4.0 of Transaction::Simple adds a new post-rewind hook so that
complex graph objects of the type in tests/tc_broken_graph.rb can correct
themselves.

Version 1.4.0.1 just fixes a simple bug with #transaction method handling
during the deprecation warning.

Version 1.4.0.2 is a small update for people who use Transaction::Simple in
bundler (adding lib/transaction-simple.rb) and other scenarios where having Hoe
as a runtime dependency (a bug fixed in Hoe several years ago, but not visible
in Transaction::Simple because it has not needed a re-release). All of the
files internally have also been marked as UTF-8, ensuring full Ruby 1.9
compatibility.

treetop A Ruby-based text parsing and interpretation DSL
trollop Trollop is a commandline option parser for Ruby that just
gets out of your way. One line of code per option is all you need to write.
For that, you get a nice automatically-generated help page, robust option
parsing, command subcompletion, and sensible defaults for everything you don’t
specify.
tzinfo TZInfo provides daylight savings aware transformations between times in different time zones.
uglifier Uglifier minifies JavaScript files by wrapping UglifyJS to be accessible in Ruby
uuidtools A simple universally unique ID generation library.
vcr VCR provides a simple API to record and replay your test suite’s HTTP interactions. It works with a variety of HTTP client libraries, HTTP stubbing libraries and testing frameworks.
wasabi A simple WSDL parser
websocket Universal Ruby library to handle WebSocket protocol
winrm Ruby library for Windows Remote Management
xml-simple A simple API for XML processing.
xpath XPath is a Ruby DSL for generating XPath expressions
ziya ZiYa allows you to easily create interactive charts, gauges and maps for your web applications. ZiYa leverages
flash which offload heavy server side processing to the client. At the root ZiYa allows you to easily generate an
XML files that will be downloaded to the client for rendering. Using this gem, you will be able to easily create great
looking charts for your application. You will also be able to use the charts, gauges and maps has a navigation scheme
by embedding various link in the graphical components thus bringing to the table an ideal scheme for reporting and dashboard
like applications. Your manager will love you for it !!

Sample site : http://ziya.liquidrail.com
Documentation : http://ziya.liquidrail.com/docs
Forum : http://groups.google.com/group/ziya-plugin
Repositories : git://github.com/derailed/ziya.git

Service Item Selection (Updated)

Here is something I get a lot,

“How can I make a service with multiple service items, but then conditional drop some during the deployment?”

Eg. You have a Service Dialog like this one here;

Download https://github.com/jonnyfiveiq/CloudFORMSNOW/blob/master/CFME/Services/Stopper/Sample_Dialog.yaml

Screen Shot 2014-10-10 at 14.11.00

Giving the user the option to select QA, Test or Production.

The decision would evaluate to one of 3 backing Service Items. Shown here in this diagram;

Slide1

Each service item here maybe RHEV, OpenStack or VMware. They might be all AWS/EC2 but different AMI’s etc!

So back to the use case,

How do I remove the service items I do not want to use from the service bundle (Nobody mention ZSTOP!)

Lets go through the solution;

1. You need to have a state machine for EACH service item. It can be the same state machine, but either way you need to specify one.

So, when you create each service item, and I am not concerned with the contents here (EC2, RHOS, RHEV, VMware etc..) You need to specify an entry point to a state machine that you have edit rights to as show here;

Screen Shot 2014-10-10 at 14.10.36

Screen Shot 2014-10-10 at 14.10.49

Screen Shot 2014-10-10 at 14.10.43

The bundle that has these three service items can be whatever you like, here is mine, its just using its own state machine as you should do, but nothing special happening here.

Screen Shot 2014-10-10 at 14.10.24

So looking at the Automate browser what do these state machines look like in file view?

Screen Shot 2014-10-10 at 20.13.35

2. The state machine has many steps, Pre1, Pre2, Pre3, Provisioin, Post Provision etc.. We want to edit Pre1 and put something for the state machine to process. We are going to call out to a method that decides to either keep this service item or to dump it out. So exactly like a conditional processor but done using some simple ruby code rather than the nice GUI’s we see in Control, Reporting and Filtering (That may come later in ManageIQ hopefully)

The code simply takes the value from the Dialog for an attribute. In my case it is “Dialog_Environment”, the possible outcomes for this attribute are “Test, QA or Production”

The next thing we do is take the value of the state machine we are running, so I have added to the state machine schema an Attribute field called “State_Environment”. Now I know what the user selected and what the current service I am running is.

So you should edit the schema of the <YourDOMAIN>/Service/Provisioning/StateMachines/ServiceProvision_Template class to include this new attribute called “State_Environment” here is a picture to show you the finished edit;

Screen Shot 2014-10-10 at 20.20.31

If the value of Dialog_Environment is NOT same as State_Environment then we want to dump this Service Item.

Here is the method; Its called Stopper, its a state on each of the Service Item State Machines, I shall show you this next;

def mark_task_invalid(task)
 task.finished("Invalid")
 task.miq_request_tasks.each do |t|
 2.times { $evm.log("info", "********************************DUMPING TASK #{t}*************************************") }
 mark_task_invalid(t)
 end
end


10.times { $evm.log("info", "*********************************************************************") }

stp_task = $evm.root["service_template_provision_task"]
miq_request_id = $evm.vmdb('miq_request_task', stp_task.get_option(:parent_task_id))
dialogOptions = miq_request_id.get_option(:dialog)

$evm.log("info", "Dialog_Environment #{dialogOptions['dialog_environment'].downcase}")
$evm.log("info", "State_Environment #{$evm.root['State_Environment'].downcase}")

if dialogOptions['dialog_environment'].downcase != $evm.root['State_Environment'].downcase
 $evm.log("info", "NO MATCH - DUMPING Service from resolution")
 task = $evm.root["service_template_provision_task"]
 mark_task_invalid(task)
 exit MIQ_STOP
end

$evm.log("info", "MATCH FOUND - Processing Service Normally")

10.times { $evm.log("info", "*********************************************************************") }

Download the Stopper method here Stopper.rb

So, the Stopper method needs to be placed somewhere, I instructed you do this on the Pre1 of EACH service item state machine. Here is an example of the QA state machine, this is the entry point for the service item QA. I have used the ON_ENTRY state, but you must leave the other states intact, you will see why in point number 3 coming next.

Screen Shot 2014-10-10 at 20.16.30

3. Now we do have a small issue to deal with, notice in the previous step the code exit is MIQ_STOP. This is great on one hand because it stops this state machine processing any further, but breaks in another as the ae_result is populated with “error” and when the bundle starts to look at its children for status it sees “error” and barfs out the entire Service Bundle, not good. So we have to fake the ae_result back to OK once we know that the reason for being “Error” is because we want it to be and not because its a genuine error. Make sense?

So we have an OOTB method called “update_serviceprovision_status”, its the job of this method to watch the service deployments and bump the return status around depending on its value. In here we simply do a check to say;

Is the item we are looking at for status, have a status value of “Invalid”, because this is unique and not a Cloudforms status, its been set specially by us for this use case. If it is “Invalid” then we know that this service needs its “ae_result” forced to be “ok” and to exit MIQ_OK. Making everyone happy, the service bundle thinks its provisioned the service when it actually did not, moving onto the next service item in the bundle.

Here is the NEW check_provision method that you need to create in your domain, I would simply copy the one from;

/ManageIQ/Service/Provisioning/StateMachines/ServiceProvision_Template/update_serviceprovision_status

And either copy this code in, just place it near the top, before the processing of the objects;

# Bypass errors for Invalid instances
if prov.message == 'Invalid'
  $evm.log("info","Skipping Invalid Services")
  $evm.root['ae_result'] = 'ok'
  message = 'Service Provisioned Successfully'
  prov.finished(prov.message)
  exit MIQ_OK
end

or download this new one from here update_serviceprovision_status.rb

I hope this all works for you, its a great use case and one we talk about all the time.

Obviously there are more than one way to do this, but most other routes either fail the bundle (bad) or require generic service items and loads more code. This is re-usable, could be prodctised and easily repeatable without a lot of effort. With the new domains in 3.1 you can have this in your tool box.

I will raise a discussion on talk.manageiq.org to share this, and get input on having the GUI conditional processor available in the Service Item designer phase.

Shell-Shock – Bash Code Injection Vulnerability via Specially Crafted Environment Variables (CVE-2014-6271, CVE-2014-7169)

UPDATED : Video version of this blog…..found here… http://youtu.be/RDcIIyYK044

 

So this Shell-Shock stuff is hitting the press quite a bit!

Fncy finding out really quickly if your Red Hat Enterprise Linux 6.5 systems are patched correctly? Even if they are turned off right now? Wow that is clever not even the virtual infrastructure players can do that…I know…its cool. Here it is..

Using Cloudforms (or ManageIQ for FREE!) download this policy and import it into Control. Then assign the policy to your targets. The policy will only check Linux systems, though it could do with a makeover to check only RHEL 6.5 systems too.

 

Download and import the following policy profile – https://github.com/jonnyfiveiq/CloudFORMSNOW/blob/master/Policies/ShellShockPolicy.yaml

To note….the policy is valid for only the fix packages as defined in article https://access.redhat.com/articles/1200223 for RHEL 6.5 systems. Feel free to modify the policy to fit your needs and share with the community at talk.manageiq.org 

1. Ensure that your VM has a recent smart state scan completed successfully. You can check by clicking on the Configuration/Packages link as follows;

Screen Shot 2014-09-28 at 20.13.12

Search the list of packages for the “bash” package. Select the package and you will be presented something like the following;

Screen Shot 2014-09-28 at 19.59.53

Ok so we have confirmed we have package detail about bash in the VMDB for this virtual machine.

2. Assign the policy, you can assign the policy anywhere you like that has coverage of the test virtual machine.

Once assigned, simply click on a VM you wish to check and select the menu “Policy” and “Check Compliance”

Screen Shot 2014-09-28 at 20.09.52

You would have noticed that your Compliance status is probably as follows;

Screen Shot 2014-09-28 at 19.51.17

 

 

 

Once the compliance check is complete the compliance area of the screen will report how old the current report is.

Screen Shot 2014-09-28 at 20.18.54

Next, Click on the Status of the compliance to drill further into the detail;

Screen Shot 2014-09-28 at 19.57.16

As you can see the policy has fail compliance check.

Now we want to remediate the issue, and re-run the compliance check.

Screen Shot 2014-09-28 at 20.21.08

So I run a “yum update bash” and as you can see “4.1.2-15.el6_5.2″ has been applied to my system, lets have cloudforms check against this now.

So, first run a smart state scan against your test virtual machine,

Screen Shot 2014-09-28 at 20.02.57

Once complete, run the compliance check once more on the virtual machine;

Screen Shot 2014-09-28 at 20.08.21

 

This time the compliance check passes, click on the status and drill further into the detail.

Screen Shot 2014-09-28 at 20.08.30

And as a last resort you can go back to the virtual machine and take a look at the package entry for “bash”

Screen Shot 2014-09-28 at 20.05.39

As you can see, Cloudforms has been updated with the latest rpm data from the yum update bash we ran.

So there you go, how simple is it to check for ShellShock using Cloudforms, really easy.

Thanks

 

Creating an OSE Service BluePrint and Ordering it

This video shows how to use the OpenShift for Cloudforms materials to create a blueprint that can deploy a multi machine setup. I demonstrate creating a catalog, two catalog items, a bundle and assigning the OpenShift policy as we desire. Finally we order the service and see it instantiate fully as a working multi machine OpenShift infrastructure.

Creating an OSE Service BluePrint and Ordering it – http://youtu.be/O05tg5kdpyQ

Monitoring the progress of an OSE Service – Part 2

In the previous post http://cloudformsnow.com/2014/09/22/deploying-openshift-part-1/ I showed the consumer use case of going into Cloudforms and requesting a service for deployment, namely a service that deploys OpenShift Enterprise.

The following link is a video that shows how you can monitor the installation. The state machine that deploys OpenShift from Cloudforms will automatically send the consumer emails to the progress of the installation as follows;

Email 1 – Verify that the Virtual Machines/Instances deployed in the service are capable of taking OpenShift Enterprise

Email 2 – Confirm that the workflow for OpenShift installer has been written and that the install as been spawned.

Email 3 – Confirmation to the finished state of the installation, if successful the email contains the links to the various components.

Along with the emails you can also monitor the installation using the tags created by the state machine on the service, these tags will allow you to easily locate the unique log file for the deployment of that service.

Monitoring the progress of OSE Deployment – Part 2 – http://youtu.be/wbcrJjpiewI