Dynamic Test/Evaluation Environment 

 Vagrant, Ganeti , Openstack are great tools for a dynamic data-driven test environment. couple them with a configuration management CFEngine3, Chef, puppet, , Ansible, or Saltstack and you will start having more time on your hand, and appreciating life around you. The possibilities are endless if you are looking for a backend highly available infrastructure Ganeti is your solution, used already by "Open Source Labs", Google, Mozilla, Greek Research and Technology Network, among others to manage cluster of virtual environments with resilience in mind. if you are looking for flexibility and providing your users with a private cloud solution Openstack will do. for testing new administration tools, policies, cookbooks, manifests, play books and blue prints than Vagrant is the way to go add the combination of these three together and you have dynamic solutions that scale in your own laptop or workstation from few virtual nodes to Amazon EC2, or your own company private cluster environment. 

Devops afternoon in Khobar- Saudi Arabia

Devops, and web operations did not pick up in the middle-east as it did in US, Europe, China, and India. We had a chance to present at the HPC Saudi 2013 user group conference that was coordinated by our technology planning engineer Khalid Chatilla, and Intel/IDC. we decided to check with CFEngine, PuppetLabs, Ansibleworks, and Opscode if they can participate, and they showed interest even though it is already end of year, and budgets already consumed, not to mention the short notice , logistics and planning that needs to take in action to secure their coming to Saudi Arabia. at the end Ansibleworks, and Puppetlabs managed to come and delivered an awesome afternoon, my colleague and friend Ahmed bu Khamessin with his limited graphical resources was able to capture some of these moments by his video camera and even though the sound quality is not great, he made it public to the world.  you can see my intro slides, and Ahmed videos below

Prezi Introduction to Saudi Devops Days  with use cases from CFEngine, and Chef.



Ansible presentation :

 Puppetlabs presentation in youtube

Software packages and repositories

Software packages and repositories is my first stop in automating the OS life cycle, the OS image including all software stacks, os, middleware, management, and application should represent a fixed state. that would difficult to track if installs were done ad-hoc outside of a packaging system. so for us we use mainly RHEL based distros. so you think the answer would be use yum, and rpms!!! well there are Java applications as jars. there are Ruby gems, there are python eggs, and there are git clones and tarballs. one answer is use fpm to convert from any format to rpm.

• so one challenge is the diversity of packaging types and how to standardise on one.  
• Second, comes the Internet isolation and state, at work we are not allowed downloads directly from the net. 

So for this second problem i need to have a way to mirror publicly accessed or Enterprise provided repos to internal repos. the easiest choice is to mirror every thing and copy it/rysnc it over to work periodically.

for Ruby Gems here is the simplest way to do it :

http://stackoverflow.com/questions/8411045/how-to-build-a-rubygems-mirror-server

$ gem install rubygems-mirror




Edit the YAML configuration file ~/.gem/.mirrorrc:




---
- from: http://rubygems.org
  to: ~/.gem/mirror

the to: filed above can be better pointing to a usb storage, where ever it points at 

$ mkdir ~/.gem/mirror

Start mirroring:

$ gem mirror

Once mirroring finishes edit ~/.gem/mirror/config.ru:

require "rubygems"
require "geminabox"

Geminabox.data = "./"
run Geminabox

Install Gem in a box:

$ gem install geminabox

Start gem server:

$ cd ~/.gem/mirror
$ rackup

Edit your application's Gemfile to use your gem server:

source "http://your.servers.ip:9292"

Virtualbox guest host NATed

After installing CentOS6.4 as guest OS in Windows 8.0 and configuring the single network interface using NAT mode, I could not from first instance ssh using putty to the guest OS DHCP ip address given as 10.0.2.15.

I had to power off the Guest and enable port forwarding first as described in NATFORWARD section under NAT networking mode on chapter 6 of the users manual.

https://www.virtualbox.org/manual/ch06.html#natforward

Below are the commands i used to configure and check port forwarding

 .\VBoxManage  listvms

 .\VBoxManage modifyvm "CentOS01"  --natpf1 "guestssh,tcp,,2222,,22"

  .\VBoxManage.exe showvminfo CentOS01 |findstr "2222"
NIC 1 Rule(0):   name = guestssh, protocol = tcp, host ip = , host port = 2222, guest ip = , guest port = 22

in putty host = localhost and port will be in this case 2222

the above was done to test the ORD OpenStack Red Hat distribution, i had several failures before i was able to install it using the quickstart successfuly. first due the disk size, the disk size should be over 22 Gbyte so that Cinder can create 20Gbyte disk by default, second the selinux needs to be enabled. and every time it fails you need to remove cinder packages and logical volume manually before restarting the installation and cleaning up the bits and pieces from old installation.

a succesfful install should not take over 20-30 mins.

How many administrators do you need for your operations?

How many administrators do you need for your operations?


Several online resources are discussing this issue. it usually depends on several factors such as :

I- Factors that could reduce the number of admins needed:

• Remote console/power and remote management tool availability
• Vitalisation
• Physical server, and rack technology (e.g. blades or skinless vs. U2 servers)
• Availability of management tools  (rack management, api such as in EC2, and cloud providers)
• Platform ( e.g. Unix, and Unix like vs. Windows )
• Configuration management and automation tools.
• Initial plan, and vision of business/data centre expansion
• Organisation requirements, maturity, stability, and adoption of the devops culture.

II- Factors that could increase the number of admins needed

• Size and diversity of data managed.
• Number, and diversity of servers and server configurations
• Number of users
• Number and diversity of applications used and  supported*. 
• Number of new technologies at the ground or acquired within  the data centre
• Complexity of the solution and infrastructure.

* used by the administration  team, and supported on behalf of others within or external to the organisation.

So what is the best practice metric that should be used? it depends on what kind of operations the business is running and how messy, or diverse it is customer or application space, as well as management approach towards operations and support from the start. not to mention what we mean by best practise is the best try, best deployment plan, however as soon as it materialised it can be improved so it becomes good practise or  could be even bad if it did not evolve

CERN did not use virtualization to help deploy and run  their HPC codes, however have chosen to adopt virtualization to ease out administration and management costs

FaceBook  230 engineers supporting data for over than more three million users, at around 130 servers per admin  [1]

Microsoft automated data center operations at around 1000-2000 servers per admin, while its new container data center will be around 10,000 server/DC employee.

IDC reports in large dominant  providers such as Google, it could be 10,000 servers/admin while in small to medium businesses it could be 30:1 for physical boxes and 80:1 for virtual machines. [2]

Gartner analyst, Errol Rasit says “We have observed that it can be, for example with a physical server, as low as 10 per admin, and for virtual servers as many as 500,”


resources :

[1] Data Center Knowledge Article "how many servers can one admin-manage"

[2] Computer World Australia IDC reference 

CFEngine3 by examples

CFEngine3 by examples : CFEngine3 3.4.x is the latest community release from CFEngine, one of the new features that was missing is the publicly available reusable easy to do/read policies. in the terminology of CFEngine i believe it is called sketches,  they are another form of describing your system desired end state using promises, however in this case you are getting them, or publishing them in Design-Center.  Nick Anderson a Motorola employee and an experienced long term CFengine and advocator have released publicly an easy to develop and test environment using vagrant in Github.  along with examples to get one started quickly. ignoring the download times of Vagrant, Git, and CFengine codes, the process should not take you more than 5-10 minutes. I have realized that CFEngine do not release binaries for OSX, so i used brew to get the latest CFEngine which was 3.4.1, while the latest at this time was 3.4.2. not bad! ha!. i had to download also the xcode command line tools to get make, gcc among other development tools to get things build and working.  No excuse on getting more familiar with CFengine in no time, especially that CFEngine senior security adviser and Orielly book "Learning CFengine 3" author sent me a chapter from his 3rd edition regarding vagrant and Behavioral testing for CFengine.