hsiliev's blog

Running Virgo on Cloud Foundry

2011-11-26T18:09:00.001+02:00

In this post I provided a VM with Virgo and Cloud Foundry. Now I'll explains how to set up such VM on your own and to use updated (2011-11-27) Cloud Foundry version forked in GitHub.

Installation steps:

1. Download VirtualBox and install it.

2. Download VirtualBox's Extension Pack and install this as well.

You may double click on the Extension Pack for Windows. Also note that the extensions are not needed if you want to setup Ubuntu on your own.

3. Download Ubuntu image and write down the user/password.

I've tested with Ubuntu 11.04. Ubuntu Server is the best choice for productive systems, and in case you like developing with VI.

Note that the recommended version is Ubuntu Server 10.04-2 LTS.

4. Start the image

Some of the archives contain only HDD image, so you may need to configure the VM and attach the downloaded HDD image.

Other images come preconfigured. For those you may need to change the network adapter settings in VirtualBox. Simply clicking on Network section and saving the auto-detected changes did the trick for me.

In general ensure the VM has at least 2GB memory. If you want to try every Cloud Foundry service and framework (or simply run the tests) you will need at least 3GB of memory.

The image may have keyboard layout you are not used to (Italy as was in my case), so it's a good idea to change this. For the Server (no GUI) you can do this with:

sudo dpkg-reconfigure console-setup

Installing Guest Additions is definitely a good idea since it allows you to copy/paste install commands in the next steps, and what's more important allows hardware and host OS acceleration.

5. Install libtool using:
    sudo apt-get install libtool

6. Install CloudFoundry on the image as explained in this GitHub vcap repository fork

You may need to reboot the VM to allow RVM (Ruby Version Manager) to pick up the configuration in vcap folder and switch Ruby versions. Sourcing the configurations did not work for me.

7. Update Cloud Foundry with the Virgo modifications

The Cloud Controller uses vcap_staging gem that does not support Virgo, so we have to update the gem with the Virgo specific code. To do this execute:
   cd ~/cloudfoundry/vcap
   cp -r staging/lib/* ~/.rvm/gems/ruby-1.9.2-p180/gems/vcap_staging-0.1.25/lib/

Finally restart the Cloud Foundry:
   bin/vcap restart

To be able to use the automatic discovery of Virgo applications in VMC we need to update the client:

   cd ~/cloudfoundry/
   git clone https://github.com/hsiliev/vmc.git
   cp vmc/lib/cli/frameworks.rb ~/.rvm/gems/ruby-1.9.2-p180/gems/vmc-0.3.13/lib/cli

How to install Virgo's Admin Console

1. Download and extract the archived console
2. In the directory where you extracted the archive, run:
vmc push virgo
3. Request virgo.vcap.me and you should see the Splash Screen with link to the Admin Console in the upper right corner.

How to update to the latest Virgo version:

1. Clone vcap-java repository

2. Go to vcap-java/virgo-setup

3. Edit virgo_manifest.yml to specify the exact version you want

4. Run rake virgo:prepare
5. Find the new version in
/tmp/virgo-<your version>/virgo.zip

6. Replace the ZIP in

~/cloudfoundry/vcap/staging/lib/vcap/staging/plugin/virgo/resources

The last two steps should be automated with the virgo:install task, but I haven't tested if this works.

Gentoo on Lenovo T520

2011-11-15T19:53:00.001+02:00

My new laptop is finally here. Lenovo T520 came with Windows 7, and if you ask me that's a bad choice for laptop with these specs. Neither the video (Intel HD), nor the 500GB HDD were made to be used by gamers where Windows is undisputed leader.

So I made up my mind to made a switch and chose Gentoo Linux since I needed better git, ruby and java support, but besides all I wanted to be in control on what's happening with the hardware and software on my system :) From previous installations I found out that Ubuntu does not support odd or older hardware without kernel recompile. On the other hand the software packages in the most popular Linux distro are not the most stable thing (especially Gnome UI stuff).

Since I had enough experience with Gentoo from setting up my development server and the surveillance system machine I decided that I'll just stick with this source based distro with rolling updates. It is also used by several cloud computing companies, so it looks it's here to stay.

I've done the basic setup almost exactly as recommended in the Gentoo Handbook. The concrete settings are detailed below.

make.conf

I used "native" architecture for GCC which leaves the optimization decision in the compiler. Since T520 has 4 cores the numer of compile/processing threads is set to 5. To speed up the emerge the download is made on background with "parallel-fetch" feature.

CHOST="i686-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe"
CXXFLAGS="${CFLAGS}"
MAKEOPTS="-j5"
FEATURES="parallel-fetch -test"
USE="mmx sse sse2 sse3 3dnow 3dnowext \
dvd dvdr cdr vcd cdda \
usb wifi laptop ieee1394 \
lm_sensors \
bash-completion symlink \
ftp \
bzip2 zip unzip 7zip rar \
jpeg svg jpeg2k png gif mng \
ffmpeg xvid x264 theora \
truetype gd \
mp3 faac faad amr \
java javascript xml ruby python \
sqlite \
ssl sni \
unicode \
daemon threads \
X xvfb dbus hal \
gtk gnome gdu extras device-mapper policykit cairo \
-kde -qt3 -qt4 \
-ipv6"
GENTOO_MIRRORS="http://mirrors.ludost.net/gentoo/ ftp://mirrors.ludost.net/gentoo/ http://distfiles.gentoo.bg/ http://ftp.gentoo.bg/ "
SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage"
ACCEPT_LICENSE="dlj-1.1 Oracle-BCLA-JavaSE skype-eula"
INPUT_DEVICES="evdev synaptics mouse keyboard"
VIDEO_CARDS="intel vesa"
LINGUAS="bg en"

I'm a Gnome user, so the flags are set to disable QT and KDE. The IPv6 is of no use to me currently so I disabled it as well.

I'm Java developer so I accepted Oracle's JDK 1.5 and 1.6 licences to enable installation of the software. Since Oracle changed the 1.6 licence and download pages it is no longer possible to install Java automatically. Fortunately emerge provides quite good hints when you have to download a package manually.

Next I inserted my hardware configuration for X - keyboard, synaptic, external mouse, and Intel HD video card.

Language configuration can also be added in make.conf to enable localization of console, X and Gnome/KDE.

Linux Kernel Configuration

I used the Gentoo's 3.0.6 kernel (gentoo-sources) with this configuration.

LAN setup

In /etc/conf.d/net you can use ifplugd to make the annoying delay on start-up go away, when there is no cable plugged (in the end this is a laptop):

ifplugd_eth0="dhcp"

Of course you will have to install ifplugd as described in the handbook.

Wireless network

The kernel configuration includes drivers for the wireless network on T520. This however is not enough to get it working.

I found out that the network is Intel Centrino Advanced-N and followed Google to this Gentoo Forum Post. The post provides quite good guide on how to identify and emerge the right driver and firmware for your wireless chip.

Gnome

As a pleasant surprise Gnome provided me with working functional keys out of the box. From changing video output through back-light to sleep buttons everything worked as expected.

Not working

Currently I don't know how to get the card reader to work.

Installing ZoneMinder in Gentoo

2011-11-08T23:18:00.005+02:00

Installing ZoneMinder in Gentoo turned out to be a relatively easy task:

emerge layman

layman -o http://gpo.zugaina.org/lst/gpo-repositories.xml
layman -a zugaina

echo "source /var/lib/layman/make.conf" >> /etc/make.conf
ACCEPT_KEYWORDS="~x86" emerge -av zoneminder

If you are interested in the details you can take a look at Overlays @ Gentoo wiki

Setting up Geovision 650 surveliance card in Gentoo

2011-10-01T23:52:00.033+03:00

I was trying to turn my old 2.2 Ghz Celeron based computer into a surveillance system. I needed a lot of computing power to be able to record from 4 cameras and to be able to integrate 2 new Vilar IP cameras as well. I chose Gentoo Linux since it does it's job as server OS quite well on my home development server.

So after 3 hours I had a customized installation compiled with -march=native and kernel stripped down to the bare minimum.

I tried to set up my Geovision 650 DVR card starting from the Linux module. I launched make menuconfig and was sure that I'll have everything ready in under an hour. As you can imagine this was not the case :) In the end it took me 2 days to have the card running and I decided to describe the steps.

Gentoo installation

I followed the handbook as usual and the installation was pretty straightforward and went smooth. The GCC configuration spared me the search for the CFLAGS of my CPU and I had to put -march=native to get everything optimized.

USE flags

I intended to use ZoneMinder so I added the needed flags right away (apache, mysql, php, perl). It turned out that to check the input of the card I had to use xawtv, so I enabled X as well.

BT878 / BTTV module

The module I needed was BTTV since Geovision 650 is equipped with two BT878 chips to allow decent capture from its 4 inputs.

However I was not able to find how to enable the compilation of the driver. I was more astonished than ashamed to find that the answer was the non-existent Infra-red Remote.... Turning the IR support in the kernel enabled the compilation of the module I desperately needed.

I tried so hard to enable BT878 support so I decided to scan all options in kernel configuration screens. I removed a lot of unneeded stuff and between the flags that were unchecked was the one ensuring that my 80GB HDD was able to host single files with size more than 2TB.

I recompiled the kernel and a few modules, replaced the old binaries, rebooted the machine...

Missing kernel features

... and I found myself with read only EXT4 file system. It turned out that the LiveCD I used to create my root partition had large files suppport turned on. In fact the kernel has this by default. Using EXT4 tools seems to pick the kernel settings so before I was able to fit 2TB files on my small HDD and now the kernel was not able to process such files anymore.

I found the solution in this blog and this was one of the reasons to describe my installation problems.

BTTV driver configuration

Gentoo now uses OpenRC to describe and configure modules (/etc/conf.d/modules). I put bttv driver in the list rebooted again to check if the driver will autodetect the card. Yes I know that modprobe would (or actually should) have done the same job but I had to be sure.

What I found was card=0, autodetecting log entry shown by dmesg. So I had to find the correct card settings. Fortunatelly there are a lot of resources out there that hinted to use "card=<number>" parameter of bttv module. The question was what to put as number. Soon I found that 150 provides me with Geovision 600 which is pretty close relative to my GV-650.

Using my Windows experience I rebooted again and after several seconds found out that nothing changed. So time for modprobe tests with different numers only to find no change in bttv output.

Fortunatelly the bttv sources were sitting quetly on my HDD. A dump of bttv.c showed that GV-600 has 0x96 value which was exactly 150 in decimal. I smiled happily and put the "correct" values in modules configuration: "card=150,150".

The next Windows-style reboot revealed that bttv was not happy with this value as well. I had the same autodetect failure in the logs.

I reported the problem in bugzilla. Meanwhile tried to search internet and found a Russian Google IRC chat that provided needed workaround. It turned out that udev kicked in too early and loaded the module before OpenRC was able to configure it correctly. OpenRC issued modprobe correctly but the module was already there.

So in the end - not really a bug :) Simply disabling coldplug with rc_coldplug="NO" in /etc.conf.d/udev did the trick.

Testing video input

I installed xawtv to test the output and after 2 hours of experiments I had no idea why the card won't work.

Google found out several configurations working fine with GV600 or GV800 but none with GV650. Strange ...

I wondered what was the difference between GV600 and GV650. After some research it turned out that GV650 has one more BT878 chip than GV600, and in turn GV800 has 4 in total.

Consulting the bttv-cards.c file revealed that one of the chips in GV800 is "master" and the rest "slaves". This and the strange mapping between the inputs and the chips in this blog led me to the conclusion that I need to change my bttv configuration to "card=157,158" (master/slave).

Another reboot and suddenly everything worked just fine.

ZoneMinder

The last time I tried to run ZoneMinder on Gentoo was 4 years ago. Too long as it turned out.

Today the ZoneMinder is nowhere to be found in the repositories and its configuration on your own is not easy. You have to hunt down the location of all dependencies, the libraries ffmpeg is build with and so on.

Well I think I managed to dig all the needed info but configure just stopped with cryptic message about openssl library. Basically it says that the library is on the system but is messed up and unusable by ZoneMinder.

Seems like I'll have to create a new blog about this :)

Virgo and the Cloud

2011-07-29T00:48:00.005+03:00

If you wanted to try your OSGi based applications in the cloud you can now do so with Eclipse Virgo and CloudFoundry using a VM image.

What you will need:

The image comes with pre-installed Ubuntu 11.04 and has:

User / Password: virgo / cloudfoundry
CloudFoundry

installation: /home/virgo/cloudfoundry
version/date: 28 July 2011
Virgo 3.0.0.RC1

Applications (Splash screen & Admin Console): /home/virgo/apps

Starting CloudFoundry
To start CloudFoundry you have to start a Terminal and issue:

cd ~/cloudfoundry/vcap
bin/vcap start

To check if the start was successful:

bin/vcap status

More information and examples can be found in the CloudFoundry VCAP Readme.

Sample applications
To push (deploy) the Virgo default applications you can use the VMC client:

cd ~/apps
vmc push virgo

where "virgo" is the name of the application we want to have running on CloudFoundry. You will have to confirm the name, URL, memory reservation and that the default applications do not use services.

As next step you can request the application at http://virgo.vcap.me. This should bring the default splash screen of Virgo with a link to the Admin Console.

Your own applications
Perhaps you will want to try your own application as well. To do this you will have to:

create a directory
copy your stuff inside
push the content with "vmc push"

There are some extras here. If you create "repository" directory inside your application's directory then the content of this repository will be copied in Virgo's repository.

Virgo on CloudFoundry

If you take a look at the provided Admin Console application you will see that it makes use of two bundles that are located inside "repository/ext". The bundles are copied in Virgo's own repository directory so the rest of the content can resolve against these bundles.

Supported artefacts
Currently supported artefacts:

WAR file (defaults to Tomcat, so you'll have to change the framework to Virgo)
PAR
Plan
Bundle (JAR)
Combination of the mentioned artefacts

Multi-application solutions

The last option provides you with the ability to have more than one artefact in your solution. All these are pushed to CloudFoundry and used in Virgo.

As an example you may have several interconnected but relatively independent applications: PAR representing some back-end, WAR that provides the end-user UI and some more artefacts (as much as you need) that can represent editors or administration logic.

Everything can run on one Virgo server for testing/development, but you may decide to create several instances of a certain component to scale out that part of your solution. All that provided that your applications are written with scalability and cloud computing in mind of course.

Where's the source code?

At the time of this writing the changes made to CloudFoundry are not final and therefore no pull request is created, but I'm up to this.

You can find the source in GitHub if you want to play with it or just check how/if it works in your case :)

Apache Gogo and SSH in Eclipse Virgo

2011-06-15T13:36:00.015+03:00

Step 1: Download Gogo shell
To run the Gogo you will need three bundles from Apache Felix downloads:

Step 2: Equinox and RFC-147
To enable RFC-147 integration in Equinox you will need some console supportability features that are provided by the Console supportability project in Equinox Incubator.

Download the Incubator build that includes the bundle we'll need. Extract only the supportability JAR (org.eclipse.equinox.console.supportability) from plugins folder.

Step 3: Enabling SSH Support
First download Apache Mina's binary distribution for SSHD, and then from the lib directory of the archive extract these two bundles:

sshd-core
mina-core

Step 4: Setting up Virgo

Copy the console supportability bundle (org.eclipse.equinox.console.supportability) in lib directory
Place the rest of the bundles (3xGogo, SSHD and Mina) in lib/kernel directory
Edit config/org.eclipse.virgo.kernel.userregion.properties file and add the bundles to base bundles list:

baseBundles = \
file:lib/kernel/org.eclipse.virgo.kernel.userregion-3.0.0.M05.jar@start,\
file:lib/kernel/org.eclipse.virgo.kernel.osgicommand-3.0.0.M05.jar@start,\
file:lib/kernel/mina-core-2.0.1.jar@start,\
file:lib/kernel/org.apache.felix.gogo.command-0.8.0.jar@start,\
file:lib/kernel/org.apache.felix.gogo.runtime-0.8.0.jar@start,\
file:lib/kernel/org.apache.felix.gogo.shell-0.8.0.jar@start,\
file:lib/org.eclipse.equinox.console.supportability_1.0.0.N20110608-2000.jar@start,\
file:lib/kernel/sshd-core-0.5.0.jar@start

In lib/org.eclipse.virgo.kernel.launch.properties remove the old Virgo shell by deleting the line ~~org.eclipse.virgo.osgi.console.telnet.hook.TelnetHookConfigurator~~. The hooks entry should look like this afterwards:

osgi.hook.configurators.include=\
org.eclipse.virgo.osgi.extensions.equinox.hooks.ExtensionsHookConfigurator

Delete (or move outside lib) the bundle lib/org.eclipse.virgo.osgi.console-3.0.0.M05.jar

Step 5: Configuring Virgo

Comment or delete osgi.console property in lib/org.eclipse.virgo.kernel.launch.properties:

#osgi.console=2401

Add the following property entries to lib/org.eclipse.virgo.kernel.launch.properties:

osgi.console.ssh=2422
osgi.console.enable.builtin=false
osgi.console.ssh.useDefaultSecureStorage=true

Add in config/org.eclipse.virgo.kernel.authentication.config file JAAS configuration for the SSH:

equinox_console {
org.eclipse.equinox.console.jaas.SecureStorageLoginModule REQUIRED;
};

Edit the bin/dmk.bat or bin/dmk.sh file to add org.eclipse.equinox.console.jaas.file and ssh.server.keystore VM system properties. After the changes the file should look as follows:

set KERNEL_JAVA_PARMS=%KERNEL_JAVA_PARMS% -Dorg.eclipse.virgo.kernel.authentication.file="%CONFIG_DIR%\org.eclipse.virgo.kernel.users.properties"
set KERNEL_JAVA_PARMS=%KERNEL_JAVA_PARMS% -Dorg.eclipse.equinox.console.jaas.file="%CONFIG_DIR%/store"
set KERNEL_JAVA_PARMS=%KERNEL_JAVA_PARMS% -Dssh.server.keystore="%CONFIG_DIR%/hostkey.ser"set KERNEL_JAVA_PARMS=%KERNEL_JAVA_PARMS% -Djava.io.tmpdir="%TMP_DIR%"

Step 6: Connecting to Virgo
Fire your favourite SSH client on the specified port (2422 for instance).

Login with the default user and password (equinox/equinox)
Create a new user and password

Step 7: Fun with Gogo
Check the features provided by GoGo and Equinox Console Supportability:

Tab Completion / Line editing
Built-in commands like lb, ls, cat to name a few
Create some scripts
Create some commands using RFC-147

Step 8: Disconnecting

Do not use "exit" since this will exit Virgo/OSGi framework. To end the session you'll have to close the console window. There is no command to close the session currently.

OSGi Bundle Dependency Analysis

2011-04-04T22:43:00.002+03:00

To find all the bundles needed by your imports you can follow these simple steps:

Step 1: Open the MANIFEST-MF of your bundle

Step 2: Go to the Dependencies tab

Step 3: Add the packages you need (if you do not already have them)

Step 4: Save the changes

Step 5: Go to Dependency Analysis and chose the hierarchy view

Step 6: Browse the hierarchical list of all dependencies

Step 7: To remove the duplicates use the Flat Layout

Please have in mind that this approach will work only for bundles that are in your Eclipse distribution or in your currently active target platform.

Git Notifier on Windows

2011-03-07T00:13:00.006+02:00

Git Notifier is a tool that can keep you up to date with what's going on in a Git repository. Unfortunately it seems that it's only for MacOS. Until this evening...

...when I decided to give Google a try and searched for Growl without removing Windows from my previous search. It turned out that Growl has Windows port so the task of having notifications for Git commits on my desktop moved from the dusty "nearly impossible" corner in my mind to "doable".

So I installed:

Cygwin

Git
OpenSSH (server and client)
Ruby

Growl for Windows

The next big step was to download Git Notifier just to find out that it utilized RubyGem. Cygwin does not provide Gem so I had to install it manually. An easy task as it turned out:

Download RubyGem
Install it using "ruby setup.rb" as described in RubyGems User Guide

Now that I had RubyGem I went straight to installing the Notifier.

gem install git-notifier --source http://gemcutter.org

Starting Git Notifier however turned out unsuccessful since this tool brings several dependencies:

ruby-growl
growl
daemons

Simply installing the dependencies did the trick:

gem install ruby-growl growl daemons

The rest is up to you. A good start would be to autostart Growl as well as git-notifier :)

At last usable Equinox console

2011-01-31T08:00:00.008+02:00

Eclipse Virgo recieved a brand new contribution that improves the usability of the Equinox console. The contribution is especially useful when connecting via telnet.

Thanks to Lazar Kirchev now Virgo has:

line editing
command history
tab completion
grep for telnet

Up to now the console port was bound to both localhost and the external IP. Therefore it was possible to just about everyone to connect to the console.

To address this issue the syntax of the osgi.console property has been extended and now you can add as well the host or address in org.eclipse.virgo.kernel.userregion.properties:

osgi.console=localhost:2401

What's more is that the telnet protocol allows negotiation between the client and the server. Taking advantage of this now there is a way for Virgo to understand which are the exact key bindings used on the client. This basically means that Virgo server running on Windows for example, can handle properly MacOS or Unix clients.

The new features can be used not only in Virgo, but also in every product using Equinox 3.7:

Virgo (3.0.0 M1)

Download Virgo from here
Enable the console as described in the User guide
Start Virgo
Connect with telnet to port 2401

Eclipse 3.7

To use the new features with Eclipse you simply need to drop the bundle in plugins and restart Eclipse.

Equinox 3.7

If you just want to try the new features you can use the install command.

If you have P2 enabled (or you like the bundle and want to keep it) you can drop the bundle in dropins and restart Equinox to enable P2 to pick up the bundle.

Troubleshooting class loading issues with Virgo

2010-12-29T17:20:00.008+02:00

My colleague Diyan Yordanov outlined the class loading issues a web developer can face in his ESE 2010 talk "Migration of existing web applications to OSGi". The classification in the talk divided the class loading issues, based on the exceptions:

ClassNotFoundException - missing class definition
NoClassDefFoundError - class definition found, but instantiation failed
ClassCastException - additional class copies

There is not much I can add, so I will reuse this classification to present the root causes we faced during our work on SAP NetWeave and the use of OSGi.

In the simple/standalone Java applications the classpath was the thing that determined the class loading and the way that we investigated what caused the exceptions above. The problems in this simple world were caused by:

ClassNotFoundException

Missing class definition (.class files)
Too restrictive file system permissions

NoClassDefFoundError

Exceptions in static block
Missing imported class
Class version mismatch (class is compiled with incompatible JVM)

JEE was the next step in the evolution of the class loading problems :) With the JEE containers we introduced complicated class loading hierarchies. The root causes in this hierarchies were the same as with the simple applications, but we faced dependency (or "reference" in NetWeaver) problems as well . Fortunately (depends on the point of view) we didn't have the time to implement versioned dependencies and get ourselves into more troubles.

The last step was made when we found out that OSGi pretty much eases the development and the support of components. It featured its own class loading that had everything we could dream of (plus some things we didn't really know we wanted). One of the extra things that were missing in our JEE world were the imports and exports of packages. And this is the thing that introduces more root causes for class loading exceptions:

ClassNotFoundException

Missing package import or export
Import version mismatch
Wrong boot class delegation property

Most of the problems described so far can be solved by simply finding which bundle:

contains a class
exports a class
can load a class

This is not an easy task in bare OSGi environment, but fortunately Virgo provides shell commands that will help you troubleshoot class loading issues by doing exactly the operations described above.The commands are:

clhas - lists all bundles that contain a class
clexport - lists all bundles that export a class or package
clload - lists all bundles that can load a class

ClassNotFoundException can be solved by checking if:

a bundle provides the class - clhas command
the class is exported - clexport command

NoClassDefFoundError cause can be determined by:

inspecting the logs for exception
trying to load the class

The result from the second step depends on the parameters of clload command and can be:

a list with bundles that can actually load the class
confirmation that a bundle can/cannot load the class

The fact that the bundle can load the class now but was unable to do so when an exception was produced would probably mean that a dependency was missing at the time of the problem.

If the bundle cannot load a class, although it should be able to do so, means that there is some problem with the dependencies - most probably there is something wrong with the:

imports of the bundle
exported packages

You can check if the needed packages are exported with the help of the clexport command.

ClassCastException is almost always caused by duplicated class(es) in some bundle. You can check the bundles that contain the class in doubt with the clhas command.

Finally you can find some example usages of the commands in the latest build of Virgo's User Guide.

The commands should be available in the next release of Virgo (the current one being 2.1.0).

Здравейте / Hi

2006-06-25T22:23:00.000+03:00

Първият ми тестов блог

This is my first test blog