Apache Karaf: no matching cipher found: client aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se server

During a recent deployment of a custom Karaf distribution via the karaf-maven-plugin, I've experienced a very strange behaviour when deployed to FreeBSD rather than my developer Mac OS rig.

Connecting thru SSH failed, so the logging has been increased:

dfi:~ doma$ ssh karaf@optiplex1 -p 8100
no matching cipher found: client aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se server 

Here's Karaf's exception logged:

16:24:03,890 INFO  8]-nio2-thread-1 125 shd.server.session.ServerSession Server session created from /192.168.0.21:64951
16:24:03,894 DEBUG 8]-nio2-thread-1 125 shd.server.session.ServerSession Client version string: SSH-2.0-OpenSSH_6.2
16:24:03,900 DEBUG 8]-nio2-thread-1 125 d.common.session.AbstractSession Send SSH_MSG_KEXINIT
16:24:03,901 DEBUG 8]-nio2-thread-1 125 d.common.session.AbstractSession Received SSH_MSG_KEXINIT
16:24:03,902 WARN  8]-nio2-thread-1 125 d.common.session.AbstractSession Exception caught
jjava.lang.IllegalStateException: Unable to negotiate key exchange for encryption algorithms (client to server) (client: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lys
ator.liu.se / server: )
        at org.apache.sshd.common.session.AbstractSession.negotiate(AbstractSession.java:1159)[125:org.apache.sshd.core:0.14.0]
        at org.apache.sshd.common.session.AbstractSession.doHandleMessage(AbstractSession.java:388)[125:org.apache.sshd.core:0.14.0]
        at org.apache.sshd.common.session.AbstractSession.handleMessage(AbstractSession.java:326)[125:org.apache.sshd.core:0.14.0]
        at org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:780)[125:org.apache.sshd.core:0.14.0]
        at org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:308)[125:org.apache.sshd.core:0.14.0]
        at org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54)
        at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:184)
        at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:170)
        at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:32)
        at java.security.AccessController.doPrivileged(Native Method)[:1.8.0_60]
        at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:30)[125:org.apache.sshd.core:0.14.0]
        at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)[:1.8.0_60]
        at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)[:1.8.0_60]
        at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)[:1.8.0_60]
        at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)[:1.8.0_60]
        at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)[:1.8.0_60]
        at java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420)[:1.8.0_60]
        at org.apache.sshd.common.io.nio2.Nio2Session.startReading(Nio2Session.java:170)[125:org.apache.sshd.core:0.14.0]
        at org.apache.sshd.common.io.nio2.Nio2Acceptor$AcceptCompletionHandler.onCompleted(Nio2Acceptor.java:135)[125:org.apache.sshd.core:0.14.0]
        at org.apache.sshd.common.io.nio2.Nio2Acceptor$AcceptCompletionHandler.onCompleted(Nio2Acceptor.java:120)[125:org.apache.sshd.core:0.14.0]
        at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:32)
        at java.security.AccessController.doPrivileged(Native Method)[:1.8.0_60]
        at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:30)[125:org.apache.sshd.core:0.14.0]
        at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)[:1.8.0_60]
        at sun.nio.ch.Invoker$2.run(Invoker.java:218)[:1.8.0_60]
        at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)[:1.8.0_60]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_60]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_60]
        at java.lang.Thread.run(Thread.java:745)[:1.8.0_60]

After trying out all sort of tips read via google, finally realized: setting the JAVA_HOME fixes the issue. To solve for all users I added this line to /etc/profile:

export JAVA_HOME=/usr/local/openjdk7

You can also put the same line to $HOME/.profile to fix it only for one user.

Karaf is beautiful but this one was ugly. Well, my fault: starting Karaf actually complains about not setting JAVA_HOME (and that results may vary), but since I used it like that for quite a while, didn't expect this.

Building and installing Apache Karaf from sources (no sudo needed)

Once upon a time I was working for a company where sudo rights weren't given to developers. So I had to spend a significant fraction of my time finding workarounds, how silly is that. The no-sudo-needed series is the result of all these efforts: I've saved the process, so next time it is needed, it's easy to get back to it.

The point here is to be able to install them just by copying and pasting the code below. Things change around us though, so whatever may build today might just stop working tomorrow. Thus I do my best to keep these code snippets up-to-date.

Like today I need Karaf again. Building from source feels very appropriate, so even though I have sudo on my computers since leaving that company, I still keep coming back here for the "easy builds". Today I updated with Karaf 2.3.11, it is currently building in another window. If succeeds, I'll create another snippet for it. If you need it, feel free to get it!

Building Apache Karaf 2.3.11 from sources

wget http://apache.belnet.be/karaf/2.3.11/apache-karaf-2.3.11-src.tar.gz
tar xvzf apache-karaf-2.3.11-src.tar.gz
cd apache-karaf-2.3.11/src
mvn clean install -DskipTests
mv assemblies/apache-karaf/target/apache-karaf-2.3.11.tar.gz ~/tools
cd ~/tools
tar xvzf apache-karaf-2.3.11.tar.gz
ln -s apache-karaf-2.3.11 karaf

Building Apache Karaf 2.3.6 from sources

wget http://apache.belnet.be/karaf/2.3.6/apache-karaf-2.3.6-src.tar.gz
tar xvzf apache-karaf-2.3.6-src.tar.gz
cd apache-karaf-2.3.6/src
mvn clean install -DskipTests
mv assemblies/apache-karaf/target/apache-karaf-2.3.6.tar.gz ~/tools
cd ~/tools
tar xvzf apache-karaf-2.3.6.tar.gz
ln -s apache-karaf-2.3.6 karaf

Building Apache Karaf 4.0 from sources

git clone https://git-wip-us.apache.org/repos/asf/karaf.git
cd karaf
git checkout karaf-4.0.0
# to avoid "Detected JDK Version: 1.6.0-65 is not in the allowed range [1.7,1.9)" on osx, can only build with 1.7
export JAVA_HOME=`/usr/libexec/java_home -v 1.7`
mvn clean install -DskipTests

Building Karaf 4.0.10 from sources, checking out given branch only, building with java 1.8 on OSX:

git clone -b karaf-4.0.10 --single-branch https://github.com/apache/karaf.git
cd karaf
export JAVA_HOME=`/usr/libexec/java_home -v 1.8`
mvn clean install -DskipTests -DskipITs

Building and installing Apache Karaf 2.2.5 or 2.3.11 from sources (no sudo needed)

In my ~/.m2/settings.xml, I have a separate profile which uses a local Nexus repository as a proxy to all the remotes. It is not active by default, only activated with the -P local switch. Thus please don't update your settings.xml like this if you don't have a local Nexus proxy (and you won't need the "-P local" switch later neither in this case). The relevant part of my settings.xml:

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                          http://maven.apache.org/xsd/settings-1.0.0.xsd">
    <profiles>
        <profile>
            <id>local</id>
            <!--<activation>-->
            <!--<activeByDefault>true</activeByDefault>-->
            <!--</activation>-->
            <properties>
                <repos.release.url>http://127.0.0.1:8081/content/groups/public</repos.release.url>
                <repos.snapshot.url>http://127.0.0.1:8081/content/groups/public</repos.snapshot.url>
            </properties>
            <repositories>
                <repository>
                    <id>nexus</id>
                    <name>public</name>
                    <url>http://127.0.0.1:8081/content/groups/public</url>
                    <layout>default</layout>
                    <snapshots>
                        <enabled>true</enabled>
                        <updatePolicy>always</updatePolicy>
                    </snapshots>
                    <releases>
                        <enabled>true</enabled>
                        <updatePolicy>daily</updatePolicy>
                    </releases>
                </repository>
                <repository>
                    <id>snapshots</id>
                    <name>snapshots</name>
                    <url>http://127.0.0.1:8081/content/repositories/snapshots</url>
                    <layout>default</layout>
                    <snapshots>
                        <enabled>true</enabled>
                        <updatePolicy>always</updatePolicy>
                    </snapshots>
                    <releases>
                        <enabled>false</enabled>
                        <updatePolicy>daily</updatePolicy>
                    </releases>
                </repository>
                <repository>
                    <id>releases</id>
                    <name>releases</name>
                    <url>http://127.0.0.1:8081/content/repositories/releases</url>
                    <layout>default</layout>
                    <snapshots>
                        <enabled>false</enabled>
                    </snapshots>
                    <releases>
                        <enabled>true</enabled>
                        <updatePolicy>daily</updatePolicy>
                    </releases>
                </repository>
            </repositories>

            <pluginRepositories>
                <pluginRepository>
                    <id>nexus</id>
                    <name>public</name>
                    <url>http://127.0.0.1:8081/content/groups/public</url>
                    <layout>default</layout>
                    <snapshots>
                        <enabled>true</enabled>
                        <updatePolicy>always</updatePolicy>
                    </snapshots>
                    <releases>
                        <enabled>true</enabled>
                        <updatePolicy>daily</updatePolicy>
                    </releases>
                </pluginRepository>
                <pluginRepository>
                    <id>snapshots</id>
                    <name>snapshots</name>
                    <url>http://127.0.0.1:8081/content/repositories/snapshots</url>
                    <layout>default</layout>
                    <snapshots>
                        <enabled>true</enabled>
                        <updatePolicy>always</updatePolicy>
                    </snapshots>
                    <releases>
                        <enabled>false</enabled>
                        <updatePolicy>daily</updatePolicy>
                    </releases>
                </pluginRepository>
                <pluginRepository>
                    <id>releases</id>
                    <name>releases</name>
                    <url>http://127.0.0.1:8081/content/repositories/releases</url>
                    <layout>default</layout>
                    <snapshots>
                        <enabled>false</enabled>
                    </snapshots>
                    <releases>
                        <enabled>true</enabled>
                        <updatePolicy>daily</updatePolicy>
                    </releases>
                </pluginRepository>
            </pluginRepositories>
        </profile>
    </profiles>
</settings>

...and the script for the build for version 2.2.5:

mkdir -p ~/tools
cd ~/tools
wget http://archive.apache.org/dist/karaf/2.2.5/apache-karaf-2.2.5-src.tar.gz
tar xvzf apache-karaf-2.2.5-src.tar.gz
cd apache-karaf-2.2.5/src
mvn clean install -DskipTests -P local
sudo mkdir -p /opt
cd /opt
sudo tar xvzf ~/tools/apache-karaf-2.2.5/src/assemblies/apache-karaf/target/apache-karaf-2.2.5.tar.gz
sudo ln -s apache-karaf-2.2.5 karaf

For version 2.3.11

mkdir -p ~/tools
cd ~/tools
wget http://archive.apache.org/dist/karaf/2.3.11/apache-karaf-2.3.11-src.tar.gz
tar xvzf apache-karaf-2.3.11-src.tar.gz
cd apache-karaf-2.3.11/src
mvn clean install -DskipTests -P local
sudo mkdir -p /opt
cd /opt
sudo tar xvzf ~/tools/apache-karaf-2.3.11/src/assemblies/apache-karaf/target/apache-karaf-2.3.11.tar.gz
sudo ln -s apache-karaf-2.3.11 karaf

Autostart on Ubuntu: to wrap or not to wrap

If you try to install the wrapper to achieve autostart, everything seems to be fine, but the wrapper segfaults on subsequent reboots, at least that's the sad situation on Ubuntu 12.04.5. This used to work before.
Still it has to work, so the workaround is to create a /etc/init.d/karaf script with this content:

#!/bin/bash

KARAF_HOME=/home/neusoft/tools/karaf

case "$1" in
  start)
    log_begin_msg "Starting Karaf..."
        $KARAF_HOME/bin/start
    log_end_msg 0
    ;;
  stop)
    log_begin_msg "Shutting down Karaf..."
        $KARAF_HOME/bin/stop
    log_end_msg 0
    ;;
  *)
  ;;
esac

To register the file with the system

sudo update-rc.d karaf defaults 98 02

Autostart on FreeBSD

Using FreeBSD, autostarting the service is different:

echo 'karaf_enable="YES"' | sudo tee -a /etc/rc.conf > /dev/null

The rc.conf entry will start the script under /etc/rc.d. And the contents of the /etc/rc.d/karaf file:

#!/bin/sh
. /etc/rc.subr
export KARAF_OPTS="-Dkey=value"
start_cmd="karaf_start"
stop_cmd="karaf_stop"
karaf_start() {
        export JAVA_HOME=/usr/local/openjdk7
        /opt/karaf/bin/start
}
karaf_stop() {
        export JAVA_HOME=/usr/local/openjdk7
        /opt/karaf/bin/stop
}
load_rc_config $name
run_rc_command "$1"

Then you just have to make it executable:

sudo chmod +x /etc/rc.d/karaf

Or even make a symbolic link for the ones expect logs to be at standard places:

sudo ln -s /opt/karaf/data/log /var/log/karaf

Gotchas

Building on OSX Yosemite with Apple's JDK 1.6 succeeds, but logging on shows that we have a crippled container:

# features:addurl http://repo1.maven.org/maven2/org/apache/felix/org.apache.felix.ipojo.features/1.12.0/org.apache.felix.ipojo.features-1.12.0.xml
# Command not found: features:addurl

Investigating /opt/karaf/data/log/karaf.log shows us the cause of the error:

"Bundle org.apache.karaf.features.command is waiting for dependencies [(objectClass=org.apache.karaf.features.FeaturesService)]"

Installing Oracle JDK 1.7 & setting JAVA_HOME appropriately fixes the problem.

export JAVA_HOME=`/usr/libexec/java_home -v 1.7`
...
(rebuild the whole shebang)
...