10/21/2018

Subpixel anti-aliasing fonts of RHEL75: running IntelliJ IDEA in the remote VM (X11 over SSH) with subpixel anti-aliasing and hinting

It's been a while since my last post. New semester, new project, new technologies; this time I have to work on RHEL 7.5, so might expect some more posts like this.

RedHat introduced the developer program, which allows developers run a properly licensed RedHat Enterprise Linux on their developer machines.

Since I wanted to achieve proper OSX-like fonts when running X11 over SSH (to mirror a problem from work: IntelliJ IDEA running in a remote RHEL box X11 forwarded has blurry fonts), I installed the downloaded DVD image into a VirtualBox VM; I noticed running IntelliJ IDEA in the Gnome3 environment does not have proper hinting and subpixel-antialiasing already, resulting in blurred, distorted fonts.

My dev machine is running MacOS Sierra, and after properly setting up X via macports and setting all the proper parameters for X11 forwarding over SSH, when idea was started from a terminal in SSH, the fonts were the same blurry (e.g. grayscale-anti-aliased, you can check this via the OSX zoom tool by zooming close; if the edges of the line are colored blue-and-red, you are using subpixel already; if the edges are grey, you are obviously using grayscale only). So let's see which version of freetype RHEL is using, that's the library drawing the fonts:

mb:~ doma$ ssh root@rhel
Last login: Sun Oct 21 15:25:07 2018 from 192.168.0.34
[root@rhel ~]# yum list installed | grep freetype
freetype.x86_64                           2.4.11-15.el7            installed    
[root@rhel ~]# 

For me, it was freetype-2.4.11-15.el7. After a bit of searching over the net, it turns out RedHat turns off the bytecode-interpreter and the hinter in freetype by default... even more searching and, I came to the following commands (adapted from https://www.centos.org/forums/viewtopic.php?t=10564):

# original red hat enterprise linux 7.5
# install some pre-requisites
yum install yum-utils libX11-devel gcc rpm-build rpm-sign bzip2-devel libpng-devel
cd ~/tools
wget http://vault.centos.org/7.4.1708/os/Source/SPackages/freetype-2.4.11-15.el7.src.rpm
rpm -ivv freetype-2.4.11-15.el7.src.rpm
cd ~/rpmbuild/SPECS
rpmbuild --sign -ba --with bytecode_interpreter --with subpixel_rendering freetype.spec
# below to avoid "file /usr/lib64/libfreetype.so.6.10.0 from install of freetype-2.4.11-15.el7.x86_64 conflicts with file from package freetype-2.4.11-15.el7.x86_64"
rpm -e --nodeps freetype-2.4.11-15.el7.x86_64
rpm -Uh --replacepkgs ~/rpmbuild/RPMS/x86_64/freetype-2.4.11-15.el7.x86_64.rpm

If you have updated the system with yum update to - I guess here - RHEL7.6, the freetype version will be different. As of March, 2019 the following works perfectly:

yum install yum-utils libX11-devel gcc rpm-build rpm-sign bzip2-devel libpng-devel
cd ~/tools
wget https://mirror.chpc.utah.edu/pub/vault.centos.org/centos/7/updates/Source/SPackages/freetype-2.8-12.el7_6.1.src.rpm
rpm -ivv freetype-2.8-12.el7_6.1.src.rpm
cd ~/rpmbuild/SPECS
rpmbuild --sign -ba --with bytecode_interpreter --with subpixel_rendering freetype.spec
# below to avoid "file /usr/lib64/libfreetype.so.6.10.0 from install of freetype-2.8-12.el7_6.1.x86_64 conflicts with file from package freetype-2.8-12.el7_6.1.x86_64"
rpm -e --nodeps freetype-2.8-12.el7_6.1.x86_64
rpm -Uh --replacepkgs ~/rpmbuild/RPMS/x86_64/freetype-2.8-12.el7.1.x86_64.rpm

If you have updated the system with yum update to 7.7, the freetype version will again be different. As of Sept, 2019 the following works perfectly:

yum install yum-utils libX11-devel gcc rpm-build rpm-sign bzip2-devel libpng-devel
cd ~/tools
wget https://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackageSource/freetype-2.8-14.el7.src.rpm freetype-2.8-14.el7.src.rpm
rpm -ivv freetype-2.8-14.el7.src.rpm
cd ~/rpmbuild/SPECS
rpmbuild --sign -ba --with bytecode_interpreter --with subpixel_rendering freetype.spec
# below to avoid "file /usr/lib64/libfreetype.so.6.10.0 from install of freetype-2.8-14.el7.x86_64 conflicts with file from package freetype-2.8-14.el7.x86_64"
rpm -e --nodeps freetype-2.8-14.el7.x86_64
rpm -Uh --replacepkgs ~/rpmbuild/RPMS/x86_64/freetype-2.8-14.el7.x86_64.rpm

To configure the nice fonts (e.g. instruct the system to use the subpixel and hinting now enabled in freetype), add the following to /etc/fonts/local.conf :

<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
  <!-- settings go here -->
  <match target="font">
    <edit name="antialias" mode="assign">
      <bool>true</bool>
    </edit>
  </match>
  <match target="font">
    <edit name="hinting" mode="assign">
      <bool>true</bool>
    </edit>
  </match>
  <match target="font">
    <edit name="autohint" mode="assign">
      <bool>false</bool>
    </edit>
  </match>
  <match target="font">
    <edit name="hintstyle" mode="assign">
      <const>hintslight</const>
    </edit>
  </match>
  <match target="font">
    <edit name="rgba" mode="assign">
      <const>rgb</const>
    </edit>
  </match>
  <match target="font">
    <edit name="lcdfilter" mode="assign">
      <const>lcddefault</const>
    </edit>
  </match>
</fontconfig>

This has fixed the blurry fonts in Gnome3 globally (as well as IntelliJ IDEA, so we're good to go) after a restart of the RHEL VM.

And the good news: this has also fixed the problem when running X11 over ssh:

mb:~ doma$ ssh -Y rhel
Last login: Sun Oct 21 15:12:05 2018 from 192.168.0.34
[doma@rhel ~]$ /opt/idea/bin/idea.sh 
Everything has changed
Everything has changed

And IntelliJ IDEA looks like this with Liberation fonts selected for UI:

That's a good start. Not quite like on osx, but close enough... This is when OSX is rendering, with the Optimus font:

With zoom it is apparent that both are using subpixel now.

This was OSX with the Optimus font. See the colorful edges? That's subpixel AA in action... And here comes the RHEL VM, rendered via Macport's X11 (but it would look the same in gnome - e.g in the native desktop environment - as earlier tests show...):

The difference is due to Retina display on my macbook; a quick xrandr reveals that XQuartz thinks we're running in 1680x1050, whilst the display's native resolution is 2880x1800. If you go to System Preferences -> Displays, under Resolution selecting Scaled -> More Space actually increases the fake resolution passed by OSX to X11 to 1920x200:

Last login: Tue Nov  6 20:01:43 on ttys004
            _     
           | |    
  _ __ ___ | |__  
 | '_ ` _ \| '_ \ 
 | | | | | | |_) |
 |_| |_| |_|_.__/
 
mb:~ doma$ xrandr -q
xrandr: Failed to get size of gamma for output default
Screen 0: minimum 640 x 480, current 1920 x 1200, maximum 3360 x 2100
default connected 1920x1200+0+0 0mm x 0mm
   2880x1800     60.00  
   1440x900      60.00  
   3360x2100     60.00  
   2560x1600     60.00  
   2048x1280     60.00  
   1650x1050     60.00  
   1280x800      60.00  
   1152x720      60.00  
   1024x768      60.00  
   840x524       60.00  
   800x600       60.00  
   640x480       60.00  
   1920x1178      1.00  
   1920x1200      2.00* 
mb:~ doma$ 

The screen looks much better like this. Some articles suggest that xrandr can actually change the resolution; in my case unfortunately there's a segmentation fault when trying to do so. So for the moment X11 will be displayed as such (not that bad anyway) - in my work, using MobaXTerm on Windows10 the fonts look quite good, not exhibiting the problem caused bby the retina display on osx.

Did it work for you? Please consider leaving a comment for others - maybe even with screenshot - , so they will be encouraged to rebuild their freetype as well given they consider the default grayscale antialiasing blurry.

5/18/2017

Building IntelliJ Idea Community edition on Ubuntu 16.10

Building IntelliJ Idea on Ubuntu is just a bit more complex than simply executing ant.

Acquiring dependencies

The commented lines below are required if you wish to use jdk1.6; in practice, things seem to build without (as of May, 2017 at least)

#sudo add-apt-repository ppa:webupd8team/java
#sudo apt-get update
#sudo apt-get install oracle-java6-installer
#sudo update-alternatives --config java
sudo apt-get install zlib1g-dev ant git
mkdir $HOME/tools
cd $HOME/tools
git clone git://git.jetbrains.org/idea/community.git idea
cd idea
./getPlugins.sh 
build/conf/install_nsis3.sh $HOME/tools/idea

If the build finishes with

...
scons: done building targets.
...the build of dependencies finished successfully.

Starting the build

If you've installed JDK1.6 (the commented lines above), here we need to refer to it (first line below). Again, IntelliJ Idea Community Edition builds without this, so left it commented out.

#export JDK_16_x64=/usr/lib/jvm/java-6-oracle
cd $HOME/tools/idea
ant

1/16/2017

Color-highlight given regex in MacOS 10.10 (Yosemite) or later

Using grep to filter commands' output is as easy as this:

mvn dependency:tree | grep selenium
Downloading: http://nexus.m44studio.com:8081/repository/maven-public/org/hpsl/common-selenium-integration-test/1.1.129-SNAPSHOT/maven-metadata.xml
Downloaded: http://nexus.m44studio.com:8081/repository/maven-public/org/hpsl/common-selenium-integration-test/1.1.129-SNAPSHOT/maven-metadata.xml (2 KB at 133.6 KB/sec)
[INFO] +- org.hpsl:common-selenium-integration-test:jar:1.1.129-SNAPSHOT:compile
[INFO] |  +- org.seleniumhq.selenium:selenium-firefox-driver:jar:3.0.1:compile
[INFO] |  |  \- org.seleniumhq.selenium:selenium-remote-driver:jar:3.0.1:compile
[INFO] |  |     +- org.seleniumhq.selenium:selenium-api:jar:3.0.1:compile
[INFO] |  \- org.seleniumhq.selenium:selenium-java:jar:3.0.1:compile
[INFO] |     +- org.seleniumhq.selenium:selenium-chrome-driver:jar:3.0.1:compile
[INFO] |     +- org.seleniumhq.selenium:selenium-edge-driver:jar:3.0.1:compile
[INFO] |     +- org.seleniumhq.selenium:selenium-ie-driver:jar:3.0.1:compile
[INFO] |     +- org.seleniumhq.selenium:selenium-opera-driver:jar:3.0.1:compile
[INFO] |     +- org.seleniumhq.selenium:selenium-safari-driver:jar:3.0.1:compile
[INFO] |     +- org.seleniumhq.selenium:selenium-support:jar:3.0.1:compile

Well as the example above is using maven to show dependencies for a given project, filtered only for selenium - what if you want to see all the dependencies, having selenium highlighted with color? Here we go:

mvn dependency:tree | grep --color -E 'selenium|$'
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ integration-tests ---
[INFO] com.doma.blueprint.cdi:integration-tests:jar:1.0.73-SNAPSHOT
[INFO] +- com.doma.blueprint.cdi:model:jar:1.0.73-SNAPSHOT:compile
[INFO] |  \- com.doma.annotations.blueprint.cdi:annotations:jar:1.0.73-SNAPSHOT:compile
[INFO] +- org.eclipse.persistence:org.eclipse.persistence.moxy:jar:2.6.0:compile
[INFO] |  +- org.eclipse.persistence:org.eclipse.persistence.core:jar:2.6.0:compile
[INFO] |  |  \- org.eclipse.persistence:org.eclipse.persistence.asm:jar:2.6.0:compile
[INFO] |  +- javax.validation:validation-api:jar:1.1.0.Final:compile
[INFO] |  \- org.glassfish:javax.json:jar:1.0.4:compile
[INFO] +- org.apache.httpcomponents:httpclient:jar:4.5.2:compile
[INFO] |  +- org.apache.httpcomponents:httpcore:jar:4.4.4:compile
[INFO] |  +- commons-logging:commons-logging:jar:1.2:compile
[INFO] |  \- commons-codec:commons-codec:jar:1.9:compile
[INFO] +- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO] +- junit:junit:jar:4.12:compile
[INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:compile
[INFO] +- org.slf4j:jcl-over-slf4j:jar:1.7.7:compile
[INFO] +- org.slf4j:log4j-over-slf4j:jar:1.7.5:compile
[INFO] +- ch.qos.logback:logback-classic:jar:1.1.3:compile
[INFO] |  \- ch.qos.logback:logback-core:jar:1.1.3:compile
[INFO] +- com.consol.citrus:citrus-core:jar:2.5.1:test
[INFO] |  +- com.consol.citrus.model:citrus-model-core:jar:2.5.1:test
[INFO] |  |  +- javax.xml.bind:jaxb-api:jar:2.2.11:test
[INFO] |  |  +- com.sun.xml.bind:jaxb-core:jar:2.2.11:test
[INFO] |  |  \- com.sun.xml.bind:jaxb-impl:jar:2.2.11:test
[INFO] |  +- commons-cli:commons-cli:jar:1.3.1:test
[INFO] |  +- commons-lang:commons-lang:jar:2.6:test
[INFO] |  +- xalan:xalan:jar:2.7.2:test
[INFO] |  |  \- xalan:serializer:jar:2.7.2:test
[INFO] |  +- xerces:xercesImpl:jar:2.11.0:test
[INFO] |  |  \- xml-apis:xml-apis:jar:1.4.01:test
[INFO] |  +- wsdl4j:wsdl4j:jar:1.6.3:test
[INFO] |  +- net.sf.jtidy:jtidy:jar:r938:test
[INFO] |  +- org.codehaus.groovy:groovy:jar:2.4.5:test
[INFO] |  +- org.codehaus.groovy:groovy-xml:jar:2.4.5:test
[INFO] |  +- org.codehaus.groovy:groovy-json:jar:2.4.5:test
[INFO] |  +- org.codehaus.groovy:groovy-jsr223:jar:2.4.5:test
[INFO] |  +- commons-logging:commons-logging-api:jar:1.1:test
[INFO] |  +- com.fasterxml.jackson.core:jackson-core:jar:2.6.3:test
[INFO] |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.6.3:test
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.6.3:test
[INFO] |  +- net.minidev:json-smart:jar:2.2.1:test
[INFO] |  |  \- net.minidev:accessors-smart:jar:1.1:test
[INFO] |  |     \- org.ow2.asm:asm:jar:5.0.3:test
[INFO] |  +- com.jayway.jsonpath:json-path:jar:2.0.0:test
[INFO] |  +- org.springframework:spring-core:jar:4.2.3.RELEASE:test
[INFO] |  +- org.springframework:spring-context:jar:4.2.3.RELEASE:test
[INFO] |  |  +- org.springframework:spring-aop:jar:4.2.3.RELEASE:test
[INFO] |  |  |  \- aopalliance:aopalliance:jar:1.0:test
[INFO] |  |  +- org.springframework:spring-beans:jar:4.2.3.RELEASE:test
[INFO] |  |  \- org.springframework:spring-expression:jar:4.2.3.RELEASE:test
[INFO] |  +- org.springframework:spring-jdbc:jar:4.2.3.RELEASE:test
[INFO] |  |  \- org.springframework:spring-tx:jar:4.2.3.RELEASE:test
[INFO] |  +- org.springframework.integration:spring-integration-core:jar:4.2.2.RELEASE:test
[INFO] |  |  +- org.springframework.retry:spring-retry:jar:1.1.2.RELEASE:test
[INFO] |  |  \- org.springframework:spring-messaging:jar:4.2.3.RELEASE:test
[INFO] |  +- org.springframework.ws:spring-xml:jar:2.2.3.RELEASE:test
[INFO] |  +- org.springframework:spring-oxm:jar:4.2.3.RELEASE:test
[INFO] |  \- org.springframework:spring-test:jar:4.2.3.RELEASE:test
[INFO] +- org.hpsl:common-selenium-integration-test:jar:1.1.129-SNAPSHOT:compile
[INFO] |  +- com.github.tntim96:JSCover:jar:1.0.24:compile
[INFO] |  |  \- org.mozilla:rhino:jar:1.7.7.1:compile
[INFO] |  +- org.seleniumhq.selenium:selenium-firefox-driver:jar:3.0.1:compile
[INFO] |  |  \- org.seleniumhq.selenium:selenium-remote-driver:jar:3.0.1:compile
[INFO] |  |     +- org.seleniumhq.selenium:selenium-api:jar:3.0.1:compile
[INFO] |  |     +- cglib:cglib-nodep:jar:3.2.4:compile
[INFO] |  |     +- org.apache.commons:commons-exec:jar:1.3:compile
[INFO] |  |     +- com.google.code.gson:gson:jar:2.3.1:compile
[INFO] |  |     +- com.google.guava:guava:jar:19.0:compile
[INFO] |  |     +- org.apache.httpcomponents:httpmime:jar:4.5.2:compile
[INFO] |  |     \- net.java.dev.jna:jna-platform:jar:4.1.0:compile
[INFO] |  |        \- net.java.dev.jna:jna:jar:4.1.0:compile
[INFO] |  +- com.codeborne:phantomjsdriver:jar:1.3.0:compile
[INFO] |  \- org.seleniumhq.selenium:selenium-java:jar:3.0.1:compile
[INFO] |     +- org.seleniumhq.selenium:selenium-chrome-driver:jar:3.0.1:compile
[INFO] |     +- org.seleniumhq.selenium:selenium-edge-driver:jar:3.0.1:compile
[INFO] |     +- org.seleniumhq.selenium:selenium-ie-driver:jar:3.0.1:compile
[INFO] |     +- org.seleniumhq.selenium:selenium-opera-driver:jar:3.0.1:compile
[INFO] |     +- org.seleniumhq.selenium:selenium-safari-driver:jar:3.0.1:compile
[INFO] |     |  \- io.netty:netty:jar:3.5.7.Final:compile
[INFO] |     +- org.seleniumhq.selenium:selenium-support:jar:3.0.1:compile
[INFO] |     +- net.sourceforge.htmlunit:htmlunit:jar:2.23:compile
[INFO] |     |  +- org.apache.commons:commons-lang3:jar:3.4:compile
[INFO] |     |  +- net.sourceforge.htmlunit:htmlunit-core-js:jar:2.23:compile
[INFO] |     |  +- net.sourceforge.htmlunit:neko-htmlunit:jar:2.23:compile
[INFO] |     |  +- net.sourceforge.cssparser:cssparser:jar:0.9.20:compile
[INFO] |     |  |  \- org.w3c.css:sac:jar:1.3:compile
[INFO] |     |  \- commons-io:commons-io:jar:2.5:compile
[INFO] |     \- org.eclipse.jetty.websocket:websocket-client:jar:9.2.15.v20160210:compile
[INFO] |        +- org.eclipse.jetty:jetty-util:jar:9.2.15.v20160210:compile
[INFO] |        +- org.eclipse.jetty:jetty-io:jar:9.2.15.v20160210:compile
[INFO] |        \- org.eclipse.jetty.websocket:websocket-common:jar:9.2.15.v20160210:compile
[INFO] |           \- org.eclipse.jetty.websocket:websocket-api:jar:9.2.15.v20160210:compile
[INFO] +- org.ops4j.pax.cdi:pax-cdi-api:jar:0.5.0:compile
[INFO] |  \- org.osgi:org.osgi.compendium:jar:4.3.1:compile
[INFO] +- javax.inject:javax.inject:jar:1:compile
[INFO] +- javax.transaction:javax.transaction-api:jar:1.2:compile
[INFO] +- org.hibernate.javax.persistence:hibernate-jpa-2.1-api:jar:1.0.0.Final:compile
[INFO] +- org.slf4j:slf4j-api:jar:1.7.7:compile
[INFO] +- org.osgi:org.osgi.core:jar:4.3.1:compile
[INFO] \- log4j:log4j:jar:1.2.17:test
Tested with Mac OSX 10.10 Yosemite. Enjoy!

10/21/2016

Gitlab on FreeBSD: Upgrading the port tree causes Shared object "libicui18n.so.55" not found, required by "charlock_holmes.so"

If you are just blindly updating FreeBDS's port tree with

portmaster --no-confirm -aD

Gitlab may fail to start due to a newer version of icui18n.so installed on the system, failing to find the old one.

This is how the error manifests itself:

# less ~/gitlab/log/unicorn.stderr.log

/usr/home/git/gitlab/vendor/bundle/ruby/2.2/gems/activesupport-4.1.11/lib/active_support/dependencies.rb:247:in `require': Shared object "libicui18n.so.55" not found, required by "charlock_holmes.so" - /usr/home/git/gitlab/vendor/bundle/ruby/2.2/gems/charlock_holmes-0.6.9.4/lib/charlock_holmes/charlock_holmes.so (LoadError)
        from /usr/home/git/gitlab/vendor/bundle/ruby/2.2/gems/activesupport-4.1.11/lib/active_support/dependencies.rb:247:in `block in require'
        from /usr/home/git/gitlab/vendor/bundle/ruby/2.2/gems/activesupport-4.1.11/lib/active_support/dependencies.rb:232:in `load_dependency'
        from /usr/home/git/gitlab/vendor/bundle/ruby/2.2/gems/activesupport-4.1.11/lib/active_support/dependencies.rb:247:in `require'
...

The fix is to reinstall the charlock_holmes gem, and then gitlab-shell:

cd gitlab
bundle exec gem uninstall charlock_holmes
bundle install
bundle exec rake gitlab:shell:setup RAILS_ENV=production

Then start gitlab (obviously with sudo rights):

service gitlab start

Feel free to drop a line below if the fix worked for you. Cheers!

References:

https://github.com/brianmario/charlock_holmes/issues/48

https://github.com/brianmario/charlock_holmes/issues/7

9/10/2016

Redmine fails to start: There was an error while trying to load the gem 'rbpdf'. Gem Load Error is: This installation of RMagick was configured with ImageMagick 6.9.4 but ImageMagick 6.9.5-5 is in use.

After redmine is installed and FreeBSD's package-tree was updated, Redmine did not start (with some nginx error in the browser), and investigating the error log:
less /var/log/nginx/error.log
...
  Message from application: There was an error while trying to load the gem 'rbpdf'.

Gem Load Error is: This installation of RMagick was configured with ImageMagick 6.9.4 but ImageMagick 6.9.5-5 is in use.

Backtrace for gem load error is:

/usr/local/lib/ruby/gems/2.2/gems/activesupport-4.2.7/lib/active_support/dependencies.rb:274:in `require'

/usr/local/lib/ruby/gems/2.2/gems/activesupport-4.2.7/lib/active_support/dependencies.rb:274:in `block in require'

/usr/local/lib/ruby/gems/2.2/gems/activesupport-4.2.7/lib/active_support/dependencies.rb:240:in `load_dependency'

/usr/local/lib/ruby/gems/2.2/gems/activesupport-4.2.7/lib/active_support/dependencies.rb:274:in `require'
...
led to the conclusion that we need to reinstall things here. So tried a remove of the concerned modules, and reinstalled the bundle:
cd /usr/local/www/redmine
gem uninstall rmagick
gem uninstall rpdf
bundle install
service nginx restart
And voila, redmine is back online.

11/26/2015

Updating FreeBSD's ports: a fully automated script to (slowly) rebuild everything

FreeBSD is a really good operating system to work with, but the daily process of keeping the port tree up-to-date could easily become a painful process.

Whenever any developer moves an important file between ports (happens a lot with Gnome3) a simple 'portmaster -a' will fail, complaining the old- and the new package willing to write to the same place.

You can always fix the issue by removing the whole package and reinstalling, but it breaks the anyway very long (e.g. hours) update procedure, and you'll have to wait another couple fo hours for the update to finish... Until you run into another package-move, and have to start over again.

After working with FreeBSD for a while, the most useful update procedure turns out to be similar to what they propose in 'man portmaster' :

     Using portmaster to do a complete reinstallation of all your ports:
           1. portmaster --list-origins > ~/installed-port-list
           2. Update your ports tree
           3. portmaster -ty --clean-distfiles
           4. portmaster --check-port-dbdir
           5. portmaster -Faf
           6. pkg_delete -a
           7. rm -rf /usr/local/lib/compat/pkg
           8. Back up any files in /usr/local you wish to save,
              such as configuration files in /usr/local/etc
           9. Manually check /usr/local and /var/db/pkg
              to make sure that they are really empty
           10. Re-install portmaster
           11. portmaster `cat ~/installed-port-list`

     You probably want to use the -D option for the installation and then run
     --clean-distfiles [-y] again when you are done.  You might also want to
     consider using the --force-config option when installing the new ports.

     Alternatively you could use portmaster -a -f -D to do an ``in place''
     update of your ports.  If that process is interrupted for any reason you
     can use portmaster -a -f -D -R to avoid rebuilding ports already rebuilt
     on previous runs.  However the first method (delete everything and rein-
     stall) is preferred.

Well, this painful (e.g. if you have lots of packages running on not-state-of-the-art hardware, could be up to 24 hours, or sometimes days) process really works. Whatever entanglement I've had with gnome packages - reinstalling everything obviously always helped.

Some time ago I bought 6 used DELL desktops to set up a home-made cluster, and today I had to update their packages, because I need to install a new port on them.

The automatic installation via 'portmaster -a' failed. As I really didn't want to go over the painful solving of the problem on each machine, I rather decided to use portmaster's documentation's suggestion, e.g. to reinstall everything on those rigs.

Which, unfurtonately turns out not to be completely automatic, and rather uncomfortable. The automatism is jeopardized by portmaster and some other tools stopping and asking for the user's confirmation over tty1, which again breaks a very long procedure.

Since the original inception of portmaster they might had different default swithches, but as of FreeBSD 10.1 and 10.2, the following script turns out to be working without interruption:

portmaster --list-origins > ~/installed-port-list
portsnap fetch update
# backup /var/db/ports with our existing options
cd /var/db
tar cvzf ports.tar.gz ports
portmaster -ty --clean-distfiles
portmaster --check-port-dbdir -y
portmaster -Faf
pkg delete -a -y
rm -rf /usr/local/lib/compat/pkg
rm -rf /var/db/pkg
cd /var/db
tar xvzf ports.tar.gz
cd /usr/ports/ports-mgmt/pkg
make install clean
cd /usr/ports/ports-mgmt/portmaster
make install clean
cd ~
portmaster -d -y --no-confirm --delete-packages --update-if-newer `cat ~/installed-port-list`

The changes needed were mainly the portmaster flags, but the /var/db/ports tree is also backed up & restored: that's the location where the answers to those blue 'dialog4ports' windows' are stored:

...
cd /var/db
tar cvzf ports.tar.gz ports
...
cd /var/db
tar xvzf ports.tar.gz
...

It is very long to do it this way, as we delete and recompile every package of the system. But as it can be completely automated with the script above, I don't mind letting the computer do it's job overnight if that saves me some trouble - e.g. solving version-issues between recently updated FreeBSD port packages...

Let us know how you manage to automatically update your FreeBSD boxes in the comment section below. Portmaster's suggestion is "brutal", but is quite effective in the end... I'm sure there are other solutions as well.


11/23/2015

Apache Karaf: no matching cipher found: client aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se server

During a recent deployment of a custom Karaf distribution via the karaf-maven-plugin, I've experienced a very strange behaviour when deployed to FreeBSD rather than my developer Mac OS rig.

Connecting thru SSH failed, so the logging has been increased:

dfi:~ doma$ ssh karaf@optiplex1 -p 8100
no matching cipher found: client aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se server 

Here's Karaf's exception logged:

16:24:03,890 INFO  8]-nio2-thread-1 125 shd.server.session.ServerSession Server session created from /192.168.0.21:64951
16:24:03,894 DEBUG 8]-nio2-thread-1 125 shd.server.session.ServerSession Client version string: SSH-2.0-OpenSSH_6.2
16:24:03,900 DEBUG 8]-nio2-thread-1 125 d.common.session.AbstractSession Send SSH_MSG_KEXINIT
16:24:03,901 DEBUG 8]-nio2-thread-1 125 d.common.session.AbstractSession Received SSH_MSG_KEXINIT
16:24:03,902 WARN  8]-nio2-thread-1 125 d.common.session.AbstractSession Exception caught
jjava.lang.IllegalStateException: Unable to negotiate key exchange for encryption algorithms (client to server) (client: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lys
ator.liu.se / server: )
        at org.apache.sshd.common.session.AbstractSession.negotiate(AbstractSession.java:1159)[125:org.apache.sshd.core:0.14.0]
        at org.apache.sshd.common.session.AbstractSession.doHandleMessage(AbstractSession.java:388)[125:org.apache.sshd.core:0.14.0]
        at org.apache.sshd.common.session.AbstractSession.handleMessage(AbstractSession.java:326)[125:org.apache.sshd.core:0.14.0]
        at org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:780)[125:org.apache.sshd.core:0.14.0]
        at org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:308)[125:org.apache.sshd.core:0.14.0]
        at org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54)
        at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:184)
        at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:170)
        at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:32)
        at java.security.AccessController.doPrivileged(Native Method)[:1.8.0_60]
        at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:30)[125:org.apache.sshd.core:0.14.0]
        at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)[:1.8.0_60]
        at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)[:1.8.0_60]
        at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)[:1.8.0_60]
        at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)[:1.8.0_60]
        at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)[:1.8.0_60]
        at java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420)[:1.8.0_60]
        at org.apache.sshd.common.io.nio2.Nio2Session.startReading(Nio2Session.java:170)[125:org.apache.sshd.core:0.14.0]
        at org.apache.sshd.common.io.nio2.Nio2Acceptor$AcceptCompletionHandler.onCompleted(Nio2Acceptor.java:135)[125:org.apache.sshd.core:0.14.0]
        at org.apache.sshd.common.io.nio2.Nio2Acceptor$AcceptCompletionHandler.onCompleted(Nio2Acceptor.java:120)[125:org.apache.sshd.core:0.14.0]
        at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:32)
        at java.security.AccessController.doPrivileged(Native Method)[:1.8.0_60]
        at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:30)[125:org.apache.sshd.core:0.14.0]
        at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)[:1.8.0_60]
        at sun.nio.ch.Invoker$2.run(Invoker.java:218)[:1.8.0_60]
        at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)[:1.8.0_60]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_60]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_60]
        at java.lang.Thread.run(Thread.java:745)[:1.8.0_60]

After trying out all sort of tips read via google, finally realized: setting the JAVA_HOME fixes the issue. To solve for all users I added this line to /etc/profile:

export JAVA_HOME=/usr/local/openjdk7

You can also put the same line to $HOME/.profile to fix it only for one user.

Karaf is beautiful but this one was ugly. Well, my fault: starting Karaf actually complains about not setting JAVA_HOME (and that results may vary), but since I used it like that for quite a while, didn't expect this.