10/21/2018

Subpixel anti-aliasing fonts of RHEL75: running IntelliJ IDEA in the remote VM (X11 over SSH) with subpixel anti-aliasing and hinting

It's been a while since my last post. New semester, new project, new technologies; this time I have to work on RHEL 7.5, so might expect some more posts like this.

RedHat introduced the developer program, which allows developers run a properly licensed RedHat Enterprise Linux on their developer machines.

Since I wanted to achieve proper OSX-like fonts when running X11 over SSH (to mirror a problem from work: IntelliJ IDEA running in a remote RHEL box X11 forwarded has blurry fonts), I installed the downloaded DVD image into a VirtualBox VM; I noticed running IntelliJ IDEA in the Gnome3 environment does not have proper hinting and subpixel-antialiasing already, resulting in blurred, distorted fonts.

My dev machine is running MacOS Sierra, and after properly setting up X via macports and setting all the proper parameters for X11 forwarding over SSH, when idea was started from a terminal in SSH, the fonts were the same blurry (e.g. grayscale-anti-aliased, you can check this via the OSX zoom tool by zooming close; if the edges of the line are colored blue-and-red, you are using subpixel already; if the edges are grey, you are obviously using grayscale only). So let's see which version of freetype RHEL is using, that's the library drawing the fonts:

mb:~ doma$ ssh root@rhel
Last login: Sun Oct 21 15:25:07 2018 from 192.168.0.34
[root@rhel ~]# yum list installed | grep freetype
freetype.x86_64                           2.4.11-15.el7            installed    
[root@rhel ~]# 

For me, it was freetype-2.4.11-15.el7. After a bit of searching over the net, it turns out RedHat turns off the bytecode-interpreter and the hinter in freetype by default... even more searching and, I came to the following commands (adapted from https://www.centos.org/forums/viewtopic.php?t=10564):

# original red hat enterprise linux 7.5
# install some pre-requisites
yum install yum-utils libX11-devel gcc rpm-build rpm-sign bzip2-devel libpng-devel
cd ~/tools
wget http://vault.centos.org/7.4.1708/os/Source/SPackages/freetype-2.4.11-15.el7.src.rpm
rpm -ivv freetype-2.4.11-15.el7.src.rpm
cd ~/rpmbuild/SPECS
rpmbuild --sign -ba --with bytecode_interpreter --with subpixel_rendering freetype.spec
# below to avoid "file /usr/lib64/libfreetype.so.6.10.0 from install of freetype-2.4.11-15.el7.x86_64 conflicts with file from package freetype-2.4.11-15.el7.x86_64"
rpm -e --nodeps freetype-2.4.11-15.el7.x86_64
rpm -Uh --replacepkgs ~/rpmbuild/RPMS/x86_64/freetype-2.4.11-15.el7.x86_64.rpm

If you have updated the system with yum update to - I guess here - RHEL7.6, the freetype version will be different. As of March, 2019 the following works perfectly:

yum install yum-utils libX11-devel gcc rpm-build rpm-sign bzip2-devel libpng-devel
cd ~/tools
wget https://mirror.chpc.utah.edu/pub/vault.centos.org/centos/7/updates/Source/SPackages/freetype-2.8-12.el7_6.1.src.rpm
rpm -ivv freetype-2.8-12.el7_6.1.src.rpm
cd ~/rpmbuild/SPECS
rpmbuild --sign -ba --with bytecode_interpreter --with subpixel_rendering freetype.spec
# below to avoid "file /usr/lib64/libfreetype.so.6.10.0 from install of freetype-2.8-12.el7_6.1.x86_64 conflicts with file from package freetype-2.8-12.el7_6.1.x86_64"
rpm -e --nodeps freetype-2.8-12.el7_6.1.x86_64
rpm -Uh --replacepkgs ~/rpmbuild/RPMS/x86_64/freetype-2.8-12.el7.1.x86_64.rpm

If you have updated the system with yum update to 7.7, the freetype version will again be different. As of Sept, 2019 the following works perfectly:

yum install yum-utils libX11-devel gcc rpm-build rpm-sign bzip2-devel libpng-devel
cd ~/tools
wget https://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackageSource/freetype-2.8-14.el7.src.rpm freetype-2.8-14.el7.src.rpm
rpm -ivv freetype-2.8-14.el7.src.rpm
cd ~/rpmbuild/SPECS
rpmbuild --sign -ba --with bytecode_interpreter --with subpixel_rendering freetype.spec
# below to avoid "file /usr/lib64/libfreetype.so.6.10.0 from install of freetype-2.8-14.el7.x86_64 conflicts with file from package freetype-2.8-14.el7.x86_64"
rpm -e --nodeps freetype-2.8-14.el7.x86_64
rpm -Uh --replacepkgs ~/rpmbuild/RPMS/x86_64/freetype-2.8-14.el7.x86_64.rpm

To configure the nice fonts (e.g. instruct the system to use the subpixel and hinting now enabled in freetype), add the following to /etc/fonts/local.conf :

<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
  <!-- settings go here -->
  <match target="font">
    <edit name="antialias" mode="assign">
      <bool>true</bool>
    </edit>
  </match>
  <match target="font">
    <edit name="hinting" mode="assign">
      <bool>true</bool>
    </edit>
  </match>
  <match target="font">
    <edit name="autohint" mode="assign">
      <bool>false</bool>
    </edit>
  </match>
  <match target="font">
    <edit name="hintstyle" mode="assign">
      <const>hintslight</const>
    </edit>
  </match>
  <match target="font">
    <edit name="rgba" mode="assign">
      <const>rgb</const>
    </edit>
  </match>
  <match target="font">
    <edit name="lcdfilter" mode="assign">
      <const>lcddefault</const>
    </edit>
  </match>
</fontconfig>

This has fixed the blurry fonts in Gnome3 globally (as well as IntelliJ IDEA, so we're good to go) after a restart of the RHEL VM.

And the good news: this has also fixed the problem when running X11 over ssh:

mb:~ doma$ ssh -Y rhel
Last login: Sun Oct 21 15:12:05 2018 from 192.168.0.34
[doma@rhel ~]$ /opt/idea/bin/idea.sh 
Everything has changed
Everything has changed

And IntelliJ IDEA looks like this with Liberation fonts selected for UI:

That's a good start. Not quite like on osx, but close enough... This is when OSX is rendering, with the Optimus font:

With zoom it is apparent that both are using subpixel now.

This was OSX with the Optimus font. See the colorful edges? That's subpixel AA in action... And here comes the RHEL VM, rendered via Macport's X11 (but it would look the same in gnome - e.g in the native desktop environment - as earlier tests show...):

The difference is due to Retina display on my macbook; a quick xrandr reveals that XQuartz thinks we're running in 1680x1050, whilst the display's native resolution is 2880x1800. If you go to System Preferences -> Displays, under Resolution selecting Scaled -> More Space actually increases the fake resolution passed by OSX to X11 to 1920x200:

Last login: Tue Nov  6 20:01:43 on ttys004
            _     
           | |    
  _ __ ___ | |__  
 | '_ ` _ \| '_ \ 
 | | | | | | |_) |
 |_| |_| |_|_.__/
 
mb:~ doma$ xrandr -q
xrandr: Failed to get size of gamma for output default
Screen 0: minimum 640 x 480, current 1920 x 1200, maximum 3360 x 2100
default connected 1920x1200+0+0 0mm x 0mm
   2880x1800     60.00  
   1440x900      60.00  
   3360x2100     60.00  
   2560x1600     60.00  
   2048x1280     60.00  
   1650x1050     60.00  
   1280x800      60.00  
   1152x720      60.00  
   1024x768      60.00  
   840x524       60.00  
   800x600       60.00  
   640x480       60.00  
   1920x1178      1.00  
   1920x1200      2.00* 
mb:~ doma$ 

The screen looks much better like this. Some articles suggest that xrandr can actually change the resolution; in my case unfortunately there's a segmentation fault when trying to do so. So for the moment X11 will be displayed as such (not that bad anyway) - in my work, using MobaXTerm on Windows10 the fonts look quite good, not exhibiting the problem caused bby the retina display on osx.

Did it work for you? Please consider leaving a comment for others - maybe even with screenshot - , so they will be encouraged to rebuild their freetype as well given they consider the default grayscale antialiasing blurry.