Faking a debian repository for package development

I use aptly to deliver my unofficial debian packages both to myself and others that might be interested.

However I’ve found that using aptly to do package development is a bad idea, because you can’t (by design, probably) overwrite packages in an aptly archive.  You can only create new versions.

For some installation tests it’s OK to use “dpkg –install”. But if your package needs to pull in depdencies, or if you wish to test a package upgrade, you need to use APT.

This article explains how to create a fake debian repository for use in package development.

Initial setup

Initial setup:

  1. Create the repository directory (this should be done as the same user you use to build the package)
    mkdir -p /tmp/repo
  2. Open /etc/apt/sources.list in a text editor and add the following (you need to be root to do this)
    # Test install apt archive
    deb file:///tmp repo/
    

Add new package to the repo

This the development cycle:

  1. Build the package (the example builds karaf)
    cd ~/git/karaf-debian/
    dpkg-buildpackage
  2. Copy the package to the fake repo (this example uses karaf, replace with your own package and package version):
    cp ~/git/karaf_4.1.5-0~9.30_all.deb /tmp/repo
  3. Generate a Packages file
    (cd /tmp; dpkg-scanpackages repo >repo/Packages)

Upgrading an existing package

This has to be done as root:

  1. First update APTs database of packages and archives
    apt-get update
  2. Run an upgrade with the following command
    apt-get dist-upgrade
  3. There will will be a question for if you wish to continue, to continue is the default, just press ENTER
    Do you want to continue? [Y/n]
  4. There will be a warning about that the package to be installed cannot be authenticated, the default here is not to install, so press “y” (without the quotes) to continue
    Install these packages without verification? [y/N]

Installing a package

This is used to e.g. test that a package is able to install its dependencies.

These operations have to be done as root:

  1. First update APT’s database of packages and archives
    apt-get update
  2. Use the following command to install a package and its dependencies, this example installs apache karaf:
    apt-get install karaf
  3. If the package pulls in new dependencies there will be a prompt for if you wish to continue. The default is to continue, so just press ENTER
    Do you want to continue? [Y/n]
  4. There will be a warning about that the package to be installed cannot be authenticated, the default here is not to install, so press “y” (without the quotes) to continue
    Install these packages without verification? [y/N]

Develop OSGi applications using karaf

Apache Karaf is a good platform for deploying OSGi based applications. Karaf is also a good platform for testing and debugging these applications. This article describes how to test and debug OSGi bundles and OSGi applications with karaf and eclipse.

The basic flow of development, is:

  1. Build the application with maven
  2. Start karaf as your own user in a way that makes it listen for remote debug connections
  3. Install the application from the karaf console
  4. Connect eclipse to karaf for a remote debug session
  5. Make karaf listen to updates to bundles with SNAPSHOT versions, and automatically load the updated bundles
  6. Make a change in eclipse
  7. Run a maven build of the project
  8. Observe that the change appears in karaf or in the debugger

For more detail, read on.

Preparations

Do the following:

  1. Install eclipse
    1. Open a web browser on https://www.eclipse.org/downloads/ and download the installer for your system
    2. Run the installer to the end, using defaults for everything
  2. Install apache maven
    1. On debian GNU/linux, just do
      apt-get install maven
  3. Download and unpack the binary karaf distribution. In bash on GNU/linux (or in git bash on Windows), do:
    mkdir -p ~/karaf
    cd ~/karaf/
    wget http://apache.uib.no/karaf/4.1.4/apache-karaf-4.1.4.tar.gz
    tar --strip-components=1 -xf apache-karaf-4.1.4.tar.gz

Clone and build the sample application

  1. Clone and build the sample application
    mkdir -p ~/workspaces/ws01
    cd ~/workspaces/ws01/
    git clone https://github.com/steinarb/hello-karaf-demo.git
    cd ~/workspaces/ws01/hello-karaf-demo/
    mvn clean install
  2. Import the sample application into eclipse:
    1. Start eclipse
    2. Open the workspace at ~/workspaces/ws01
    3. Import the existing maven project hello-karaf-demo into the workspace (File->Import… then select Maven->Existing Maven Projects and click the wizard through to the end)

Install the application into karaf

  1. Open a command line window and start karaf with an argument that makes karaf listen for remote debug connections
    cd ~/karaf/
    bin/karaf debug
  2. From the karaf console prompt:
    1. Install the run-time requirements for the test application
      feature:repo-add mvn:no.priv.bang.osgi.service.adapters/service-adapters-karaf/1.0.0/xml/features
      feature:install adapter-for-osgi-logservice
      feature:install pax-http-whiteboard
    2. Install the sample application with the following commands
      bundle:install mvn:no.priv.bang.demos/hello-karaf-demo/1.0.0-SNAPSHOT
      bundle:start mvn:no.priv.bang.demos/hello-karaf-demo/1.0.0-SNAPSHOT
  3. Open a web browser on the following URL http://localhost:8181/hello

Debug into the application running in karaf

  1. Set up a remote debug connection in karaf
    1. Open the menu Run->Debug Configurations…
    2. In the debug configurations dialog:
      1. Select “Remote Java Application”
      2. Click the “New launch configuration”
      3. In the Name field, write
        Remote karaf debug
      4. In the Port field, write
        5005
      5. Select the “Source” tab
      6. Click the button “Add…”
      7. In the “Add Source” dialog:
        1. Select “Java Project”
        2. Click the button “OK”
        3. In the “Project Selection” dialog
          1. Click the checkbox of “hello-karaf-demo”
          2. Click the button “OK”
      8. Click the button “Debug”
  2. Set a breakpoint in the code
    1. Open the HelloServlet.java file (press Ctrl-Shift-t to open the “Open Type” dialog, type HelloServlet and select the HelloServlet class)
    2. Go to line 60 (press Ctrl-l, then type 60, and press Enter), i.e. this line
                  response.setStatus(200);
    3. Press Ctrl-Shift-b to set a breakpoint on line 61
  3. Reload the web page in the web browser
  4. Observe that the debugger stops at the breakpoint
  5. Press F8 to make the code continue to run

Make a modification in the code, picked up by karaf

    1. In the karaf console, give the following command to make karaf listen in the local repository for new versions of bundles with SNAPSHOT version
      bundle:watch *
    2. Modify the code in eclipse
      1. In the HelloServlet.java file, change
            private static final String TITLE = "Hello world!";
            private static final String PARAGRAPH = "This is sent via PAX Web Whiteboard Extender.";
        

        to

            private static final String TITLE = "Hello karaf world!";
            private static final String PARAGRAPH = "Powered by Apache Karaf.";
        
      2. Save the HelloServlet.java file
    3. Build the project with maven
      1. In eclipse, select the menu Run->Run Configurations…
      2. In the dialog Run Configurations:
        1. Select “Maven Build” in the list on the left
        2. Click the “New launch configuration” icon (top left in the dialog)
        3. In the “Name:” field, type:
          mvn install hello-karaf-demo
        4. In the “Base directory” field, type:
          ${project_loc:hello-karaf-demo}
        5. In the “Goals” field, type:
          install
        6. Click the button “Run”
    4. Reload the web page in the web browser and observe that the change now is present

Installing with a karaf feature

In the installation example earlier in this article the runtime requirements of the sample application were first installed into karaf (pax-http-whiteboard and adapter-for-osgi-logservice).

It is possible to install this example application in a way that also pulls in the dependencies:

  1. The first thing to do, is to remove the existing installation in karaf:
    1. Stop karaf
      logout
    2. Remove the “data” subdirectory
      rm -rf data
    3. Start karaf
      bin/karaf debug
  2. Removing the data directory removes all state in karaf, and karaf is back to a fresh installation, this can be verfied by:
    1. listing the bundles in the karaf console
      bundle:list
    2. Verifying that http://localhost:8181/hello results in a 404 Not Found
  3. The first thing to do, is to add the feature repository for the hello world application. The feature repository is an XML file containing one or more “karaf features”. A karaf feature can be a list of bundles to load, and it can also contain references to other features.  The feature file for the hello world application is attached to the maven bundle artifact and can be installed with the following command:
    feature:repo-add mvn:no.priv.bang.demos/hello-karaf-demo/1.0.0-SNAPSHOT/xml/features

    Note: This is the same karaf commando as the first command of the dependencies install. The pax-http-whiteboard feature is built-in to karaf and doesn’t require a feature repository install

  4. Now the feature for the hello world application can be installed
    feature:install hello-karaf
  5. Verify that the feature install has pulled in dependencies
    bundle:list
  6. Verify that the application is running on http://localhost:8181/hello

Some things to try on your own:

  1. Remove the data directory again and try installing the bundle without the dependencies and see what happens
  2. Start karaf watching for bundle modifications in the local maven repository
  3. List bundles on different levels with the bundle:list command with the threshold argument (hint: all karaf commands take the “–help” argument)
  4. Try launching the “mvn install” for the bundle using eclipse hotkeys after modifying the java file and observe that karaf loads the rebuilt module

Installing apache karaf on debian

Until the RFP (Request For Packaging) bug for karaf in the debian bug tracker is resolved, here is an APT archive with a karaf package for debian (architecture “all”).  The package is created using native debian packaging tools, and built from a source tarball and the APT archive itself is created, using aptly.

The package has been tested on Debian 9 “stretch” (the current stable), amd64.

Do the following commands as root on a debian GNU/linux system:

  1. Add the key for the APT archive
    wget -O - https://apt.bang.priv.no/apt_pub.gpg | apt-key add -
  2. Open the /etc/apt/sources.list file in a text editor, and add the following lines:
    # APT archive for apache karaf
    deb http://apt.bang.priv.no/public stable main
  3. Install karaf with apt-get
    apt-get update
    apt-get install openjdk-8-jdk karaf
  4. Log in with SSH (password is “karaf” (without the quotes)) and try giving some commands:
    ssh -p 8101 karaf@localhost

 

Packaging karaf with native debian packaging tools

Note! This is an improvement over the packaging in  Installing apache karaf on debian stretch, this package is packaged using native debian packaging tools instead of fpm, and is built from the karaf source tarball instead of the karaf binary tarball.

Apache karaf is an OSGi container and application server that is provisioned from maven, and has an ssh server. Basically it is possible to start an empty karaf, ssh in and give some commands to install an application using maven.

There still isn’t a native .deb package on maven (see  the RFP (Request For Packaging) bug for karaf in the debian bug tracker), but this package can be installed from my own maven repository.

The packacing projecct can be found on github: https://github.com/steinarb/karaf-debian

Procedure to build the package

  1. Install the required build tools
    apt-get update
    apt-get install openjdk-sdk git maven-debian-helper devscripts
  2. Clone the karaf package repository
    mkdir -p ~/git
    cd ~/git/
    git clone https://github.com/steinarb/karaf-debian.git
  3. Build the package
    cd ~/git/karaf-debian/
    dpkg-buildpackage

After this, there will be a karaf-*.deb package in the directory above the karaf-debian directory.

Setting up a debian package archive with aptly

This article describes how to set up a debian archive with aptly on a debian 9 “stretch” computer, served by an nginx web server.

Initial setup

  1. Add a DNS alias for your virtual nginx web site (outside of the scope of this blog post). The examples below assume that apt.mydomain.com is the DNS alias
  2. Install the required software, logged in as root, give the following command
    apt-get install gnupg pinentry-curses nginx aptly
  3. Logged in as your regular user, do the following:
    1. Create a gpg keyNote! It is a good idea to do the key generation when logged into a debian desktop and move the mouse about during generation, to get good random values for the key generation.Giving the following command at the command line
      gpg --full-generate-key
      1. At the prompt for key type, just press ENTER to select the default (RSA and RSA)
      2. At the prompt for key size, type “4096” (without the quotes) and press ENTER
      3. At the prompt for how long the key should be valid, type “0” without the quotes and press ENTER
      4. At the prompt for “Real name”, type your real name and press ENTER
      5. At the prompt for “Email address”, type your email address and press ENTER
      6. At the prompt for “Comment”, type the host name of your archive web server, e.g. “apt.mydomain.com” and press ENTER
      7. At the prompt “Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit?”, type “O” (without the quotes) and press ENTER
      8. At the prompt for a passphrase, give a passphrase you will remember. You will be asked for this passphrase every time every time the repository is published
    2. Export the public key of the gpg key pair in a form that the web server can use
      1. First list the keys to find the id of the key
        steinar@lorenzo:~$ gpg --list-keys
        /home/steinar/.gnupg/pubring.gpg
        ---------------------------
        pub   rsa4096 2017-12-27 [SC]
              6B7638490707CCE365DF5415D2BA778731DC1EAC
        uid           [ultimate] Steinar Bang (apt.mydomain.com) <sb@dod.no>
        sub   rsa4096 2017-12-27 [E]
        
      2. Then use the id of the key to export the public key of the gpg key pair in a form that the web server can return
        gpg --output apt_pub.gpg --armor --export 6B7638490707CCE365DF5415D2BA778731DC1EAC
      3. Publish the key with the default GPG keyserver
        gpg --send-keys 6B7638490707CCE365DF5415D2BA778731DC1EAC
    3. Create a local repository “stable”
      aptly repo create -distribution="stable" stable
    4. Configure an architecture in the archive: open the ~/.aptly.conf file in a text editor, and change the line
      "architectures": [],

      to

      "architectures": ["amd64"],

      Note! Without a concrete architecture in place, aptly refuses to publish. So add an architecture here, even you are going to publish packages with architecture “all” (e.g. java, python, shell script). In the example I’m using “amd64” which, despite its name, is appropriate for modern 64 bit intel chips (i5 or i7 of various generations).

    5. Import a package into “stable” (the example uses the package built in Installing apache karaf on debian stretch)
      aptly repo add stable git/karaf-deb-packaging/karaf_4.1.4-1_all.deb
    6. Publish the archive (switch the gpg-key with the id of your actual repository key):
      aptly publish repo --gpg-key="6B7638490707CCE365DF5415D2BA778731DC1EAC" stable

      Note! If you get a time out error instead of a prompt for the GPG key passphrase, and you’re logged in remotely to the server, the reason could be that gpg tries to open a GUI pinentry tool. Switch to explictly using a curses-based pinentry and try the “aptly publish” command again. Do the command:

      update-alternatives --config pinentry

      and select “pinentry-curses” in “Manual mode”

  4. Log in as root and do the following
    1. Create a root directory for a new web server and copy in the public key used to sign the published achive
      mkdir -p /var/www-apt
      cp /home/steinar/apt_pub.gpg /var/www-apt/
    2. In a text editor, create the file /etc/nginx/sites-available/apt with the following content
      server {
      	listen 80;
      	listen [::]:80;
      
      	server_name apt.mydomain.com;
      	root /var/www-apt;
      	allow all;
      	autoindex on;
      
      	# Full access for everybody for the stable debian repo
      	location /public {
      		root /home/steinar/.aptly;
      		allow all;
      	}
      
      	# Allow access to the top level to be able to download the GPG key
      	location / {
      		allow all;
      	}
      }
      

      Note! I actually started out with also serving HTTPS and signing with letsencrypt, but as it turns out APT doesn’t support HTTPS out of the box, so there was no point in including it in this HOWTO

    3. Enable the site by creating a symlink and restarting nginx
      cd /etc/nginx/sites-enabled
      ln -s /etc/nginx/sites-available/apt .
      systemctl restart nginx

Your APT artchive is now up and running.

Use the new APT archive

To start using the APT archive, do the following on a debian computer:

  1. Log in as root
  2. Add the archive key
    wget -O - https://apt.mydomain.com/apt_pub.gpg | apt-key add -
  3. Add the archive by adding the following lines to /etc/apt/sources.list
    # My own apt archive
    deb http://apt.mydomain.com/public stable main
  4. Update APT to get the information from the new archive
    apt-get update
  5. Install a package from the archive
    apt-get install karaf

Future additions and updates of existing packages can be done as your regular user, with no need to log in as root during the process.

Publish a new version of a package

To update an existing package:

  1. Build a new version of the package
  2. Add the new version of package to the package archive
    aptly repo add stable git/karaf-deb-packaging/karaf_4.1.5-1_all.deb
  3. Update the publish the package archive (i.e. the package archive served by nginx)
    aptly publish update --gpg-key="6B7638490707CCE365DF5415D2BA778731DC1EAC" stable
  4. Do “apt-get update” on the computers using the archive
  5. Do “apt-get dist-upgrade” on the computers using the archive, and the package should be upgraded

Remove old versions of a package

To delete old versions of a package:

  1. Do a query to verify that the expression matches only the packages you want to delete
    aptly repo search stable 'karaf (<=4.1.4-4)'
  2. Remove all packages matching the query
    aptly repo remove stable 'karaf (<=4.1.4-4)'
  3. Remove old versions of the database (this is where the disk usage of the repository is reduced)
    aptly db cleanup

 

Installing apache karaf on debian stretch

Edit: It is now possible to install karaf on debian without building it yourself, the package installed is not the one described here, but the new and improved package built from source with native debian packaging tools, that can be found here  https://github.com/steinarb/karaf-debian

Apache karaf is an OSGi container/application server with some nice properties:

  1. It has an SSH server you can log into and a command line where you can inspect and configure the karaf instance
  2. It can be provisioned using apache maven, basically you can start with an empty karaf, ssh into the SSH server and pull in and start your application using “maven magic”
  3. It is much simpler to get an OSGi application up and running in apache karaf, than in any other system I have tried, since I first was to introduced to OSGi in 2006
  4. Karaf can also be used to run non-OSGi applications packaged as jar or war files
  5. In a development setting is very simple to deploy new versions of the code using maven and remote debug the deployed code frome eclipse or IntelliJ

Running karaf on a debian GNU/linux system is a little hampered by there not being a native .deb package. I have opened an RFP (Request For Packaging) bug for karaf in the debian bug tracker. When/if that issue is ever resolved as done, karaf will be easily availabel on debian and also on all of the distros that are based on debian (e.g. ubuntu and mint).

Until then do my own debian packaging. I forked the packaging I found at https://github.com/DemisR/karaf-deb-packaging and made some changes:

  1. Switched from oracle JDK 8, to openjdk 8
  2. Updated to karaf version 4.0.7 (the currently newest stable release at the time of forking), later upgraded to karaf 4.1.1 and again upgraded to karaf 4.1.2
  3. Use /var/lib/karaf/data instead of /usr/local/karaf/data
  4. Use package version “-1” instead of “-3”
  5. Switched from using the the service wrapper (karaf-wrapper) to plain systemd start using the scripts and config from bin/contrib in the karaf distribution
  6. Made the stop of running services more robust

The resulting .deb package will follow the usual service pattern of a debian service: the service will run with a user named after the service (i.e. user “karaf” which is the single member of group “karaf” and the owner of all files the service need to touch). The service will log to the regular debian syslog. The configuration will end up in /etc/karaf and all files not part of the installation will be left untouched on a .deb package uninstall and upgrade.

My fork of the packaging, lives at https://github.com/steinarb/karaf-deb-packaging

To create the package and install karaf, do the following steps:

  1. Log in as root on a debian system
  2. Install the prequisites for building the package, debian packages and ruby gem:
    apt-get update
    apt-get install git maven openjdk-8-jdk postgresql ruby ruby-dev build-essential
    gem install fpm
  3. Clone the packaging project and build the deb package:
    cd /tmp
    git clone https://github.com/steinarb/karaf-deb-packaging
    cd karaf-deb-packaging
    ./dist_karaf.sh
    mkdir -p /root/debs
    cp *.deb /root/debs
  4. Install the .deb package:
    dpkg --install karaf_4.1.4-1_all.deb

After karaf has been installed it is possible to log in as user “karaf”, with the following command

ssh -p 8101 karaf@localhost

The password is also “karaf” (without the quotes).

This opens the karaf console command line

        __ __                  ____
       / //_/____ __________ _/ __/
      / ,<  / __ `/ ___/ __ `/ /_
     / /| |/ /_/ / /  / /_/ / __/
    /_/ |_|\__,_/_/   \__,_/_/

  Apache Karaf (4.1.4)

Hit '<tab>' for a list of available commands
and '[cmd] --help' for help on a specific command.
Hit 'system:shutdown' to shutdown Karaf.
Hit '<ctrl-d>' or type 'logout' to disconnect shell from current session.

karaf@root()>

At this command line, you can eg.

  1. install an application
  2. start, stop and list running applications
  3. set the configuration used by the applications

But all of these except for the first, will be items for later posts.

Til minne om kaninen Daisy

Jeg tar et avbrekk fra postinger om datateknikk til å minnes kaninen Daisy.

Siste bilde av kaninen Daisy
Fortsatt en glad kanin som koste seg med løvetann.

En liten svart og hvit løvehodekanin.

Hun ble født i november 2009 og døde 29. august 2017, litt over 7,5 år gammel. Ungene fikk først hilse på henne den 22. desember 2009. Fra rett over nyttår 2010 og fram til hun døde bodde hun hos oss.

Da hun kom til oss var hun verdens mykeste og nusseligste lille kanin. En bitteliten pelsball som var redd for å hoppe ned fra sofaen. Vi måtte legge puter på golvet for at hun skulle tørre å hoppe ned. Silkemyk pels var en egenskap hun beholdt helt fram til sin død.

Ungene var 7 og 8 år gamle da vi fikk henne, så hun har vært en del av familien gjennom en stor del av oppveksten deres. Vi hadde Daisy da bestemor og bestefar på Mo solgte huset og første gang ungene kjørte Trollstigen og Geiranger og gikk Besseggen.

Hun fikk være med oss på høstferier og vinterferier og påskeferier. Hun har vært på Gålå og på Sjusjøen og på Femundsenden og masse andre plasser. Hun var ikke så glad i reiseburet men hun så til å trives når hun fikk hoppe rundt og undersøke et nytt sted.

De fleste feriene tilbrakte hun imidlertid hos Heidi på Rotnes katte- og kaninpensjonat og der hadde hun det fint.

Hun fikk sin egen instagram-konto, kaninen Daisy.

Jeg fikk min egen nisselue🎅

A post shared by ❤️Daisy❤️ (@kaninen_daisy) on

Instagram

Daisy var glad i jula men aller mest glad i juletrærne. Hun var også glad i snø som lot henne grave huler og ganger på verandaen hun tilbragte mye tid på.

Ikke alle kaniner er like hyggelige, men Daisy var en kjælen og vennlig kanin som gjerne hoppet opp i fanget til folk og elsket en god ørekos.

Hun fikk ingen kaninvenner men hun fikk mye oppmerksomhet fra menneskene sine og jeg tror hun hadde et fint lite kaninliv hos oss.

Daisy elsket kvister fra hengebjørk, mislikte tørr fjellbjørk, men gumlet både løvetann og kløver med stort velbehag. Hun var en ganske renslig og pertentlig liten kaninfrøken, og hun holdt seg på matta (dvs. hun var ikke veldig glad i parketten), noe som gjorde at hun fikk gå mye fritt. Ikke spiste hun ledninger heller om vi ser bort ifra et par-tre hodetelefoner tilhørende Eirik.

Det er trist at vi aldri mer skal få se kaninen danse foran verandadøra for å bli sluppet inn, og aldri mer få se kaninen sitte på den lille matta foran buret og stirre forhåpningfult mot matskapet. Og aldri mer få se kaninen være “crazy Daisy” og sprinte rundt på teppet og opp i sofaen.

Det er trist at Daisy er død men jeg er veldig glad for at akkurat hun ble vår kanin og at hun var hos oss i disse årene. Farvel lille venn!