[Hackable1-dev] The automated build system and upcoming release

Pierre Pronchery ppronchery at bearstech.com
Fri Dec 12 05:30:23 CET 2008 

			Dear fellow developers,

I am really glad to announce that the automated build system has made
real good progress. It can currently produce useful filesystem images
for both users and developers. But first, let's introduce how we are
working on Hackable:1 (and therefore, how you can join).


The beta release
----------------
As you probably know, we have released a first beta in the middle of
November. It was engineered solely by Marcus Bauer, using the
instructions found there:
http://wiki.debian.org/DebianOnFreeRunner

He has then customized, extended, fine-tuned and released it on our
website:
http://www.hackable1.org/hackable1/
where you will also find the instructions to install it.


Limitations
-----------
While the version released is a good preview and testbed environment for
what we want to achieve, it does not scale to community development:
- we would have to reproduce the work manually for different devices, or
  even dedicated configuration,
- we would have to go through Marcus for every single fix,
- he would have to keep a user-neutral, personal-data free version at
  all times ready for release,
- every release would weight 300MB+

This can even lead to security concerns, like forgetting an SSH key in
the root folder :)


An automated process
--------------------

In reality, work is only valuable in the long term if it can be
understood and reproduced by anyone. It is even better if it can be
kept to a minimum of discrete, atomic steps. This is what I tried to
keep in mind while designing the scripts to produce our future
releases, and assist in this procedure.

You will find them in our SVN source code management system:
$ svn co svn://svn.hackable1.org/hackable1
This will also download code from Openmoko and Opened Hand's
repositories: it is used when creating Debian packages of some of their
software, for inclusion in Hackable:1.

To automatically build a filesystem image, it is recommended to run a
recent flavour of Debian. The architecture does not matter. The tools
involved are:
- dpkg
- dropbear (not for long)
- mtd-tools (for JFFS2 images)
- sudo (optional)
- wget
Then, place yourself in the trunk/build sub-directory like this:
$ cd hackable1/trunk/build
$ ./build.sh image
The Openmoko Freerunner is currently used by default, as it is our only
target officially supported so far.

After about 10 minutes, you will be able to flash your Openmoko with
dfu-util this way:
$ dfu-util -a kernel -D destdir-Openmoko-Freerunner/boot/uImage.bin
$ dfu-util -a rootfs -R -D \
Hackable1-Openmoko-Freerunner-2008.12a-rootfs.jffs2

More detailed instructions about this process can be found there:
http://wiki.openmoko.org/wiki/Dfu-util

Alternatively, to build a developer tarball archive, you can run:
$ ./build.sh PURPOSE=developer archive

You can then use the instructions already available from our current
release.


Content of the image
--------------------

The image is currently generated from three repositories:
- Debian Lenny (still in "testing" phase)
- FSO unstable (under heavy development)
- Hackable:1 beta

Unfortunately, both Debian and FSO repositories are subject to changes
that we do not directly influence. Therefore, things may break there,
with the possibility to file bug reports at best. On the other hand, we
control our repository, but it is currently managed from our beta
release (based on Debian "unstable"), with binaries that do not match;
the packages will be appropriately rebuilt ASAP.

The package list is generated from a series of (sub-)profiles, found
under "profiles". Dependencies are cached, stored in SVN, and can be
refreshed this way:
$ ./build.sh depends
$ ./build.sh PURPOSE=developer depends

They are currently approximative, because my development station runs
Debian Etch (the "stable" release), while the current repository in
Hackable:1 is Lenny; this will be also sorted out ASAP.

For portability reasons, maintainer scripts of the software installed
cannot be executed. Instead, configuration is done on a per-package
basis, with sub-scripts found in the "packages" directory under "build".

My first experiment at reducing the footprint of the image is:
- removing all package documentation data,
- excluding manual pages as well,
- erasing locale data,
- wiping the (otherwise functional) packaging database.

All of them combined make me gain a lot of space and time at the
moment. This is a rather blunt and arbitrary procedure, which is
actually configurable per-profile though. As always, I will welcome
differing opinions and improvements in this area.


Known issues
------------
- the image needs to reboot once to fix an issue with fonts (but it can
  be done otherwise)

- dropbear's host SSH keys should be generated during the first boot
  (may raise an issue with entropy though)

- functionality does not match the beta release yet:
  * GPS needs to be manually turned on
  * the GSM daemon is not started automatically (the muxer is)
  * the user interface is not polished
  * phone applications are not tested yet

- boot time is still slower than expected:
  * nodm should be started earlier
  * non-critical daemons (sshd, gsmd, gpsd...) should be started once
    the UI is ready (and maybe only accordingly to their last known
    state)

- services are not fault-tolerant (not restarted after crashes)

- Hackable:1's Debian packages are somewhat simple, we will also fix:
  * upstream references
  * proper attribution
  * run-time and build-time dependencies
  * versions (with relevant SVN revisions)
...as soon as time constraints permit it.

- the jffs2 driver is complaining about "Empty flash ... end at ..."
  on my neo, which slows down booting but is actually harmless

- I am sometimes hit by the White Screen Of Death (I heard there is a
  fix upstream)


Packaging
---------

About packaging, you will find scripts in the "trunk/packages"
directory:
- package.sh: creates tarball from upstream, applies Debian packaging
  patches, and creates the Debian packages respectively.
- upload.sh: creates the "Release" and "Packages.gz" files and uploads
  them to an external server together with the packages.

Unfortunately, not all upstream sources and packaging patches are
available in the SVN currently, or referenced in the script. The
patches are currently stored directly in "trunk/packages", in
uncompressed format.

Here's an example of how I proceed currently:
$ export package=jana
$ export version=0.0
$ export revision=1
$ tar xzvf "$package-$version.tar.gz" && \
(cd "$package-$version" && patch -p1 < \
"../${package}_$version-$revision.diff" && \
dpkg-buildpackage -us -uc -rfakeroot)

If all goes well, you get one or more Debian packages built.

The packages currently available have been extended as found there:
http://www.hackable1.org/debian/dists/beta/main/binary-armel/

The following line in /etc/apt/sources.list should reference them
accordingly:
deb http://www.hackable1.org/debian beta main
(I haven't tested it yet)


Daily builds
------------

We will setup a daily build system with all of this ASAP (again). Test
releases and package upgrades will be pushed as improvements hit the
SVN.


Contributing
------------

You are all welcome to send bug reports, even without patches, either
to this mailing-list or (preferably) there:
http://trac.hackable1.org/trac/newticket (an account is necessary)

Please bear with us as we are still in an early stage, rushing to match
our internal deadlines :)

Your help is precious though, and will be appreciated in many areas.
Feel free to let yourself known!

This is also true if you are stuck somewhere, and it certainly means my
explanations are not good enough. This will all go in respective Wiki
pages and tickets.


Conclusion
----------

I can tease you with my current results:

> Current size: 260M      destdir-Openmoko-Freerunner
> Deleting documentation...
> Removing package database...
> Removing locales...
> Final size: 186M        destdir-Openmoko-Freerunner
> Creating image Hackable1-Openmoko-Freerunner-2008.12a-rootfs.jffs2...
> 83M     Hackable1-Openmoko-Freerunner-2008.12a-rootfs.jffs2

I can upload an image manually if that's useful.

With all of this said, I am really looking forward to integrating our
latest telephony fixes, and finalizing the configuration :)

Good last bit of the night,
-- 
Pierre Pronchery, Hackable devices R&D, Bearstech
email: ppronchery at bearstech.com         42 boulevard Sébastopol
PGP:   AE49 5F7D D56A 4BD6 7B1F         75003 Paris, France
       8655 125C 0FE6 5566 EBD8         Web: http://bearstech.com
Phone: +49 177 472 7481                 Phone: +33 6 71 62 42 74
Fax:   +49 304 208 1861                 Fax:   +33 1 42 72 20 03

More information about the Hackable1-dev mailing list