Sitara Linux Training: Getting Started with Openembedded

Introduction

The following labs will help new users become familiar with OpenEmbedded and the various terminology and files. This will be done by teaching users how to create their own "layer" which will contain any new recipes or tweaks to recipes found in other layers.

The layer you're creating, meta-custom, is intended to represent your own Yocto layer. In the lab context it is being cloned from a TI server, but normally this repository would reside on your own build servers, where you would have full ability to patch it and update it. You'll be patching and updating locally in the labs as a very close approximation of a typical build flow.

Note that there is also a training series which walks through much of this material and provides some other details and commentary along the way:

https://training.ti.com/customizing-yocto-for-production-series

Setup

The Processor SDK Linux configuration will be used as the foundation for this training. The reference machine/device will be the AM335x.

Before getting started with this training you need to ensure that your computer is OpenEmbedded ready. First off, this training will require over 33 GB of hard drive space. If you plan on using OpenEmbedded beyond this training then it is recommended to have at least 60 GB of free space available. Furthermore, if you're planning to build the full file system as shipped with Processor SDK Linux, you would need over 200 GB of space.

Since OpenEmbedded downloads everything from source you need to ensure that the computer you're using to run OE builds has internet access. OpenEmbedded uses several methods to download source files such as git, wget, svn. Also to install additional dependencies applications apt-get is used. Therefore, if a user is behind a firewall (i.e. workplace network) then proxies for the various applications need to be set.

1. Follow the steps listed at Prerequisites to properly configure your machine, setup proxies, and install the toolchain. Stop before the section Build Steps.
2. Create a directory in which to perform all this work. We will refer to this base directory as <oe-dir>.
3. cd <oe-dir>
4. Begin running through the Build Steps, but do not execute the final command (bitbake)!

   Note: If you close your shell and later come back to this, you'll need to re-run the . conf/setenv step and the export PATH=... step.

5. At this point your shell path should be <oe-dir>/tisdk/build.
6. We're going to make a customized version of the image called arago-base-tisdk-image. Before continuing you should successfully build this target:

   MACHINE=am57xx-evm bitbake arago-base-tisdk-image

7. Now we will clone an additional layer called meta-custom for training purposes.

   cd ../sources

   git clone git://git.ti.com/sitara-linux/meta-custom.git -b proc-sdk-linux

So at this point, you should see something like this:

<oe-dir>/tisdk/sources
├── bitbake
├── meta-arago
├── meta-custom
├── meta-linaro
├── meta-openembedded
├── meta-processor-sdk
├── meta-qt5
├── meta-ti
└── oe-core

Notes on the Lab Flow

1. In the labs, all of the build steps are using MACHINE=am57xx-evm. Naturally, if that's not the board you're using, you should substitute accordingly. See the table of machines here.
2. There are tags to start each lab to facilitate being able to jump into any of these with a known-good starting point. The tags all have "2018-04" in them to distinguish them from earlier tags, e.g. Lab1-2018-04, Lab2-2018-04, etc.
3. If you jump into a specific lab, note that you must be sure bblayers.conf has been configured to add the meta-custom layer. Your builds aren't going to work without meta-custom. In other words, <oe-dir>/tisdk/build/conf/bblayers.conf is not part of the meta-custom repository, so checking out the solution to any given lab is not going to properly configure this file. That must be done manually (or you must modify the setup script/cfg to include that layer).
4. There are also tags for all of the answers for each lab. Each of those is appended with -Answers, e.g. Lab1-2018-04-Answers, Lab2-2018-04-Answers, etc.
5. You are encouraged to commit your work within your working branch after each lab. That will facilitate comparisons with the answers, etc.

Lab 1: Creating Your Own Layer

It is recommended that users use meta-arago as a foundation when building for any Sitara base device such as am335x. However, tweaking recipes, deleting and adding recipes within layers you don't own or contribute to causes several problems. It becomes difficult to update the various layers since your tweaks will likely cause git merge/rebasing issues when fetching the latest sources and also sharing your changes will be difficult since your changes can't be pushed publicly since you likely don't have the proper permissions.To solve the above issues a seperate layer should be created which allows a "sandbox" to be created without altering any files outside of the user's layer. In this training a new layer called meta-custom will be created and used through out the labs.

Modifying layer.conf

When creating a new layer the conf/layer.conf file must be created. Various variables within this file expect a common "unique name" to be used and a proper priority to be set.

Open a terminal window and execute the following steps:

cd <oe-dir>/tisdk/sources/meta-custom
git checkout -b MyLab1Branch Lab1-2018-04

This creates a new branch called "MyLab1Branch" using the tag "Lab1-2018-04" as its starting point.

Before you make any changes, we can first see the existing layers in the project with the following commands:

cd <oe-dir>/tisdk/build
MACHINE=am57xx-evm bitbake-layers show-layers

We can see the highest priority layer (meta-processor-sdk) is priority 15.

In <oe-dir>/tisdk/sources/meta-custom/conf/layer.conf you will see something similar to the following:

BBPATH .= ":${LAYERDIR}"
# We have a recipes directory, add to BBFILES
BBFILES += "${LAYERDIR}/recipes-*/*/*.bb ${LAYERDIR}/recipes-*/*/*.bbappend"

BBFILE_COLLECTIONS += "<unique name>"
BBFILE_PATTERN_<unique name> := "^${LAYERDIR}/"

BBFILE_PRIORITY_<unique name> = "<priority value>"

If a layer depends on another layer then its priority should be higher. Meta-custom will sit on top of meta-processor-sdk, whose priority is 15. Therefore, a priority of 16 can be used for meta-custom.

Replace <unique name> with meta-custom and <priority value> with 16. Your updated <oe-dir>/tisdk/sources/meta-custom/conf/layer.conf should look like below.

BBPATH .= ":${LAYERDIR}"
# We have a recipes directory, add to BBFILES
BBFILES += "${LAYERDIR}/recipes-*/*/*.bb ${LAYERDIR}/recipes-*/*/*.bbappend"

BBFILE_COLLECTIONS += "meta-custom"
BBFILE_PATTERN_meta-custom := "^${LAYERDIR}/"

BBFILE_PRIORITY_meta-custom = "16"

Commit your changes:

git commit -am "Configured name and priority for meta-custom"

Modifying bblayers.conf

The second file that needs to be tweaked is <oe-dir>/tisdk/build/conf/bblayers.conf. This configuration file determines which layers bitbake should parse. The BBLAYERS variable requires the full path to each layer that should be used. The current value of BBLAYERS was automatically configured by the oe-layersetup script. Since a new layer is being created outside of the script, bblayers.conf needs to be manually modified.

Add the highlighted line to the BBLAYERS variable.

# Layers configured by oe-core-setup script
BBLAYERS += " \
        /absolute/path/to/oe-dir/tisdk/sources/meta-custom \
        /absolute/path/to/oe-dir/tisdk/sources/meta-processor-sdk \

After modifying bblayers.conf you should now be able to re-run this command and see meta-custom:

cd <oe-dir>/tisdk/build
MACHINE=am57xx-evm bitbake-layers show-layers

Testing things out

Now it is time to try building something from your new layer! Hidden within your new layer is a recipe called "hello-world" that will be discussed in detailed later. Open a new terminal window. This window will be referred to as the "build window".

When using bitbake (ie building a package, building a filesystem, checking bitbake environment variables etc...) the first three steps must be accomplished when a new terminal window is used.

Execute the following four steps in the "build window".

1. Use the change directory command to go to the <oe-dir>/tisdk/build directory in the "build window"
   

   cd <oe-dir>/tisdk/build

2. If your build environment is not setup already, make sure you've completed these steps:

   source conf/setenv

   export PATH=<tool chain bin dir>:$PATH

3. Check for the "hello-world" recipe being visible in your environment:

   MACHINE=am57xx-evm bitbake-layers show-recipes hello*

4. Build Something!
   

   MACHINE=am57xx-evm bitbake hello-world

If no error message are seen then your layer has been successfully setup and you have built a mysterious recipe!

Lab 2: Creating and Customizing Your Own Output Image

You can continue building on your branch from Lab 1, or alternatively you can start a new branch:

cd <oe-dir>/tisdk/sources/meta-custom
git checkout -b MyLab2Branch Lab2-2018-04

In this lab we're going to further customize the arago-base-tisdk-image. We've provided the following recipe as a starting point:

• meta-custom/recipes-core/images/custom-base-image.bb

The intent is that this would be a recipe fully under your control that allows you to add/subtract anything you want to/from the file system.

The way we know it's customizing arago-base-tisdk-image is from the first line:

require recipes-core/images/arago-base-tisdk-image.bb

This approach is commonly used when a user wants to generate a "new" recipe that has a lot in common with the "old" recipe. By using the standard shell script "require" statement a recipe can "include" all of text from another recipe. At this point the new recipe only has to modify what it wants. To accomplish this the relative path from the layer that contains the recipe you want to "require" must be used with the require function.

The full path for arago-base-tisdk-image.bb is <oe-dir>/tisdk/sources/meta-arago/meta-arago-distro/recipes-core/images/arago-base-tisdk-image.bb. The only portion of the path that is needed for the require function is "recipes-core/images/arago-base-tisdk-image.bb".

Trimming the filesystem

The custom-base-image has no need for packagegroup-arago-base, packagegroup-arago-console and packagegroup-arago-base-tisdk which was added by arago-base-tisdk-image. By using some handy python functions that bitbake has defined variables can be manipulated in various ways.

Since the IMAGE_INSTALL variable specifies what will be installed in the image this variable will need to be tweaked.

The statement "${@oe_filter_out('<substring to be removed>', '<string>', d)}" will return a new string that had the specified sub string removed.

For example, these lines are removing several package groups from our custom-base-image:

IMAGE_INSTALL := "${@oe_filter_out('packagegroup-arago-base', '${IMAGE_INSTALL}', d)}"
IMAGE_INSTALL := "${@oe_filter_out('packagegroup-arago-console', '${IMAGE_INSTALL}', d)}"
IMAGE_INSTALL := "${@oe_filter_out('packagegroup-arago-base-tisdk', '${IMAGE_INSTALL}', d)}"

Build this customized image using the following command (from the "build" directory):

MACHINE=am57xx=evm bitbake custom-base-image

Adding to the filesystem

Let's add the hello-world recipe to our package. You'll need to add it to this file:

• meta-custom/recipes-core/images/custom-base-image.bb

Add the following statement to the above image recipe

# Add any packages or packagegroups that we want added to our filesystem
IMAGE_INSTALL += "\
    hello-world \
    "

Rebuild your customized image using the following command (from the "build" directory):

MACHINE=am57xx=evm bitbake custom-base-image

Now you should have a hello-world utility in your file system. You can do a quick sanity check:

cd arago-tmp-external-linaro-toolchain/deploy/images/am57xx-evm
tar tf custom-base-image-am57xx-evm.tar.xz | grep hello-world

The above tar command is just doing a "test" where it lists the files inside, and then we are grepping for hello-world which we expect to find in the /usr/bin directory.

We can commit our changes if everything looks good:

git commit -am "Add hello-world to custom-base-image"

Lab 3: Updating a Recipe to Use the Latest Commit

You can continue building on your branch from Lab 2, or alternatively you can start a new branch:

cd <oe-dir>/tisdk/sources/meta-custom
git checkout -b MyLab3Branch Lab3-2018-04

There's a really annoying issue with the hello world recipe. There's no line feed! When you run it, it looks like this:

root@am57xx-evm:~# hello-world
Hello worldroot@am57xx-evm:~#

Luckily the developers have fixed the issue. The goal of this lab is to pull in the latest updates from that repository so that we don't have that issue!

1. Look at the commit history for the hello-world source code.
2. You should see the newly added commit d8e6b51 "add a line feed".
3. If you click the d8e6b51 link, you can get the full commit id: d8e6b51902e8dd97e716fed78b3c774f38c4d2e7
4. Open the recipe: meta-custom/recipes-core/hello-world/hello-world_1.0.bb
5. Update the SRCREV variable to use this new commit id and update PR to r3. This is needed to inform the build system that the recipe has been updated.
6. From the "build" directory, rebuild your image: MACHINE=am57xx-evm bitbake custom-base-image

Now when you run your hello-world utility it will work better:

root@am57xx-evm:~# hello-world
Hello world
root@am57xx-evm:~#

If everything looks good, go ahead and commit your fix:

git commit -am "Update to latest commit (add line feed)"

Lab 4: Applying a Custom Patch at Build Time

You can continue building on your branch from Lab 3, or alternatively you can start a new branch:

cd <oe-dir>/tisdk/sources/meta-custom
git checkout -b MyLab4Branch Lab4-2018-04

Up till now, we've had a pretty good relationship with the developers of hello-world. We had reported a bug with the line feed and were happy to see they fixed it. However, we disagree on an important point. Specifically, we feel very strongly that there should be an exclamation point ("Hello world!"), but they refuse to fix it. So from now on, when we build hello-world we want to apply our own patch.

Creating the Patch

1. We can find the existing code in the work area at build/arago-tmp-external-linaro-toolchain/work/armv7ahf-neon-linux-gnueabi/hello-world/1.0-r3/git. The r3 corresponds to the PR variable in the hello-world_1.0.bb recipe.
2. Edit hello-world.c to add the exclamation point!
3. If you want to force bitbake to recompile this code so you can try it out, you can (from the "build" directory) run MACHINE=am57xx-evm bitbake -f -c compile hello-world.
4. You can manually copy the executable from hello-world/1.0-r3/git/Release/hello-world to your target and confirm it's working as expected. Or in this case it's so simple you can just run strings hello-world | grep Hello on the executable and see that it contains the proper string.
5. Let's commit the change to this temporary git repo and output a patch.

   git commit -am "Adding exclamation point!"

   git format-patch -1 will output 0001-Adding-exclamation-point.patch.

Don't lose that patch!

Applying the patch at build time

Ideally any patches should be sent to the maintainers of the source code repository. However, if they can't or won't accept it, or perhaps there's a time crunch then you'll need to apply that patch manually at build time. That's what we're going to do here.

1. The generated patch should be placed at sources/meta-custom/recipes-core/hello-world/files/0001-Adding-exclamation-point.patch.
2. Add the file to the meta-custom git index, e.g. from inside the meta-custom directory run git add recipes-core/hello-world/files/0001-Adding-exclamation-point.patch
3. Update sources/meta-custom/recipes-core/hello-world/hello-world

LICENSE = "BSD"
LIC_FILES_CHKSUM = "file://COPYING;md5=8d30c2460fbf3f17135994e5c6ead1f3"

PR = "r4"

FILESEXTRAPATHS_prepend := "${THISDIR}/files:"

# Commit id/hash associated with the specific git commit that you want to use.
SRCREV = "d8e6b51902e8dd97e716fed78b3c774f38c4d2e7"

SRC_URI = "git://git.ti.com/sitara-linux/hello-world.git;protocol=git \
           file://0001-Adding-exclamation-point.patch"

S = "${WORKDIR}/git"

Finally, go back to the build directory and rebuild your custom image:

MACHINE=am57xx-evm bitbake custom-base-image

Assuming everything looks good, go ahead and commit your changes in the meta-custom repository:

git commit -am "exclamation mark patch for hello-world"

Lab 5: Modifying a Recipe from Another Layer

You can continue building on your branch from Lab 4, or alternatively you can start a new branch:

cd <oe-dir>/tisdk/sources/meta-custom
git checkout -b MyLab5Branch Lab5-2018-04

Sometimes there are cases where the recipe itself needs to be patched. If possible, it's preferred to have the owner of that recipe make any corrections. However, there may be times where the owner of the recipe cannot or will not make a change for one reason or another. In this cases, you need a way of making a modification to that recipe. This is done through a bbappend.

For example, we're going to look at lmbench which has its main recipe in meta-oe, plus a pre-existing bbappend in meta-arago-distro:

• ./meta-openembedded/meta-oe/recipes-benchmark/lmbench
• ./meta-arago/meta-arago-distro/recipes-benchmark/lmbench

In order to implement a bbappend, you must use the same directory structure, i.e. both of the above are seen in recipes-benchmark/lmbench. You'll find there's already a "starter file" located here:

• ./meta-custom/recipes-benchmark/lmbench/lmbench_%.bbappend

The '%' character is a wild card, meaning that even if the lmbench version gets updated, this recipe will still be attempted.

Since this is an append, bitbake by default will not search this directory for any files. Therefore, this has been entered into the file:

FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"

${THISDIR} returns the directory the recipe or bbappend is found in. The ${PN} variable relates to the "package name" (lmbench), so it will search for any files in a subdirectory called lmbench. You'll notice there's already a patch there for you to apply:

• ./meta-custom/recipes-benchmark/lmbench/lmbench/0001-inconsequential-change.patch

This patch is modifying a program called "hello" that is part of the lmbench suite. It is a trival change if you look at it.

Since a new patch needs to be added, the file name using the file:// syntax needs to be added to the SRC_URI variable.

Add the following statement to the bbappend file.

SRC_URI += "file://0001-inconsequential-change.patch"

Finally you'll need to define a PR_append. Since our layer is called meta-custom you might choose to define something such as:

PR_append = ".custom1"

Pay attention to what's happening when you build your image:

MACHINE=am57xx-evm bitbake custom-base-image 

You'll see it's only rebuilding lmbench, and also the file name should have concatenated the "custom1":

0: lmbench-3.0-a9-r2.arago3.custom1 do_compile - 3s (pid 16209)

You can correspondingly looked at the patched source code at this location:

• /build/arago-tmp-external-linaro-toolchain/work/armv7ahf-neon-linux-gnueabi/lmbench/3.0-a9-r2.arago3.custom1/lmbench-3.0-a9/src/hello.c

You can see that the "inconsequential patch" has been applied, i.e. the first 'h' is lower case. And similarly if you execute "hello" on your target you'll see the same.

Assuming everything looks good, you can go ahead and commit your changes to meta-custom:

git commit -am "lmbench bbappend"

Lab 6: Package Groups

You can continue building on your branch from Lab 5, or alternatively you can start a new branch:

cd <oe-dir>/tisdk/sources/meta-custom
git checkout -b MyLab6Branch Lab6-2018-04

Packagegroups provide a convient way to logically group packages that need to be installed in a file system.

Creating a packagegroup

We are able to specify run-time dependencies via the RDEPENDS variable. For the first time a recipe will be used to ensure when its <package> is installed that the other package(s) set in the RDEPENDS variable.

The standard location for a packagegroup recipe is within a layer's recipe-core/packagegroups folder. The recipe name also follows the naming convention of packagegroup-<unique name>.

A ready-to-use packagegroup file called packagegroup-meta-custom.bb has already been created at <oe-dir>/tisdk/sources/meta-custom/recipes-core/packagegroup.

DESCRIPTION = "Collection of meta-custom's packages"
LICENSE = "MIT"
PR = "r0"

inherit packagegroup

META_CUSTOM_EXAMPLES = " \
    hello-world \
"

RDEPENDS_${PN} = "${META_CUSTOM_EXAMPLES}"

If you wished to add more recipes to this group, you would simply list them after hello-world.

Building the new packagegroup recipe

To ensure that everything is working as expected execute the following commands in the "build window"

MACHINE=am57xx-evm bitbake packagegroup-meta-custom

If no errors are seen then your new packagegroup has been created! It's not necessary to run this independently, i.e. if the packagegroup is included in an image then it will get built automatically when you build the image.

Include the packagegroup in your image

It's incredibly easy to include a packagegroup in your image. We just need to make a quick edit to meta-custom/recipes-core/images/custom-base-image.bb. You can replace hello-world with your packagegroup packagegroup-meta-custom:

# Add any packages or packagegroups that we want added to our filesystem
IMAGE_INSTALL += "\
    packagegroup-meta-custom \
    "

Now you can rebuild your image:

MACHINE=am57xx-evm bitbake custom-base-image

In this particular instance it's not really helping us since there's only a single package in the packagegroup. However, it's easy to see how one could leverage this capability in more complex build environments.

Commit your update to meta-custom:

git commit -am "include packagegroup-meta-custom in custom-base-image"