Long time, no see!

In this corner of our Collabora blogosphere we're talking about USB gadgets. This post is about the cmtp-responder, but before you read on you might want to look at our other posts about USB gadgets: In part 1 I introduced the concept of USB gadgets, their configfs composition interface, available open source tools, and basic systemd integration. In part 2 I wrote about one particular USB gadget function - FunctionFS - and its integration with systemd. Then I presented the cmtp-responder, a permissively-licensed MTP responder implementation and showed how to play with it on your PC with dummy\_hcd driver. Last time we met, you received instructions about how to run cmtp-responder on an ARM board. All these topics revolve around USB gadgets, configfs interface to manipulate them, and useful libraries/tools created to ease the said manipulation. This time we have several new elements for you, so without further ado let's get started!

Goodies for you

In this post I will cover six different areas:

• Docker for (cross-)compiling cmtp-responder & its friends 
• Example strategy for building and deploying to a target
• debos for building a QEMU image to try cmtp-responder on your PC without affecting your system
• Tests arriving in cmtp-responder
• cmtp-responder changes
• mtp-tools & fixes and gio

Docker

Compiling a new piece of software, or even more cross-compiling, might sometimes be too difficult and quite often inconvenient. These days you solve the issue of replicating the build environment (differences between somebody else's and your own build environment are the root of so many problems!) with a Docker container. Your first piece of good news is that this time in the cmtp-responder repo there is now a dedicated directory, which contains a Dockerfile and scripts to both build and use the containers for each target architecture: x86\_64, armhf, and aarch64. You get a separate container for each architecture. Teaching you Docker is far beyond the scope of this post, but the provided scripts are pretty self-explanatory: you either invoke them with "run" or "build" arguments. The latter needs to be run once to build the Docker image, while the former you run each time you want to enter your (unified) build environment. Optionally, you can provide a WORKDIR environment variable, which defaults to your $HOME. The WORKDIR becomes your current directory when you enter the container's command line. Here's an example on how to enter your containerized build environment to cross-compile for aarch64:

$ git clone https://github.com/cmtp-responder/cmtp-responder.git  # once
$ cmtp-responder/Docker/aarch64.sh build                          # once
$ WORKDIR=${PWD} cmtp-responder/Docker/aarch64.sh run
aarch64@cmtp-responder-build:/home/build$

If you prefer not to build your Docker image locally, you can optionally explicitly download it from ghcr:

$ docker pull ghcr.io/cmtp-responder/cmtp-responder-build:aarch64

or simply use REGISTRY environment variable when calling the script:

$ WORKDIR=${PWD} REGISTRY=ghcr.io/cmtp-responder cmtp-responder/Docker/aarch64.sh run

which will download the image on the first use.

You enter the other two build environments similarly.

Another related positive note is that you can actually use this same build environment to compile libusbgx and gt on top of compiling cmtp-responder. So, inside the correct container you can use dedicated scripts. They are written in such a way that they "detect" the target architecture depending on the user name, and the user name inside the container is deliberately set to a correct value in the Dockerfile.

libusbgx

Invoke build-libusbgx.sh to build e.g. for aarch64:

aarch64@cmtp-responder-build:/home/build$ cmtp-responder/Docker/build-libusbgx.sh

If everything goes well, then you get libusbgx directory and its install-${ARCH} subdirectory with build artifacts installed. To build for other architectures, you simply need to enter the appropriate build environment; the script is invoked the same way, and it is smart enough to know the differences.

gt

Invoke build-gt.sh to build e.g. for aarch64:

aarch64@cmtp-responder-build:/home/build$ cmtp-responder/Docker/build-gt.sh

The build artifacts end up in gt/install-${ARCH}. To build for other architectures, the same comment applies as for libusbgx.

cmtp-responder

Invoke to build e.g. for aarch64:

aarch64@cmtp-responder-build:/home/build$ cmtp-responder/Docker/build-cmtp-responder.sh

The build artifacts end up in cmtp-responder/install-${ARCH}. To build for other architectures, the same comment applies as for libusbgx.

If you're curious, Docker images contain minimal dependencies needed at compile time to build:

• libusbgx
• gt
• cmtp-responder

libusbgx is checked out at container image build time to its /opt/src/libusbgx. It is then built and installed into the system (multiarch, in case of armhf and aarch64) location (still inside the container) so that it is picked by the build system when gt and cmtp-responder are being built by the container user. This copy of libusbgx is used, so to say, behind the scenes. If you want to explicitly build libusbgx then follow the instructions above.

What do I do with the build artifacts?

To deploy cmtp-responder to a target system all 3 (libusbgx, gt, and cmtp-responder) must be built, collected, and transferred. The most basic way of doing it is to configure their build so that they are installed to a user-writable location (just like in the examples above), then make a tar.gz and unpack it in the target system. ATTENTION! Recently distributions are switching to /bin and /lib being symlinks, so --keep-directory-symlink MUST be passed to tar when unpacking on the target, ohterwise /bin/ and /lib will be lost. The repository now contains an example make-dist.sh. It assumes that libusbgx, gt, and cmtp-responder directories are next to each other in the current directory and that they contain install-${ARCH} directories with files installed at the build stage, and generates cmtp-responder-${ARCH}.tar.gz, etc-${ARCH}.tar.gz, gt-${ARCH}.tar.gz and libusbgx-${ARCH}.tar.gz, also in the current directory. Example invocation (which does not need to happen inside a container):

$ ARCH=aarch64 make-dist.sh

You then transfer the .tar.gz files to your target and there you can use deploy-tar-gz-to-target.sh to deploy and remove-from-target.sh to revert the installation from the target.

Obviously, Linux distributions are more than welcome to do their own packaging. The examples above are just that - examples which you can follow to make your life easier while building and/or developing cmtp-responder.

I wanna try it! (and I don't have a suitable board)

There's already a blog post for you (mentioned in the first paragraph) about running on top of dummy\_hcd. To make it even easier (your definition of "easier" might be different than mine; what I mean here is that you can get away without having to mount an installation media image in qemu and manually install a Linux distro to your qemu image), you can take advantage of a great tool called debos, which builds a complete image for you to be run in qemu. And, obviously, you can use another Docker container to run debos - please follow the instructions at that project's site to obtain the image. This is the recipe you want.

Invoke debos and convert the resulting image to QEMU's native qcow2 with generate-img.sh.

Please note that the resulting image is an UEFI image, so you need to tell QEMU to use appropriate "bios", that is already handled by run-vm.sh.

From this moment on you have a suitably equipped virtual machine to try your freshly built cmtp-responder. The login and password are cmtp-responder. Just follow the instructions from the "What do I do with build artifacts" section. Oh, and ssh will be your friend to copy the *.tar.gz and "deploy/remove" scripts. Now is the time! Please note that you (first) need to modprobe libcomposite followed by modprobe dummy_hcd to actually run dummy\_hcd. If you don't then, at deployment time the systemd will (rightfully) complain.

I don't need to mention that for this option you want the build artifacts for x86\_64.

Testing, testing

The trouble with any MTP responder is that to test its functionality and/or regressions, you actually need to connect it to a USB host and perform tests from there. What makes it even more difficult is that there are different kinds of USB hosts, each with its own implementation of the USB stack. We are addressing exactly this! In cmtp-responder's repo now there are tests. Let me cite the README:

The tests are written in bats.

They are built according to a layered architecture: the ["driver"] test cases are architecture-agnostic and they run using os-setup, usb-utilities, and mtp-utilities. The os-setup, usb-utilities, and mtp-utilities are wrappers around a specific variant, selected in config.bash.

As of this writing there exist 3 such backends:

• bash dummy (for debugging purposes, no actual interaction with any MTP device)
• Linux gvfs (for running on top of gvfs on Linux)
• Windows Power Shell (for running in WSL2 on Windows)

os_setup can create an OS-specific "cookie", which is then passed unchanged to the toplevel "driver" bats tests and then to the actually running backend. As of this writing, the Linux variant stores the USB bus ID and device number, the expected mountpoint of a gvfs directory corresponding to the MTP device contents, and a gio handle of the MTP device under test. The Windows Power Shell variant stores locations of helper directories and the "product" string as seen by the Windows OS and MTP store name.

The tests are run like this:

$ bats 01_enum.bats
$ bats 02_file_operations.bats

Please note that in config.bash you may want to configure the backend variant and numbers of iterations for several kinds of activities performed during testing. You may also need to configure your device identification in mtp-device-id.bash. In particular, to run on top of dummy\_hcd you need to say:

MTP_DEVICE_HCD_DRIVER=/sys/bus/platform/drivers/dummy_hcd
MTP_DEVICE_HCD=dummy_hcd.0

NOTE 1: Now that we have testing facilities, if you add/change functionality of cmtp-responder, you _need_ to write appropriate tests. If you ask nicely, next time I might want to blog about how to write such tests

NOTE 2: The Windows Power Shell backend generally works, but is not great, we can probably do better, but I don't know Power Shell and Windows enough. This is definitely a room for one of you, dear readers, to step in and help us improve.

NOTE 3: The Linux gvfs variant is ok, but perhaps a bit convoluted, a more straightforward backend would use mtp-tools without having to mount/unmount with gvfs (but then in exchange handling mtp-tools output properly would be complex). Room for another brave volunteer!

NOTE 4: There is no Mac variant of tests. We do want our MTP devices to be connectable to various USB hosts, don't we? Mac-specific knowledge and Mac machine donations are welcome!

cmtp-responder changes

Certain changes have been applied to the repo. Let me list them here:

• obviously, tests have been added
• CMake 3.0 is now required, which seems not too strict a requirement
• cmtp-responder now prints a better message if sd_listen_fds() returns less fds than expected - before this change the message was that inheriting the descriptors simply failed. Instead, the number of received fds is printed, unless sd_listen_fds() fails completely, in which case the old message is output 
• CMakeFile.txt has been reworked to use better syntax and to install the build artifacts in a more configurable way utilizing GNU install dirs 
• Remnants of Tizen heritage have been removed and some cleanups have been applied.

Tools of the trade

Finally, let me briefly talk about accessing your cmtp-responder from command-line on a Linux host. One of the options is to use gio:

$ gio mount mtp://Collabora_MTP_Gadget_000000001
$ ls /run/user/1000/gvfs/
'mtp:host=Collabora_MTP_Gadget_000000001'
$ gio mount -u mtp://Collabora_MTP_Gadget_000000001

gio mount is accessible in /run/user//gvfs.

Another option is mtp-tools, e.g.:

$ mtp-detect

Please note that there used to be certain bugs in mtp-tools, but we have fixed them. However, until your distro picks the fixed version you might want to build from source on your host:

$ git clone https://git.code.sf.net/p/libmtp/code libmtp-code
$ cd libmtp-code
$ ./autogen.sh
$ mkdir install
$ ./configure --prefix=${PWD}/install
$ make && make install

If you happen to run the virtual machine built with debos, you can just copy the binaries from ${PWD}/install to your virtual machine, no need to build _inside_ the VM.

Summary

It's been a while since we last visited this topic, but we believe we offset this with lots of good stuff for you: you are receiving a unified build environment with Docker, an example strategy for building and deploying to a target, an easy/automated option to create a virtual environment to test cmtp-responder even without appropriate hardware, you are getting cmtp-responder tests to be run from the USB host perspective for two operating systems and we make sure we speak the same language in terms of the tooling used to interact with MTP devices. That's a lot for you to try!