As usual, I’m writing reports of the sessions I’m attending at Open Source Summit / Embedded Linux Conference / IoT Summit Europe 2018. This time, however, I’m doing it on the Mind website:
https://www.mind.be/en/2018/10/22/OSS-EU-2018.html
You can also find other reports there, e.g. of Kernel Recipes in Paris.
Posted in Uncategorized
This is an update of what changed in ftrace since Steven’s presentation about its state in v3.18.
What ftrace already did: function tracing, function graph tracing, snapshots, trace events, triggers, and debugging. It uses the hooks installed by gcc’s profiling. Since gcc 4.6, gcc has the -mfentry modifier to call fentry() (instead of mcount()) before the stack frame is set up, so that you get access to the parameters. This is used by live kernel patching to replace a function. The mcount/fentry call adds about 13% of overhead, so they are replaced by NOPs so the overhead becomes unmeasurable. When tracing is enabled for a function (set_ftrace_filter), the NOPs are replaced by the original jump.
In addition to filtering on function name, you can filter on PID or children. Triggers define additional things that get done when a trace point is hit: saving a stack trace, save the trace buffer (= snapshot), turn trace off or on. Also profiling info can be gathered: hit counts, stack usage. max depth limits how deep you trace (starting from the switch from userspace, i.e. syscall or page fault or interrupt).
Instead of tracing on functions, you can also trace on trace events (trace events are added by ftrace to each trace point; the trace point is defined by the source code and gets used by ftrace, perf, …).
For debugging: trace_printk prints to the trace buffer instead of the log buffer, requires no locking etc. A sysctl allows to dump the trace buffer on panic.
Since 4.0: when two different tracers are attached to different functions, a separate trampoline is instantiated for each. This avoid iterating over the tracers which is quite expensive. Before, the trace function would be called directly but only if only a single tracer is in use (globally). Now, it can still be called directly as long as it’s on a different function/tracepoint. To be able to do this, the ftrace infra needs to allocate space for the new trampoline and make it executable, so it’s a bit complicated.
Also added in 4.0 is NOT in the trace event filter logic. It existed for functions but not for events.
Also added in 4.0, the very dangerous tp_printk kernel command line option. It will use printk() instead of tracing to a buffer.
[missed some things that went to fast]
Since 4.4 it is possible to filter triggers on CPU# or PID.
Since 4.4 it is possible to set tracing options before the tracer is enabled, so you can do e.g. filtering immediately.
Since 4.4 it is possible to filter functions by module (or not module) with :mod:modulename. Including *:mod:!* to trace all functions not in modules. Since 4.14 it is also possible to do this before the module is loaded.
Since 4.4 there is a separate filter file for PIDs, which overcomes the limit on the string size on the generic expression filter.
Since 4.9 the event PID filter is extended to also trace children of that PID, so you don’t need to add each child as it appears. Since 4.14 also for functions.,
Since 4.14, the filters have full glob support, also * in the middle and ? and [].
Since 4.14, a hardware latency detector is added to detect SMIs/NMIs, by running loops with interrupts disabled. You give it a width and an interval (called window).
What’s coming: module init functions, …
Steven ran out of time so not everything is covered. Check the slides.
Maxime worked on several projects involving Preempt-RT:
Real-time = deterministic behaviour: bounded latencies, absolute priorities for tasks (SCHED_FIFO, _RR and _DEADLINE), handle complex cases like priority inversion (rt-mutex with priority inheritance), starvation, …. Most of this is already in upstream Linux. What Preempt-RT still adds: full kernel preemption; various optimisations for worst-case scenario instead of common-case scenario.
Full kernel preemption consists of forcing threaded interrupts (so we get priorities for interrupts as well), making locks sleepable (spinlock normally doesn’t allow anything else on the same CPU; sleepable lock will yield when it doesn’t get the lock). Nothing else changes, so all the normal Linux OS is still there. Only the non-RT tasks will have to live with what is left over by the RT tasks.
To analyse the effect of the Preempt-RT patch, use tools like vmstat, mpstat and pidstat. E.g. mpstat shows how many interrupts each core handles. However, take care because they show results differently. For example, without threaded interrupts, interrupts are not counted as context switches in these tools, while with threaded interrupts each interrupt gives 2 (non-voluntary) context switches (one to the interrupt and one back).
As a benchmark, use stress-ng with a fixed number of operations and measure execution time. Just CPU makes no difference. “fault” (that triggers page faults) is significantly slower. So you need to test this. Note that stress-ng contains cyclictest as well.
In addition to applying preempt-RT, you need to do more things to improve predictability:
Clearly, you need to know the system. E.g. DMA can give latencies on the SoC bus. SMI is not maskable (it does thermal management…) so measure how long it takes. Hardware resource sharing (e.g. SIMD unit shared between different cores).
Bean and Zoltan (the speaker) work at Micron in the embedded business unit, in storage software, often in automotive. As part of this they have quantified the storage system overhead in embedded systems for access to eMMC, UFS and NVMe, i.e. how much of the speed provided by these storage technologies can actually be achieved by userspace. Also quantify the overall performance improvement of NVMe over UFS. But comparing is difficult since you can’t have a NVMe device that is fully equivalent to a UFS device.
All three technologies are a NAND chip with a controller and firmware. eMMC can get 400MB/s at its interface, may go up to 566 in the next generation. UFS Gear3 can have two lanes of each up to 728MB/s. NVMe Gen3 1000MB/s per lane.
For tresting they use Fio in single and multi-threaded mode, always using DirectIO or sync IO. Using function_graph tracer and blktrace. The trace points allow to measure latency: from user space submission to BIO submission; from BIO submission to storage device submission; for data transfer up to completion algorithm; from completion interrupt to block layer completion.
They did experiments on two boards: a somewhat older 2xCortex-A9 Zedboard, and a newer 4xCortex-A57 Nvidia board. On the Zedboard, eMMC performance is completely dominated by software overhead, ranging from 63 to 92% of the total latency spent in software. To some extent this is caused by the cache invalidation which takes a long time in Cortex-A9. On the Nvidia board, performance is a lot better for large sizes (12% to 39%), but stil significant for small 4KB request sizes (up to 72%).
Experiments for UFS and NVMe have to be done on different boards. For 4K write there is still significant (60-74%) overhead. The graphs showed that with 8 threads the latency is significantly decreased; someone in the audience suggested that this is due to interrupt coalescing, which amortizes the interrupt time over all threads. Even for 128K accesses the overhead is non-negligible. The results show that the overhead of NVMe is indeed significantly lower, e.g. for 4K random write the overhead of NVMe is only 66% that of UFS. To estimate the system-level performance, they used some formulas that I didn’t understand but that result in 8-25% speed difference between UFS and NVMe. This difference is not so much because of faster NVMe, but mostly because the Linux stack is better. However, the hardware queue size is also a factor: NVMe can support a lot more outstanding tasks. For very high thread counts, UFS performance starts to drop while NVMe sustains.
Sergey made the patches, Steven is the reviewer, so Sergey did the presentation.
printk() is complicated. It takes a number of locks, which ones exactly depends on your .config. In addition, you can do printk() from an NMI that interrupted an NMI. It is easy to deadlock, e.g. printk() may take the scheduler spinlock, so you can’t use it in the scheduler when that lock is taken already. lockdep as well: reporting a deadlock will call printk() again. Therefore, for a long time printk() idd lockdep_off() and disables the RCU validator.
So printk_safe was created. This allows printk() to be reentrant, enables lockdep again in printk(), and generally make printk() less deadlock prone. But it’s not quite reentrant yet. printk_safe() can’t be called from sleeping context.
The fundamental prolbem is that printk depends on two different types of locks: locks internal to printk, and locks that come from somewhere else (e.g. locks in the serial driver). A solution would be to do printk_deferred() everywhere, which means just one internal lock. But the actual printing has to be done somewhere, and in bad lockup scenarios there is no guarantee that e.g. IRQ context will ever arrive again. Alternatively, we could fallback on early_printk which doesn’t do locking but breaks e.g. dmesg. For example, add a write_on_panic() callback to the console struct that is completely lockless and can’t be called from any context except panic().
There used to be zap_locks() in printk that would drop locks when it detects recursion, but that only looked at the internal locks while the external ones are actually the tricky ones. Since there is now a better way of handling recursion, the function is removed. However, the zap_locks() approach could be used in console drivers. Add a zap_locks() member and call it from panic context so that you can re-enter the write() function.
It could also be possible to remove locking from the console drivers, i.e. the console functions don’t do any locking themselves, the callers do, by calling a lock()/unlock() member.
console_sem is used for a lot more than printk(). printk() uses it to make sure the print happens only on a single CPU. The console also uses it to handle line wrapping, UTF8 encoding, avoid mixing with TTY processing, cursor blinking, avoiding race between printk() from user context and printk() from IRQ, … Also for non-printk() related things: power management, adding/removing consoles. Some of these can schedule() with console_sem held. This means that in a livelock situation, printk() won’t come out again because it won’t acquire the console_sem.
The problem is that printk() is a mix of different subsystems: framebuffers, serial ports, TTY, sched, timekeeping, …. Maybe it’s time for a new printk() API. Change into a polling API: printk just goes into a buffer, and the consoles poll the log buffer. This allows to remove console_sem. This is in fact how serial drivers work already: they transmit xmit characters out of their buffer. This might even work with not all consoles working in polling mode. The problem is that there is no immediate flush, so when printk() returns the message is not printed yet.
printk() makes sure there is just one CPU printing the buffer, the other CPUs just append to the buffer. The printing CPU continues printing until the buffer is empty. However, if the printing CPU is in atomic context, IRQ, you’re adding unbounded latency which is some kind of lockup. So you’d like some preemption points in there. But that doesn’t solve all problems, and also slows printk() down which is an issue for e.g. OOM print which slows down OOM killer. So for some people it’s actually making things worse. Again, polling could be a solution, but now from printk_thread, but that didn’t work. People really want to have direct printk(), because you want it when your system dies.
Krzysztof is in a team that mostly works on persistent memory programming. syscall_intercept is a satellite project. Source on https://github.com/pmem/syscal_intercept.
libpmemfile is a fully userspace filesystem (with persistent memory as backend), so not FUSE-based, nothing goes to the kernel. syscall_intercept is part of libpmemfile. It patches all the system calls and replaces them with jumps to a hook function. So it’s like LD_PRELOAD but then for syscalls instead of libc functions.
syscall_intercept patches the code. To be able to do that, it first disassembles the code to find the syscalls using libcapstone, then find their context (not always trivial/possible), and hotpatch the code with a jump. It only patches libc – in most cases that’s the only one doing syscalls, but it’s also possible to patch the entire .text in the binary (except libsyscall_intercept itself and libcapstone). There is a single syscall hook function that checks the syscall number argument to decide what to do.
Capstone is an open-source disassembly framework. It is used to iterate through all instructions and evaluate if it is a syscall. Also the next instruction has to be evaluated to see if it is relocatable, and if it depends on the instruction pointer. The call can’t be replaced with a direct call or jump to a C function due to the argument and stack prologue, so there is a wrapper routine to set that up. For each syscall instance, a wrapper is instantiated that jumps back directly to the original address, which avoids problems with stack etc. Since on x86_64 a syscall is 2 bytes while a long jump is 5 bytes, you need to make space. If the subsequent instructions can be relocated, then they are put in the wrapper instance. Else, they look for a nearby hole of 5 bytes and issue a short jump to it. And there are a few other solutions too.
SYS_clone is a special case because you have two processes with a different stack pointer, simply restoring registers doesn’t work. So there is a complicated workaround. Also problems with rt_sigreturn and ptrace that don’t have a workaround.
syscall_intercept is just an SDK. You need to write a library that is loaded with LD_PRELOAD and that does something useful in the syscall hooks. To avoid loops, any actual syscall made by the library has to use syscall_no_intercept() instead of syscall().
This can be used for example to make a replacement of strace() that doesn’t make any extra syscalls, just logs every syscall. This is one of the examples in the repo.
Problem when running the program under GDB: you don’t want to instrument gdb.
Code is patched only once, so generated code or dynamically loaded code is not hooked. Also handwritten assembly that uses some tricks or non-standard ways of issuing a syscall could be problematic.
Other things you can do with this library: Error injection, a faster strace, userspace device emulation (which is basically the libpmemfile use case). Also, the same approach could be applied to other instructions than syscalls, as long as they are recognisable in assembly.
syscall_intercept is currently x86_64 only. It could be extended with other arches supported by libcapstone, but that would require supporting their syscall interface.
Interesting question from the audience: could the vDSO approach have been used instead of hotpatching? The speaker nor the audience knew an answer to this.
Yann works for Orange and develops set-top boxes in three teams in two locations. Most are application developers, not Linux or embedded experts. The main part of the firmware comes from third parties. To put this together, they need a generic build system that is not dependent on the target and middleware. It should be easy to use and not take too much build time. A home-grown build system was tried before but no success. The build system provided by the provider of the middleware is very specific, for a specific target, not generic enough.
Evaluated build systems:
Buildroot is simple (package in a few lines) and efficient (doesn’t take longer than absolutely necessary), is entirely community-driven (no companies behind it), community resources like website and manual.
Build process: make .config file, build toolchain, build packages, run finalize step that cleans up unnecessary cruft, generate filesystem images (with hooks at various steps). Package build process: download, extract, patch, configure, build, install (with hooks before and after each step).
External is the place for costumisations. Use Buildroot as a git submodule.
Config files are saved as defconfigs under the configs/ directory. Some people use defconfig snippets and stack them together to make different variants of the board. However, that way you can’t use the Kconfig UI, it only allows to save a total defconfig.
New packages go in packages/, new filesystems use go in fs/, all use the same Buildroot syntax.
For fine-tuning, you can use a custom skeleton and overlays. However, don’t use overlays too heavily. Preferably create a package for them, even if it just copies stuff to the target. That allows to use all of the possibilities of Buildroot, e.g. taking into account size.
In the external tree, it is also possible to add extra logic, e.g. a custom make rule to check that there are no circular dependencies. Or add a hook that is run after every package build or just before creating the filesystem. Preferably move things to helper scripts and don’t do it directly in make, much easier to maintain (e.g. syntax highlighting). It is even possible to add infrastructure. Orange added a orange-package infrastructure that adds some features, like installation of documentation.
To avoid creating too many make variables (which makes ‘make’ slower), avoid defining new variables in the infrastructure. Instead, use as much as possible generic variables. In addition, it is more readable since you don’t have to double-dollar to escape the call to eval. Adding such internal infra makes it much easier for developers to add packages.
UIDs can be created automatically by Buildroot. However, that means that adding a package could change the UID chosen. To avoid that, Orange packages must declare the UID explicitly. The orange infra checks that it is explicit.
For D-Bus, there have to be authorisation files that allow a specific application to access specific objects on the bus. That is tedious to declare. So Orange has a script that scans the source code and looks which access each package needs, and creates the authorisation file automatically. There are some exceptions that have to be handled, that is done with extra variables in the package .mk file. AppArmor uses a similar approach. This is also a reason to not put things in an overlay but in a package: that way, all this magic can work.
Orange has added a lot of infrastructure to automate things without need for the developer to take (much) action. It makes sure that these things are done systematically, reproducibly and maintainable.
Ann researches how to manage contributors in open source projects. Before that she worked as a developer and as a community manager. This presentation is a summary of research results on casual contributors.
Casual contributors are important to a project because they are often the majority of contributions. Also non-patch contributions: testing, help at a conference, … They increase innovation and software quality. But they also diffuse knowledge about the project to their social network.
Casual is not a good word, because they are often committed. They’re not necessarily one-off. Also they are often habitual contributors in other communities. Perhaps “Episodic” is a better term.
Many of the community management techniques also apply to casual contributors. But without a strategy, how do you know you’re managing them effectively? If you’re only looking at converting drive-by contributors to habitual contributors, you miss the valuable episodic contributors that also remain committed to your project.
Five factors influence the intention to remain (intention to remain is used as a predictor for actually remaining, turns out to be the best one). Motives: enjoyment, socialising, personal benefit (this is a negative motive, once the itch has been scratched there is no reason to stay). Technical barriers discourage enjoyment and socialising and are affecting episodic contributors more.
Social norms: how a contributor perceives the response of their social network to their participation (peer pressure). For open source contributions, the social network often knows less of the involvement than in other volunteer work so it is less relevant. However, they are responsive to personal invitations – this is true for volunteers in general and also for OSS contributors. Especially for non-code contributors.
Psychological sense of community: do they feel welcome? Inclusivity is often mentioned as an important factor.
Satisfaction: did the experience match the expectation? This is one of the strongest factors towards intent to remain.
Organisational commitment: identify with the community, feel part of something bigger. People who talk about their involvement with family and friends are more likely to remain (not a causal relation).
Create a strategy for episodic contributions:
[Note: slides that are online have a lot more text than what was presented at the conference, so it should be quite readable.]
Software gets bigger and bigger, this is certainly the case for Linux as well. In addition, Linux is configurable so not all code is built. Different kinds of developers are involved: casual contributors, maintainers and janitors.
Janitors clean up other people’s messes. They know coding style conventions and API changes. However, they don’t know the subsystem they affect deeply, and often they don’t have the possibilities to test well. There is a risk of a silent compiler failure: the janitor modifies some code, compilation succeeds, but the compiler actually didn’t build the modified code because it was configured out.
JMake handles the silent compiler failure to improve the reliability of janitor code. This grows out the coccinelle work: when doing this kind of tree-wide change, it is hard to make sure that you’re actually testing what you changed. People want to have immediate feedback of what they do, so an online tool that sends a mail is not appropriate. Even under allyesconfig, some parts are not built.
JMake looks at a diff and what gets built, and reports any modified line that did not get built. JMake can also find that the line can be built by compiling allyesconfig for a different architecture.
Tools available to JMake: make.cross and allyesconfig, in-tree defconfigs. Trying all that would take too long, so JMake has heuristics. For files in arch/, allyesconfig is used. For drivers etc. it is alsways x86/allyesconfig. If that fails, look in the Makefile if there is a CONFIG variable associated with the C file. A final heuristic is to use the same arch as for the other lines in the patch.
Extra challenge for .h file, because you don’t know where it will be used. Additional complication that a header file with the same name is often used for different arches. Also conditional complication is especially heavily used for .h files, i.e. not including unless some symbol is defined.
To find out which lines are actually compiled, you could look at the line numbers in the compiled code (.lst file) (but this doesn’t work for macros); introduce a syntax error in the modified line and check that an error is reported (but not reliable and compiler-specific); mutate source code and verify that the mutation is in the .i file (preprocessed code), and if yes also verify that the (unmodified) source file actually gets built. The mutation is done by adding a string (which never gets modified by preprocessor) surrounded by characters that are not valid in C. If the mutation is in a macro, it will end up on a different line but it will end up somewhere.
To run jmake, you give it a commit ID or range of commit IDs to look at. It then goes through the above process. It looks at changes in blocks that are not interrupted by #ifdef. It reports for each file how it managed to build the modification: “make” = x86/allyesconfig; “make.cross ARCH=…” = ARCH/allyesconfig; “make.cross ARCH=foo:bar_defconfig” = with a different defconfig than allyesconfig; Failure = needs to be looked at manually. For a commit that affects 83 files it took about 8.5 minutes on Julia’s laptop.
Julia ran it on 11K commits. 96% of the modified non-arch files are visible in x86/allyesconfig. 365 .c files and 75 .h files outside of arch are not visible in x86 but are in some other arch, typically arch. 415 of .c files doe compile but not all modified lines are compiled; 54 of these can be found in other arches, but 361 cases JMake fails.
Some issues:
Julia made an objective definition of what a janitor is based on some metrics about the type of commits people make. Basically a janitor makes changes in a lot of different files, in different subsystems. She detected 21 janitor commits that do not get built on x86/allyesconfig.
The tool works well when you’re reacting to dependencies, e.g. adding new arguments to a function. It does not work when you create dependencies, e.g. adding const to a declaration – the latter would need to build all the users of that function, but JMake doesn’t do that.
Gilad is the maintainer of the ARM TrustZone CryptoCell Linux device driver.
Smart devices are used for everything, so we need to be able to trust them. However, we also want a frictionless user experience and be able to do anything with it. This is guaranteed to fail, so we need a second line of defence, a trusted way of failing. If someone gets hold of our device, we don’t want them to have access to all our secrets, to get access to additional resources, and we want to know about it and be able to get them out again. We want trusted boot: reboot the device and it is safe again.
All the components are there, we have to make them fit together.
Secure boot (Android style, but others are similar): chain of trust through the boot process, each component verifies the next one. ROM uses a public key in e.g. eFuses to verify bootloader. Bootloader verifies kernel and boot fs. OS verifies the full rootfs. Root key can also be in flash with just a hash in eFuses, or it can be a certificate chain with just hte hash of the root in eFuses.
Checking rootfs is done with DM-verity. It prevents a persistent rootkit: if the persistent storage is changed, we will know. DM-verity adds hashes and signatures to a readonly filesystem using device-mapper. Check is done every time we access the filesystem, not at boot. It uses a Merkle tree of hashes of blocks to arrive at a root hash that can be verified through a signature. The Merkle tree is stored on the device, so we need to verify log4096(device size) hashes. Cfr. figure in the slides.
This works only for readonly devices: when a block changes, the entire Merkle tree changes. For read-write data, the simplest is using full-disk encryption (dm-crypt) which implicitly does authentication. dm-crypt is a device-mapper layer between the actual filesystem (e.g. ext4) and the block device (e.g. eMMC) so neither of these knows about the encryption. This uses a single key for everything accessing the device, and the key is kept in memory all the time. The key is password-protected.
Problem with whole-disk encryption: multiple users, not possible to avoid encryption for some use cases. For example, alarm clock app is in encrypted storage, if the device reboots during the night, you have to give the password before the alarm clock can start running…. fscrypt solves this by pushing encryption into the fs layer, which allows different or no encryption keys for different directories and files. So e.g. the alarm clock app may be encrypted with a key that is stored in the rootfs, while the sensitive information is encrypted with a user-provided password. Limitation of fscypt: doesn’t hide all metadata, e.g. file size is not encrypted. Multiple keys can be loaded separately into the kernel. When the key is available in the kernel, you can see the file. When the key is not loaded, you can see there is a file but not its name or content.
The problem is the key: it has to be put in the kernel and stay there, so it is vulnerable it the kernel is compromised. Solution is some trusted execution environment, e.g. TrustZone in ARM. TrustZone is a hypervisor mode (called TEE, Trusted Execution Environment) that has access to memory that the normal OS has no access to. The OS then asks the TEE to store the key in memory that is not accessible to the kernel. It is never possible to get it out again; to do encryption, the kernel asks the TEE to put the key in a hardware crypto engine.
Instead of a TEE, you can also use a Trusted Platform Module (TPM) discrete from the CPU. Keys are directly stored in there and never go to flash; they are even generated in the TPM so they really never ever go to memory. But of course the TPM can still have bugs that can be exploited. The TPM can also do attestation: give access to certificates only if a certain set of hashes (of the HW and SW state) is provided in a certain order. This is done with Integrity Measurement Architecture (IMA) subsystem in Linux. Attestation is a way to check a sequence of hashes without needing to store all the hashes.
Mind's Embedded Linux Blog 