Farewell packages.namniart.com

2023-07-05 00:00:00 +0000

Farewell packages.namniart.com

The time has come to retire packages.namniart.com. There hasn’t been a need to upload any updated packages in years, and traffic has been steadily declining, from 200GB/year when I first set up the server, to just over 500MB over the past year; with all of the recent traffic requests for the directory index and web crawlers. I haven’t added any new ROS releases or Ubuntu releases in many years, and traffic on the old releases is non-existent at this point.

The Open Source Robotics Foundation (OSRF) has been maintaining the ARM builds for ROS over the past 6 or 7 years, and continues to maintain builds of the latest ROS 1 LTS, ROS 2 LTS and ROS 2 latest release.

I’ve hosted packages.namniart.com out of my personal budget since the beginning, and it’s time to retire it and reclaim a small part of that budget. I don’t expect very many people to notice, but figued I’d make a quick blog post to announce the shutdown.

I have a complete backup of the server if it becomes necessary. If you have urgent need of the files that were hosted there, please email me and we can work something out.

Pitfalls and Traps in Linux Multicast

2020-06-01 00:00:00 +0000

Pitfalls and Traps in Linux Multicast

IPv4 multicast seems like a wonderful idea: you send one packet, your network processes it and sends a copy of it to each host that needs it. Over the years, wiser network engineers than myself have learned that multicast on a routed network can be difficult to configure and may not be worthwhile, but multicast has found adoption on local networks as an alternative to broadcast, and can be efficient if the majority of network switches support IGMP snooping. There are, however, many pitfalls on the path to using multicast on a local network.

If you want to use multicast on a Linux host with multiple network interface, in addition to the usual pitfalls and traps, there are several well-hidden tar pits where you can get stuck for days or weeks.

Background on IGMP and Multicast Group Membership

IGMP and Membership

The IGMP protcol is used by hosts to inform the routers and switches on their local network that they wish to receive multicast traffic. End devices send IGMP membership advertisement, and routers and IGMP-aware switches listen for these IGMP membership advertisements and ensure that multicast traffic is forwarded to those end devices. The IGMP protocol is defined by RFC-1112 (IGMP v1), RFC-2236 (IGMP v2) and RFC-3376 (IGMP v3).

IPv4 Multicast to Ethernet Layer 2 address mapping

To allow switches to handle IPv4 multicast traffic with minimal knowledge of IPv4, each IPv4 multicast address is mapped to an Ethernet MAC address by Section 6.4 of RFC-1112. The lower 23 bits of the IPv4 multicast address are placed in the lower 23 bits of the 01:00:5E:00:00:00 MAC address to create the Ethernet multicast address. (This results in multiple IPv4 multicast addresses mapping to the same Ethernet address).

Multicast vs IGMP

When working with multicast, it is important to recognize that IPv4 multicast, Ethernet multicast, and IGMP are related but separate protocols. IGMP is used to dynamically define and manage multicast groups on a network, but the actual packet forwarding is handled by the network switches based on the packet’s Ethernet multicast address, and by routers based on the packet’s IPv4 multicast address.

In many switches, you may only be able to inspect IPv4 or Ethernet multicast address, and it is important to know how they are related in order to debug forwarding issues.

Some switches may also be configured with static multicast group memberships based on the IPv4 or Ethernet address of the group. The implementation of this is left entirely to the network switch vendor and varies significantly between switches.

IPv4 Multicast Address Allocation

IPv4 multicast addresses are allocated by the IANA. Generally, these addresses are allocated to protocols that operate on local networks and which need unique identifiers.

Of special note is the 239.0.0.0/24 range, which is allocated for use within an organization and which should not be routed across the internet. This range is defined by RFC-2365. multicast address allocations.

IGMP snooping switches without an IGMP Querier

IGMP snooping is a feature that can be enabled on many higher-end switches. The switch listens for IGMP group membership advertisements, and for each incoming group membership, it adds that port to the advertised multicast groups, so that any multicast packets to those groups will be sent to that port. This prevents multicast traffic from being delivered to ports that aren’t interested in it. This saves bandwidth, and in some cases can prevent small devices from being overwhelmed by large amounts of multicast traffic.

Switches with IGMP snooping also implement a group membership timeout: if the IGMP membership report hasn’t be received recently, then the group memberships are removed from that port. This is necessary for proper handling of devices that are attached through multiple switches, but there’s a trap here: devices only send multicast membership reports when they join a group, or when they receive an IGMP membership query.

The network device that sends IGMP membership queries is a IGMP Querier. All of the devices that I’ve seen so far either don’t include an IGMP Querier, or have it turned off by default. This results in the very fun “multicast works for a few minutes and then stops” issue. Of course, this happens because the multicast group memberships time out.

Of course, now that you know all of this, the fix is simple: set up an IGMP Querier! All of the IGMP-aware switches that I’ve seen include one, and turning it on is usually simple. The only configuration parameter for it is a querier address, which leads to the next issue…

Choosing a poor IGMP Querier Address

Some posts on the network tell you that you can choose any address for your IGMP querier, and some even suggest choosing 1.0.0.0, 2.0.0.0, etc so that it is easy to control the priority of IGMP querier election. They are WRONG.

On multi-homed Linux machines (and probably other operating systems too), the OS will drop any packets which originate from an implausible IP address. If your Linux machine has multiple interfaces, and receives a packet from, say, 1.0.0.0 on an interface which only has local connectivity, Linux correctly deduces that this packet could not have possibly come from that network, and drops it. Since the IGMP querier is sending packets from the querier address, this will cause Linux to drop the IGMP queries instead of responding to them.

Here is an example network that exhibits this issue. The Linux Laptop is attached to two networks, one wifi and the other wired. Its internet connection and default route go through wifi so its default route is 192.168.1.1 (the wifi router).

Since we have not added any additional routes to the Linux Laptop’s routing table, it knows that the only traffic that can come from the wired network should be from the 192.168.2.0/24 address range, or multicast traffic.

If the switch’s IGMP querier is configured with an IP address such as 1.0.0.0, when the Linux Laptop receives that traffic on its wired network port, it will drop it because it is impossible.

Again, the fix here is pretty simple: allocate an IP address to your switch, and set the IGMP to the switch’s IP address, or allocate a dedicated IP address on the same subnet that is explicitly for use by the IGMP querier.

In the example above, the switch should be allocated an IP address from the 192.168.2.0/24 range; perhaps 192.168.2.3, since this address is not currently used.

Sniffing Multicast traffic

Since switches with IGMP snooping only send multicast data to the ports that are joined to those groups, common packet capture tools such as tcpdump and wireshark will not capture any multicast traffic if they device where they are run is not joined to the multicast groups of interest.

The best solution that I have found for this is to either write custom python tools to join the relevant multicast groups and capture the traffic, or to configure the switch to forward all multicast traffic to the port in question; most switches call this “router mode.” isn’t sending you any packets!

Inbound Multicast on Multi-homed Hosts

When joining a multicast group with setsockopt IP_ADD_MEMBERSHIP, Linux allows you to specify a local IP address to bind to, or if you specify INADDR_ANY, it will choose an address for you. If you choose INADDR_ANY, Linux chooses “an appropriate interface”! On a computer with a single network interface this is fine, but if you have more than one interface, there’s a chance that INADDR_ANY will choose the wrong interface.

The fix here is a bit painful: you have to write your software to detect all of the configured IP addresses and either ask the user to pick one, or join the multicast group on all of them. Luckily, you can join multiple multicast groups on the same socket, so this isn’t too hard to manage.

You can do this in python with the following code (using psutil to detect interfaces):

import socket
import psutil

# Multicast address and group
MCAST_GROUP = "239.0.0.1"
MCAST_PORT = 2020

# Create an IPv4 UDP socket.
listen = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

# Set REUSEADDR so that we can also bind transmit sockets on the same port.
listen.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR)

# Bind to our multicast port.
listen.bind((socket.INADDR_ANY, MCAST_PORT))

# For each interface.
for interface, addrs in psutil.net_if_addrs().items():
    # For each address on the interface.
    for addr in addrs:
        # If the address is an IPv4 address and not localhost.
        if addr.family == socket.AF_INET and not addr.address.startswith("127."):
            # Join our multicast group on this address.
            listen.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP,
                socket.inet_aton(MCAST_GROUP) + socket.inet_aton(addr.address))

# Receive data from our socket!
data = listen.recvfrom(1500)

Sadly, outbound multicast behaves differently.

Outbound Multicast on Multi-homed Hosts

Linux chooses the outbound interface for a multicast packet, not based on the the interface that has the multicast subscription(s), but instead based on the system’s default route. This means that even if you’ve joined a multicast group and explicitly specified an interface, Linux will ignore that interface and send your multicast packets out the default route.

It also doesn’t matter if you have multiple memberships on the same socket, Linux still only sends one packet.

Linux provides setsockopt IP_MULTICAST_IF to specify the IP address of the outbound interface for a socket, but this option can only have one option on a socket, so if you want to send multicast packets to many interfaces, you have to open a separate socket for each interface!

You can do this in python with the following code (using psutil to detect interfaces):

import socket
import psutil

# Multicast address and group
MCAST_GROUP = "239.0.0.1"
MCAST_PORT = 2020

transmit = []

# For each interface.
for interface, addrs in psutil.net_if_addrs().items():
    # For each address on the interface.
    for addr in addrs:
        # If the address is an IPv4 address and not localhost.
        if addr.family == socket.AF_INET and not addr.address.startswith("127."):
            # Create an IPv4 UDP socket.
            tx = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

            # Set the multicast interface for this socket.
            tx.setsockopt(socket.SOL_IP, socket.IP_MULTICAST_IF,
                socket.inet_aton(addr.address))

            # Set REUSEADDR so that we can bind all of our transmit sockets
            # on the same port.
            tx.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR)

            # Add to our list of transmit sockets.
            transmit.append(tx)

data = bytes(24)
# Use each socket to send data to our multicast group on the interface
# associated with that socket.
for tx in transmit:
    tx.sendto(data, (MCAST_GROUP, MCAST_PORT))

Dissecting Ubuntu Kernel Packaging

2019-03-20 00:00:00 +0000

WHY???

For a project at my day job, I need to build a custom version of the linux kernel and package it in a way that is compatible with Ubuntu. The Linux kernel includes some build rules for debian packages, but these don’t seem to work very well ( make bindeb-pkg works, but make deb-pkg doesn’t). The debian packaging rules also omit the kernel debug symbols and the perf tools packages, which we need.

I did a similar packaging effort when I worked for Willow Garage, but that was over 5 years ago, and all I remember was that it was complicated, so I’m going to take detailed notes this time through.

This is written mostly as notes and a roadmap for myself. It assumes a relatively deep understanding of debian packaging, and a familiarity with the kernel and it’s build and configuration systems.

I’m going to start with the directions from https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel . Roughly the plan is:

Check out git://kernel.ubuntu.com/ubuntu/ubuntu-precise.git
Get a rough understanding of the packaging rules in that repository
Transplant those packaging rules onto my kernel
Trim out most of the kernel configuration variants
Build one (or maybe two) kernel variants, and the supporting packages.

`debian` and `debian.master`

The upstream repository contains debian and debian.master

Listing of debian:

$ tree debian
debian
├── commit-templates
│   ├── bumpabi
│   ├── config-updates
│   ├── external-driver
│   ├── missing-modules
│   ├── newrelease
│   ├── sauce-patch
│   └── upstream-patch
├── compat
├── control-scripts
│   ├── headers-postinst
│   ├── postinst
│   ├── postinst.extra
│   ├── postrm
│   ├── postrm.extra
│   ├── preinst
│   └── prerm
├── debian.env
├── docs
│   └── README.inclusion-list
├── gbp.conf
├── rules
├── rules.d
│   ├── 0-common-vars.mk
│   ├── 1-maintainer.mk
│   ├── 2-binary-arch.mk
│   ├── 3-binary-indep.mk
│   ├── 4-checks.mk
│   └── 5-udebs.mk
├── scripts
│   ├── abi-check
│   ├── config-check
│   ├── control-create
│   ├── link-headers
│   ├── misc
│   │   ├── getabis
│   │   ├── git-ubuntu-log
│   │   ├── insert-changes.pl
│   │   ├── insert-mainline-changes
│   │   ├── insert-ubuntu-changes
│   │   ├── kernelconfig
│   │   ├── retag
│   │   └── splitconfig.pl
│   ├── module-check
│   ├── module-inclusion
│   └── sub-flavour
├── source
│   ├── format
│   └── options
├── stamps
│   └── keep-dir
├── tests
│   ├── check-aliases
│   └── README
└── tools
    └── generic

Of these files, the following are part of a normal debian package, and I will ignore them for now:

debian/compat

The debian/rules file is the other file that I recognize as being part of a normal debian package, but it is interesting because it is the primary makefile which performs the steps to build the deb.

I’m going to start there and walk through these files to figure out what they’re doing.

`debian/rules`

Looks like this loads the version from debian/debian.env (which in this case appears to be debian.master )

Inclues debian/rules.d/0-common-vars.mk which:

sets up variables for the ubuntu series, release, revisions, and previous revisions. Despite being a repo for precise, series is set to oneiric. Odd…
loads an environment file for the series (in this case, debian/.oneiric-env
runs some logic to determine whether to do build modules, abi checks, build debug modules, do a full build, and a few other flags that look like control WHAT is built.
Reads the kernel version, and generates variables that appear to hold the standard kernel package names ( linux-image-$(abi\_release) and friends )
Generates other package names for debug, tools, etc if enabled above.
Generates a kernel make command and stores it in the kmake variable.

Has -include $(DEBIAN)/rules.d/$(arch).mk which looks like it will include the architecture-specific file from debian.master/rules.d/ if it exists.

For all of the architectures, this appears to set the following variables:

human_arch, build_arch and header_arch, which describe the architecture
defconfig appears to be a default config target to run to generate the default kernel config.
flavours appears to be a list of configurations to build the kernel in. Note that this is the British spelling of flavour.
build_image appears to be the name of the final kernel binary
kernel_file appears to the path to the kernel binary
install_file appears to be the name of the kernel binary when installed
loader appears to be the name of the bootloader
no_dumpfile appears to be true or false. not sure what this controls

Inclues debian/rules.d/1-maintainer.mk, which appears to be additional build targets to be used by the debian kernel maintainers.

Checks DEB_STAGE and appears to disable most targets for a stage1 (?) build.

Defines a few basic targets.

Defines the clean target, which deletes a number of files from debian and copies their equivalents from debian.master into the debian directory.

Includes debian/rules.d/2-binary-arch.mk, which mostly seems to define rules to prepare the source tree for in-tree or out-of-tree builds, perform the build and some kind of build stamping action, and run install commands to copy built artifacts into the correct directory for their destination debian package. Most of this appears to be generic (make pattern rules) and is controlled by variables set in the previous inclusions. It looks like this also instantiates a copy of most of these rules for each flavour in the flavours variable.

Includes debian/rules.d/5-udebs.mk, which appears to be for building micro-debs. These are only used by the debian live CD and install CD, so I’m going to completely ignore this for now. If it looks like they’re problematic later, it looks like they can be disabled by setting the disable_d_i variable to any value.

Includes the debian/rules.d/3-binary-indep.mk file, which appears to have the build rules for creating the kernel header, source and tools packages.

Includes debian/rules.d/4-checks.mk, which appears to have generic ABI and module check rules.

The main debain/rules then does some rather interesting things:

It generates debian.master/control.stub with a script that iterates over flavours (and sub-flavours?) and does some find and replace on debian.master/control.stub.in
It generates debian/control from debian.master/control.stub, creates a bunch of build directories and files, and then runs a program called kernel-wedge to generate another control file in the build directory. Many of the inputs and flavour-specific files for this step appear to come from various places inside the debian.master/d-i directory.

It seems quite unusual to generate the debian/control file in this manner. This is likely a good part of the magic of this build and may deserve more study later.

Other Files

debian/commit-templates do not appear to be reference from the rules. These are probably for manual use by the maintainers.
debian/gpb.conf is a configuration file for git-buildpackage. Hopefully can be ignored.
debian/control-scripts are used by 2-binary-arch.mk when assembling packages. Probably don’t need to be modified.
debian/docs appears to contain some documentation justifying the existence of the module inclusion list. Since I don’t plan to drastically change the structure of the packaging system, I probably don’t need to read this.
debian/scripts look like scripts that are invoked from the package build rules.
debian/source files look like config files for the deb build system. Probably not something I’ll need to change.
debian/stamps looks like some part of the build stamping process.
debian/tests appears to be a dedicated subsystem for running tests on built kernel package. It appears to contain a single test :P
debian/tools/generic appears to be a generic entrypoint for tools that are tied to the kernel version. It reads the name that it is invoked as, looks for an executable of the same name within a kernel-specific directory, and tries to invoke it. Clever, and almost certainly will not require modification.

`debian.master`

Listing of debian.master:

$ tree debian.master
debian.master
├── abi
│   ├── 3.2.0-125.168
│   │   ├── abiname
│   │   ├── amd64
│   │   │   ├── generic
│   │   │   ├── generic.compiler
│   │   │   ├── generic.modules
│   │   │   ├── virtual
│   │   │   ├── virtual.compiler
│   │   │   └── virtual.modules
│   │   ├── armel
│   │   │   ├── omap
│   │   │   ├── omap.compiler
│   │   │   └── omap.modules
│   │   ├── armhf
│   │   │   ├── highbank
│   │   │   ├── highbank.compiler
│   │   │   ├── highbank.modules
│   │   │   ├── omap
│   │   │   ├── omap.compiler
│   │   │   └── omap.modules
│   │   ├── fwinfo
│   │   ├── i386
│   │   │   ├── generic
│   │   │   ├── generic.compiler
│   │   │   ├── generic.modules
│   │   │   ├── generic-pae
│   │   │   ├── generic-pae.compiler
│   │   │   ├── generic-pae.modules
│   │   │   ├── virtual
│   │   │   ├── virtual.compiler
│   │   │   └── virtual.modules
│   │   └── powerpc
│   │       ├── powerpc64-smp
│   │       ├── powerpc64-smp.compiler
│   │       ├── powerpc64-smp.modules
│   │       ├── powerpc-smp
│   │       ├── powerpc-smp.compiler
│   │       └── powerpc-smp.modules
│   └── perm-blacklist
├── changelog
├── changelog.historical
├── config
│   ├── amd64
│   │   ├── config.common.amd64
│   │   ├── config.flavour.generic
│   │   └── config.flavour.virtual
│   ├── armel
│   │   ├── config.common.armel
│   │   └── config.flavour.omap
│   ├── armhf
│   │   ├── config.common.armhf
│   │   ├── config.flavour.highbank
│   │   └── config.flavour.omap
│   ├── config.common.ports
│   ├── config.common.ubuntu
│   ├── enforce
│   ├── i386
│   │   ├── config.common.i386
│   │   ├── config.flavour.generic
│   │   ├── config.flavour.generic-pae
│   │   └── config.flavour.virtual
│   ├── powerpc
│   │   ├── config.common.powerpc
│   │   ├── config.flavour.powerpc64-smp
│   │   └── config.flavour.powerpc-smp
│   └── ppc64
│       ├── config.common.ppc64
│       └── config.flavour.powerpc64-smp
├── control.d
│   ├── flavour-control.stub
│   ├── vars.generic
│   ├── vars.generic-pae
│   ├── vars.highbank
│   ├── vars.omap
│   ├── vars.powerpc64-smp
│   ├── vars.powerpc-smp
│   ├── vars.virtual
│   └── virtual.inclusion-list
├── control.stub.in
├── copyright
├── deviations.txt
├── d-i
│   ├── exclude-firmware.armel-omap
│   ├── exclude-firmware.armhf-highbank
│   ├── exclude-firmware.armhf-omap
│   ├── exclude-modules.amd64-virtual
│   ├── exclude-modules.armel-omap
│   ├── exclude-modules.armhf-highbank
│   ├── exclude-modules.armhf-omap
│   ├── exclude-modules.i386-virtual
│   ├── exclude-modules.ia64
│   ├── exclude-modules.powerpc
│   ├── exclude-modules.sparc
│   ├── firmware
│   │   ├── nic-modules
│   │   ├── README.txt
│   │   └── scsi-modules
│   ├── kernel-versions.in
│   ├── modules
│   │   ├── block-modules
│   │   ├── crypto-modules
│   │   ├── fat-modules
│   │   ├── fb-modules
│   │   ├── firewire-core-modules
│   │   ├── floppy-modules
│   │   ├── fs-core-modules
│   │   ├── fs-secondary-modules
│   │   ├── input-modules
│   │   ├── ipmi-modules
│   │   ├── irda-modules
│   │   ├── md-modules
│   │   ├── message-modules
│   │   ├── mouse-modules
│   │   ├── multipath-modules
│   │   ├── nfs-modules
│   │   ├── nic-modules
│   │   ├── nic-pcmcia-modules
│   │   ├── nic-shared-modules
│   │   ├── nic-usb-modules
│   │   ├── parport-modules
│   │   ├── pata-modules
│   │   ├── pcmcia-modules
│   │   ├── pcmcia-storage-modules
│   │   ├── plip-modules
│   │   ├── ppp-modules
│   │   ├── sata-modules
│   │   ├── scsi-modules
│   │   ├── serial-modules
│   │   ├── speakup-modules
│   │   ├── squashfs-modules
│   │   ├── storage-core-modules
│   │   ├── usb-modules
│   │   ├── virtio-modules
│   │   └── vlan-modules
│   ├── modules-powerpc
│   │   ├── block-modules
│   │   ├── message-modules
│   │   ├── nic-modules
│   │   ├── scsi-modules
│   │   └── storage-core-modules
│   ├── modules-sparc
│   │   ├── block-modules
│   │   └── message-modules
│   └── package-list
├── etc
│   ├── getabis
│   └── kernelconfig
├── NOTES
└── rules.d
    ├── amd64.mk
    ├── armel.mk
    ├── armhf.mk
    ├── i386.mk
    ├── powerpc.mk
    └── ppc64.mk

Off the top, a few of these directories and files look familiar from looking through previous rules:

debian.master/abi ABI symbol definitions for kernel variants. I think these are checked in to make sure that new kernels are ABI compatible with previous builds of the same kernel.
debian.master/changelog and debian.master/changelog.historical: the standard debian package changelog.
debian.master/config directory appears to contain common and flavor-specific configs. I suspect this is where the majority of the work will happen.
debian.master/config/enforce appears to be some kind of validation that every flavor has some config options set. debian/scripts/config-check appears to use this file.
debian.master/control.d appear to be debian control file chunks that are included for each kernel flavor.
debian.master/control.stub.in is the template for the control file; covered in previous section.
debian.master/copyright copyright. Read it, respect it.
debian.master/deviations.txt appears to be a partial list of changes. Not sure what these are against…
debian.master/d-i I’ll come back to this one.
debian.master/etc scripts appear to be used by debian/scripts/misc. Not quire sure what these do, but they feel old and maybe a little crusty.
debian.master/NOTES someone left their notes here. Seems to be mostly notes about which patches are applied to this kernel.
debian.master/rules.d arch-specific build rules; see debian.rules section above.

`debian.master/d-i`

I said I’d come back to it!

This directory contains the kernel-versions.in file which seems to contain metadata about the arch/flavour combinations, covered above when going through debian/rules

The rest of this directory seems to contain files that map kernel modules to modules packages, and blacklists for some of those modules packages on some arch/flavour combinations. This looks completely unfamiliar to me; I don’t think these packages are part of my current Ubuntu distribution.

DOH! I’ve been looking at the kernel build system for Ubuntu Precise (12.04), released in 2012! No wonder these flavors look a little old.

Switch gears to Ubuntu 14.04 and 18.04

Ok; now checking out the 14.04 kernel to see how it has evolved:

git clone git://kernel.ubuntu.com/ubuntu/ubuntu-trusty.git

The debian directory and debian/rules appear largely unchanged. There are a few more scripts in debian/scripts, but the overall structure appears unchanged.

The debian.master directory has substaintially more flavours, but again appears otherwise unchanged.

Comparing with 18.04 ( git://kernel.ubuntu.com/ubuntu/ubuntu-bionic.git ), most of the structure again appears the same, but the structure of debian.master/d-i/firmware and debian.master/d-i/modules appears to have a bunch of new sub-directories for each architecture.

The Transplant

Now that I have the lay of the land, I think it’s time to try transplanting these changes onto my kernel and attempting a build.

I copied over all of debian and debian.master from ubuntu’s kernel into my kernel tree. Since I’m only focusing on an amd64 architecture, I removed the files for all of the other architectures. I now have the following directory structure alongside the rest of the kernel source:

debian
├── cloud-tools
│   ├── hv_get_dhcp_info
│   ├── hv_get_dns_info
│   └── hv_set_ifconfig
├── commit-templates
│   ├── bumpabi
│   ├── config-updates
│   ├── external-driver
│   ├── missing-modules
│   ├── newrelease
│   ├── sauce-patch
│   └── upstream-patch
├── compat
├── control.d
│   └── flavour-buildinfo.stub
├── debian.env
├── docs
│   └── README.inclusion-list
├── gbp.conf
├── linux-cloud-tools-common.hv-fcopy-daemon.upstart
├── linux-cloud-tools-common.hv-kvp-daemon.upstart
├── linux-cloud-tools-common.hv-vss-daemon.upstart
├── rules
├── rules.d
│   ├── 0-common-vars.mk
│   ├── 1-maintainer.mk
│   ├── 2-binary-arch.mk
│   ├── 3-binary-indep.mk
│   ├── 4-checks.mk
│   └── 5-udebs.mk
├── scripts
│   ├── abi-check
│   ├── config-check
│   ├── control-create
│   ├── helpers
│   │   ├── close
│   │   ├── open
│   │   └── rebase
│   ├── link-headers
│   ├── misc
│   │   ├── final-checks
│   │   ├── find-obsolete-firmware
│   │   ├── fw-to-ihex.sh
│   │   ├── gen-auto-reconstruct
│   │   ├── getabis
│   │   ├── get-firmware
│   │   ├── git-ubuntu-log
│   │   ├── insert-changes.pl
│   │   ├── insert-mainline-changes
│   │   ├── insert-ubuntu-changes
│   │   ├── kernelconfig
│   │   ├── retag
│   │   └── splitconfig.pl
│   ├── module-check
│   ├── module-inclusion
│   ├── retpoline-check
│   ├── retpoline-extract
│   ├── retpoline-extract-one
│   └── sub-flavour
├── source
│   ├── format
│   └── options
├── stamps
│   └── keep-dir
├── templates
│   ├── extra.postinst.in
│   ├── extra.postrm.in
│   ├── headers.postinst.in
│   ├── image.postinst.in
│   ├── image.postrm.in
│   ├── image.preinst.in
│   └── image.prerm.in
├── tests
│   ├── control
│   ├── rebuild
│   └── ubuntu-regression-suite
├── tests-build
│   ├── check-aliases
│   └── README
└── tools
    └── generic
debian.master
├── abi
│   └── perm-blacklist
├── changelog
├── changelog.historical
├── config
│   ├── amd64
│   │   ├── config.common.amd64
│   │   ├── config.flavour.generic
│   │   └── config.flavour.lowlatency
│   ├── annotations
│   ├── config.common.ports
│   ├── config.common.ubuntu
│   └── enforce
├── control.d
│   ├── flavour-control.stub
│   ├── generic.inclusion-list
│   ├── vars.generic
│   └── vars.lowlatency
├── control.stub.in
├── copyright
├── d-i
│   ├── exclude-modules.amd64-virtual
│   ├── firmware
│   │   ├── kernel-image
│   │   ├── nic-modules
│   │   ├── README.txt
│   │   └── scsi-modules
│   ├── kernel-versions.in
│   ├── modules
│   │   ├── block-modules
│   │   ├── crypto-modules
│   │   ├── fat-modules
│   │   ├── fb-modules
│   │   ├── firewire-core-modules
│   │   ├── floppy-modules
│   │   ├── fs-core-modules
│   │   ├── fs-secondary-modules
│   │   ├── input-modules
│   │   ├── ipmi-modules
│   │   ├── irda-modules
│   │   ├── kernel-image
│   │   ├── md-modules
│   │   ├── message-modules
│   │   ├── mouse-modules
│   │   ├── multipath-modules
│   │   ├── nfs-modules
│   │   ├── nic-modules
│   │   ├── nic-pcmcia-modules
│   │   ├── nic-shared-modules
│   │   ├── nic-usb-modules
│   │   ├── parport-modules
│   │   ├── pata-modules
│   │   ├── pcmcia-modules
│   │   ├── pcmcia-storage-modules
│   │   ├── plip-modules
│   │   ├── ppp-modules
│   │   ├── sata-modules
│   │   ├── scsi-modules
│   │   ├── serial-modules
│   │   ├── speakup-modules
│   │   ├── squashfs-modules
│   │   ├── storage-core-modules
│   │   ├── usb-modules
│   │   ├── virtio-modules
│   │   └── vlan-modules
│   └── package-list
├── etc
│   ├── getabis
│   └── kernelconfig
├── NOTES
├── reconstruct
├── rules.d
│   └── amd64.mk
└── tracking-bug

As a quick check that things are not completely insane, I ran fakeroot ./debian/rules clean and was pleasantly surprised to find that it succeeded.

Since I’m only trying to build one flavor (my custom flavour), I made the following hacks:

Modified debian.master/rules.d/amd64.mk to list only that flavour in the flavours
Removed the contents of debian.master/config/config.common.ubuntu and debian.master/config/amd64/config.common.amd64
Deleted debian.master/config/amd64/config.flavour.generic and debian.master/config/amd64/config.flavour.lowlatency
Created debian.master/config/amd64/config.flavour.my-flavour with my entire kernel config.

At this point, I should be able to run the build and get all the packages out of it. Let’s see if it works…

CONCURRENCY_LEVEL=24 fakeroot ./debian/rules binary

(yes, I have a 24-core machine. Set CONCURRENCY_LEVEL appropriately for your computer)

Unsurprisingly, I’m not quite that lucky:

/home/austin/linux/scripts/Makefile.headersinst:63: *** Some mandatory headers (poll.h) are missing in arch/x86/include/uapi/asm.  Stop.
make[3]: *** [asm] Error 2
make[2]: *** [headers_install] Error 2
make[2]: Leaving directory `/home/austin/linux/debian/tmp-headers'
make[1]: *** [sub-make] Error 2
make[1]: Leaving directory `/home/austin/linux'
make: *** [install-arch-headers] Error 2

Also of note, the version reported during the build is completely wrong. Adding the appropriate entry to debian.master/changelog quickly fixes that.

It looks like the missing poll.h header has moved from arch/x86/include/uapi/asm/poll.h to arch/x86/include/generated/uapi/asm/poll.h as some point. It looks like the Bionic hwe-edge branch currently has a similar kernel version and some build support for that, so I’ll try to port that in.

… hours pass …

I finally tracked down the build issue. It seems to be some kind of issue if the local kernel build tree is not completely clean. I removed all of the generated headers, and the build seems to be past that step.

I encountered a few other errors that were also resolved by cleaning my source directory (I had been doing normal builds in it before I added the debian rules). make mrproper will clean out EVERYTHING, but unfortunately this also includes the debian directory that I’m working on. Luckily I habitually check things into git so all I had to do to recover was a git checkout. I also added this patch to skip that step in the future (I’m on a 4.19.2 kernel. YMMV):

diff --git a/scripts/package/Makefile b/scripts/package/Makefile
index 73503ebce632..d96ac1649aea 100644
--- a/scripts/package/Makefile
+++ b/scripts/package/Makefile
@@ -82,7 +82,7 @@ bindeb-pkg: FORCE
 intdeb-pkg: FORCE
        +$(CONFIG_SHELL) $(srctree)/scripts/package/builddeb
 
-clean-dirs += $(objtree)/debian/
+#clean-dirs += $(objtree)/debian/
 
 # snap-pkg
 # ---------------------------------------------------------------------------

The config/enforce step also found a few options that I should enable, so I used fakeroot ./debian/rules editconfigs to edit my flavours and enable those configs. (The search operator, /, in menuconfig makes finding and enabling the missing options easy). This also moved all of my config options from my flavour to the common options, but that seems like the desired behavior so I left it. If I find myself needing to add move flavours later, this is probably the right path to take.

… building again … more hours pass …

Modules packaging error because I had CONFIG_LOCALVERSION set. Reset it to the empty string in my config.

… building … minutes pass …

Module check failed:

Debug: module-check-my_flavour
II: Checking modules for my_flavour...     prev_abidir : /home/austin/linux/debian.master/abi/4.19.2-0.0/amd64previous or current modules file missing!
   /home/austin/linux/debian.master/abi/4.19.2-1/amd64/my_flavour.modules
   /home/austin/linux/debian.master/abi/4.19.2-0.0/amd64/my_flavour.modules

Really, since I’m bootstrapping a branch new kernel, I don’t have a previous ABI version to check against so I’d like to either disable this check or bootstrap it with itself.

It looks like this check can be skipped by adding an ignore file for the “previous” version:

$ mkdir -p debian.master/abi/4.19.2-0.0/amd64
$ touch debian.master/abi/4.19.2-0.0/amd64/ignore

I had to add similar ignores for modules:

$ touch debian.master/abi/4.19.2-0.0/amd64/ignore.modules

Finally, the retpoline check also failed for a similar reason, so I fixed it by duplicating the new retpoline to the “previous” version:

$ cp debian.master/abi/4.19.2-1/amd64/my_flavour.retpoline debian.master/abi/4.19.2-0.0/amd64/

… more time passes …

Another stray build error appears:

dpkg-gencontrol: error: package linux-image-unsigned-4.19.2-1-my_flavour not in control info
dh_gencontrol: dpkg-gencontrol -plinux-image-unsigned-4.19.2-1-my_flavour -ldebian/changelog -Tdebian/linux-image-unsigned-4.19.2-1-my_flavour.substvars -Pdebian/linux-image-unsigned-4.19.2-1-my_flavour -Vlinux:rprovides= returned exit code 255
make: *** [binary-my_flavour] Error 25

Looks like this is due to the debian/control file not being generated correctly; it still has entries for the original generic and lowlatecy flavours, instead of my_flavour.

Removed the old lowlatency control variables and renamed the other one to my flavour:

rm debian.master/control.d/vars.lowlatency
mv debian.master/control.d/vars.generic debian.master/control.d/vars.my_flavour

debian.master/d-i/kernel-versions.in also needed an update, to change the flavour from generic to my_flavour.

Luckily, these changes are easy to verify by running fakeroot ./debian/rules clean and then inspecting the resulting debian/control file.

… another interminable wait for the build …

Further progress; now Debug: binary-udebs fails:

find: `debian/serial-modules-4.19.2-1-my_flavour-di': No such file or directory
serial-modules-4.19.2-1-my_flavour-di will be empty
find: `debian/ppp-modules-4.19.2-1-my_flavour-di': No such file or directory
ppp-modules-4.19.2-1-my_flavour-di will be empty
find: `debian/firewire-core-modules-4.19.2-1-my_flavour-di': No such file or directory
firewire-core-modules-4.19.2-1-my_flavour-di will be empty
find: `debian/plip-modules-4.19.2-1-my_flavour-di': No such file or directory
plip-modules-4.19.2-1-my_flavour-di will be empty
find: `debian/pcmcia-storage-modules-4.19.2-1-my_flavour-di': No such file or directory
pcmcia-storage-modules-4.19.2-1-my_flavour-di will be empty
find: `debian/irda-modules-4.19.2-1-my_flavour-di': No such file or directory
irda-modules-4.19.2-1-my_flavour-di will be empty
find: `debian/parport-modules-4.19.2-1-my_flavour-di': No such file or directory
parport-modules-4.19.2-1-my_flavour-di will be empty
find: `debian/nic-pcmcia-modules-4.19.2-1-my_flavour-di': No such file or directory
nic-pcmcia-modules-4.19.2-1-my_flavour-di will be empty
find: `debian/pcmcia-modules-4.19.2-1-my_flavour-di': No such file or directory
pcmcia-modules-4.19.2-1-my_flavour-di will be empty
find: `debian/squashfs-modules-4.19.2-1-my_flavour-di': No such file or directory
squashfs-modules-4.19.2-1-my_flavour-di will be empty
find: `debian/speakup-modules-4.19.2-1-my_flavour-di': No such file or directory
speakup-modules-4.19.2-1-my_flavour-di will be empty
find: `debian/fs-core-modules-4.19.2-1-my_flavour-di': No such file or directory
fs-core-modules-4.19.2-1-my_flavour-di will be empty
make[1]: *** [do-binary-udebs] Error 1

Looks like most of those are due to my choice of kernel config. I think I can suppress them by listing those modules (or module groups?) in debian.master/d-i/exclude-modules.amd64-my_flavour

… build continues …

BUILD SUCCESS!

Figuring out if it actually works will be a task for another day…

Nixie MultiMeter, Part 18

2019-03-12 00:00:00 +0000

More Power Analysis

I was curious and I already had a pretty decent simulation of my linear regulator in LTSpice, so I spent some more time playing with it to see how the filtering between the digital and analog sides would work.

What I found surprised me a bit: under a 100kHz, 0-200mA sinusoidal load, the output voltage has 40mV of ripple. (Increasing the frequency reduces the ripple). I’m not surprised that there’s ripple under load, but the magnitude of it surprised me a bit.

I also modeled TI’s recommended filter and its output into a 10mA resistive load. It definitely helps reduce the ripple, but it also drops almost 0.2V, so I suspect a decent part of the ripple reduction comes from DC resistance of the ferrite bead (15 ohms) forming a low-pass filter with the 22uF capacitor. If I choose a ferrite bead with lower DC resistance, the voltage drop disappears but the ripple gets larger.

Trying to put a filter on the primary load doesn’t seem to help either. The filter’s resistance increases the voltage swing at the primary load, but doesn’t seem to reduce the voltage ripple at the regulator or on the analog side.

Adding 70uF of ceramic capacitors improves ripple significantly, but it’s still around 0.8mV (800uV), and that’s a lot of capacitance!

Replacing the electrolytic capacitor in the analog filter side with a ceramic capacitor further reduces the ripple on that side, to about 10uV.

With a 10uF ceramic capacitor on the analog filter side, removing the 70uF of capacitance on the load side results in about 20uV of ripple on the analog side. Good enough!

Nixie MultiMeter, Part 17

2018-12-05 00:00:00 +0000

Power

For this post, I will be focusing on the 3.3V digital and analog power supplies.

3.3V Power Supply

The first three steps when choosing a power supply are to establish basic requirements for voltage, current, and noise. For the 3.3V power supply, the voltage was chosen since it is a common voltage, and is compatible with the MSP430 (although the MSP430F6736 can run from voltages between 3.6 and 1.8V). The current requirement will be driven primarily by the worst-case current for each component which is powered from 3.3V. For many of these componets, the worst-case current is a combination of the maximum output current and the device’s internal worst-case current. The exact formulas can usually be found on the device’s datasheet.

3.3V Load	Worst-case Current
MSP430F6736	9.5mA, plus modules and outputs
LTC6078 (x2)	160uA each, plus outputs
LMV358	340uA, plus outputs
Resistor drive	100mA load on LMV358
Virtual ground (200k)	16.5uA
Reference Current	80uA
Total	110mA

Clearly, the largest component of the 3.3V load comes from the drive circuit for resistance measurement. That’s also the part of the circuit where we’re most likely to see overloads, so that power supply should probably have significantly more capacity. Sizing the power supply at 200mA (almost 2x the required current) seems like a safe bet, and it shouldn’t be hard to find a switching (and maybe linear!) regulators that can do that.

Finally, the choice of power supply must consider noise, to help choose the type of regulator (switching or linear) and possibly choose several power supplies, if components need to have low-noise voltage supplies.

The MSP430F6736 datasheet strongly suggests powering the analog and digital supply pins from the same power supply. It seems like the obvious solution then is to use a single supply for all 3.3V loads, and add extra filtering on the supply for the high-precision part of the circuit. The amount of filtering required will be determined by the required output precision.

The ranges for both ADCs are +/- 14mV, over an 18-bit range, which comes out to about 107nV per bit. Any noise below that level will be undetectable by the ADC. The two components that will be affected by power supply noise will be the ADC within the MSP430, and the buffer op-amp.

Background on Decibels

I see decibels on a regular basis, but I rarely have to do math with them, so I’ll rehash the formulas here:

PSRR (dB) = 20 * log( power ripple / signal ripple )

Or, to go the other way:

power ripple / signal ripple = 10 ^ ( PSRR / 20 )

ADC Power Noise

The ADC has a power supply rejection ratio of -79dB or 1/8913; this means that any noise on the power supply will apear on the ADC reading, reduced by a factor of about 8913. Working backwards, the power supply ripple would have to be above 950uV ( 107nV * 8193 ) to affect the ADC reading.

This is a bit bogus, because the ADC only specifies a PSRR at 50Hz, but the average switching power supply operates between 100kHz and 2MHz. PSRR tends to be heavily frequency-dependent, with more coupling from power supply to ADC at higher frequencies, so this value is at best a lower bound, and at worst completely useless.

It may be more useful to look at the integration time of the ADC to determine how much power supply noise will affect the ADC readings. If the power supply noise is highly periodic with one or two major frequency components, and the ADC integration time is longer than several cycles, it seems likely that the power supply noise will average out before it appears on the ADC.

The ADC parameters that I selected (all the way back in part 6) were borrowed from TI’s reference design; they are a 2MHz modulator frequency and a 64x oversampling, for 19 bits of ADC resolution at 32k samples per second. Getting a switching power supply that runs several times faster than 32kHz should be easy. Given that the Sigma-Delta convertor works by sampling at high speed, in order to avoid aliasing the switching power supply frequency should not be close to any of the harmonics of the 2MHz sampling rate.

Aliasing occurs when two processes overlap at constant intervals; for example a 500Hz switching regulator frequency would be 1/4 of the 2MHz modulator frequency, and could cause 1 in every 4 ADC samples to land on a significant peak or trough of the power supply ripple, which would cause the power supply noise to be over-represented in the final ADC sample.

Typically, ADCs will have an analog low-pass filter at half the sample rate to filter out high-frequency noise; this filter would also help filter out power noise from other components, but it probably won’t help with noise induced within the ADC. For an ADC sampling rate of 32k samples per second, the Nyquist–Shannon sampling theorem states the the cutoff frequency should be set at half of the sampling rate; in this case that will be 16kHz.

LTC6078 Power Noise

The LTC6078 has a power supply rejection ratio of 97 dB, but it drops to zero around 4MHz:

Since the ADC will have a low-pass filter on the input, we only need to worry about noise rejection in the LTC6078 up to the cutoff frequency of 16kHz.

At 16kHz and above, the PSRR is only about 25dB, or a reduction of about 17x between ripple on the power supply and ripple in the op-amp’s output. This means that a power supply ripple as small as about 1.7uV would be output as ripple of 100nV, and could appear in the ADC output.

Reducing the cutoff frequency to 10kHz would result in a PSRR of about 40dB, which is a 100x reduction in power supply ripple, and puts the maximum power supply ripple at 10uV.

Power supply ripple for switching supplies is typically in the 250kHz to 1MHz range, so either of these filters will probably be acceptable.

Power Filter

Based on the previous sections, the power supply and filters need to produce a 3.3V analog rail with no more than 10uV of ripple between a few hertz and the ADC sampling frequency of about 2MHz. This seems like a rather difficult target to hit, so I’m just going to try to create the best low-pass filter(s) for the power supply that I can, and see how close I can get.

The 3.3V supply will look something like this:

12V -> filter + regulation -> 3.3V -> filter -> 3.3V analog

Switching regulators can produce a significant amount of output noise, so using a switching regulator to produce the 3.3V rail, and then trying to remove all of the switching noise for the 3.3V analog supply doesn’t seem like it will work very well.

Instead, I should probably try to produce a 3.3V supply with the lowest ripple possible, and then use additional filtering between the 3.3V and 3.3V analog supplies to further suppress ripple and remove any ripple that might be introduced by the other components on the 3.3V rail.

From reading through The Art of Electronics, it looks like the best way to reduce any incoming ripple is with a combination of tranditional filtering and a circuit called a capacitance multiplier. After the filtering stages, a linear regulator can remove some of the remaining ripple and produce a regulated output voltage.

12V -> filter? -> capacitance multiplier -> linear regulator -> 3.3V -> filter -> 3.3V analog

(An interesting alternative is the LT1533 and described in Linear Tech App Note 70, but the IC is $10 and requires a separate transformer and inductors).

Linear Tech App Note 101 has some fantastic advice and characterization of the noise produced by switching regulators, and how to reduce or eliminate it. Mostly, the takeaway here is that switching regulators produce high-frequency transients (up to several hundred megahertz!) and that the parasitics in typical capacitors and regulators aren’t effective at blocking these frequencies, but that ferrite beads can be quite effective.

As as starting point, I’m going to try to create a noise simulation in LTSpice, and then simulate a circuit which can remove that noise (and probably substantially more). For a basic starting point, I’ve chosen a LT3062 since it is a linear regulator that can take the full 12V input voltage, has a model in LTSpice, and is relatively cheap. Unsurprisingly, it doesn’t have great power supply rejection at higher frequencies (10MHz), but at low frequencies it advertises an output ripple of 30uV!

As a starting point, I’ve created a simulation of the input (green, upper plot) and output (blue, lower plot) of the LT3062 (the adjustable version, hence the voltage-setting resistors). I’m simulating the ouput of a rather nasty switching power supply with a 0.4Vpp sine wave at 500kHz, a 1Vpp exponential with a fundamental frequency around 10MHz, and a healthy dose of white noise. I’m using a resitive load of 150mA, and the simulated regulator’s output is about 3.3V, with about 15uVpp of ripple. Not bad!

Adding a ferrite bead knocks down the random, high-frequency noise considerably, and adding a capacitance multiplier (completely untuned!) knocks down the remaining noise to a value that LTSpice doesn’t seem able to calculate correctly, but which appears to be less than 1uVpp. I’ll take it!

The only remaining consideration for this circuit is the power dissipation in the pass transistor and the regulator. Combined, all of the pass elements will need to drop 8.7V at up to 200mA, and dissipate the resulting 1.75W of heat. With the simulation at 200mA, the regulator needs to dissipate about 1.35W, and the pass transistor dissipates about 0.5W. I’d rater disspate that power in different part of the circuit, so I’ve added a resistor between the capacitance multiplier and the regulator. I’ve chosen the resitor value so that it drops about 5V and disspates about 1W at maximum current. At lower currents, it will have a lower voltage drop and disspate less current. At higher currents, it will limit the total current through the circuit.

For the final analog filter, it looks like the analog portion of the MSP430 can draw as much as 10mA. Each 24-bit ADC can draw a maximum of 1mA, not counting the current required to run the voltage reference. The 10-bit ADC can draw a maximum of 185uA, including the current required to run the voltage reference. Combined with 160uA for the LTC6078, the total current for the precision analog side of the circuit is about 11mA.

TI has a nice whitepaper on Filtering Techniques: Isolating Analog and Digital Power Supplies in TI’s PLL-Based CDC Devices, which describes common power filtering techniques and recommends a ferrite bead and ceramic capacitor filter between the digital and analog supplies. Given that this will only be filtering out noise induced by other components in the 3.3V circuit, this will probably be more than adequate.

Nixie MultiMeter, Part 16

2018-08-26 00:00:00 +0000

Design Completion Checklist

I’m coming up on the end of this project; partly because I’m close to having the schematic complete, and partly because I’m ready to be done. I’ve put together a checklist of the things that I need and want to get done before I start doing PCB layout.

Must Have
1. Higher-power op-amps for resistance drive and virtual ground
2. Prevent floating leads/inputs in 4-wire mode
3. LEDs for decimal point, range and mode indication
4. 3.3V regulator
5. Power input jack
6. Power supply decoupling for MSP430
7. Power supply decoupling for op-amps
8. MSP430 programming and debug connector (and cable?)
9. MSP430 crystal
10. Inter-board connector or cable
11. Mode select method (switches, buttons, rotary switch?)
Optional
1. Resistance Calibration Mode(s)
2. Try to reduce number of relays to reduce cost
3. Look for other easy ways to reduce cost.
4. Computer uplink for automated measurement (Serial, USB, or other)
5. 3D models for the remaining components that do not currently have models. Not a big deal for small components, but will help verify footprints on more complex items and help visualize the soldering and assembly process.

1.1 Higher Power op-amps

The Art of Electronics (and at least one post on Elecronics StackOverflow) suggest that the easiest ways to get more output current from an op-amp are to either add a pair of BJT transistors in a push-pull output stage, or add a second op-amp to act as a buffer on the output. The only caveat with this kind of design is that the output stage needs to have a higher gain-bandwidth product, to avoid causing delays and phase shifts in the feedback loop.

The first op-amp in the chain provides most of the precision, and the second op-amp (or transistors) provides the desired output current.

For the overall circuit, I have the following requirements from previous posts:

Input Offset 10uV or less
Input leakage 1pA or less
Output Current 100mA or more
Single-ended supply voltage of 3.3V
Output voltage swing from 0.65V to 2.65V
No serious bandwidth requirement; circuit is designed for steady-state operation

The input op-amp satisfies the input offset and input leakage requirements, but since it’s only driving the second op-amp, it doesn’t need to satisfy the output current requirement.

The output op-amp satisfies the output current requirement, but since it’s driven from the relatively low-impedance output from the first op-amp, doesn’t need to meet the input leakage requirement. Since the feedback loop is wrapped around both op-amps, the first op-amp can also compensate for substantial input offsets in the second op-amp. The output op-amp should have a higher slew rate and gain-bandwidth product (GBW) to avoid oscillations.

Since both stages have unity gain, the both need to meed the supply voltage and output voltage swing requirements.

The existing LT6078 that I’ve chosen for other parts of the circuit is still one of the best choices for meeting the requirements for the first stage op-amp.

With the reduced requirements on the output stage, there are lots of op-amps which fit the current and supply voltage requirements. The ON Semi LMV358 looks like a reasonable choice, with an output current of 185mA, GBW of 1MHz, rail-to-rail outputs and a supply voltage range of 2.7V to 5.5V. (There were too many choices here, so I rolled the dice and picked one).

That wraps up the selection of the high-power op-amp. Future posts will focus on the remaining items to complete the design.

Rust on MSP430, Part 3

2018-07-21 00:00:00 +0000

It Builds!

Wrapping up the previous posts about Rust on the MSP430

Following up on the issue I filed against msp430-rt last time, I followed the directions for the panic_implementation update, submitted my changes as a pull request and they were accepted and uploaded to crates.io.

With those fixes, I can now build the blinky example in msp430-quickstart with xargo build --target msp430-none-elf --release --example blinky

Unfortunately, I have a MSP430FR2433 demo board, but the examples are written for a MSP430G2553 board, so I needed to update them for my variant of the microcontroller and demo board.

Porting to MSP430FR2433

There were three major steps to porting the existing quickstart demo to the board that I’m using:

Generating the board package with Svd2Rust
Updating the pin usage in the demos
Debugging compile issues and memory layout

Svd2Rust

The Svd2Rust tool is pretty amazing; it generates a Rust package with all of the register and bit definitions and access functions for a microcontroller, directly from a register description file from the manufacturer.

The SVD format was developed for the ARM microcontrollers, but the MSP430 isn’t an ARM core. For the MSP430 series we can generate the SVD file with msp430_svd: cargo run -- msp430fr2433 | xmllint -format - > msp430fr2433.svd

From the generated SVD file, we can run svd2rust to generate the Rust support package: svd2rust -i msp430fr2433.svd --target msp430 > lib.rs

As recommended by svd2rust, I followed up the code generation by running form -i lib.rs -o src and cargo fmt to split the generated file up into modules and format it according to standard Rust style.

I’ve uploaded the resulting msp430fr2433 package to github.

Updating the Demo

I made three major changes to the demo to get it ready for testing on my board. I updated the dependencies so that it depends on my new package, I updated the demo to use the new register ownership model that svd2rust generates, and I updated the pin definitions to use the LED pins on my board.

Since I have my msp430fr2433 package checked out locally, I replaced the msp430g2553 dependency in the Cargo.toml with:

[dependencies.msp430fr2433]
features = ["rt"]
path = "../msp430fr2433"

I’m not quite sure what the ownership implications are, but the ownership model for peripherals has changed. The old code to access the watchdog timer was:

let wdt = msp430g2553::WATCHDOG_TIMER.borrow(&cs);

The new version of this is:

let p = msp430fr2433::Peripherals::take().unwrap();
let wdt = p.WATCHDOG_TIMER;

(Similar changes apply to the Port access register)

Since my board is different from the one originally used by the quickstart, my LEDs are on pins P1.0 and P1.1 instead of the original pins. The old pin setup was:

// set P0 high and P6 low
port_1_2.p1out.modify(|_, w| w.p0().set_bit().p6().clear_bit());

And the new pin setup is:

// set P1.0 high and P1.1 low
port_1_2.p1out.modify(|_, w| w.p1out0().set_bit().p1out1().clear_bit());

I’ve made a fork of the msp430-quickstart projet and pushed my changes there. The full diff of my changes is available on github. (spoiler alert, this also includes some of my fixes from debugging)

Debugging

To start, with my changes the quickstart demo builds, but I get a linker error:

/home/austin/ti/msp430-gcc/bin/../lib/gcc/msp430-elf/7.3.1/../../../../msp430-elf/bin/ld: address 0x100ce of /home/austin/msp430/msp430-quickstart/target/msp430-none-elf/debug/examples/blinky-652d021404ae3401 section `.vector_table' is not within region `VECTORS'
/home/austin/ti/msp430-gcc/bin/../lib/gcc/msp430-elf/7.3.1/../../../../msp430-elf/bin/ld: address 0x100ce of /home/austin/msp430/msp430-quickstart/target/msp430-none-elf/debug/examples/blinky-652d021404ae3401 section `.vector_table' is not within region `VECTORS'
/home/austin/ti/msp430-gcc/bin/../lib/gcc/msp430-elf/7.3.1/../../../../msp430-elf/bin/ld: 
The section .vector_table.interrupts appears to be wrong. It should
end at address 0xFFFE
collect2: error: ld returned 1 exit status

I’ve seen a few embedded linking errors before; this one looks like the memory layout specified in the linker script isn’t correct for this library and this microcontroller. With a new processor and new support library, the obvious thing to do is to check the memory layout in the datasheet against the memory.x file in the project, and look for inconsistencies.

Page 40 of the datasheet lists the vector table (interrupt handlers) as having a reserved region starting at 0xFF88, with the vector table starting at 0xFFDA and ending at 0xFFFE.

A search through the generated msp430fr2433 package finds the vector table in src/interrupt/mod.rs, and it includes the reserved region. With the reserved addresses, it totals 59 entries but does not include the reset handler.

The original memory.x had the VECTORS region starting at 0xFFE0; updating it to 0xFF88 better matches the documentation. To get the same end address, I had to update the size to 60 entries at 2 bytes each, for a total size of 120 bytes (or 0x78 in hex).

With that update, I get a new linker error:

/home/austin/ti/msp430-gcc/bin/../lib/gcc/msp430-elf/7.3.1/../../../../msp430-elf/bin/ld: address 0x10076 of /home/austin/msp430/msp430-quickstart/target/msp430-none-elf/debug/examples/blinky-652d021404ae3401 section `.vector_table' is not within region `VECTORS'
/home/austin/ti/msp430-gcc/bin/../lib/gcc/msp430-elf/7.3.1/../../../../msp430-elf/bin/ld: address 0x10076 of /home/austin/msp430/msp430-quickstart/target/msp430-none-elf/debug/examples/blinky-652d021404ae3401 section `.vector_table' is not within region `VECTORS'
/home/austin/ti/msp430-gcc/bin/../lib/gcc/msp430-elf/7.3.1/../../../../msp430-elf/bin/ld: 
The section .vector_table.interrupts appears to be wrong. It should
end at address 0xFFFE
collect2: error: ld returned 1 exit status

The end address is now 0x10076, which indicates that the vector table is 118 bytes longer than it should be. 118 bytes is 59 x 2, which is suspiciously similar to the vector table size. Looking at the vector table source code carefully, the Vector object that defines each entry in the vector table is:

pub union Vector {
    _handler: unsafe extern "msp430-interrupt" fn(),
    _reserved: u32,
}

It’s union’ed with a 32-bit field, which means the minimum size is 32 bits, but the address size of the MSP430 is only 16 bits. Changing this to a 16-bit field fixes the remaining linker errors.

pub union Vector {
    _handler: unsafe extern "msp430-interrupt" fn(),
    _reserved: u16,
}

While I’m here, I also updated the RAM and ROM start addresses and offsets in memory.x to match the datasheet.

Success!?

I was able to program my board and run the blinky demo with mspdebug tilib 'prog target/msp430-none-elf/debug/examples/blinky'! However, as soon as I powered the board off and back on again, it no longer blinks.

After digging through the original TI demo, I found that there’s an additional initialization step needed for this microcontroller.

Adding this initialization step to the blinky demo fixes the initialization:

let pmm = p.PMM;
pmm.pm5ctl0.modify(|_, w| w.locklpm5().clear_bit());

And with that, it finally works reliably!

Nixie MultiMeter, Part 15

2018-07-09 00:00:00 +0000

Various Updates

Multi-board Schematic

I found a nice write-up about how to use a single set of schematics to design a multi-board project in Altium, and by extension, CircuitStudio: https://resources.altium.com/pcb-design-blog/multi-board-design-projects-in-altium-designer . The key trick is to create one project which contains all of the schematics, and then create another project in the same directory for each board, but only add the schematics for that board to that project.

Setting this up for my project was pretty easy; the hardest part was assigning each schematic sheet to the right board.

Resistance Measurement Circuit

The basic premise for measuring resistance is to apply a known current, and measure the resulting voltage. The ICL7103 reference design that I studied in part 2 re-uses the precision resistor ladder from the voltage divider in combination with an adjustable voltage source to generate the test current. The reference design doesn’t state the test current, but it looks like it was 100uA, generated by a voltage of 0.1V applied across a 1k resistor. The same voltage is applied across the precision resistor ladder, which generates a precise current through the device under test. Measuring the voltage across the device and dividing by the current gives the measured resistance.

The ADD3501 uses a similar principle, but a very different circuit. Instead of using a single op-amp, it uses two op-amps, once to develop the reference voltage, and another to drive the output. It drives the output through a discrete BJT instead of driving it directly, and uses a separate set of precision resistors instead of re-using the voltage divider ladder. This design uses the precision voltage reference, but since the precision voltage is applied across an adjustable resistor, it doesn’t seem like the precision of the voltage reference is very important; it just looks like it’s used as a stable voltage source.

My existing schematic has a precision resistor ladder for voltage measurements, similar to the other designs, but I’ve added some extra protection resistors to the back of the ladder in a way that at least 1M of resistance is always present between the input and the rest of the circuit, and the added resistance makes it difficult to use as part of the precision current source.

From Part 6, I wanted to have 100 Ohm, 1k, 10k, 100k, 1M and 10M resistance measurement ranges, and I have a ADC full-scale range of 10mV. Since I don’t need to to concurrent power measurements, I can potentially adjust the ADC gain for a different range. The low 10mV sensing voltage implies a lower applied current, so I’m going to run with it for now.

To start, a basic table of the test currents required for each resistance range. This is mostly a quick check that the test current is in the right ballpark. (I’ve added a few lower resistance ranges for good measure).

Resistance Range	Test Current	Resistance at 0.2V
0.1 Ohm	100 mA	2 Ohms
1 Ohm	10 mA	20 Ohms
10 Ohm	1mA	200 Ohms
100 Ohm	100 uA	2k Ohms
1k Ohm	10uA	20k Ohms
10k Ohm	1uA	200k Ohms
100k Ohm	0.1uA	2M Ohms
1M Ohm	10nA	20M Ohms
10M Ohm	1nA	200M Ohms

That’s a lot of resistor values! I’d like to have most of that range, but that’s too many resistors and relays for my taste. I looked at using a current mirror to generate the current reference, but given the wide range of operating voltages and currents, I don’t think a current mirror is a good idea (and The Art of Electronics agrees).

If I can generate two voltage references (probably 0.1V and 1V), then I can use the same resistor to generate two test currents.

Resistance Range	Test Current	Resistance at 0.1V	Resistance at 1.0V
0.1 Ohm	100 mA	1 Ohms	10 Ohms
1 Ohm	10 mA	10 Ohms	100 Ohms
10 Ohm	1mA	100 Ohms	1k Ohms
100 Ohm	100 uA	1k Ohms	10k Ohms
1k Ohm	10uA	10k Ohms	100k Ohms
10k Ohm	1uA	100k Ohms	1M Ohms
100k Ohm	0.1uA	1M Ohms	10M Ohms
1M Ohm	10nA	10M Ohms	100M Ohms
10M Ohm	1nA	100M Ohms	1G Ohm

If I choose 10, 1k, 100k and 10M resistors, I can get 0.1, 1, 10, 100, 1k, 10k, 100k and 1M measurement ranges.

The LTC6078 has maximum source and sink currents of 5mA and 7mA respectively, so it won’t be able to generate the 10mA and 100mA currents needed for the 1 Ohm and 0.1 Ohm ranges by itself. If I want those ranges, I’ll need to use a different op-amp or a transistor to generate enough output current. I suspect this is why the ADD3501 design uses a BJT in the drive portion of the current measurement stage.

Since the virtual ground in my circuit is also driven by an LTC6078, I’ll need to update the virtual ground or find another way to sink the matching test current.

Section 4.3.1, subsection E provides a couple of suggestions for increasing the output drive current of an op-amp. The easy suggestions are the LT1010, BUF633 and BUF634 power boosters, which are just unity-gain amps with an output current capability of 200mA. The other suggestion is a BJT push-pull follower. This also suggests a third option: using more generic op-amps with high-current capability for the virtual ground and constant current source. I’m already using a dual op-amp IC for the virtual ground and current source; swapping it for a different part number isn’t much work (and it doesn’t have the same input leakage and offset requirements as the input buffer op-amps).

The only thing I’ll need to be careful with, regardless of what kind of op-amp I use, is the power dissipation in the virtual ground, the reference resistor, and the op-amp or transistor providing the constant current source. At 3.3V and 100mA, that’s 0.33W total (probably spread across several components). My general rule of thumb is that anything under 1W is within the thermal envelope of most IC, so it’s probably not enough to overheat anything, but the resulting heating should be taken into account along with the other tolerances on components.

Most of the op-amps that are able to source more current have a higher input offset. The input current leakage and voltage offset just needs to be low enough to avoid introducing additional error into the test voltage. For the 0.1V reference voltage, 1 lsb corresponds to 1/10000 * 0.1V = 10uV. At the 1M range, the input leakage current needs to be below 1/10000 * 10nA = 1pA. Based on previous specs, I suspect it will be quite tricky to find an op-amp that meets all of these criteria.

Instead, I can take advantage of the low leakage current of the buffer op-amp, and use the output from the buffer as the negative input to the current stage.

4 Wire Mode

For resistance measurements where the resitance of the wires may be a significant percentage of the test resistance, a 4-wire resistance test applies the test current with one set of leads, and uses separate test leads to sense the resulting voltage. There may be some voltage drop in the leads that apply the test current, but since the current in the test leads is nearly zero, there’s no voltage drop to affect the measurement.

Precision Voltage/Current Source

The current source in the ICL7103 uses a JFET as as stable current source across a wide voltage range, and I’m intending to roughly duplicate this design, but with two output resitors to provide a switchable output voltage range. Since the output resitors are adjustable, the current source needs to be stable across the measurement voltage range and across temperature. Since the JFET or current source may have some internal heating and poor internal heat dissipation, it’s best to assume that it may be operating within spec but well above the ambient temperature range.

The voltage across the current source will probably be between 1.65V (virtual ground voltage) and the maximum input voltage of 10mV + 1.65V = 1.66V. Similarly, the output voltage of the current source op-amp will be 0.1V or 1.0V above the sense voltage, so the total op-amp voltage will be between 1.75V (0.1V + 1.65V) and 2.66V (1.0V + 1.65V + 10mV). Specifically, this means that the op-amp will need to operate within 0.6V of the high rail.

I could re-use the MMBF4117 that I’m using in the input protection circuit as the current source. According to the MMBF4117 Datasheet, the drain current at Vgs = 0 varies from 80uA to 60uA over a temperature range of -55C to 125C (0.2% / C). Similarly, the drain current varies from 52uA to 55uA over a Vds range of 1.0V to 2.0V. (5.7% / V or 0.006% / mV).

As an alternative, I could use an adjustable current source like the LM234. Over the same temperature range, the reference voltage varies from 48mV to 84mV (0.4% / C). According to the datasheet, the current doesn’t vary above about 1.0V for a resistor value of 68 Ohm and a current of 1mA. (This is obviously very good, but may also be a modest exaggeration by the manufacturer).

A third, possibly more accurate current source is the REF200, but the minimum compliance voltage is 2.5V. That’s too far above our minimum current source compliance voltage of 1.65V to be useful.

Given that the MMBF4117 is about $0.50 and the LM234 is $1.34 for similar performance, the MMBF4117 seems like the obvious choice.

Trimmer Potentiometers

The minimum current through the MMBF4117 is 30uA. Rounding down to 20uA to give a bit of headroom, the maximum resitances required to generate 0.1V and 1.0V are 3.3k and 33k. (3.7k and 33k in parallel produce about 3.3k). I’d like to use a multi-turn potentiometer for both adjustments. On digikey, there aren’t any 3.7k or 33k multi-turn potentiometers, but a larger resitor would also be acceptable. Digikey has 4.7k, 5k, 10k, 47k and 50k, and choosing the 5k and 50k models gives me a bit of extra margin for variation in the JFET current.

Rust on MSP430, Part 2

2018-06-15 00:00:00 +0000

Continuing the previous post about Rust on the MSP430

I found that it looks like I should use the MSP430 compiler from TI instead of the version from the Ubuntu repository, so I downloaded TI’s MSP430 compiler. This required signing up for a TI account (I already had one) and agreeing to some fairly substaintal export restrictions (I’m a US citizen so not a big deal for me, but probably a big deal for people outside of the US).

The TI compiler installed into ~/ti/msp430-gcc, so I edited my bashrc to add the bin directory within the install to my path: export PATH=$PATH:$HOME/ti/msp430-gcc/bin. TI’s compiler includes an executable named msp430-elf-gcc, so I also removed my previous hack to make msp430-gcc into msp430-elc-gcc.

Running xargo rustc --target=msp430-none-elf --release succeeded!

It doesn’t seem like that compiled the msp430-quickstart project; the target directory in the project exists, but there aren’t any binaries in it. I think there’s something else that I need to do to finish the compilation process. I think xargo rustc only built a rust cross-compiler for the MSP430; I think I still need to find a way to invoke that compiler to build the project.

Looking at a more complex project, the AT2XT, they have their source code in the src directory, so I copied examples/blinky.rs to src/main.rs (not sure if the name is important or not). Now, doing xargo build --target=msp430-none-elf --release seems to make more progress and fails with:

   Compiling msp430-rt v0.1.2
error[E0522]: definition of an unknown language item: `panic_fmt`
 --> /home/austin/.cargo/registry/src/github.com-1ecc6299db9ec823/msp430-rt-0.1.2/src/lang_items.rs:3:1
  |
3 | #[lang = "panic_fmt"]
  | ^^^^^^^^^^^^^^^^^^^^^ definition of unknown language item `panic_fmt`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0522`.
error: Could not compile `msp430-rt`.
warning: build failed, waiting for other jobs to finish...
error: build failed

This looks like the msp430-rt crate is using something that doesn’t exist in the nightly version of Rust. I’ll have to check their bug tracker to see if there are any other reports of this.

There are not currently any issues on the msp430-rt repo on github, but there was an announcement about changing panic_fmt to panic_implementation two weeks ago. I suspect this is the issue, and msp430-rt hasn’t caught up yet.

I filed a bug report on msp430-rt about this, so that the next person who encounters the same issue has a bit more information: https://github.com/pftbest/msp430-rt/issues/2

That’s it for the day. See y’all next time.

Rust on MSP430

2018-06-13 00:00:00 +0000

This is mostly just my collected notes on getting Rust to build code for the MSP430.

I’m building on Ubuntu 14.04 which is quite old, so there are a probably a few quirks in my setup compared to newer Ubuntu or other Linux distributions.

Install MSP430 gcc toolchain: sudo apt-get install binutils-msp430 gcc-msp430 gdb-msp430 msp430-libc msp430mcu mspdebug
I already have rust installed, but I upgraded to nightly rust because the examples only build on nightly:
- rustup install nightly
- rustup default nightly
After googling for msp430 and rust I found msp43-quickstart
- There are some additional instructions in PR #3 that were quite useful. Most of these instructions are borrowed from there.
I installed rust-src: rustup component add rust-src
Installed xargo: cargo install xargo
- Installing xargo from inside the msp430-quickstart project fails. I’m not sure why or how, but it’s trying to pick up settings from that project.
- cd ~, cargo install xargo works.
Cloned msp430-quickstart: git clone https://github.com/japaric/msp430-quickstart.git
cd msp430-quickstart
Tried to build with xargo rustc --target msp430-none-elf --release -- -C link-arg=-Tlink.x -C link-arg=-mmcu=msp430g2553 -C link-arg=-nostartfiles -C linker=msp430-elf-gcc -Z linker-flavor=gcc; failed to find msp430-elf-gcc
- Build command from the msp430_rt crate
My compiler is named msp430-gcc so I created a shell script on my path named msp430-elf-gcc that does exec msp430-gcc "$@"

Retried build. Failed again with:

 note: "msp430-elf-gcc" "-mcpu=msp430" "-c" "-o" "/tmp/xargo.XqgEMtdDHVan/target/msp430-none-elf/release/deps/core-2ec70cc69e40cb54.core0.rcgu.o" "/tmp/xargo.XqgEMtdDHVan/target/msp430-none-elf/release/deps/core-2ec70cc69e40cb54.core0.rcgu.s"
    
 note: Assembler messages:
 Fatal error: unrecognized cpu type msp430

I am not able to find any results online for an error message that matches unrecognized cpu type msp430 but my best guess is that I need a newer version of msp430-gcc

No luck, but it’s time to call it a day. When I come back to this later I’ll probably either try to upgrade to a newer version of Ubuntu (14.04 is ANCIENT!) or look for a PPA that has built a newer version of msp430 gcc for 14.04

Farewell packages.namniart.com