[development] Docker and CoreOS support


#1

Continuing the discussion from [need feedback] CoreOS support:


[need feedback] CoreOS support
#2

Currently working on getting an armhf CoreOS install image built, will let you know how it comes along.


#3

\o/ thank you !


#4

Sadly doesn’t work due to the CrOS chroot :confused:


#5

Managed to get Docker working. Needs a custom kernel with bridge and loop modules enabled but other than that, the stock Ubuntu image seems enough! Will write up a tutorial if you want.

Some of the things that need fixing to make it work out of the box:

  1. Stock ubuntu image has a kernel with no bridge or loop kernel
    modules.
  2. Ubuntu image doesn’t create /dev/loopX devices on boot, and they need to be created. Can be created with pushd /dev && MAKEDEV loop && popd
  3. bridge-utils is missing and needs to be installed.

root@c1-docker:/dev# docker info
Containers: 0
Images: 0
Storage Driver: devicemapper
 Pool Name: docker-0:13-3517461-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 291.5 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 0.7 Mb
 Metadata Space Total: 2048.0 Mb
Execution Driver: native-0.2
Kernel Version: 3.17.0-87

#6

Any update on this? I’m looking into using it but can’t even find a beta image.


#7

I don’t think CoreOS is worth the hassle given distributions like RancherOS.


The chroot environment is Gentoo. Compiling CoreOS is not complex. You will need to go through the packages, though, to find out what patches they need for armhf and adjust the bootloader for C1s (I don’t know what Scaleway is using, uboot like back then?). Given that there are already “templates” from Ubuntu (use their source’s patches) this shouldn’t be surprisingly hard.

Patches go to /etc/portage/patches/<category>/<name>[:version]/. Modify file(s) make.conf like you did for Gentoo. By default CrOS uses -mtune=generic (and the wrong arch…). Replace the binhosts with your own.

You then start the first run in order to obtain binaries, which you then upload someplace (hello s3ql and object storage, which still doesn’t work for me due to CORS, @Benedict). It is important that you select --binaries=false with CrOS/CoreOS scripts, or they will be pulled for arch amd64. That’s all there is, and waiting.

On Amazon EC2. for the target amd64 you would start like this (takes about 60 minutes @ 8 ECUs). You won’t be able to test the resulting QEMU images, but you can easily download them locally and do that (the files are disk images and compress using plzip to 160~200 MB within 6 minutes). As there is no development image (one won’t be downloaded) just use a Gentoo’s stage3! (Prune the contents of the image for amd64 and extract the stage3 on partition 9 etc. similarly.)

# become root
# we will need this later
(apt-get -qq -y install repo htop qemu-kvm qemu libvirt-bin plzip) &

# prepare space on the instance-storage SSD
mkfs.ext4 -L dev-root -m 0 -O ^has_journal /dev/xvdc
tune2fs -o journal_data_writeback /dev/xvdc
fsck.ext4 -f /dev/xvdc
mkdir -p /usr/src/coreos
mount -o defaults,data=writeback,lazytime /dev/xvdc /usr/src/coreos
chmod 1777 /usr/src/coreos

wait
# become a normal user! exit 'su'

cd /usr/src/coreos
repo init -u https://github.com/coreos/manifest.git && repo sync
sudo ./chromite/bin/cros_sdk
echo "core" | ./set_shared_user_password.sh

# this is where you modify make.conf, and pull patches
(cd /etc; mv portage portage.vanilla; git clone --depth=1 https://…/my-portage.git portage)
# do this for every make.conf you find if you encounter more than one!
# Or adjust repo's Manifest when you know which bits you need to replace.

./setup_board --default --board=… --usepkg=false
./build_packages --usepkg=false --getbinpkg=false
…

Disclosure: I’ve been using Gentoo for 12+ years, ran a binhost for the SheevaPlug some years, and sporadically contributed patches for porting software to ARM. Until Calxeda went out of business. The CoreOS images on AWS EC2 on the community marketplace are from me.

Once you have that first image you will need an update server which talks the reduced Omaha protocol. There’s already a mockup in the SDK.


#8

Hello,
personnaly with CoreOS i use Fleet also.
I’m really looking forward a coreOS image to use scaleway :slight_smile:


#9

Hi there

Any more news on this? With the release of x64 servers I’m keen to set up a cluster of CoreOS instances to run some workloads on as the low cost and hourly billing is a perfect combination for an elastic Kubernetes cluster.

James


#10

Hello,

Waiting for the official image to be available, you should also be able to boot in network using ipxe.


After this, you can use their installer and setup your node into a new volume.

Then create a snapshot, and boot more nodes


#11

Waiting for CoreOs too ^^ .


#12

Like @james I’m trying to achieve too a Kubernetes cluster on top of the CoreOS base system running on multiple C2s servers (~> x86)
So a definitive big +1 for CoreOS :smile:


#13

But from what I read in the forum, once you need to reboot you still need to do some manual thing in iPXE, otherwise you will not land in the CoreOS.

That would be a disaster scenario as CoreOS reboot very often.


#14

Hi there,

Seems the CoreOS topic is quite inactive for months - is it officially abandonned or still some WIP ?

Thanks,
Nicolas


#15

You’re referring to this one:

(Which is not about support for CoreOS on armhf.)