I never understood how to use Docker, what makes it so special? I would really like to use it on my Rapsberry Pi 3 Model B+ to ease the setup process of selfhosting different things.
I’m currently running these things without Docker:
- Mumble server with a Discord bridge and a music bot
- Maubot, a plugin-based Matrix bot
- FTP server
- Two Discord Music bots
All of these things are running as systemd services in the background. Should I change this? A lot of the things I’m hosting offer Docker images.
It would also be great if someone could give me a quick-start guide for Docker. Thanks in advance!
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
Fewer Letters More Letters CA (SSL) Certificate Authority DNS Domain Name Service/System Git Popular version control system, primarily for code HA Home Assistant automation software ~ High Availability IP Internet Protocol LXC Linux Containers NAS Network-Attached Storage SBC Single-Board Computer SSD Solid State Drive mass storage SSL Secure Sockets Layer, for transparent encryption
9 acronyms in this thread; the most compressed thread commented on today has 15 acronyms.
[Thread #592 for this sub, first seen 11th Mar 2024, 17:25] [FAQ] [Full list] [Contact] [Source code]
IMHO with docker and containerization in general you are trading drive space for consistency and relative simplicity.
a hypothetical:
You set up your mumble server and it requires the leftpad 3.7 package to run. you install it and everything is fine.
Now you install your ftp server but it needs leftpad 5.5. what do you do? hope the function that mumble uses in 3.7 still exists in 5.5? run each app in its own venv?Docker and containerization resolve this by running each app in its own mini virtual machine. A container running mumble and leftpad 3.7 can coexist on host that also has a container running a ftp server with leftpad 5.5.
Here is a good video on what hole docker and containerization looks to fill
https://www.youtube.com/watch?v=Nm1tfmZDqo8Doesn’t that mean that docker containers use up much more resources since you’re installing numerous instances & versions of each program like mumble and leftpad?
Kinda, but it depends on the size of the dependencies, with drive space bing so cheap these days do you really worry about 50Mb of storage being wasted on 4 different versions of glib or leftpad
Docker containers aren’t running in a virtual machine. They’re running what amounts to a fancy chroot jail… It’s just an isolated environment that takes advantage of several kernel security features to make software running inside the environment think everything is normal despite being locked down.
This is a very important distinction because it means that docker containers are very light weight compared to a VM. They use but a fraction of the resources a VM would and can be brought up and down in milliseconds since there’s no hardware to emulate.
To put it in simpler terms, I’d say that containers virtualise only the operating system rather than the whole underlying machine.I guess not then.
It virtualises only parts of operating system (namely processes and network namespaces with ability to passthru devices and mount points). It is still using host kernel, for example.
I wouldn’t say that namespaces are virtualization either. Container don’t virtualize anything, namespaces are all inherited from the root namespaces and therefore completely visible from the host (with the right privileges). It’s just a completely different technology.
The word you’re all looking for is sandboxing. That’s what containers are - sandboxes. And while they a different approach to VMs they do rely on some similar principals.
I started self-hosting a bit prior to when Docker took off, and getting multiple services running was much harder. Service A wants a certain version of PHP installed with certain plugins while Service B wants a different version. You’d follow a tutorial for installing Service C and desperately hope that it wouldn’t somehow break Service A or B. You installed Service D for a bit despite all the installation pain and now want to uninstall it - I hope you tracked exactly what config changes you made throughout the system so you can undo it.
Docker fixed all of this by making each service independent through containers which made self-hosting 10x easier. I’d also add that I love how easy it is to transfer my setup to a new server - I keep all of my container volumes in a specific directory and my docker-compose files in another and that’s all I need to backup / transfer. Without Docker you’d have to specifically handle each & every configuration file and database location, and if you later upgrade to a newer version of the OS or a different distro you’d have to handle possible conflicts between your versions and what the distro expects.
There have been some great answers on this so far, but I want to highlight my favourite part of Docker: the disposability.
When you have a running Docker container, you can hop in, fuck about with files, break stuff as you try to figure something out, and then kill the container and all of the mess you’ve created is gone. Now tweak your config and spin up a fresh one exactly the way you need it.
You’ve been running a service for 6 months and there’s a new upgrade. Delete your instance and just start up the new one. Worried that there might be some cruft left over from before? Don’t be! Every new instance is a clean slate. Regular, reproducible deployments are the norm now.
As a developer it’s even better: the thing you develop locally is identical to the thing that’s built, tested, and deployed in CI.
I <3 Docker!
What about your preferences/configs/files (when you spun up a fresh one)?
The most popular way of configuring containers are by using environment variables that live outside the container. But for apps that use files to store configuration, you can designate directories on your host that will be available inside the container (called “volumes” in Docker land). It’s also possible to link multiple containers together, so you can have a database container running alongside the app.
Install Portainer, it helps you get used to managing docker images and containers before going full command line.
I actually prefer dockge, I only have a few containers and its a lot simpler while still able to do all the basics of docker management. Portainer was overkill for me.
I feel that a lot of people here are missing the point. Docker is popular for selfhosted services for a few main reasons:
- It is one package that can be used on any distribution (or even OS with a Linux VM).
- The package contains all dependencies required to run the software so it is pretty reliable.
- It provides some basic sandboxing against non-malicious services. Basically the service can’t scribble all over your filesystem. It can only write to specific directories that you have given it access to (via volumes) other than by exploiting security vulnerabilities.
- The volume system also makes it very obvious what data is important and needs to be backed up or similar, you have a short list.
Docker also has lots of downsides. I would generally say that if your distribution packages software I would prefer the distribution’s package over the docker image. A good distribution package will also solve all of these problems. The main issue you will see with distribution packages is a longer delay before new versions are made available.
What Docker completely dominates was previous cross-distribution packaging options which typically took one of the previous strategies.
- Self-contained compiled tarball. Run the program inside as your user. It probably puts its data in the extracted directory, maybe. How do you upgrade? Extract and copy a data directory? Self-update? Code is mutable and mixed with data, gross.
- Install script. Probably runs as root. Makes who-knows what changes to your system. Where is the data, is the service running? Will it auto-start on boot. Hope that install script supports your distro.
- Source tarball. Figure out the dependencies. Hope they don’t conflict with the versions your distro has. Set up users and setup scripts yourself. Hope the build doesn’t take too long.
Sorry if I’m about 10 years behind Linux development, but how does Docker compare with the latest FlatPak trend in application distribution? How you have described it sounds somewhat similar, outside of also getting segmented access to data and networks.
For desktop apps Flatpak is almost certainly a better option than Docker. Flatpak uses the same core concepts as Docker but Flatpak is more suited for distributing graphical apps.
- Built in support for sharing graphics drivers, display server connections, fonts and themes.
- Most Flatpaks use common base images. Not only will this save disk space if you have lots of, for example GNOME, applications as they will share the same base but it also means that you can ship security updates for common libraries separately from application updates. (Although locked insecure libraries is still a problem in general, it is just improved over the docker case.)
- Better desktop integration via the use of “portals” that allow requesting specific things (screenshot, open file, save file, …) without full access to the user’s system.
- Configuration UIs that are optimized for the desktop usecase. Graphically tools for install, uninstall, manage permissions, …
Generally I would still default to my distro’s packages where possible, but if they are unsuitable for whatever reason (not available, too old, …) then a Flatpak is a great option.
Try to run something that requires php7 and something else that requires php8 on the same web server; or python 2 and python 3.
You actually can, but it’s not pretty.
(The thing about a declarative setup isn’t much of a difference, you can do it for any popular Linux distro.)
Doesn’t that mean that docker containers use up much more resources since you’re installing numerous instances & versions of each program like PHP?
Oh, sure, the bloat on your images requires resources from the host.
There is the option of sharing things. But, obviously that conflicts a bit with maintaining your environments isolated.
The thing with Docker is that people don’t want to learn how to use Linux and are buying into an overhyped solution that makes their life easier without understanding the long term consequences. Most of the pro-Docker arguments go around security and that’s mostly BS because 1) systemd can provide as much isolation a docker containers and 2) there are other container solutions that are at least as safe as Docker and nobody cares about them.
Companies such as Microsoft and GitHub are all about re-creating and reconfiguring the way people develop software so everyone will be hostage of their platforms. We see this in everything now Docker/DockerHub/Kubernetes and GitHub actions were the first sign of this cancer. We now have a generation that doesn’t understand the basic of their tech stack, about networking, about DNS, about how to deploy a simple thing into a server that doesn’t use some Docker BS or isn’t a 3rd party cloud xyz deploy-from-github service.
Before anyone comments that Docker isn’t totally proprietary and there’s Podman consider the following: It doesn’t really matter if there are truly open-source and open ecosystems of containerization technologies. In the end people/companies will pick the proprietary / closed option just because “it’s easier to use” or some other specific thing that will be good on the short term and very bad on the long term.
Docker may make development and deployment very easy and lowered the bar for newcomers have the dark side of being designed to reconfigure and envelope the way development gets done so someone can profit from it. That is sad and above all set dangerous precedents and creates generations of engineers and developers that don’t have truly open tools like we did. There’s LOT of money into transitioning everyone to the “deploy-from-github-to-cloud-x-with-hooks” model so those companies will keep pushing for it.
Note that technologies such as Docker keep commoditizing development - it’s a negative feedback loop that never ends. Yes I say commoditizing development because if you look at it those techs only make it easier for the entry level developer and companies instead of hiring developers for their knowledge and ability to develop they’re just hiring “cheap monkeys” that are able to configure those technologies and cloud platforms to deliver something. At the end of the they the business of those cloud companies is transforming developer knowledge into products/services that companies can buy with a click.
Most of the pro-Docker arguments go around security
Actually Docker and the success of containers is mostly due to the ease of shipping code that carries its own dependencies and can be run anywhere. Security is a side-effect and definitely not the reason why containers picked-up.
systemd can provide as much isolation a docker containers and 2) there are other container solutions that are at least as safe as Docker and nobody cares about them.
Yes, and it’s much harder to achieve the same. In systemd you need to use 30 different options to get what using containers you achieve almost instantly and with much less hussle. I made an example on my blog where I decided to run
blockyin Systemd and not in Docker. It’s just less convenient and accessible, harder to debug and also relies on each individual user to do it, while with containers a lot gets packed into the image and therefore harder to mess up.Docker isn’t totally proprietary
There are a many container runtimes (CRI-O, podman, mirantis, containerd, etc.). Docker is just a convenient API, containers are fully implemented just with Linux native features (namespaces, seccomp, capabilities, cgroups) and images follow an open standard (OCI).
I will avoid comment what looks like a rant, but I want to simply remind you that containers are the successor of VMs (virtualize everything!), platforms that were completely proprietary and in the hands of a handful of vendors, while containers use only native OS features and are therefore a step towards openness.
Docker and the success of containers is mostly due to the ease of shipping code that carries its own dependencies and can be run anywhere
I don’t disagree with you, but that also shows that most modern software is poorly written. Usually a bunch of solutions that hardly work and nobody is able to reproduce their setup in a quick, sane and secure way.
There are a many container runtimes (CRI-O, podman, mirantis, containerd, etc.). Docker is just a convenient API, containers are fully implemented just with Linux native features (namespaces, seccomp, capabilities, cgroups) and images follow an open standard (OCI).
Yes, that’s exactly point point. There are many options, yet people stick with Docker and DockerHub (that is everything but open).
In systemd you need to use 30 different options to get what using containers you achieve almost instantly and with much less hussle.
Yes… maybe we just need some automation/orchestration tool for that. This is like saying that it’s way too hard to download the rootfs of some distro, unpack it and then use
unshareto launch a shell on a isolated namespace… Docker as you said provides a convenient API but it doesn’t mean we can’t do the same for systemd.but I want to simply remind you that containers are the successor of VMs (virtualize everything!), platforms that were completely proprietary and in the hands of a handful of vendor
Completely proprietary… like QEMU/libvirt? :P
but that also shows that most modern software is poorly written
Does it? I mean, this is especially annoying with old software, maybe dynamically linked or PHP, or stuff like that. Modern tools (go, rust) don’t actually even have this problem. Dependencies are annoying in general, I don’t think it’s a property of modern software.
Yes, that’s exactly point point. There are many options, yet people stick with Docker and DockerHub (that is everything but open).
Who are these people? There are tons of registries that people use, github has its own, quay.io, etc. You also can simply publish Dockerfiles and people can build themselves. Ofc Docker has the edge because it was the first mainstream tool, and it’s still a great choice for single machine deployments, but it’s far from the only used. Kubernetes abandoned Docker as default runtime for years, for example… who are you referring to?
Yes… maybe we just need some automation/orchestration tool for that. This is like saying that it’s way too hard to download the rootfs of some distro, unpack it and then use unshare to launch a shell on a isolated namespace… Docker as you said provides a convenient API but it doesn’t mean we can’t do the same for systemd.
But Systemd also uses unshare, chroot, etc. They are at the same level of abstraction. Docker (and container runtimes) are simply specialized tools, while systemd is not. Why wouldn’t I use a tool that is meant for this when it’s available. I suppose bubblewrap does something similar too (used by Flatpak), and I am sure there are more.
Completely proprietary… like QEMU/libvirt? :P
Right, because organizations generally run QEMU, not VMware, Nutanix and another handful of proprietary platforms… :)
but I want to simply remind you that containers are the successor of VMs
Successor implies replacement. I think containers are another tool in the toolkit of servers/hosting, but not a replacement for VMs
Avoid Docker as much as possible. It gives you a quick way to run apps, but in a way that you can’t see what is going to be installed.
It is also double resource heater.
And it has many frequent vulnerabilities.
in a way that you can’t see what is going to be installed.
You can look at the Dockerfile and see every single step that goes into building a particular image.
It is also double resource heater.
That’s patently false.
And it has many frequent vulnerabilities
Dawg have you ever actually used docker?
