Nagios. It does depend on what you mean by monitor though. Nagios is good at telling you that “service A on host B” is down" but less useful for looking at things like performance trends. I particularly like being able to setup dependencies between services, so I get the alert for the root cause, and not all of the services that have gone down because of it.

You dismiss the data you recorded because it doesn’t seem to support your hypothysis tgat there is greater lag in wayland, but that’s not really the right approach, and I think it points to a different conclusion.

You recorded a lag of 5 or 6 frames at 90 frames per second in both Xorg and wayland, which suggests that the lag is the same to within 0.011 seconds, and I don’t think that you can say that’s a huge difference. However, what you didn’t test is the acceleration curve on mouse movement. If that curve is different under wayland it could easily feel infuriatingly laggy without actually showing any extra delay on the movement starting or ending.

I’m not sure how you’d accurately test that, a HID device just sending mouse move events wouldn’t do it as wouldn’t mimic you accelerating the mouse from stationary, so wouldn’t exercise the acceleration curve in wayland. You might need a physical device that moves your actual mouse a fixed dustance and then measure the distance the cursor moves on screen. Repeat for different movement speeds and you might have sone useful data.

… and half the country is extremely enthusiastic about that.

There’s the reason nothing is done about it. It’s probably not actually half, but enough people didn’t speak up early enough, and so this has become the loudest voice in the room. Unless, and until that changes, the whole world is in for a rough ride.

Good grief! You can’t just go posting that sort of thing in public, at least tag it NSFW.

Now please excuse me while I go and clutch my pearls in horror.

The problem with releasing them on day one is that you then can’t gather more. If you’ve only just exposed the edges of the malfeasance you need time to get the rest before exposing it. Go too early and the rest of the evidence can be destroyed, covered up or those holding it coearsed into silence.

Having a dead man’s switch is a way to ensure whatever you’ve gathered gets released if you’re no longer in a position to gather more. As such I disagree with the poster about making it public knowledge before release. Keep it secret until you have everything, then release it.

Ha! Faking key presses, truly an elegant weapon for a more civilized age. If it works, it works.

The thing that got me to switch was being able to maintain my pane layout between connections. The various window and pane management niceties (naming, swapping, listing and the like) got me to stay. Now you can keep your screen, but you’d have to pry tmux from my cold, dead, tty.

It’s likely that you’re using a systemd based system and the admin hasn’t enabled linger for your user.

I agree with the sentiment regarding being woken up, but I used to look forward to being on call. I could go to bed happily, knowing I was earning a significant premium and I’d still get a good night’s sleep because the systems just didn’t go down. I had the advantage that most of the customers I supported had similar requirements, so I had their systems locked up pretty well. Minor problems (disk space. Why is it always disk space?) would self heal, catastrophic failures (hardware failures or the engineer who supposed to replace a component unplugging the wrong server) would fail over to the rest of the cluster. I never had much trouble with logging either, it was typically one of the first things set up, and I had most of the setup automated to avoid missing anything. I suppose the thing was I was supporting systems I’d built, and I’d built them to ensure I didn’t have to be woken up.

I do a lot more troubleshooting and rescue type work nowadays, and the number of times I run into systemd components just not doing what they should is frustrating to say the least. Being able to pull the logs by knowing the service name would be nice, but a) you could already do that because you set up different services to log to different places and b) you don’t always know the service name in question. Being able to just grep the log directory is a lifesaver. You can still do that, but only because distros set systemd up to log to file as well as it’s binary format. I loathe the way systemd ends up spreading it’s unit files over about a dozen different directories, with overrides increasing that even further. I just want to know what services I’ve got and what will start up, in exactly what order, on the next reboot, dammit! The last one is particularly tricky as, due to services being started in parallel, you can’t predict exactly what order things will actually start between targets. That shouldn’t matter, units should have all their dependencies properly listed, but it’s no fun tracking down a race condition that only happens once every x reboots when a particular network service takes a few hundred milliseconds longer to come up. Give me sequential boot any day. It might take a few tens of seconds longer, but it happens the same way each time, and I only need to look in one place to know what that is.

As to systemd’s dominance, once Redhat, where Mr Pottering worked, chose it, it became hard for other distros not to. Derivative distros obviously went with it, and if you look back through the various email discussions, it was far from a unanimous choice for distros like Debian to choose it. They did so eventually mostly, as far as I can see, because it would theoretically make packaging easier. Fortunately they still support sysvinit, so all is not lost for those of us who want a mainstream distro without systemd bloat.

Shifting stuff to kube is definitely goot for making things more robust, so long as you’ve got the underlying clustering working, and I quite like working with it too. Once you realise it’s basically just a database and message queue with a bunch of controllers for managing storage, networking, containers and the like, and the ability to extend that, you can do all sorts of fun things with it.

Anyway, I’ve gone on for long enough. If you’re a sysadmin and the number of trouble calls is going down, then you probably don’t hear this often enough: well done, you’re doing a great job.

Ok, fair point on the capital D, I must have read it like that years ago and it stuck. I shall have to make an effort to unlearn it.

As to the rest, systemd has been a constant thorn in my side ever since L. Pottering published “Rethinking PID 1” back in 2010 or so. I found, and still find, that most of the assertions and actions in that document either don’t really hold, or just aren’t really relevant. Basically it’s trying to solve a problem that really wasn’t an issue in the real world, and does so in such a massively overbearing way that everything actually becomes more laborious than it otherwise would be. From my perspective it’s an unnecessarily complex and poorly architected attempt to answer a need that was better served in different ways. That it’s become a near mono-culture is deeply concerning.

I’ve also run into all sorts of awkward edge cases and misfeatures over the years, from the automounter that occasionally didn’t to race conditions that only manifest at the worst moments, none of which would have occured had the basic tenet of “do one thing and do it well” been followed. The extreme verbosity of the configuration, and unnecessarily large number of places it can be spread just serve to make it even more unpleasant to deal with compared to the simplicity of init scripts, crontabs and the like.

The sad thing is, there’s undoubtedly some good ideas buried in it, but they could all have been implemented much more lightly and in a way that worked with the rest of the ecosystem rather than fighting it. Things like starting daemons in what is essentially a repeatable sandbox, or being able to isolate logging per service. They could, and had both been implemented already, but systemd has a real “not invented here” problem, so everything was built again, with all the attendant bugs, and design issues that inevitably brings.

Ultimately clients pay good money for me to look after their systems, systemd or not, so I probably shouldn’t grumble, but I miss the days when Linux was a clean and elegant system, without this multi-tentacled thing sitting on top of it.

SystemD is far too much of a poorly thought through mess to have anything like a sane GUI configuration, it doesn’t even have a sane textfile based configuration. We’re going to have to wait fir SystemD to crumble under it’s own weight and be replaced with multiple, simple, cleanly designed components before we have any hope of a sane config again. Sort of like we used to have before a certain someone/some company (depending on how conspiratorial you’re feeling) decided to come along and muck it all up.

/rant

Thank you for coming to my Ted Talk Rant. You may gather I dislike SystemD quite a lot.

A dinosaur skeleton in a spacesuit.

It’s an old phrase, “lions lead by donkeys”, originally used to describe British soldiers being lead by incompetent generals, possibly during the Crimean war. The concept goes way further back though; apparently an Athenian general in the 4th century said (translated) “an army of deer commanded by a lion is more to be feared than an army of lions commanded by a deer”.

Vim is running as you, rather than root, so you wont be able to edit other files as root, and any rogue plugins wont be able to either, which is good.

Sudoedit has various guards around what it’ll let you edit, in particular, you can’t edit a file in a directory you already have write permission on as doing so allows the user to bypass restrictions in the sudoers setup (there’s more detail in their issue tracker. If the directory is already writable though, you don’t need sudoedit anyway.

Be cautious of doing this with security sensitive files. When it copies the file and gives you ownership, any sensitive data in it is exposed to any process running as your user id, and and temporary fil£s the editor creates may also contain the sensitive content and be owned by you.

This is fundamentally true. However it is possible to limit the bandwidth of data the employee can exfiltrate.

That’s fair, but the OP was talking about having the sensitive data directly on the laptop, which rather limits your ability to control access to it, and why I was suggesting keeping the data on a server and providing access that way, so limits can be put in place.

Assuming a privileged employee suddenly becomes a bad actor.

Your threat model probably isn’t the employee who suddenly goes rogue and tries to grab everything, but the one who spends and extended period of time, carefully, extracting key data. As you, the former can be be mitigated against, but the latter is very much harder to thwart.

But I couldn’t for example download our entire customer database, I could get a specific record, I could maybe social engineer access to all the records of a specific customer, but there is no way I’d be able to extract all of our customers via an analog loophole or any standard way. The data set is too big.

That’s well set up, but, whilst your competitor would love the whole database, what they’re really interested in is the contact details and contract information for maybe your largest three customers. When the dataset to extract is small enough, even quite advanced rate limiting can’t really help much. Making sure no one person has access to all of the most valuable data is a good start, but can present practical problems.

And this is what you are trying to limit. If you trust your employees (some you have to), you can’t stop them from copying the keys to the kingdom, but you can limit the damage that they can do, and also ensure they can’t copy ALL the crown jewels.

I think we’re basically saying the same thing. The OP talked about putting all the sensitive information on the employee’s laptop, and that’s what I was trying to steer them away from.

In the past I’ve been asked if we can provide our developers access to pull the full source tree, but not copy it anywhere, and, to quote Charles Babbage, “I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.”

I enjoy the security side of sysadmin work, but I find it rather depressing, as all you can hope to do is lose slowly enough that it’s not worth attacking you.

I agree that you should assume you’re being monitored, but, while that helps against malware type exfiltration, it does little to stop someone who is determined to exfiltrate the data as there are a myriad of ways to do so that aren’t possible to monitor, such as simply taking a video of the screen whilst displaying the information.

Ultimately, the company has to trust the employee or not give them access to the sensitive data, there’s no middle ground.

There is a fundamental issue with this approach: the rogue employee has already copied the data to a USB drive by the time you try to wipe it.

If the data is confidential, you either need to set up standard disk encryption and trust the employee, or not let them access it in a way it can be bulk copied. For instance, might it be possible for them to use a webapp that you control access to or a remote desktop type setup?

It depends what you want to do with it. If it’s just for storing files/backups then encrypt them before uploading and make sure the key never goes anywhere near the VPS. If it’s for serving up something like a simple website, you probably care more about data integrity than exfiltration, so make sure you have the security, including selinux or equivalent, locked down, and regularly run integrity checks. If it’s for running something interactive, or where data will be generated or downloaded to the machine, you’re out of luck, there’s no even theoretical way of securing that against an adversary with that much access.

Not morons, just not educated enough about them to understand exactly what the implications of that action are.