admin log
fd.o admin tasks and planned outages



Sat, 20 Nov 2004

brave new wor^Wgabe

As you may have noticed, sort of got compromised a few days back. By 'sort of', I do, of course, mean 'totally'. Adam Conrad noticed a few thousand bounces in his inbox courtesy of being on www-data, and that they were all for spams being sent as www-data. Whoops. We started hunting for an insecure, but when we took a look at lsof and discovered an IRC proxy running, we decided it was something more insidious. From there, the machine got killed to all access but ours, and we started tracking down the point of entry. It turned out that it was compromised via a hole in TWiki, but no news was to be found on the TWiki site about this hole, nor was there a new release. How not to do security 101.

At this point, we came to the conclusion that all we could do from here was reinstall, so Keith got a call (from an Australian mobile, roaming into the UK, to a US mobile; I fear to think how much that cost) letting him know the score. Local muscle on site, we dug in and prepared for a reinstall. Most people familiar with the setup (and my writings on here about 'ill') know that the setup was accumulated, not designed, and was horrifically out of control. It was a mess, and probably incredibly insecure. Very few things were done properly to scale to where it was. So, we took a deep breath, and noted that this was a blessing in disguise as we got to sit back and have a think about what we were doing this time. I got out some pieces of paper and started scribbling (across six of them, actually), and we all got chatting on what we could do when we rebuilt it all.

LDAP is already running on a separate machine, using Debian's userdir-ldap. We have a separate source machine on our hitlist; hosting only CVS/SVN/Arch repositories, and various web downloads. These downloads would have to be signed for somehow, and all provided in a common download area. Three huge hits: we're mirrorable, there's an audit trail and security on source, and the general access machine and the source server are totally separate. Rock on.

SSH access is open to the general public, with the old home directories in /home/compromised. If you administer a project with CVS or whatever, please check that it hasn't been tainted. You can compare the repositories in /cvs and /compromised-cvs to see the difference; /cvs contains the repositories as they were on 15th Oct.

Administering your account requires a GPG key. Admins will be rather loathe to perform menial duties (e.g. changing SSH keys) on a regular basis, so if you ask us for anything, make sure it's to add a GPG key to your account. This way, it's the same amount of work for us, and it ensures that you can take care of your own account in future: less work for both of us, and less time spent waiting.

Did I mention you should all have GnuPG keys? No, really. They're incredibly useful. If we had signed copies of everything, verification would be an utter doddle. But we don't, so it isn't.

Enjoy the new -daniels
[16:31] | [daniels] | # | TB

more drm repocopies

Copied some sis stuff missed in the previous repocopies.
[15:42] | [anholt] | # | TB

cvs on hold

Adam, cvs is on hold because we run it read-only, and we need a patch to shut it up on read-only mode (lest clients get confused and disconnect), so I put in that patch and drag it repeatedly through security updates. Ugh. -daniels
[15:42] | [daniel] | # | TB

planet love

Gave Planet some love -- fixed Mike A. Harris's feed, and removed rml's as was timing out. Also made the admin feed not look like crap by not escaping the HTML. Novel! -daniels
[15:42] | [daniel] | # | TB

welcome, welcome

Welcome to the fd.o admin blog. Here, we will announce impending downtime, actions taken, anything of interest admin-related. It is syndicated by RSS and Atom. Enjoy!
[15:42] | [daniel] | # | TB

apt-get upgrade, how fun

Yet another apt-get upgrade. Everything updated, except for cvs (which was on hold, does anyone happen to know why?) and mailman (which I've put on hold because the preinst script is braindead... I have pasc looking into this). Things seem reasonably sane, but if something suddenly seems broken, feel free to blame new versions (and mail me to complain).

... Adam (adconrad)
[15:42] | [adconrad] | # | TB

viewcvs love

Today, with the very, very able assistance of James Henstridge, I upgraded ViewCVS. James set upstream up in a vendor branch that we can check out, make our own modifications in HEAD, etc. Some stuff *may* be broken, but it should be just fine. Thanks James! -daniels
[15:42] | [daniel] | # | TB

new committer

Added Fabio Massimo Di Nitto, X co-maintainer for both Ubuntu and Debian, to the xorg group to commit random build fixes, and changes to the Debian section, et al. -daniels
[15:42] | [daniel] | # | TB

fixing postfix, crazy debugging, oh my!

Wow. newaliases on gabe has been segfaulting for quite some time now, so Pasc Hakim, myself and Adam Conrad settled down to take a proper look at it just before. After a crazy amount of debugging various insane problems and a mindnumbing time spent on deep excursions through gdb, we finally tracked down the issue (ldap must not be in alias_databases, only alias_maps) through a hell of a lot of guesswork (not after significant debugging), and thus the problem was solved. The new accessibility list should now have all its aliases present.

This one was especially bizzare to track down: some unknown function somewhere was getting called and overflowing like crazy, so the third letter of /etc/postfix/ was getting corrupted; either to /emc/postfix/ or /eoc/postfix/ But it's all fixed now, and man was that insane. Postfix got upgraded to 2.1.5-1.0.1 somewhere along the way, and also a standard upgrade with new versions of tla, debconf, blah, blah. Phew! -daniels
[15:42] | [daniel] | # | TB


I have mentioned a few times in the past that I am planning to move LDAP and other privileged services over to tycho (aka, to get it off the machine we have around 300 accounts on. To that end, I've been writing 'ill', a mail interface for administration. It's Python-powered, and categorised into project managers, and account managers. All project managers can submit requests for account creation, which are then approved (which is expected to be largely a rubber stamp, as it is today) by the account managers. This offers two really compelling advantages over the current situation: we don't need to grant people root for just creating accounts (there are far too many sudoers currently, for any system; not a slight on anyone at all, just a reflection on the fact that no project needs seventeen administrators).

Combined with moving the LDAP server somewhere else, this should hopefully allow us to scale far beyond where we are -- including into the realm of translators, which has sort of been pending getting the box far more secure than it is today.
[15:42] | [daniel] | # | TB


Matthias Clasen has his fd.o account working now, with commit access to mime. He's a Red Hatter, and while he already had an account, we just had to fix his authorized_keys file, and add him to a new group. And no, we don't have an admin blackhole :P --byte

Then added him to shared-mime-info and xdgmime as well
[15:42] | [byte] | # | TB

Random fd.o admin work

There's now an Accessibility list on fd.o. Some more random account creation, with people being added to new groups and folks getting some cvs accounts. Seems we might've missed someone for months too, sorry about that! Accounts created today:
  • clahey added to dbus group
  • newren account created and added to icccm-extensions, startup-notification group
Otherwise, looks like accounts creation should be sped up, and who has an easy fix for the cvsweb issue? Where the silly message You don't have permission to access /doc/viewcvs/images/logo.png on this server. appears. As an example, look at the broken logo here, where it says Powered by ViewCVS. The path exists, already... --byte
[15:42] | [byte] | # | TB

joukj key

Added another pub key for joukj at his request (access to the previous machine is limited). I wish I had a procedure for deciding when to accept emailed requests for stuff like this.
[15:42] | [anholt] | # | TB

Added Dieter Nuetzel

Added a new Mesa/DRI developer, Dieter Nuetzel. He's been finding breakages in DRI stuff for far too long and is being punished appropriately.
[15:42] | [anholt] | # | TB

DRM Repocopies

Been doing some repocopies in the dri project for bsd-core. I don't think I've broken anything, but I'm not quite done yet. --anholt
[15:42] | [anholt] | # | TB

Bugzilla upgrade

Upgraded bugzilla from 2.18.rc2 to 2.18.rc3.
[15:42] | [anderson] | # | TB

Fixed viewcvs for

All of the links to on the cairo website were broken. Fortunately, /etc/viewcvs/viewcvs.conf is under revision control, so it was easy to see that in the cairo-general section "default_root = cairo" had been changed to "default_root = /". But that change hadn't been committed so I don't have anyone to blame at this point.
[15:42] | [cworth] | # | TB

About Administrators




Web Sites