admin log
fd.o admin tasks and planned outages



Sat, 22 Jan 2005

fd.o updates

So, in the past few weeks, a lot has happened with In the manic pace of reconstruction that followed (take down machines for basic forensics, sit down and scribble out plans, attempt to acquire hardware, back up, reinstall, restore, build up new infrastructure), it can be said that we haven't been fantastically communicative. Most of the things that happen get mentioned on IRC shortly before or after they get done, as that's easier. So, sit back and enjoy the ride.

What happened?
On Nov 15th at approximately 00:07 PST, an intruder got access to fd.o via a simple TWiki shell injection attack. They were running PsyBNC and some other IRC proxy as www-data, in /var/tmp (/var/tmp/.cache, /var/tmp/.tmp, and a couple of others). We do not believe (indeed: really quite sure), in retrospect, that they gained root. However, at the time, we did not know this, and the Debian compromise showed us that root compromises aren't necessarily known at the time. So, following due paranoiaprocess, we decided to do it the hard way -- take it down and start from scratch.

And the recovery?
Myself and keithp did the initial installation, and Adam Conrad helped get a whole bunch of services up and configure them beyond that point. The existing services were quite typical of the problems we've had with fd.o -- it was a whole sort of general mish-mash that didn't scale well. Around twenty hacked-together 'solutions', nothing of which really worked well on its own, let alone coherently. This gave us an opportunity to sit down, and so we did -- projects have their own space under /srv/, and most services are segregated under there also (CVS, Bugzilla, et al). Because the structure was so radically different and at the time we didn't know if /home had been compromised, we decided to restore the old home directories as /home/compromised, and leave the rest as is. This means that if you had an account previously, your home directory -- including public_html -- has not been restored. This also holds true for projects.

Authentication is being handled with Debian's userdir-ldap. We have segregated the LDAP server on to kara, another machine located on an internal network. I originally set up OpenLDAP with SSL/TLS across the network, but that fell over within an hour; the load there being me setting some stuff up, and keithp poking around. Clearly, it was unsuitable for general use. We're now using the excellent userdir-ldap system, which is also deployed within Debian, which lets users manage their own records to a degree. The admin interface is far, far nicer than we ever had. The previous system spat out '/usr/bin/dialog is unhappy with your terminal' all the time for no apparent reason, would return a blank dialog for success on any operation, and a dialog saying 'Successfully [did stuff] to [object]' upon failure. It was also pretty incomplete.

While we were at it, we dealt with migration to Apache 2, new ViewCVS, a newer version of Subversion (and hopefully the fsfs backend), newer Bugzilla, a chrooted pserver for CVS (/srv/, a chrooted BIND 9, et al.

Where are we now?
Right now, we're standing on a pretty strong grounding, I believe. The standards are back up, thanks to Waldo Bastian and Chris Lee, and we now have a far stronger system for dealing with users/projects; translators are now properly on the cards. I am in the middle of writing a small daemon to deal with download.fd.o. The idea is that people upload tarballs and whatever along with a GnuPG-signed .changes file, which gets processed by the queue reaper, and put into the download.fd.o structure. With proper segregation of projects and write access only through a given daemon which logs an audit trail, we believe this will be a good bit more secure. Not least, it also gives us something we can properly present to mirrors, so we can get all the important bits mirrored out to the world.

Where are we going?
Brrrrr. Many member projects -- GStreamer, modular X, others -- are going really cool places. As an organisation, we still have several works in progress. download.fd.o is a large part of that, and we also have other infrastructural work (mainly separation of services) that are pending getting some more machines. The most important work has been done; mainly reinstallation, getting a proper, scalable, setup together to deal with many projects, separating out the LDAP server, download.fd.o is almost done, etc. The main thing that remains to be done is for projects to deal with migrating to the new structures (if you need to be active, email, and for pretty much everyone in the project to get GnuPG keys. Yes, you.

That aside, there's not a terrible lot to be done; most of my personal wishlist for infrastructure and services has been dealt with, which makes me happy. As does the prospect of a (relatively) long and relaxing Christmas/New Year holiday period. Cheers.

Daniel, on behalf of the sitewranglers
[02:03] | [daniel] | # | TB

About Administrators




Web Sites