IRL Status Page

January 2003 Archive

• 1/30/03
  • Today's project is to get albert completely up to snuff.
  • NTP has been installed and configured without crypto on albert. Current versions of SSH and SSL on albert are 3.4p1 and 0.9.6g, I believe. I cannot find a newer package for SSL.
  • Root login from pogo to albert is enabled. Albert still doesn't let normal users log in with host-based auth. Dunno why.
  • NTP is up with crypto on albert, synchronizing to pogo.
• 1/29/03
  • Bunnylou, cowbird, pogo, malarky, bridgeport, baldwin, and snavely are all running exactly the same sshd_config now. Our machines should be configured identically.
  • I am working on a network install of redhat 8. I guess we'll see how it goes.
  • Network installs of redhat 8 and redhat 7.3 have both failed in the same manner. I am going to attempt a debian net install, but have to wait for the 80mb boot cd image to download.
  • Debian didn't work right. I managed to get redhat 8 to do a minimal install.
• 1/28/03
  • Mills upgraded openssl and openssh on snavely and hepzibah last night, apparently without a hitch. Porkypine is on hold.
  • The core on the lock on the door was changed. It works much better now. Old keys work. This contributed to my late start today.
  • Emails have been dispatched to Mills to ask about what to do about albert.
• 1/27/03
  • Created /var/empty with root:root ownership and mod 0400 on malarky, bridgeport, and baldwin so they can support OpenSSH 3.5p1's privilege separation user dealie.
  • Tried to upgrade albert to redhat 8. Was stymed. RPM is a terrible system.
  • Tried to install NTP on albert with redhat 7. Was stymed. OpenSSL on there is ancient and NTP apparently won't compile against its libraries.
  • Thinking about messing with hepzibah. Do I want to risk blowing up another machine?
  • The latest version of OpenSSL won't compile on porkypine out of /m/pogo/dist/openssl. I don't think it's going to happen, eh?
  • Bill says he's going to put in a ticket to get the lock fixed. Finally!@#!#@
  • I have FIXED bunnylou and cowbird! PrivSep was to blame. UseLogin is turned off, and root and users are all passwordless now. The prefs file is cleaned up, but it's not the same as pogo. The goal of having the same sshd_config for all the machines will not happen. I am especially proud of this since it consolidates a lot of the information for the various systems into simpler configuration data, and root login without password never worked on bunnylou, and now it works on both!
  • The following machines have confirmed working ssh-as-root login from pogo: malarky, bridgeport, baldwin, cowbird, bunnylou, snavely.
• 1/24/03
  • I have upgraded OpenSSL and OpenSSH on cowbird and bunnylou. They are at the same versions as pogo and the rest of the sun crew. Sadly, passwordless root login from pogo doesn't work. We shall see what's up with that.
  • Serial consoles are enabled as per the /etc/motd on the lab critters. Albert is temporarily called butterfly.udel.edu (128.4.1.68) if you're curious about why you can't connect.
• 1/23/03
  • Pogo (and all of the suns on 128.4.1.0/24) have been upgraded to OpenSSH 3.5p1 and OpenSSL 0.9.7 (Dec 31 2002). All configuration files remain unchanged. Please inform me of any difficulties.
  • Albert has gone through a hell of a day. I installed windows on another drive, and he still doesn't respond to ARPs from boombah, using a different NIC and a different IP address (butterfly.udel.edu 128.4.1.68). MDavis says the next step is to restart boombah and see if that fixes things but I am highly doubtful.
• 1/22/03
  • Albert is back up, running very few net services. Serial consoles are not activated on albert yet.
• 1/20/03
  • Happy MLK day, everyone.
  • SAMBA issues on the network
    • SAMBA doesn't include a means by which to mount a SMB share as a filesystem entity. Therefore, SMB is not, using SAMBA alone, an acceptable replacement for NFS on our network.
    • There is a software package called Sharity that claims to be able to solve our problem. It will not compile using simple install instructions on Baldwin, but depending on whether it's an attractive option to the management, I may investigate it for more than ten minutes.
    • Sharity comes with a precompiled binary package which should work, but I'm not going to su and install it without the OK from the BossMan(tm).
  • NTP on albert
    • Things are frustrating. I'm getting a pretty specific error when I run ntpd, but I have no idea how to interpret the error message. An email has been dispatched to Mills.
    • I am so desperate for error code documentation that I have subscribed to the NTP NNTP feed just so I can see if anyone's posted this before. No one has.
    • The NTP problem is something with the PEM password, but I never set up a PEM password. Maybe I should?
  • Albert ARP problems
    • I am making a new kernel for albert to address this, maybe.
    • Albert is down until tomorrow so I can secure it with its new install.
  • SSHd on porkypine
    • Things are interesting here. I'm getting a very specific error but I don't know what to do about it in order to make things work.
• 1/17/03
  • Project status:
    • Albert
      • NTP: I am blocking awaiting the functioning tunnel to 128.4.2 so that I can use whimsy as the testbed NTP server. I don't want to mess with any of the stratum 1 servers here as far as testing my install goes.
      • NTP update: At the end of the day I have got the keys going but I am getting error messages from ntpd when I try to start up with crypto enabled. I have been troubleshooting for some hours but haven't managed to figure out the problem. It keeps asking me for a PEM password when it start up, though, then gets pissed when I don't supply it within a few seconds.
      • SSH: MDavis says that the version of sshd that comes with linux is broken as far as host-based authentication goes. It will work via shosts from pogo, baldwin, and dilbert, but won't work for anyone else for root login without password. I am blocking on compiling a new version pending the completion of the ARP problem.
      • 128.4.2.0/24: This problem is vexing. I had to get it so that MDavis could log in before he could help me troubleshoot. I just did that. Killing gated on baldwin still lets baldwin talk to boombah, so obviously that isn''t the problem on albert. For some reason, albert's ARPs are getting chomped to and from boombah only, which is quite bizarre. MDavis thinks it may be a hostmask problem or some other issue.
    • Hepzibah
      • I want to get a serial console going from hepzibah to albert. I will give this consideration today.
    • Porkypine
      • This critter is on hold while I work out albert's problems.
    • NTP
      • I have been reading the NTP documentation pretty carefully but I'm not sure I'm covering it all. I will report on this in person.
    • Lab
      • I am experimenting with WindowMaker as a window manager since I think it may encourage Harish to use the suns if he has a more user-friendly enviornment than fvwm2.
      • Things have gone well. WindowMaker is a great window manager, but Harish seems unimpressed. He has installed VNC on bridgeport so he can use bridgeport from the windows machine.
      • Harish has somehow managed to kill bridgeport. Blame him. I didn't do it.
• 1/16/03
  • Today's big project is figuring out why albert can't talk to 128.4.2.0/24. I installed gateD finally on albert, after copying it from one of the department linux machines. gateD works fine, but albert is having an awful time getting ARP replies from boombah. After a huge amount of troubleshooting, I've narrowed the problem down reasonably well and I am assuming at this point that it's something funky with the router or switch configuration on Mike Davis' end. Therefore, the problem has been handed to him. He'll get back to me when he has time to figure it out.
  • I want to use whimsy on the 128.4.2.0/24 network to test ntp crypto with, since I don't know of any other machines that I have an account on that are set up as a secure ntp server, and I need to copy files off the server in order to get things to work according to the ntp-keygen page. ntp keys are set up on albert, and once we get this arp nonsense figure out, I will be able to test with whimsy. Until then, ntpd is not running on albert.
  • http://www.eecis.udel.edu/~mills/database/brief/autokey/autokey.pdf linked from http://www.eecis.udel.edu/~mills/autokey.html is 404.
• 1/15/03
  • Today was spent talking to mills and trying to get RIP working on albert. The reason that routed doesn't work is that it doesn't understand RIPv2, and now zebra doesn't want to package-install on albert. I will compile and hand-install zebra tomorrow from its latest source files and make things work.
• 1/14/03
  • Due to the depressing events surrounding snavely yesterday, I'm going to try to focus on making NTP work with crypto on albert today.
  • NTP on Albert
    • Ok, so once again I'm fighting with the NTP documentation page. The color scheme on the documents makes them pretty poorly, and Mozilla for Solaris hates to print in ANY regard. Going to a windows machine in the building fixed that but made me feel dirty.
    • I have grown to rather hate the NTP docs. Terms go undefined, abbreviations are used liberally, and I am, in general, having a terrible time finding the information than I need even though I know I'm looking at the right documents.
    • Frustration has set in about albert's NTP. Email has been sent to Mills. I await reply and am doing my daily sensor sweeps instead.
  • General lab stuff
    • I got so tired of not feeling like I was getting anything done that I got us a new tape dispenser and asked Melissa to get our lock on the door looked at. Feh.
    • I am considering installing the NextStep-like window manager "WindowMaker" on the millslab suns. Does anyone care?
    • Porkypine keeps rebooting himself. I'm confused as to why. How do I read the syslogs on these machines?
    • Work has commenced on getting a router onto albert.
  • Security
    • Daily nmapping is taking place. Whee.
    • I have gotten the evil-eye via email from Mills about sequestering snavely. I stand by my decision. I can find NO reference to tcsh opening any ports in its manpage, on google, or whatnot. It's just not good behavior, unless this is some proprietary HP-UX thing, which I doubt it is. I've spent a few hours today just researching that point, and I can't feel that it's justifiable to ignore this.
    • Bunnylou and cowbird have a lot of udp ports open. I can't see anything overly strange in their lsof output, though.
    • Snavely has been reconnected to the network.
  • ISDN router security
    • Because I was sidetracked on NTP and such today I started poking at pup-dog and boombah. Ended up messing up pup-dog. Oops.
• 1/13/03
  • I think I'm going to take a little different tack with today's stuff. I'll make a bullet list per task and put my comments about each task under it.
    • lab security
      • Daily nmapping has been done. No new open ports were found on tcp. I'm going to start closing down UDP ports soon. I do note, however, the yp and NFS use a HUGE number of ports. It's staggering how many of our open ports are due to yp, rpc, and nfs.
      • Argh. Bunnylou still had snmpd running with an open UDP port. It's been killed and won't come back.
      • Cowbird did, too. Killed.
      • Why do cowbird and bunnylou have different executables installed? Having to build my own lsof is lousy.
      • kdebugd running on cowbird is reported to be a serious security hole for remote access. I am disabling it. If anyone complains, let me know, but there is no patch for it.
      • cfgmgr has been disabled on cowbird. It opens a port and http://sabernet.home.attbi.com/papers/Tru64.html recommends closing it.
      • "rwho" has been disabled on cowbird. It's in /sbin/rc3.d and /sbin/rc2.d.
      • That above page recommends removing the rpc entries from /etc/inetd.conf on cowbird. I think I'll try it and see how it goes.
      • rwho and those rpc entries are now disabled on bunnylou as well.
      • Another page at the site listed above from attbi gives HP-UX security hints. It's depressing. Snavely will never be secure.
      • Snavely's tcsh listens on a UDP port. This is NOT kosher.
      • "Aserver", HP-UX's audio server, has been disabled on snavely. It's an /sbin/rc service.
      • There's some very weird stuff going on with services on snavely keeping lots of UDP ports open. Something inside me says snavely is compromised.
      • I cannot reconcile tcsh having UDP ports open. Snavely is going off the network until I can figure out what is wrong with it.

        # ./lsof | grep tcsh | grep UDP; ps -e | grep tcsh
        tcsh      29848 conner    3u  inet 0x00e88180       0t0    UDP *:2122
        tcsh      29862   root    3u  inet 0x00fe8680       0t0    UDP *:782
        tcsh      29894 conner    3u  inet 0x0103f780       0t0    UDP *:2123
         29894 pts/1     0:00 tcsh
         29848 pts/0     0:00 tcsh
         29862 pts/0     0:00 tcsh
        

      • Ok, so bad things are afoot, security-wise. Snavely has been sequestered, and I've locked out all but mills, harish, and conner on NIS on albert. My own password, due to interactions with snavely, may be compromised. We need to do a few things to lock down the network. Namely, we need to wipe snavely and reinstall, we need to change the root password and maintain a few different password domains (In which critical machines get different root passwords) and institute a root password rotation policy, and we need to cut down on all the open ports and NFS traffic going on, I think. If the network was more closed, it would be easier to keep secure and to notice security violations. I will discuss my comments with Mills on Wednesday, but I have the feeling that our security situation is looking pretty grim. One compromised machine takes down our whole network, and snavely may have been compromised for years.
    • sshd2 running on porkypine
      • I am getting errors where the daemon can't open /dev/tty. I'll look into it.
      • I have sent email to M. Davis asking about the error. A script file of the output is also available at sshd.log.
    • ntp.org documentation page
      • I am waiting for feedback from anyone who can read this page, namely Prof. Mills. I hope to be able to contribute the page to the www.ntp.org webmaster soon, and I am seriously considering asking if I should be considered as a new webmaster for ntp.org.
      • Mills has contacted the NTP webmaster and pointed him in my direction.
    • Secure NTP on albert
      • Not much work has been done on this lately. I don't really know what I'm doing and I've been sidetracked by other stuff going on like ssh on porkypine.
    • Hepzibah serial console
      • It is my intention to contact harlan stenn in regard to this so that I don't get clobbered by his wacky kernel antics. Maybe I'll do that today.
      • I have emailed Harlan to ask him about hepzibah's kernel. Included was a note that Harish has been messing with the kernel for IPv6 multicast routing, and that we need to make sure that doesn't get clobbered.
    • SSH running passwordless on all machines
      • I got bunnylou and cowbird working with this. Can anyone find any other machines that it doesn't work on besides porkypine?
• 1/10/03
  • I am pleased to announce the release of my proof of concept replacement for the NTP doc page. You can view my replacement here. Please send any feedback to conner@eecis.
  • Important characteristics of my replacement page:
    • It is fully XHTML1.0 compliant, and validates at two separate XHTML validators and at a proper XML validator.
    • It is accessable to those with browsers that don't render standard layouts. The page is well-linked internally, so it is easy to skip down to the bottom of the page when necessary, if someone with a blindness-assisting reader were to go to the page, for instance. I validated the page through an accessability-validator and it returned fine except for the most stringent accessability level, which only complains about tab-order and command-key shortcuts for links.
    • It organizes information by task and audience. I haven't read everything on the page, so I don't know if I got everything in the right place, but I do believe that this method of organization presents the information in a more logical, focused manner. Of note, pay attention to the fact that setup/install information is separated from theory-of-operation information, and copyright and whatnot are separated. Little-accessed-but-important stuff like Y2K information is at the bottom, instead of being in the way at the top. Also, because the "manpages" aren't the primary focus of the page, they are grouped together at the bottom instead of eating up a ton of attention at the top of the list.
    • There is still a lot of redundancy in the links, and links that point to the same page don't always have the same title. This is bad, but I haven't taken the time to correct it like I should.
    • This page looks fine without CSS at all. It will, quite literally, render properly in NCSA mosiac (Unless mosiac chokes on the minor xhtml stuff, which it shouldn't.)
    • The page also renders properly in lynx. Due to the complete separation of style and content, it actually is just as usable in lynx as it is in mozilla. I also think it happens to look quite good in lynx, but that's just me.
    • Because of the simplicity of the code needed to generate the page, it is reasonable to edit it in just about any editor, instead of needing something like GoLive. I created the page in VIM, and it took a lot of time (several hours) for me, but I had to organize all the information and check everything. Editing individual parts of the page is very simple now. If you look at the source, note that there are no tables, no images, no frames, no nothing. It's just text at the top with LOGICAL HEADERS that actually define parts of the page, and that the information is presented as a nice bulleted HTML list instead of the old way of doing it on ntp.org -- A table with an image for a bullet in each row.
    • As an aside, I am certain that the old ntp.org version of this page is completely inaccessable to a blind reader.
  • At least snavely and porkypine have root exploits that are trivial to make use of for any local user. I'm not sure what I should do about it.
  • It LOOKS, to the uninitiated, like I haven't done much yet today, but I've been spending some time at http://www.insecure.org/ reading about security exploits for older unix systems. HP-UX seems to be particularly full of holes, but we're not susceptible to the remote root exploit that they mention, because of the daemon clamping down.
  • Also, the web site mentioned above notes that LPD is a security risk on older UNIX systems. This makes me feel good that I disabled it on all of our systems yesterday which don't have a printer attached.
  • The problem with cockatoo yesterday was a router, not anything I did.
  • porkypine is now known to be running /usr/local/sbin/sshd as commercial sshd version 1, and /usr/local/sbin/sshd{1,2} as openssh. I didn't know that, and it explains some of the wackiness.
  • Login works with sshd2 on porkypine as far as getting a tty, then it dies. I'm not sure how to rectify that, but I'll get to work on it on Monday, since I think I'm in the home stretch here with porkypine.
• 1/9/03
  • Today's first project is to get ssh working on all of the requisite systems. I have already enabled ssh host-based authentication on bunnylou and cowbird. Snavely works ok already, and porkypine is so angry at me that it won't do it with the same config options as all the other hosts. I label porkypine, "Not a team player."
  • I have decided that OpenSSH 3.5p1 will not work on porkypine without editing the source code. I have located the place that it dies in xmmap.c, but that does no good because I have not been given the go-ahead to fix the source by hand.
  • I am closing down more ports. bunnylou's LPD has been shut down. To undo this, just rename the files in /sbin/rc3.d and /sbin/rc2.d that are relevant.
  • cowbird's LPD has been shut down. See /sbin/rc3.d and /sbin/rc2.d
  • cockatoo stopped working today. The printer is fine, but something software-wise is munged. I am submitting an eecis gripe.
  • "dtspc" -- The common desktop enviornment subprocess control daemon -- Has a known remote exploit and is active on several of our machines (snavely, bunnylou, cowbird). Given that those machines no longer run X or anything, I am going to disable that service on all of them. mdavis has been notified.
  • lpd has been disabled on porkypine by commenting out a block in /etc/rc
  • I am more confident now that we have few security holes in the TCP services offered by our machines. Due to the fact that UDP is less well documented in terms of well-known services, I'm going to hold off on blasting the UDP configs out and trying to lock them down, too. I am also more confident that I know what the ports that we have open are. A lot of the things that nmap identifies as weird services is actually ypbind. I hate ypbind.
  • My last project for the day, which should eat up my last few hours here, is to document my experiences in NTP-land more carefully and send off a review to the web team for www.ntp.org.
  • As an aside, I went over to Chuck and physically located mizbeaver, so I know where she is now.
  • I've made a proof-of-concept web page to replace the ntp.org documentation page. I'll flash it around tomorrow.
• 1/8/03
  • Yesterday wasn't very eventful in terms of changing the state of the lab. I spent a lot of time reading the documentation at www.ntp.org, discussing what Harish has been doing in the lab here, nmapping a few of the machines and discussing with Ben Miller what the open ports are, (It's bad when the linux box has a port open that lsof can't find) and the like. The SNMP daemons have been turned off on snavely, though they didn't appear to actually have a port open. I feel like my new policy for daemons running in the lab will be to keep things turned off unless they're shown to be necessary, with the obvious caveat of not turning off daemons that will prevent login and the like.
  • Today's task is to give a status report to Mills when he comes in, and to build the current development version of NTP on albert. I download NTP 4.1.72 from www.ntp.org, which was listed as the current development tarball. It's assuredly not as current as the bitkeeper repository, but I have not been instructed to install BK on albert, and I don't feel like waiting another six hours to get any sort of feedback from Mills since he does all his email in batches at the most bizarre times of the day.
  • Also on the agenda for the day is to find out what has the 'netinfo' port open on bunnylou. It's a mystery to me, and since netinfo is a NextStep service, I am wondering what else uses that port. I'll probably do that if I get tired of trying to build NTP.
  • The report of my adventures in linux-ntp land are at: albert_ntp.html.
• 1/6/03
  • I've done some work without noting it in here promptly recently. Notably, albert was down over Winter break. He's back up now.
  • A Nmap scan of 128.4.1.* has been done. I'm investigating any funky ports that I found open.
  • Work is continuing today on getting openssh to work on porkypine. Hilarity ensues.
  • Work has stopped for the day on getting openssh running on porkypine. I hate porkypine now.
  • A new item on the agenda: getting hepzibah set up with a serial console.

Maintained by John Conner (conner@eecis.udel.edu). For IRL internal use. Please do not distribute this URL.