After the excitement and the fun that is the Red Hat Summit, I had some time this week to work with Jason on updating the design. Before I dive into the design process and the mockups, I think it’d be best to do a review of how cgroups work (or at least how I understand them to) so that the rest makes more sense. (And maybe I’ve got some totally incorrect assumptions about cgroups that have resulted in a flawed design, so hopefully my calling out the current understanding might make it easier for you to correct me ).

A designer’s understanding of cgroups via diagram

So cgroups, which are sometimes referred to as containers (I think because a similar Solaris feature, zones, is sometimes called containers) can be used to slice an entire operating system into buckets, similarly to how virtual machines slice up their host system into buckets, but without having to go so far as replicating an entire set of hardware.

So this diagram kind of shows how four of the system resources that cgroups can control – CPU, memory, network, and storage I/O – could be cut into slices that are then combined into two groups – the yellow and the purple one – which make up virtual OSes. Say I gave cgroup #1 (yellow) to Sally, and cgroup #2 (purple) to Joe. Whenever Sally starts a process, you could set it to only run on the CPUs that are members of cgroup #1 (via cpuset), at whatever priority level is set for those CPUs (via cpu). It’ll only be able to use as much memory as was allocated to cgroup #1, only be able to use as much network and I/O bandwidth as cgroup #1 is able to use. When Joe starts a process, because he’s part of cgroup #2, he won’t be using the same CPUs as Sally. He may have more or less memory, I/O, and network bandwidth allocated to him.

It’s kind of / sort of like Joe and Sally are using different computers, on the same operating system. Cool, right?

Cgroups don’t have to go that deep, though. You don’t need to slice across an entire system.

You can have a cgroup that *just* deals with controlling access to the CPU. Or *just* controls memory. Or maybe only deals with two of the four (CPU, Memory, Network, I/O) resources, or just three of the four.

You can have a cgroup that *just* deals with one resource (say CPU), and that group only deals with specific processes. Or users. Or a combination thereof. (More later on that.)

Depending on the resource you’re looking to control and the cgroups module you’re using, you can configure access to that resource in different ways. I believe one of the more common ways of controlling CPU usage via cgroups is to assign ‘shares’ for various groups’ usage of the CPU (using the ‘cpu’ module.) I’m not sure what scale/units these shares are on, but they are relative to each other, so if I give group #1 a weight a 1024, and group #3 a weight of 2048, then group #3 will get scheduled for CPU time twice as much as group #1 will.

I don’t think this share system is particularly intuitive, which is still an open problem in the current draft of the UI design.

Other resources and modules let you control access in different ways. For example, the memory module lets you configure an upper bound of memory usage, I believe via providing the maximum number of bytes of memory that could be used by members of the group.

You can create and apply cgroups to processes and users on-the-fly or on a longer-term / persistent basis. Say some process is running amok and is starving other processes on your system… you can change the process’ cgroup membership on-the-fly to provide it a more limited set of system resources so that other processes on the system can run. However, this change would be temporal and may be based, for example, on a specific pid number that won’t apply if the process is restarted or the system is rebooted. If you’d like more persistent cgroup membership, you can create a set of rules (cgrules.conf). A neat simple thing you could do with cgroup rules, for example, is something mentioned by Linda Wang in her Red Hat Summit talk on cgroups; you could arrange your cgroups rules such that the sshd always gets a dedicated chunk of CPU time so that if a process runs amok on a server system, you still might be able to ssh in remotely to diagnose the problem.

It’s via these rules that you can set up persistent groups on the system. It would be kind of a pain to have to set them up every time a system is rebooted, especially the more processes and users you’re managing, and the more groups you need to create in order to manage them. You can use multiple cgroups modules (for example, cpu, cpuset, mem, net) within a single cgroup, and then write rules to place processes run by particular users and/or user groups into the cgroup, or write rules to place processes matching particular attributes into the group.

Above is a diagram demonstrating a cgroup that restricts only CPU usage for processes matching firefox-*, npviewer.bin run by users that are in the ‘guest’ group or whose usernames are ‘student1′ or ‘student2′. Kind of putting it all together, showing a single group and its resource allocation, and its associations with particular users and processes.

Who would use a UI for this, and why?

Tthe UI design mostly focuses on setting up persistent rules, and doesn’t really allow for on-the-fly cgroups rearrangement of currently-running processes and currently-logged in users. The thinking behind this is that there might be a couple of main reasons you’d be using a UI for cgroups:

• Proactive: Initial system resource allocation planning
• Reactive: In response to a complaint – ‘My processes are getting capped,’ or ‘Such-and-such process isn’t running right.’

Thinking about the way my previous experience with system administrators has typically gone, there’s sadly not usually a lot of time for proactive planning and organization – a majority of time tends to get spent on reacting to client concerns. If someone is calling you up on the phone to tell you that a process they are relying on has gone awry, it seems the most effective way to get them to stop calling you would be to change the rules to make a more persistent change to help them, rather than to just apply a change on the fly and wait for their next phone call. So I think you’d probably prefer to change rules in reaction to a client complaint, not just do an on-the-fly change (although you could.) When you’re being proactive and initially setting up a system, I think you’d not want to do that on-the-fly at all, because you’d have to keep reapplying. It’d be better to craft a set of rules to persist on the system.

So it seems like these two main use case types – proactive resource allocation, and reactive allocation adjustments based on client feedback – are both best served by focusing on rules, so that’s why the UI design only focuses on those.

Maybe there is a case for doing the on-the-fly stuff though. It does make for some pretty awesome demos, though, like the one Bob Kozdemba gave at the Red Hat Summit last Friday, moving multiple copies of a graphics rendering tool between cgroups and vms and changing their configuration on the fly so you could visually see the affect of resource caps on the processes.) You could also use the on-the-fly configuration changes to test out a theory about how you should set the rules.

So, specifically, what kinds of problems could we imagine a user looking to solve with this interface? Our thoughts were that most likely a system administrator would find it most useful:

Under the ‘Proactive’ Category:

• I have a system running a business-critical app that sadly has a memory leak, and I want to make sure that app doesn’t screw up the other apps running on the system. I’d like to put a cap on its memory usage so when it goes down it doesn’t take the rest of the system down with it.
• I have a thin-client lab meant for the use of students in the science department, but students from other departments are allowed access via guest accounts. I’d like to make sure the guest accounts don’t block any of the science department students’ work on the server.
• I’m an administrator at an ISP and we provide virtual machines to end users in different pricing tiers. I’d like to enforce the limits set by our pricing tiers so that customers’ VMs aren’t getting more power than they are paying for and aren’t starving customers that are paying for more power.

Under the ‘Reactive’ Category:

• A student from the science department has called helpdesk to complain that his simulations don’t have very much oomph and are taking far too long to run. I need to figure out what system resource policy might apply to this user, and make modifications to his policy as needed to help him out if possible. (For example, perhaps he was previously outside of the science department and just transferred in, so he’s still running in the guest group.)
• Uh-oh, something’s gone wrong. A business critical process keeps getting OOM’ed! I need to figure out what resource allocation policy applies to it to see if there a rules that are causing the issue / that could be adjusted to get the app running consistently again.
• Our bi-weekly payroll processing application is still going full-steam ahead but it’s Monday morning, the business day is starting, and that server needs to be used for other things. Whoops, that’s never happened before, it usually finishes on Sunday. I need to see what policy applies to it, and modify our rules so that the payroll process tones things down a bit if it hasn’t finished over the weekend.

The mockups

So after a lot of sketching, crossing out, sketching, crossing out, and thinking about the above types of use cases, we thought to break up the main window in this way:

• A users tab: For when you have a specific user/group in mind you’d like to limit, or if you’ve got a specific user on the phone who you’re trying to troubleshoot with;
• A processes tab: For when you have a specific process in mind you’d like to limit, or if you’ve got someone on the phone upset about a particular process you need to help debug;
• A containers tab: For when you need to initially create your cgroups, or if you’d like to tweak the configuration of a particular group, or if you’d like to see how the processes within the group are performing.

So here’s what they look like:

The users tab

The processes tab

The containers tab

Some issues / further work

Here’s a bit of a braindump of where these need more work:

1. So I mentioned in one of the example use cases a payroll application, that could run full steam during the weekend but had to be dialed back during the work week. Well – this UI design doesn’t account for scheduling in the rules, yet.
2. There’s no mockups for the various dialogs needed for adding and modifying rules.
3. It would be cool to do a mockup showing how this might integrate into the GNOME system monitor – maybe an additional filter on it?
4. It would also be cool to have some kind of integration with the user accounts dialog. Maybe it’d show the policy that applied to a user and let you modify it from that dialog?
5. Right now in the containers tab, we had the idea to list the containers by the resource they are managing – so groups involving cpu, cpuset, cpuacct for example would all be listed under a ‘CPU’ category. However, Jason had an idea about showing logical cgroups that span resource types – e.g., you might have a cgroup tuned for DB usage with different settings for CPU in combination with memory & I/O.
6. The users and processes tab doesn’t really let you monitor resource usage per user / per process. Is that needed? Not sure.
7. Units – shares vs upper limits vs… the actual numbers you’re configuring for each rule need to be fleshed out. If cpu / cpuset / cpuacct for example can be used in conjunction, how would that be shown in the UI?
8. What about a mechanism to compare effective resource allocation vs actual resource usage? So you could see, ‘hey, this process is starved, maybe we should consider upping it,’ or ‘hey, this user isn’t really coming anywhere near their limits, should we dial them down?’ An earlier draft of the mockups showed graphs of each for comparison.
9. (Edit: idea from Bill Nottingham) Support multiple systems in one UI over the network

The wiki page for this design (including Inkscape SVG sources) is here:
http://fedoraproject.org/wiki/Design/CGroupsUI

Feedback

Is this nuts? Does it make sense? Is this the wrong approach? Do you use cgroups? Would this be a useful tool, or does it suck?

I hope you’ll let us know.

The problem

Background

See the screenshot above? Scary little bugger, right? Yep, there’s a few issues with this screen. We’ve received bugs on it requesting that the text be changed to be more accurate. There’s bigger problems than that, though.

Let’s go a bit into the background of this dialog. It occurs early on in the screen flow:

[ lang selection ] => [ keyboard selection ] => [ basic or special storage ] => [ REINIT DIALOG ] => [ hostname selection ]

Here’s the full order of things in the anaconda code.

The screen that asks if you have basic or specialized storage must come towards the beginning of the screen set, because it is at this point anaconda has to scan to see if any pre-existing installations are on the system in order to know if we can upgrade a pre-existing system or not. However, we don’t want to scan specialized storage devices which may be mounted to the machine if we know for sure that the user isn’t interested in using them as part of the install – it’ll take a while, it may spins up a lot of devices that were not active using more electricity, putting wear on equipment, etc. hitting a lot of networks.

• If you pick basic, anaconda just scans local devices.
• If you pick specialized, anaconda scans local devices & network storage devices. This route also gives you the option of adding iscsi and other kinds of disks that require user input to be able to discover.

There may be a false assumption here – someone may want to use specialized storage devices as install targets, in which case scanning them for upgradable bits makes sense. However, someone may also simply want to use specialized storage devices as mounted data storage, in which case we shouldn’t bother scanning them for upgradable bits.

Anyhow, once we know which devices to scan (local if basic, local+network if specialized), they are scanned and if any of them appear to have missing partition tables, this dialog is popped up.

Anaconda may not know how to read the partitions on the drive, another operating system may be able to. There *could* be data on that drive, but we just don’t know because we don’t know how to read it. The drive may very well be blank (if it’s brand-new / baremetal) or it could be a virtual drive and have no data either. So our dilemma is:

• we can’t mount the drive as a data drive if we can’t read the partitions, so it’s useless.
• we can’t use the drive as an install target unless we completely format it.
• there may be data on the drive, we just don’t know. if we reinitialize a drive that was only meant to be a data drive… we’re erasing data for no good reason.
• there may not be any data at all in which case popping up the dialog instead of just reinitializing the disk and being quiet about it without causing so much bother.

Problems

Here are the problems you can see just looking at a screen surface level here:

• The titlebar says warning. The text says error. The icon indicates a question. These are all in conflict with one another. This dialog is more of a warning dialog than anything else.
• The metadata about the drive in question is strewn all over the dialog and hard to read.
• The dialog uses the word ‘reinitialized’ and ‘re-initialized’ multiple times without explaining what it means.
• THE DIALOG USES ALL CAPS
• The dialog says that re-initializing will cause all data to be lost, but actually, it will only cause data on the drives being reinitialized to be lost, and only if there was any data in the first place (which there might not have been.)
• Four buttons across the bottom of the screen is a bit of an overload, yet if we had say 100 such devices attached to the system (slices of a network drive maybe) then we would see this dialog 100 times without those extra ‘all’ buttons.
• What does ignore do? Ignore what? Ignore this warning and go ahead an re-initialize? Wait a minute… (It actually ignores the drive in question, removing it from the set of drives considered in the install process)
• Overall, the dialog is scary, and this fright is brought up in situations that should not be frightful – e.g., you’re simply installing on a virtual machine with a virtual disk. No need for the scare!

Design Challenges

Here’s some of the things that make resolving these problems a bit dicey:

• Two classes of users who simply don’t care about this dialog and for whom it’s a waste of times / click effort are users installing virtual machines and users who have brand new hard drives. The default choice of reinitializing all soothes these folks.
• This dialog might come up towards the beginning of the installer screenflow, or closer to the partitioning screen. There are two storage device scans that happen – because some drives (like iscsi drives where you need to manually input address information) can’t be scanned for until you hit later screens.
• We need to prevent users from clicking next-next-next and destroying a drive (That they might not necessarily own if it’s networked!) by accident. This is one reason the dialog is currently a dialog rather than a regular screen. It’s difficult to remove the next & back buttons in the middle of the screen flow.
• There’s a big difference between what this reinitialization thing means for data drives vs install targets. Data drives are added in anaconda so automounts can be set up. We can’t mount a drive that we can’t read. While it sucks, it’s far less sever than the other case, in which an install target can’t be used unless we wipe it.
• We can’t move the basic vs. specialized selection screen to be with the rest of the storage device screens, because it’s the scan that takes place here that determines if an upgrade is possible (it examines the drives, looking for pre-existing & upgradeable installations.) The problem here is that it’s really not smart to just go scanning any network drives available for pre-existing installs unless we know for sure the user intends us to do that.

Questions

So looking all of this over, here’s some questions that might come to mind:

• Is the default choice of ‘reinitialize all’ the right choice? (Not sure. We make a choice here between making life easier for virt installs or making life easier for folks who don’t want to lose data. We don’t know how large these populations are relative to each other.)
• This dialog is separated from other storage-related screens. Can we move it closer to the partitioning screen? (Maybe.)
• Reinitializing currently affects all drives scanned, without any filtering by the user (besides the local or local+network drives selection). Can additional filtering be added?
• Reinitializing currently affects drives meant to be part of the install target set AND drives merely meant to be automounted to the system. Can we limit it to only install-target drives?
• If a user really cared about the data on a drive, would they really select it as an install target? (Probably not.) In that case, do we need to warn them that data might be lost if they use a drive with an unreadable partition table as an install target? We already warn them that any drive used as an install target will be wiped!

Mockups & Ideas

The radio-button approach

In this approach, we think about the different types of users who might encounter the dialog and offer those users who have any data to lose an easy way to avoid the entire reinitialization screen and its related stresses.

We only display the radio button dialog If one or more missing partition table drives are detected. If the user says the device is virtual or blank, we reinitialize it without asking. If the user says the drive had been used before or they weren’t sure, we give them the scary reinitialization screen.

Some issues with this idea is that it’s not very scalable – you have 100 drives you can’t read, you get this radio button dialog for 100 drives. Ouch. We could potentially provide a list view of all unreadable drives, but right now anaconda pops them up as it goes. It may be possible to poll all the drives first, cache the data, and then present a list, though. It’s simply not possible in anaconda right now. But with a list of multiple drives, the multiple choice selection won’t work any more – each drive might be different. Um, so maybe not the best idea – or at least, this approach needs more work.

If you remove all the drives from the install to protect them you get the last dialog – we can’t install if there’s nowhere to install to. :)

The text-massaging Approach

This is a less-weighty approach to apply in the meantime, without a lot of underlying code change. Except, oops, it does require too many underlying code changes. Right now there’s no caching of drive data, so the dialog pops up as anaconda scans drives and detects missing partition tables. So anaconda doesn’t know, until it knows, that a drive is unreadable. So the multiple list view won’t work here.

It’s a better approach than the next one though, which is more compromising to the current state of anaconda’s functionality. One thing to note here – if you have one drive that can’t be read, you see the first screen. If you have more than one drive that can’t be read, you see the second screen. So the mockups in this screen are either/or, they are not a sequence.

The text-massaging approach – simpler

This approach caused the least code churn so it’s the one we’ll go with in the meantime we figure out the larger problems here.

Note the button text has been changed – it says ‘Yes, discard any data’ and ‘No, protect any data.’ We say any data because there may not be any data at all. Saying ‘discard data’ makes it sound like there actually is data to be discarded. We say ‘discard’ rather than ‘destroy’ because it’s a little less frightening of a word I think.

Rather than having four buttons along the bottom, we have two that apply to the single device in question. For our virt-installing friends, we have a checkbox that will enable the button to apply to any such devices.

So while we have a new layout and new language, the functionality is unchanged.

Where to go from here

One idea could be to show the list of drives and ask the user if there is any data they care about on those drives, because we can’t read them and will need to erase them to use them in install. Then give users the options to remove them from the install process / protect them one-by-one with the option to check off the drives they care about the data on. (or vice-versa, check off which drives they want to use for the install.)

We originally had mocked up the latter during the Anaconda storage UI redesign – showing all the drives we found and allowing users to check off the ones they wanted. This was removed at the last minute because of complaints that we ask for too much information about storage devices in the UI. I think part of the problem is because we only worked on the storage UI, not the entire anaconda ui, so there was some redundancy or at least the appearance of such. This time around hopefully we’ll get this right since we’re looking at the entire UI.

Matt Jadud made a blog post yesterday offering up free interaction design for free & open source projects.

Wait, wait. Let me repeat that and make it a little bigger, because in the brevity of that sentence you might have missed it’s sheer awesomeness:

Matt Jadud is offering up free interaction design for free & open source projects.

This is a great opportunity.

Matt is a computer science professor at Allegheny College, and he’s teaching an interaction design course this coming semester (which starts next Thursday!) Matt is an alumni of the Teaching Open Source POSSE program (POSSE standards for “Professors’ Open Source Summer Experience”) and a writer for opensource.com, and he is passionate about getting students involved in free & open source software and improving the connections between FLOSS and academia. So when you read his blog post, where he’s offering up the students of his Fall 2010 interaction design class to help FLOSS projects, you can be assured he means business.

So here’s what Matt is asking of you:

• You need to be a developer or manager directly involved with the project you’re submitting for interaction design help. While I am sure we all have our pet bugs in Firefox or Nautilus, if you’re not directly involved in these projects, you’ll not be able to provide Matt’s students the upstream contacts and guidance they need for their work to be effective.
• You need to be open to UI changes, and not of the mindset your UI is already perfect. Matt is looking for folks who want the help, and who are willing to consider the data and the suggestions for improvement his students will be providing.
• You have to commit to respond to students in a timely manner. These students will be working within a 14-week schedule. If you take 2 weeks to get back to a student, you’ll really impact their ability to get their work done.

This is not so much to ask given what Matt is offering here:

• Your project will be part of a list Matt will provide to his students to choose from. If a student ends up working on your project, they will have picked it from other projects and likely be genuinely interested in working on it. (And motivated too, to get a good grade )
• The students will work within your existing community infrastructure. Already have a mailing list, IRC channel, etc.? Matt’s students will engage with you in these places the same as any other contributor.
• You will receive a usability evaluation and/or mockups and suggestions for improving your interface from an interaction designer in-training! What you choose to do with that data is up to you, but please consider it seriously!

Now, I know there are tons of you out there who are ready for interaction design help and are definitely willing to work with designers, because I hear from you all the time! There’s not enough interaction designers in the FLOSS community, and I believe programs like Matt’s that engage up-and-coming designers in the FLOSS community early on will help build up our interaction designer population. They provide a wonderful mutual benefit – the design students get to work on real-life projects, not just throw-away designs that are abandoned forever at the end of the semester – and the developers involved get the design help they desire but have such a hard time finding because of the dearth of designers. Help provide these students a great experience in interacting with our community, and maybe they’ll stick around!

Note that you won’t become a surrogate professor here – Matt will be teaching and leading these students throughout the process – if you have any concerns or questions throughout the process, if something isn’t working and you’d like it addressed, just get in touch with Matt.

So what are you waiting for? Don’t be shy! Pop on over to Matt’s blog and sign up!