When is a bug report useful?

Every now and then we get the same complaints about bug reports again: Either it is a complaint by the user who feels her/his report doesn’t get enough attention from the developers or it is the developers who complain that there are too many bad and useless reports making them spending too much precious coding time on triaging bug reports.
Disclaimer: I am not a developer but do bug triaging, documentation writing and application testing.

I think personally that bug reports are an important feedback from the users, provided those reports are indeed useful to the developers. But what is actually a good bug report? I will restrict my explanations to KDE and its Bugzilla bug tracker, but this applies of course to all bug reports.

1. It should be about the most recent version available

I know that not everybody is updating their computer regularly, some people only do security updates which is probably well enough for their use case, but they should also be aware that Free Software is fast-paced and most developers are already working on newer versions and may even be some versions ahead. In general, bug fixes are not backported to previous version by the developers, simply due to lack of time and manpower.

If you use an older version you can still report it to your distribution if they provide support for it, though. Some distributions even do backports, but also rarely to older versions as most do not have enough resources either.
Don’t forget to specify the exact versions of the product and of KDE (available in the Help menu of all GUI applications).

2. It has to be reproducible

A one-time bug is not reproducible, so it is impossible to fix as the developers need to be able to reproduce it to test and search the location in the code where the problem lies. It is also very important to give a short step-by-step guide how to reproduce the bug.

3. The subject and comment must be in English, easily understandable, and searchable

If you can’t report a bug in English, please don’t. The KDE community is international and the working language is English as it is the one most people in the community understand.
We often get reports with subjects like “Doesn’t work” or ” Your application sucks, fix it!”. This is of course not very clear and highly subjective. For a bug report to be useful it must be well described so it can be searched in the database and so it is actually understandable by the developers.
The Subject should be a short sentence summing up the important points of the bug, for example “Product X crashes when clicking on Icon Y”.
You can and should give more details in the following comment, but keep it short and with a clear structure.

4. The report should be unique

Some bugs will hit many people so it is not unlikely that many people will report it. This leads often to duplicate bug reports which need to be eliminated and collected into one unique report. Knowing that more than half of the reports we get are duplicates this will of course cause quite some triaging work.
When reporting a bug a well written subject line will allow to find duplicates in Bugzilla quite easily. It is even easier if it is a crash as usually Dr. Konqi will suggest duplicates by querying the database. In doubt, don’t submit the report but ask in IRC, search the forum or the mailing list of the project if somebody has already seen it.
Bugzilla provides a very good advanced search query one can get used to quite easily.

5. If it is a crash, it should come with a valid backtrace

But what is a valid backtrace? Most distributions will strip the applications files of their debugging symbols to gain space, especially if they ship a CD and/or DVD of their versions. Those debugging symbols can be obtained separately and have file name endings like -dbg (Debian and Debian-based distros), -debuginfo (OpenSUSE) or -debug (Mandriva) for example. You will find more information and more distributions listed in this KDE Techbase summary.
Dr. Konqi will rate the backtrace quality with stars as you can see here:

A duplicate of Bug 282875

As a general rule: do not submit crash reports that do not have 3 stars as something is missing. If you don’t have debugging symbols installed, Dr. Konqi will warn you and provides the opportunity to install them directly from within Dr. Konqi.

Of course, a backtrace should ideally always be posted in the comment and not as an attachment, so it will be searchable in the database. Hint: the important thread is the one that has [KCrashHeader] after the thread header, so you can most of the time strip the other threads.

6. Be available to give feedback if needed

By submitting a bug report you get in touch with the KDE bugsquad and developers. Make sure you will be around to give feedback and you should also be available for testing if needed. If you are not willing to contribute in that way, please do not submit reports as it will be causing more work for other people to test.

7. Be patient and polite

It is not always possible for the bugsquad and/or the developers to get back to you immediately. We get an average of 50 to 100 reports a day, of which most will be either duplicates or invalid, so there is quite some stuff to get through in a day.

Don’t forget that this is Free Software and almost all KDE people work on their projects as volunteers in their free time. So shouting at people working as volunteers is totally counterproductive as they will probably prefer to do something else instead.
Make sure you stay on topic and be polite and helpful in the same manner as you would expect others to behave towards you, this is more likely to make people work on solving the problem.
Another point to consider is that your use-case is not necessarily the unique one, there are often many ways to do things in an application and you know that KDE is highly configurable for the users. Don’t expect to be the one who does use an application in “the standard way” and be ready to accept a workaround.

If you want to help, you can of course also give a hand in triaging to clean up the database. We welcome all volunteers who want to contribute to make KDE even better! Feel free to come to the #kde-bugs channel on IRC and have a look at the links given below.

UPDATE: Just to put this in perspective of the work ahead I asked Ben Cooksley, one of our sysadmins, to give me some numbers:

With the sheer number of reports in KDE’s bugzilla and not enough people to triage, a valid report can well get drowned: as of today, October 20th 2011 the database has

  • 278,805 total bugs
  • 64,469 attachments
  • 56,301 bugs were resolved as DUPLICATE
  • 20,250 were resolved as INVALID.
  • 40,571 open with the status ASSIGNED/ NEW/ REOPENED/ UNCONFIRMED
  • 3,830 are in NEEDSINFO status
  • 26,174 are UNCONFIRMED and need triaging

Thanks to Ben Cooksley, Martin Grässlin and Will Stephenson for suggestions and spotting typos :)

Useful links:

How to Report Bugs Effectively by Simon Tatham
Bug Writing Guidelines in Bugzilla
Quick Introduction to Bugzilla in KDE’s Techbase wiki
Guide to Bug Triaging by Dario Andrès in KDE Techbase

Be Sociable, Share!

flattr this!

This entry was posted in Amarok, Bugs, Free Software, KDE and tagged , , , . Bookmark the permalink.

28 Responses to When is a bug report useful?

  1. Huy says:

    Would you please if I can translate your article and repost it in my blog?

    • myriam says:

      Yes, I would be pleased as long as you give credit, but you will have to translate it yourself, I don’t master your language.

  2. mgraesslin says:

    Great post, hardly anything I could add. May I syggest to move it to userbase and add a link from bug creation wizard and DrKonqui to it?

    One additional point I have which became a small issue recently: Report the bug in English.

  3. Artem S. Tashkinov says:

    It all sounds and looks nice in theory.

    In reality the second if not the first most important factor in filing a bug report, is the developer’s concern and interest.

    In different open source bugzilla’s I have over 50 bugs reports which are obviously abandoned because developers simply don’t care.

    • myriam says:

      I updated the blog with some numbers just so you can relate. It is very rarely a lack of interest or concern, trust me, I am in daily contact with the developers and they really do care.
      Make sure all your reports are up-to-date, with all necessary information and for the latest available version. You can send a polite reminder and then just be patient.
      And of course, we talk about bugs here, not feature requests :)

  4. r i p says:

    After installing the debugging symbols, will I have an increase of ram and cpu usage? how much?

    • myriam says:

      Only during the time it takes to produce the backtrace. After that it will just use some space on your hard disk.

  5. r i p says:

    what means “invalid”? When a bug is “unconfirmed”?

    • myriam says:

      A bug is invalid if it is not usable and there is no further information given when asking for feedback, or if it is not reproducible. These are just examples though.
      A bug is unconfirmed until it has been triaged and confirmed. For that it needs to be reproducible and have all the necessary information. Then its status will change to NEW.

  6. Laerte says:

    > 3. The subject and comment must be in English…

    That excludes most international users as possible bug reporters. I know the developers cannot do much a respect, but its a pity.

    • myriam says:

      Well, we simply can’t change that. Also, a user must be able to give feedback in English. As I said, English is the language most used in computing and all developers can read and understand it.

    • David Edmundson says:

      The alternative is to have developer’s getting reports they can’t understand. Which is going to be even more frustrating both for the developer as well as the user doing the reporting.

  7. Thijs says:

    Good post, thank you. With the maturing of KDE4 as far as it has been in the past few years, in my opinion bug reducing should be a top priority right now. In a perfect world, every interval of 1 KDE release cycle should see the number of bugs decreasing, at least for the more or less stable components (including, e.g., Plasma, KWin, Amarok, KDevelop, Dolphin, Solid). Stuff like Sebas’ Nepomuk bugsprint is something that I admire a lot, and I hope that once cleaned out, the bug numbers can be kept under control. However, most of the KDE software seem to have a more or less linear upward trend in number of open bugs, with once in a while a sprint to mitigate the effect.

    To state the painful: I often feel guilty that I hardly contribute a lot to these things, but this has a few reasons (beyond time). That and the fact that I believe that free software volunteers should work on their projects for fun (and at least I myself enjoy building new stuff better than bughunting), make that I could never blame anyone for not fixing my pet annoyances right this instant.

    There are a few things that are discouraging me right now from posting bugs, as well as going into bug triaging.

    1) There are too many bugs in the tracker. This means that it is hard for me as a bug filer to find out whether a bug is a duplicate or not. Since I do not want to add to that pile (and my time during bugfiling is usually limited), I often end up not filing at all.

    2) Because there are too many bugs in the tracker, and because of misbehaviour of some in the comments, prolific devvers like Martin or Aaron have declared the bugtracker of limited use. This I understand well when looking at the bug tracker, but it also keeps me from (for instance) joining last months KWin triaging sprint. Is there a way to make bugs.kde.org usable again? This would include a commitment of devs to using BKO (not to suggest they are not using it, but the blogs are not really communicating that right now). It might also help to go to more austere measures, like asking people to confirm their bugs every half year (and close a bug if it doesnt happen). In the current situation, where bug reports do not fulfill the requirements, it should be set to invalid/needsinfo. Perhaps some of this can be done automatically?

    3) I only have a production system, and am not willing to move that to something beyond the (non-rolling) distro’s. In my case it is Fedora, but the vast bulk of ‘normal’ users only start using KDE SC 4.7 this fall, right when 4.8 is entering beta. If this means I should not file my bugs at bugs.kde.org, but only through my distro, that is fine. But that needs to be communicated. Perhaps by simply not allowing bugreports against older versions (e.g. anything but the current SC release).
    It would also mean that e.g. Dr. Konqi should be switched off by those distro’s, or redirected to their own bug trackers.

  8. Andy says:

    Most developers may do care. But with Linux distributors I have a different experience. For example, I reported a bug with a wlan hardware. I was asked to do some tests which I did but there was no easy solution, somebody had to look at driver code.

    Having reached that point, there was no further progress. Every few months when a new distro release was available, they wanted to have it retested, just to see if they could close it. No intention to work on the issue.

    I gave up after 2 years and will not test anymore. It would have been easiest if they said that they don’t have resources to do something about it. And I will not report bugs to that company anymore.

  9. Allen Winter says:

    Also it helps is to resolve your bug if you notice that it has been fixed.

  10. The User says:

    Dr Konqi checks if there are “enough” debug symbols, it is actually impossible to send the bug report if there are too few stars. That is very annoying, sometimes I look at the backtrace and only some unimportant debug symbols are missing, but Dr Konqi will not report it for me.

    • myriam says:

      That was changed on purpose, simply to avoid getting invalid crash reports.
      Of course, if you are able to judge by yourself that the backtrace is valid, you can report it manually. But make sure it is not a duplicate!

      • The User says:

        Well, the Bugzilla web interface is very ugly and Dr Konqi is very comfortable, it is just bad not to be able to use a better interface.

    • teprrr says:

      Actually in that case this might be a bug in the rating system, if you are sure the backtrace is useful. Don’t know the background about how DrKonqi checks for the validity, but in case the symbols missing are for example for basic libraries (meaning glib, libc) but still available for Qt and KDE stuff, it would be useful to allow submissions.

  11. skierpage says:

    Good points. But everything breaks down with “It should be about the most recent version available”. (K)ubuntu doesn’t ship the most recent version available!

    For reporting, what would be great is if the bug ecosystem was both version- and distribution- aware. I should be able to search for a bug, and even if it’s fixed find relevant bug reports because it still applies to my version and/or distro. Ubuntu’s Launchpad tries to handle this with its “(+) Also affects project (+) Also affects distribution” and upstream linking, but in my limited understanding their implementation creates another fjord in which tens of thousands of bugs languish underwater. The problem also goes in the other direction: these days if you Google for problems with any Linux feature or program, the top search results are invariably cluttered with detailed bug reports and forum threads about Ubuntu 8.04 and Fedora 9… from 2008.

    Another issue with reporting is the interconnected layers in any modern Linux distro. “KDE System Settings picks the wrong sound card” turned out to be PulseAudio providing identical description strings for them (“Internal Audio Analog Stereo”) because the kernel’s PCI probing is messed up on my hardware, with a workaround that requires Alsa config file changes. Likewise “Firefox prints blank images” turned out to be bugs in the interaction of Ghostscript and libpoppler exposed by Ubuntu’s configuration of CUPS to still use a PostScript workflow even when apps generate PDFs for printing. Both involved four web sites, four bug systems, four wikis, and four forums to wander around in after looking in my distro’s bug system. Many people would call this the typical Linux clusterf**** and give up in despair, but it shows that bug systems have to become smarter about relationships.

    In order to reproduce a bug with latest version, if I’m lucky I can download a standalone nightly binary from the project, unpack it into a test directory and retest with that. If I can’t, then sometimes I can switch my package manager to install unstable packages, but often that has cascading dependencies that require updating half a dozen other packages to unsupported latest. I’ve had problems with VLC and Wine and Pulseaudio and no easy way to try the latest short of compiling from source or installing a rolling bleeding-edge distro in a VM.

    Here’s an idea: what if projects provided a runnable version of their nightly build? Each could have a test server with compiled code and you could either use X’s DISPLAY to have the remote binary display on your screen, or you could use a remote desktop protocol, some of which can even run in a browser. This “Try the latest” facility could take screenshots and link to bug reports. And imagine how many more people would use your cool program if they could test-drive it!

    I really appreciate anyone who triages bug reports in the current situation, it’s a tough mission.

    • It is a tough mission, but it helps to triage the reports as they come in, not at a later point when you are fighting against 10,000 existing bug reports.

      Amarok is one of the few Free Software projects with 100% triaging rate. This means that no bug report will sit unanswered for longer than two weeks on average. Only this makes the reports really useful for us, and it also creates a higher user satisfaction. Making a bug report and having it sit there forever without getting any feedback is quite frustrating, so this is important.

      Treat your bug triagers well, and make them feel as important as the developers, which they really are.

  12. Pingback: computers: reporting bugs in a sea of versions and projects | Wuff

  13. zoki says:

    One more point probably missed. All of the bugs are not solved in the latest version. When new version is released it is very welcome to test if the bug is solved or not, and if not just write “I have tested the problem in new version 5.2.1 and problem still persist.” – if there is no feedback for old bugs, developers can take them as less important and so solving all other bugs that do have a feedback.

    • myriam says:

      We do that when triaging on a regular basis, shortly before a new release comes out as well as about 1 month after the release.

  14. Pingback: Getting Acquainted With Bug Reporting. | Wisdom Health Prosperity