8

My company has been running a trial of Atlassian Crucible for some months now. For repositories where it's working properly, users have given very positive feedback about the tool. The problem I'm having is that we have several different projects, each with its own repository, and some of those repositories are very large. One repository in particular has a large number of branches and probably around 9,000 files per branch. Browsing that repository in Crucible is extremely slow.

Crucible is running on a CentOS VM. The VM has 4GB of RAM, and I've set Crucible's maximum at 3GB, of which it is currently using 2GB. I've brought this up in a support ticket with Atlassian, and they suggested the following:

In particular because you have a rather large SVN repository you will likely find that Fisheye will be creating a large index file on disk. To help improve performance a few things you can try are:

I've tried all of these things to an extent, but so far none have helped greatly. I was originally running Crucible on a Windows box with 2GB of RAM using the built in HSQL DB. Moving to MySQL on CentOS saw a performance increase for some repositories, and made Crucible much more stable, but did not seem to help much with our biggest repository. There are only so many files/branches I can exclude from indexing while maintaining the tool's usefulness.

That being the case, does anyone have any tips on how to speed up Crucible on large repositories, without investing in insanely powerful hardware?

Thanks!

Edit: To clarify, since I didn't mention it explicitly above, I am using FishEye.

Edit 2: Since I originally posted this, performance has improved somewhat with new Crucible releases, but it's still not great by any means. It seems that this issue affects many users, including some with far more powerful hardware than we are using. Thus, I do not believe it is a hardware issue, but rather an issue with inherent inefficiency in Crucible. Atlassian is aware of the issue and will be including further performance improvements in future releases, so hopefully those changes solve our problems.

Edit 3: I'd forgotten how long ago I'd asked this question, so in my previous edit I neglected to mention that our hardware situation has also changed since it was originally asked. We're now running Crucible on a dedicated, physical server, still using CentOS. The hardware is still modest (4GB RAM, quad core CPU and dual 500GB disks in RAID 1 with external backup), but we did see a slight performance increase when we moved away from the VM.

Community
  • 1
Mitch Lindgren
  • 183
  • 1
  • 8
  • I know this is an old question but FYI to anyone finding this via search, in my (very limited) recent experience just moving the database to an external PostgreSQL instance will give you a major speedup for large repos (obviously, this assumes that machine is sufficiently powerful to run a decent sized postgres instance; I also tweaked the vaccume settings a bit for my hardware, but just out of the box it was faster). This will drastically reduce disk access times, and the performance and usability is far better than mysql (or at least, it appears to be for fecru) – Sam Whited Dec 25 '15 at 01:28

3 Answers3

2

Since migrating to MySQL made a noticeable difference for some repositories, consider tuning the database for further improvements. Changing some my.cnf values from the defaults can make a huge difference. See InnoDB Performance Optimization Basics for more information. Also check for slow queries by enabling the slow query log and add indexes where appropriate.

My next guess would be network speed: is your Crucible instance on the same wired local network as your SVN repositories? You might also try giving Crucible a trial run on the same machine as your primary repository if possible to eliminate network latency as the culprit.

And I know it might be difficult depending on your work environment, but running Crucible in a VM probably isn't helping things; Atlassian makes a note of this on their very brief Best Practices for Crucible Configuration page. I'm sure you've already come across it, but I'll also mention the Tuning FishEye page for other readers.

I also have performance issues for large projects, but attribute a lot of the slowness to Crucible's heavy web interface. This is especially true after clicking around for a bit (previously viewed pages in a review remain in the browser window, even when hidden from sight). Our developers have noticed a slight speed increase by switching to Google Chrome. Also check out the Atlassian IDE Connector if a compatible plugin exists for your development environment. The Eclipse IDE Connector had issues of its own the last time I used it (many months ago), but it could at least handle large file sets without hanging up.

Depending on your company's development practices, you could stop scanning a large number of code branches (assuming many of them are no longer active), and disable repositories for completed/dead projects until they're needed. My company utilizes very small teams on a large number of projects, so most of the time we work primarily on trunk, making branches the exception; we therefore explicitly add branches to scan instead of including all branches by default. Also make sure you're not accidentally scanning tags.

How is your CPU usage on the Crucible box? If you're using SVN behind Apache HTTPD, examine how many connections are consumed by Crucible during a big repository scan. Aside from that, I'm not sure what else you could look at (maybe disk speed? Repository scan frequency?), but hopefully the above tips will help a bit.

Dave
  • 156
  • 3
  • Thanks for the detailed response. I updated my original question as I forgot to include updated hardware info in my previous edit. Network speed is probably an issue in initial indexing but should not cause problems once the indexes have been created (which is where we see a lot of pain - when browsing indexed files.) We did end up moving Crucible to a dedicated physical box and saw a modest performance increase. Most of our devs use Chrome, and I've warned everyone using Crucible NOT to use IE as its JavaScript engine (before IE9 anyway) is very slow. – Mitch Lindgren Jul 19 '11 at 15:48
  • I wasn't aware previously viewed files were kept in memory, so I'll be sure to tell everyone to refresh when things get slow. FYI, though, Atlassian has completely dropped support for the Eclipse connector, which I got a lot of complaints about. They do have connectors for other IDEs but Eclipse is a big one for us. As for branches, some of our teams do not use them and some do extensively. The teams that don't use branches are fine. The teams that do encounter serious slowness. Unfortunately asking them to change their process is kind of out of the question. – Mitch Lindgren Jul 19 '11 at 15:53
  • I've looked into MySQL performance previously and will do so again. It looks like the big bottleneck is just traversing the index files, though. A faster disk might help here, but our disks are already pretty fast (albeit not top of the line. Before moving to the new server I saw a lot of IO wait, but I don't see too much any more.) I think all I can do now is to wait for Atlassian's touted performance improvements. I'm going to mark your answer as accepted, though, as I think it contains a lot of valuable information for others who might be in the same situation. – Mitch Lindgren Jul 19 '11 at 16:28
1

>4 G of RAM isn't "insanely powerful" hardware. Assuming you've got 25 users and you're using Fisheye (which you mention), you're spending $4400 on just the software. $4k at Dell could buy you a server with 48G of RAM.

Also, are you using a 64-bit JVM? The docs suggest that you'll see better memory footprint (as in, less of it) on a 32-bit JVM.

Bill Weiss
  • 10,782
  • 3
  • 37
  • 65
  • Thanks for the info. We are using a 64-bit JVM. I'll see if I can switch to 32-bit and see if it helps. Edit: Oops - hit enter and it saved my comment instead of adding new lines. My bad. Regarding hardware, it's something of a catch-22: the hardware situation is somewhat out of my control, and it's hard for me to justify increased expenditure until we know that the tool will work for all of the teams who need to use it. I'll see if anything can be done to the existing setup (allocate more memory to that VM, for example.) – Mitch Lindgren Oct 04 '10 at 22:38
  • Apologies for double-commenting, but one other question: are you confident that memory is the problem? I realize 4GB isn't a lot, but Fisheye/Crucible aren't even maxing out the 3GB maximum I've set on the JVM. – Mitch Lindgren Oct 04 '10 at 22:45
  • I'm not sure if that's your issue, but I was just pointing out that that's not "insanely powerful". While it's performing poorly, could you collect some system statistics? Run `top`, `iostat` and whatnot, and see what's hurting. – Bill Weiss Oct 05 '10 at 16:13
  • "Insanely powerful" was a poor choice of words. I just feel that a $4,000 server with 48GB of RAM is an inordinate requirement for a web app which is used by so few developers. – Mitch Lindgren Nov 18 '10 at 22:22
  • Ugh, sorry again - I keep forgetting what the enter key does in Stack Exchange comments. I will see what I can do about getting some system stats during a high using period and append them to my original question. – Mitch Lindgren Nov 18 '10 at 22:23
  • 3
    $4400 / 25 users / 2 years == $88/dev/year. How many dev hours a year does it have to save you? – Bill Weiss Nov 19 '10 at 23:12
  • Bill, despite my personal feeling that a web app with such a small user base should not require hardware like that, I agree with your cost-benefit analysis. The issue is that I am unwilling to throw money at the problem when I am not sure that it IS a hardware problem. As it turns out, numerous other users have reported this issue and some of them are running with 12 cores and 48GB of RAM. see this Jira ticket: https://jira.atlassian.com/browse/CRUC-4518 - since I first posted this, performance has improved for us somewhat with new Crucible releases. It is still not great, though :( – Mitch Lindgren Jul 18 '11 at 22:01
0

While I haven't actually tried this myself, I'm experiencing precisely the same symptoms as you.

I am currently considering turning off stored diff info for the offending repositories. I asked the question on Atlassian's Q&A site and got some promising advice.

My issue is the same - indexing isn't the problem, it's a huge disk footprint running on a poorly performing disk array in a VM. Since I cannot upgrade the disk at present, I need to find another way around it. The answerer in my post above says that removing diff info will reduce disk footprint at the expense of losing the ability to search added/removed lines. Though he also suggests that it won't affect the speed of browsing files with long histories.

If someone else sees this & can report success / failure with this switch, please comment here.

Oh and I'm running 2.7.13 with the same performance issues.

Mark McDonald
  • 576
  • 1
  • 4
  • 12