Login | Register
My pages Projects Community openCollabNet

Discussions > issues > [Issue 1540] New - Refresh SVN status cache takes up to an hour

subclipse
Discussion topic

Hide all messages in topic

All messages in topic

[Issue 1540] Refresh SVN status cache takes up to an hour

Author mralwasser
Full name Patric R.
Date 2016-04-29 06:00:55 PDT
Message http://subclipse.tig​ris.org/issues/show_​bug.cgi?id=1540






------- Additional comments from mralwasser at tigris dot org Fri Apr 29 06:00:55 -0700 2016 -------
Any news from the developers regarding this issue ?

We also experience high cpu usage during a long lasting "Refresh SVN status
cache" job.

[Issue 1540] Refresh SVN status cache takes up to an hour

Author stefane
Full name Stefan Eggerstorfer
Date 2016-03-05 13:49:54 PST
Message http://subclipse.tig​ris.org/issues/show_​bug.cgi?id=1540






------- Additional comments from stefane at tigris dot org Sat Mar 5 13:49:54 -0800 2016 -------
Created an attachment (id=438)
FileModificationManager.patch

[Issue 1540] Refresh SVN status cache takes up to an hour

Author stefane
Full name Stefan Eggerstorfer
Date 2016-03-05 13:49:35 PST
Message http://subclipse.tig​ris.org/issues/show_​bug.cgi?id=1540






------- Additional comments from stefane at tigris dot org Sat Mar 5 13:49:35 -0800 2016 -------
Created an attachment (id=437)
svn_update_timings.png

[Issue 1540] Refresh SVN status cache takes up to an hour

Author stefane
Full name Stefan Eggerstorfer
Date 2016-03-05 13:46:58 PST
Message http://subclipse.tig​ris.org/issues/show_​bug.cgi?id=1540






------- Additional comments from stefane at tigris dot org Sat Mar 5 13:46:57 -0800 2016 -------
I am experiencing a similar performance issues on a huge workspace (100+ projects).
I did some research and have found one suspicious behavior when Subclipse gets notified about changed
resources after a refresh:

Assume you have an Ant build directory which is svn-ignored containing following hierarchy:
- build/com
- build/com/company
- build/com/company/product

When Eclipse detects changes in all 3 folders, Subclipse updates the SVN status for each of them with
an algorithm like this:
- find the nearest parent folder with a SVN status (in this case I guess build, it has status IGNORED)
- update the SVN status of the whole folder recursively
In our example the whole build folder is update 3 times.

I also collected some timings of my workspace, see attached file "svn_update_timings.png" (it contains
aggregated values of 3 loops of a clean ant build and Eclipse startup with "refresh on startup" setting
enabled).
The first line is the aggregation of the whole workspace, the second line is one huge project with a
build folder which was updated 656 times for these 3 runs, summing up to 48s spent during SVN status
update just in this project (and more than 3 minutes for the whole workspace).

I applied one change to the update algorithm, see attached patch "FileModificationMan​ager.patch". I
changed the SVN update loops of folders to remember and skip already updated folders. The time needed
to update the project was down to 1/4 (~13s), the whole workspace took now 73s (-60%).

What do you think about this change? Is it a valid modification of the algorithm?

[Issue 1540] Refresh SVN status cache takes up to an hour

Author janmae
Full name Jan Mäser
Date 2013-08-09 00:59:47 PDT
Message http://subclipse.tig​ris.org/issues/show_​bug.cgi?id=1540






------- Additional comments from janmae at tigris dot org Fri Aug 9 00:59:46 -0700 2013 -------
I did some more research and this happens as well with the same project but on a
Windows XP machine using Eclipse Helios (SR2) and Subclipse Version 1.10.1 - It's
the exact same behavior and stack traces.

[Issue 1540] New - Refresh SVN status cache takes up to an hour

Author janmae
Full name Jan Mäser
Date 2013-08-02 01:08:26 PDT
Message http://subclipse.tigris.org/issues/show_bug.cgi?id=1540 Issue #|1540 Summary|Refresh SVN status cache takes up to an hour Component|subclipse Version|1.10.x Platform|JavaHL OS/Version|other URL| Status|NEW Status whiteboard| Keywords| Resolution| Issue type|DEFECT Priority|P3 Subcomponent|core Assigned to|issues@subclipse Reported by|janmae ------- Additional comments from janmae@tigris.org Fri Aug 2 01:08:25 -0700 2013 ------- The problem i experience is that after various working copy modifications the 'Refresh SVN status cache' process kicks in. In my case most of the time this takes up to an hour which is quiet difficult to work with. While the cache is being refreshed it uses up a whole CPU core. The odd thing about it is that most of the CPU time gets lost in the kernel according to the task manager. The hardware though is not an issue, it's a state of the art notebook (Core i7, 16GB memory and a SSD for the hard drive). For the operating system i'm bound to Windows7 64bit. I'm usig the current Kepler version of Eclipse and following plugins Subclipse (Required) 1.10.1 Subversion Client Adapter (Required) 1.10.0 Subversion JavaHL Native Library Adapter 1.8.1 Subversion Revision Graph 1.1.1 SVNKit Client Adapter (Not required) 1.8.0 I've to add that i've experienced the same issue with the latest Juno version (4.2.2) of Eclipse and the latest 1.8.x Suclipse plugin version. With the older 1.6.x plugins i haven't had the same issues. What information i can give about the project working copy is that it's unusually big which has always been an issue but i can't change that. The working copy (including the .svn directory) consists of about 62k files in about 4k7 folders. I've made several dumps of the thread working the cache update and always came up with one of the following 3 in various recursion depths. "Worker-18" prio=6 tid=0x00000000130fd000 nid=0x1234 runnable [0x000000001803f000] java.lang.Thread.State: RUNNABLE at java.io.WinNTFileSystem.checkAccess(Native Method) at java.io.File.canWrite(Unknown Source) at org.tigris.subversion.subclipse.core.resources.LocalResourceStatus.(LocalResourceStatus.java:91) at org.tigris.subversion.subclipse.core.status.StatusCacheManager.updateCache(StatusCacheManager.java:121) at org.tigris.subversion.subclipse.core.status.StatusCacheManager.updateCache(StatusCacheManager.java:96) at org.tigris.subversion.subclipse.core.status.StatusCacheManager.refreshStatus(StatusCacheManager.java:270) at org.tigris.subversion.subclipse.core.resourcesListeners.FileModificationManager.refreshStatus(FileModificationManager.java:250) at org.tigris.subversion.subclipse.core.resourcesListeners.FileModificationManager.access$2(FileModificationManager.java:223) at org.tigris.subversion.subclipse.core.resourcesListeners.FileModificationManager$2.run(FileModificationManager.java:188) at org.tigris.subversion.subclipse.core.util.JobUtility$1$1.run(JobUtility.java:22) at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2345) at org.tigris.subversion.subclipse.core.util.JobUtility$1.run(JobUtility.java:20) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:53) Locked ownable synchronizers: - None "Worker-18" prio=6 tid=0x00000000130fd000 nid=0x1234 runnable [0x000000001803e000] java.lang.Thread.State: RUNNABLE at java.io.WinNTFileSystem.list(Native Method) at java.io.File.list(Unknown Source) at java.io.File.listFiles(Unknown Source) at org.tigris.subversion.subclipse.core.status.StatusUpdateStrategy.processUnversionedFolder(StatusUpdateStrategy.java:90) at org.tigris.subversion.subclipse.core.status.StatusUpdateStrategy.processUnversionedFolder(StatusUpdateStrategy.java:98) at org.tigris.subversion.subclipse.core.status.StatusUpdateStrategy.processUnversionedFolder(StatusUpdateStrategy.java:98) at org.tigris.subversion.subclipse.core.status.StatusUpdateStrategy.processUnversionedFolder(StatusUpdateStrategy.java:98) at org.tigris.subversion.subclipse.core.status.StatusUpdateStrategy.processUnversionedFolder(StatusUpdateStrategy.java:98) at org.tigris.subversion.subclipse.core.status.StatusUpdateStrategy.collectUnversionedFolders(StatusUpdateStrategy.java:76) at org.tigris.subversion.subclipse.core.status.RecursiveStatusUpdateStrategy.statusesToUpdate(RecursiveStatusUpdateStrategy.java:73) at org.tigris.subversion.subclipse.core.status.RecursiveStatusUpdateStrategy.statusesToUpdate(RecursiveStatusUpdateStrategy.java:46) at org.tigris.subversion.subclipse.core.status.RecursiveStatusUpdateStrategy.statusesToUpdate(RecursiveStatusUpdateStrategy.java:46) at org.tigris.subversion.subclipse.core.status.RecursiveStatusUpdateStrategy.statusesToUpdate(RecursiveStatusUpdateStrategy.java:46) at org.tigris.subversion.subclipse.core.status.RecursiveStatusUpdateStrategy.statusesToUpdate(RecursiveStatusUpdateStrategy.java:46) at org.tigris.subversion.subclipse.core.status.RecursiveStatusUpdateStrategy.statusesToUpdate(RecursiveStatusUpdateStrategy.java:46) at org.tigris.subversion.subclipse.core.status.RecursiveStatusUpdateStrategy.statusesToUpdate(RecursiveStatusUpdateStrategy.java:46) at org.tigris.subversion.subclipse.core.status.RecursiveStatusUpdateStrategy.statusesToUpdate(RecursiveStatusUpdateStrategy.java:46) at org.tigris.subversion.subclipse.core.status.StatusCacheManager.refreshStatus(StatusCacheManager.java:270) at org.tigris.subversion.subclipse.core.resourcesListeners.FileModificationManager.refreshStatus(FileModificationManager.java:250) at org.tigris.subversion.subclipse.core.resourcesListeners.FileModificationManager.access$2(FileModificationManager.java:223) at org.tigris.subversion.subclipse.core.resourcesListeners.FileModificationManager$2.run(FileModificationManager.java:188) at org.tigris.subversion.subclipse.core.util.JobUtility$1$1.run(JobUtility.java:22) at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2345) at org.tigris.subversion.subclipse.core.util.JobUtility$1.run(JobUtility.java:20) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:53) Locked ownable synchronizers: - None "Worker-18" prio=6 tid=0x00000000130fd000 nid=0x1234 runnable [0x000000001803e000] java.lang.Thread.State: RUNNABLE at java.io.WinNTFileSystem.getBooleanAttributes(Native Method) at java.io.File.isDirectory(Unknown Source) at org.tigris.subversion.subclipse.core.status.StatusUpdateStrategy.processUnversionedFolder(StatusUpdateStrategy.java:97) at org.tigris.subversion.subclipse.core.status.StatusUpdateStrategy.processUnversionedFolder(StatusUpdateStrategy.java:98) at org.tigris.subversion.subclipse.core.status.StatusUpdateStrategy.processUnversionedFolder(StatusUpdateStrategy.java:98) at org.tigris.subversion.subclipse.core.status.StatusUpdateStrategy.processUnversionedFolder(StatusUpdateStrategy.java:98) at org.tigris.subversion.subclipse.core.status.StatusUpdateStrategy.processUnversionedFolder(StatusUpdateStrategy.java:98) at org.tigris.subversion.subclipse.core.status.StatusUpdateStrategy.processUnversionedFolder(StatusUpdateStrategy.java:98) at org.tigris.subversion.subclipse.core.status.StatusUpdateStrategy.collectUnversionedFolders(StatusUpdateStrategy.java:76) at org.tigris.subversion.subclipse.core.status.RecursiveStatusUpdateStrategy.statusesToUpdate(RecursiveStatusUpdateStrategy.java:73) at org.tigris.subversion.subclipse.core.status.RecursiveStatusUpdateStrategy.statusesToUpdate(RecursiveStatusUpdateStrategy.java:46) at org.tigris.subversion.subclipse.core.status.RecursiveStatusUpdateStrategy.statusesToUpdate(RecursiveStatusUpdateStrategy.java:46) at org.tigris.subversion.subclipse.core.status.RecursiveStatusUpdateStrategy.statusesToUpdate(RecursiveStatusUpdateStrategy.java:46) at org.tigris.subversion.subclipse.core.status.RecursiveStatusUpdateStrategy.statusesToUpdate(RecursiveStatusUpdateStrategy.java:46) at org.tigris.subversion.subclipse.core.status.RecursiveStatusUpdateStrategy.statusesToUpdate(RecursiveStatusUpdateStrategy.java:46) at org.tigris.subversion.subclipse.core.status.RecursiveStatusUpdateStrategy.statusesToUpdate(RecursiveStatusUpdateStrategy.java:46) at org.tigris.subversion.subclipse.core.status.RecursiveStatusUpdateStrategy.statusesToUpdate(RecursiveStatusUpdateStrategy.java:46) at org.tigris.subversion.subclipse.core.status.StatusCacheManager.refreshStatus(StatusCacheManager.java:270) at org.tigris.subversion.subclipse.core.resourcesListeners.FileModificationManager.refreshStatus(FileModificationManager.java:250) at org.tigris.subversion.subclipse.core.resourcesListeners.FileModificationManager.access$2(FileModificationManager.java:223) at org.tigris.subversion.subclipse.core.resourcesListeners.FileModificationManager$2.run(FileModificationManager.java:188) at org.tigris.subversion.subclipse.core.util.JobUtility$1$1.run(JobUtility.java:22) at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2345) at org.tigris.subversion.subclipse.core.util.JobUtility$1.run(JobUtility.java:20) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:53) Locked ownable synchronizers: - None
Messages per page: