Login | Register
My pages Projects Community openCollabNet

Discussions > users > Re: [Subclipse-users] Calculating Decoration - why so slow?

subclipse
Discussion topic

There will be a brief maintenance window every Friday at 17:00 Pacific.
For further details, see CollabNet's maintenance and upgrade policy.

Hide all messages in topic

All messages in topic

Re: [Subclipse-users] Calculating Decoration - why so slow?

Author Tom Walter <tom dot walter at hitwise dot com>
Full name Tom Walter <tom dot walter at hitwise dot com>
Date 2009-01-15 16:44:51 PST
Message Just an update for anyone who finds themselves having the same kind of
problems as me in future.

I moved all my files locally, and installed an eclipse plugin called
FileSync to automatically transfer changed files to the network share.
http://andrei.gmxhom​e.de/filesync/index.​html

It works like a dream! As promised, Eclipse refreshes happen in a couple
of seconds. As a result, Subclipse switches, updates, and merges all
happen nearly as fast.

So the problem comes down to Eclipse being extremely slow to refresh a
project over a network share. I'm not certain why this would be. I have
a 1gbit ethernet connection to the fileserver. And the fileserver has
plenty of grunt (8 cores or something, and a HDD to match). I suspect it
is down to Samba and having to check file modified times/CRCs on so many
small files. There must be some kind of system hooks that Eclipse can
use to make this extremely fast from local disk that it can't use over
samba, is my guess. Any eclipse devs on the list care to weigh in?

Anyway, it probably will not make much difference for subclipse to not
do the full project refresh.

Thanks for all you help and advice Mark. And thanks for making a great
plugin! I'm certainly very happy now I can take full advantage of it.

Tom Walter wrote:
>> I have to think that Eclipse is reasonably efficient at doing a
>> refresh. You can have a large project on local disk and if you
>> refresh it and it is already in synch it runs in just a second or two.
>> So I do not think it can be doing a full crawl of all of the files
>> (although I do not see how else it could do it).
>>
>>
> A second or two! Wow if it should be running that fast there must be
> something seriously wrong with my setup. Perhaps Eclipse itself has a
> problem working on network drives? Or perhaps its a windows thing...
> eclipse might be able to hook into some OS-level thing when working
> locally that it can't use when accessing a network drive.
>
> Its so hard to tell with Eclipse, since it could be any one of a number
> of plugins that are causing the refresh to be slow, and I have ended up
> complaining about subclipse simply because that is the most visibly
> affected function for me.
>
>> So the question is will it be significantly faster for us to crawl the
>> structure and gather all the .svn folders and then refresh each of
>> those, versus just letting Eclipse do it all itself? Given that in
>> the switch case, more than half the total amount of files still need
>> to be processed, it seems likely to not be much faster to me. I think
>> it is worth trying if someone wants to setup the tests. The patch is
>> fairly easy.
>>
> Is it just a matter of checking out the subclipse source, applying the
> patch. Running some java compile command, and dropping a JAR somewhere?
> If it's that simple I can give it a shot, as long as you can tell me
> what the compile command should look like and where to put the JAR.
>
> --------------------​--------------------​--------------
> http://subclipse.tig​ris.org/ds/viewMessa​ge.do?dsForumId=1047​&dsMessageId=101​2570
>
> To unsubscribe from this discussion, e-mail: [users-unsubscribe@s​ubclipse.tigris.org]​.
>
>
>
Attachments
attachment-1327776

Re: [Subclipse-users] Calculating Decoration - why so slow?

Author Tom Walter <tom dot walter at hitwise dot com>
Full name Tom Walter <tom dot walter at hitwise dot com>
Date 2009-01-08 16:31:24 PST
Message > I have to think that Eclipse is reasonably efficient at doing a
> refresh. You can have a large project on local disk and if you
> refresh it and it is already in synch it runs in just a second or two.
> So I do not think it can be doing a full crawl of all of the files
> (although I do not see how else it could do it).
>
A second or two! Wow if it should be running that fast there must be
something seriously wrong with my setup. Perhaps Eclipse itself has a
problem working on network drives? Or perhaps its a windows thing...
eclipse might be able to hook into some OS-level thing when working
locally that it can't use when accessing a network drive.

Its so hard to tell with Eclipse, since it could be any one of a number
of plugins that are causing the refresh to be slow, and I have ended up
complaining about subclipse simply because that is the most visibly
affected function for me.
> So the question is will it be significantly faster for us to crawl the
> structure and gather all the .svn folders and then refresh each of
> those, versus just letting Eclipse do it all itself? Given that in
> the switch case, more than half the total amount of files still need
> to be processed, it seems likely to not be much faster to me. I think
> it is worth trying if someone wants to setup the tests. The patch is
> fairly easy.
Is it just a matter of checking out the subclipse source, applying the
patch. Running some java compile command, and dropping a JAR somewhere?
If it's that simple I can give it a shot, as long as you can tell me
what the compile command should look like and where to put the JAR.

Re: [Subclipse-users] Calculating Decoration - why so slow?

Author markphip
Full name Mark Phippard
Date 2009-01-08 16:05:47 PST
Message
Attachments
attachment-1311004

Re: [Subclipse-users] Calculating Decoration - why so slow?

Author Tom Walter <tom dot walter at hitwise dot com>
Full name Tom Walter <tom dot walter at hitwise dot com>
Date 2009-01-08 15:24:56 PST
Message I'm not a Java coder so I don't think my skills are up to
patching/compiling and testing that out.

If it's unlikely to make a difference, then it sounds like a waste of
effort anyway.

Just something to add to the wishlist I guess... full project refreshes
are slow for some people so a way to avoid that would be good.


> It would not be difficult to only update the .svn folders but I would
> be skeptical it would make that big of an improvement. We'd still
> have to walk the tree and discover each of these folders and then
> issue the Eclipse API to refresh each of these. It is conceivable
> that the process of walking the tree could trigger Eclipse to see it
> is out of date and refresh anyway, we have certainly run into cases
> like that in the past with certain Eclipse API calls.
>
> Unfortunately, there are no easy ways to test what kind of difference
> it would make. If you want to setup an environment where you have
> Subclipse checked out and run and test it in an Eclipse runtime, I
> could send you patches to try and see what kind of difference it
> makes.
>
>
Attachments
attachment-1310970

Re: [Subclipse-users] Calculating Decoration - why so slow?

Author jcompagner
Full name jcompagner
Date 2009-01-08 08:32:32 PST
Message
Attachments
attachment-1310244
attachment-1310245

Re: [Subclipse-users] Calculating Decoration - why so slow?

Author markphip
Full name Mark Phippard
Date 2009-01-08 07:57:17 PST
Message
Attachments
attachment-1310151

Re: [Subclipse-users] Calculating Decoration - why so slow?

Author Tom Walter <tom dot walter at hitwise dot com>
Full name Tom Walter <tom dot walter at hitwise dot com>
Date 2009-01-07 16:28:41 PST
Message I've done as you suggest and timed it. It seems that a manual refresh on
the project (with no modified files) takes about 5 mins, which is the
same length of time as an svn switch or an svn branch/tag. This remains
the same whether I run the refresh immediately prior to the svn switch
or not.

So clearly it is just that eclipse takes a long time to refresh my
particular project. I'll see what I can do about that. I suspect it is
because it is a large project accessed from windows over a samba
connection to a linux fileserver.

I think you've mentioned before that subclipse has to do the full
refresh to ensure eclipse refreshes the hidden .svn directories. It
would be great to see an enhancement to subclipse that could trigger
eclipse to refresh only the files changed by svn (and the hidden .svn
directories), without requiring a full project refresh. I think that
would provide substantial performance increases for people who develop
on remote servers, which is not uncommon. As it is, a 5 minute wait
after any project level svn process makes subclipse impractical for use
in this situation. Can I file a feature request?

Anyway, thanks for taking the time to respond Mark.

Mark Phippard wrote:
> On Mon, Jan 5, 2009 at 12:40 AM, Tom Walter <tom dot walter at hitwise dot com> wrote:
>
>
>> I have a project with 10k odd files. When using 'compute deep outgoing
>> state' and doing any non trivial thing like switching branches or
>> opening a new project, the 'SVN Decoration Calculation' task seems to
>> never end. I am not sure if it crashes or what, but generally I get
>> bored and kill the eclipse process after 10 mins or so. Because also
>> when this task is running it prevents other user tasks from occurring.
>> Also occasionally it seems to run out of memory and shut down eclipse at
>> about 1 GB or so.
>>
>> I can get around it by turning off deep outgoing state of course... but
>> this makes it difficult to tell the status of the project... which is a
>> major feature I use from subclipse.
>>
>> Running a command line svn status on the project finishes in
>> milliseconds, so what exactly is taking all the time in eclipse?
>>
>> I am on windows, the files are hosted on a linux machine that I connect
>> to over LAN using samba. Eclipse 3.4.1, Subclipse 1.4.7, using JavaHL.
>>
>> I've tried reinstalling eclipse and subclipse from scratch, and
>> recreating the project, starting with eclipse - clean and all that. I
>> don't think it is a problem with the state of my environment. It seems
>> to be something that subclipse particularly doesn't like about either
>> the files in the project or perhaps the network setup...
>>
>> Any suggestions would be appreciated.
>>
>
> What happens if you right-click on project and choose the Refresh
> option before you do this? It sounds like your workspace might be out
> of synch.
>
> The switch option does a refresh after it runs, so if your workspace
> was out of synch, this can take a long time to run. If you do the
> refresh first, that can give you an idea if that is the problem or
> not.
>
>
Attachments
attachment-1308808

Re: [Subclipse-users] Calculating Decoration - why so slow?

Author markphip
Full name Mark Phippard
Date 2009-01-05 07:23:13 PST
Message On Mon, Jan 5, 2009 at 12:40 AM, Tom Walter <tom dot walter at hitwise dot com> wrote:

> I have a project with 10k odd files. When using 'compute deep outgoing
> state' and doing any non trivial thing like switching branches or
> opening a new project, the 'SVN Decoration Calculation' task seems to
> never end. I am not sure if it crashes or what, but generally I get
> bored and kill the eclipse process after 10 mins or so. Because also
> when this task is running it prevents other user tasks from occurring.
> Also occasionally it seems to run out of memory and shut down eclipse at
> about 1 GB or so.
>
> I can get around it by turning off deep outgoing state of course... but
> this makes it difficult to tell the status of the project... which is a
> major feature I use from subclipse.
>
> Running a command line svn status on the project finishes in
> milliseconds, so what exactly is taking all the time in eclipse?
>
> I am on windows, the files are hosted on a linux machine that I connect
> to over LAN using samba. Eclipse 3.4.1, Subclipse 1.4.7, using JavaHL.
>
> I've tried reinstalling eclipse and subclipse from scratch, and
> recreating the project, starting with eclipse - clean and all that. I
> don't think it is a problem with the state of my environment. It seems
> to be something that subclipse particularly doesn't like about either
> the files in the project or perhaps the network setup...
>
> Any suggestions would be appreciated.

What happens if you right-click on project and choose the Refresh
option before you do this? It sounds like your workspace might be out
of synch.

The switch option does a refresh after it runs, so if your workspace
was out of synch, this can take a long time to run. If you do the
refresh first, that can give you an idea if that is the problem or
not.

--
Thanks

Mark Phippard
http://markphip.blogspot.com/

Re: [Subclipse-users] Calculating Decoration - why so slow?

Author jcompagner
Full name jcompagner
Date 2009-01-05 02:59:04 PST
Message can you use a profiler when starting eclipse (like yourkit)
to see what happens exactly?

On Mon, Jan 5, 2009 at 06:40, Tom Walter <tom dot walter at hitwise dot com> wrote:

> Hi
>
> I have a project with 10k odd files. When using 'compute deep outgoing
> state' and doing any non trivial thing like switching branches or
> opening a new project, the 'SVN Decoration Calculation' task seems to
> never end. I am not sure if it crashes or what, but generally I get
> bored and kill the eclipse process after 10 mins or so. Because also
> when this task is running it prevents other user tasks from occurring.
> Also occasionally it seems to run out of memory and shut down eclipse at
> about 1 GB or so.
>
> I can get around it by turning off deep outgoing state of course... but
> this makes it difficult to tell the status of the project... which is a
> major feature I use from subclipse.
>
> Running a command line svn status on the project finishes in
> milliseconds, so what exactly is taking all the time in eclipse?
>
> I am on windows, the files are hosted on a linux machine that I connect
> to over LAN using samba. Eclipse 3.4.1, Subclipse 1.4.7, using JavaHL.
>
> I've tried reinstalling eclipse and subclipse from scratch, and
> recreating the project, starting with eclipse - clean and all that. I
> don't think it is a problem with the state of my environment. It seems
> to be something that subclipse particularly doesn't like about either
> the files in the project or perhaps the network setup...
>
> Any suggestions would be appreciated.
>
> Cheers
> Tom
>
> --------------------​--------------------​--------------
>
> http://subclipse.tig​ris.org/ds/viewMessa​ge.do?dsForumId=1047​&dsMessageId=100​4347
>
> To unsubscribe from this discussion, e-mail: [
> users-unsubscribe@su​bclipse.tigris.org].​
>
Attachments
attachment-1302219

[Subclipse-users] Calculating Decoration - why so slow?

Author Tom Walter <tom dot walter at hitwise dot com>
Full name Tom Walter <tom dot walter at hitwise dot com>
Date 2009-01-04 21:40:58 PST
Message Hi

I have a project with 10k odd files. When using 'compute deep outgoing
state' and doing any non trivial thing like switching branches or
opening a new project, the 'SVN Decoration Calculation' task seems to
never end. I am not sure if it crashes or what, but generally I get
bored and kill the eclipse process after 10 mins or so. Because also
when this task is running it prevents other user tasks from occurring.
Also occasionally it seems to run out of memory and shut down eclipse at
about 1 GB or so.

I can get around it by turning off deep outgoing state of course... but
this makes it difficult to tell the status of the project... which is a
major feature I use from subclipse.

Running a command line svn status on the project finishes in
milliseconds, so what exactly is taking all the time in eclipse?

I am on windows, the files are hosted on a linux machine that I connect
to over LAN using samba. Eclipse 3.4.1, Subclipse 1.4.7, using JavaHL.

I've tried reinstalling eclipse and subclipse from scratch, and
recreating the project, starting with eclipse - clean and all that. I
don't think it is a problem with the state of my environment. It seems
to be something that subclipse particularly doesn't like about either
the files in the project or perhaps the network setup...

Any suggestions would be appreciated.

Cheers
Tom
Messages per page: