create working solution for globals issue

[kotowicz]

Problem statement: #2
Possibly related Matlab issues:
http://www.mathworks.com/matlabcentral/newsreader/view_thread/163362
http://www.mathworks.com/matlabcentral/newsreader/view_thread/287891
http://mathforum.org/kb/message.jspa?messageID=7445456

I'm happy to receive any feedback from those of you working with globals.

[kotowicz]

detailed problem demo:

>> global a
>> a = 10

a =

    10

>> matlabpool open local
Starting matlabpool using the 'local' profile ... a
connected to 12 workers.

a =

    10

>> a

a =

    10

>> ParforProgressStressTest2

  >> execution time was 1.64s.

ParforProgressStressTest2 running time: 1.6267s.
>> a
Undefined function or variable 'a'.

the global a disappears because JAVAADDPATH clears all globals.

[kotowicz]

The 0.2.5 release https://github.com/kotowicz/matlab-ParforProgress2/releases/tag/v0.2.5 should address all issues. In case there are still problems with global variables, please report them here.

[kotowicz]

demo to show that globals do not get cleared anymore:

>> global a
>> a = 10

a =

    10

>> matlabpool open local
Starting matlabpool using the 'local' profile ... connected to 12 workers.

>> ParforProgressStressTest2(10000, 0)

  >> execution time was 1.36s.

ParforProgressStressTest2 running time: 1.3521s.

>> a

a =

    10

>> matlabpool close
Sending a stop signal to all the workers ... stopped.

>> ParforProgressStressTest2(10000, 0)

  >> execution time was 3.18s.

ParforProgressStressTest2 running time: 3.1779s.

>> a

a =

    10

>> matlabpool open local
Starting matlabpool using the 'local' profile ... connected to 12 workers.

>> ParforProgressStressTest2(10000, 0)

  >> execution time was 1.3s.

ParforProgressStressTest2 running time: 1.2994s.

>> matlabpool close
Sending a stop signal to all the workers ... stopped.

>> a

a =

    10

[chinasaur]

I haven't had a chance to look into the code yet; will try this week. Maybe you can just tell me; was the problem with my approach that if you close and open the matlabpool then it doesn't realize it needs to add the Java path again in the child processes?

If that's the issue, I think the best solution would be to have a runonall that does the same kind of check for whether Java path is already set on all the child processes. Then just run that every time and it will normally do nothing but if it's the first time it will add the java path on all, and if the pool has been restarted it will add java path on just the children but not the master.

[chinasaur]

Actually, perhaps you should consider an additional change that I have made in my own branch. In my version, in addition to trying to avoid the globals issue, I have also eliminated the Client Java class. Instead, the Client class is just a Matlab handle and then the increment method looks like:

function increment(o, i) %#ok<INUSD>
    % i is a fake input so we stay compatible with
    % "ParforProgressConsole2.m"
    s = java.net.Socket();
    s.connect(o.ISA);
    s.close();
end

Of course, it's a bit constraining to make the Client Matlab instead of Java, but since its job is simple I consider this an improvement. Additionally, you don't have the Java path issue on the child processes anymore, since only the Server node actually needs any extra Java path. (This is probably why I didn't catch the matlabpool restart bug when I submitted my change to you; apologies for debugging in an inconsistent context!)

(I believe I went through some iterations trying some fancier things with the java.net.Socket like setting some additional flags, but in the end this simple thing seemed best.)

Let me know if you want a PR with these more aggressive changes.

[kotowicz]

thanks for your feedback. At this moment I would try to avoid making too many changes at the same time. For now I'd like to see whether the current implementation (v0.2.5) does work (in theory) for everybody. Once we have a stable, working solution, I suggest we try to look at different options for the future.

[chinasaur]

That's fine; I'll try to find time to do some tests. Take a look at my fork if you are interested in the simplified approach; I think my fork is up to date.

[mtompkins]

I noticed I'm current with your release and my globals are getting dropped. I use them to init an SMTP send email address. I'll see if I can widdle it down to something you can replicate locally.

That said, unfortunately the problem still exists.

[chinasaur]

@mtompkins, do your globals get stomped repeatedly, or just the first time parforprogress is loaded?

[mtompkins]

@chinasaur I'm not quite sure what you are asking so I'll answer as follows:

I init a single global variable which holds a simple email address.

I then use the bar during a parametric sweep (lasts a few days).
On a failure of the sweep it should send an email

When the script ends an email should be sent notifying a successful completion.
My latest sweep ended today (successfully) but there was an error at the email send routine as the global variable had been dropped.

If I test the sections that init the variable and send the email (bypassing the parametric sweep itself) everything works as expected.

[chinasaur]

(Answer below is based on my fork, which is somewhat different, but I believe should apply equally here; kotowicz can correct if I'm mixed up.)

I believe the globals clearing happens when parforprogress adds itself to your javaclasspath. This used to happen every time parforprogress was called, but the new changes should make it only happen the first time it's called, and only if it's not on the classpath already.

You could try initiating something that uses parforprogress, interrupting it early, init your global and then rerun; it shouldn't stomp the global then. Try with something shorter than a few days first :).

If this fixes your problem, then you could arrange to have the appropriate classes added to javaclasspath on startup and then I believe you won't have to think about it anymore.

[mtompkins]

Gotcha, thank you. So init / paint the progress bar briefly then carry on basically until there is a larger rework.

[kotowicz]

@mtompkins do you run your code like this? Please notice that you need to use the run_javaaddpath = 0 setting for it to work with globals.

 >> run_javaaddpath = 0;
 >> s = 'dummy task';
 >> n = 100;
 >> percentage = 0.1;
 >> do_debug = 0;
 >> ppm = ParforProgressStarter2(s, n, percentage, do_debug, run_javaaddpath)
 >> for j = 1 : n
 >>     your_computation();
 >>     ppm.increment(i); 
 >> end
 >> delete(ppm);

In case this doesn't work, could you please post the Matlab output showing which globals disappear after running ParforProgressStarter2? Thanks!