Re: help! Octave, connecting laptops to "supercomputers"

[Pedro]

On Thu, Feb 21, 2013 at 2:06 AM, Pedro <address@hidden> wrote:
> On Sun, Feb 3, 2013 at 3:33 PM, Pedro <address@hidden> wrote:
>> On Sun, Feb 3, 2013 at 12:19 PM, Thomas Weber <address@hidden> wrote:
>>>
>>> Hi,
>>>
>>> reading through your mails gives me the impression that you have no idea
>>> what you actually want, sorry.
>>
>> It doesn't matter ;)
>> I think that every response is giving interesting information about
>> that field. And helps me, and I hope also other people that is having
>> the same problems.
>>
>>> If you process data on a host and want to look at a graphics from this
>>> data processing, then this image must be sent to your local machine. It
>>> doesn't matter at all what kind of connection you use; the image must be
>>> sent to your machine. If you need to sync lots of data (like > 10 MB)
>>> over a slow connection, then rsync is the best general purpose solution.
>>>
>>> Yes, there are options that are better then forwarding an X connection
>>> to your laptop via ssh. But then it depends on what you want. Because,
>>> if you just want to run a script via the Octave command line, then you
>>> wouldn't even forward your X session, but just run Octave inside GNU
>>> screen or tmux.
>>
>> Yes, I see screen or tmux really useful for keep running processing
>> when you disconnect session with ssh [1] [2]
>>
>>> On Thu, Jan 31, 2013 at 02:00:33AM +0100, Pedro wrote:
>>> > First of all, Writing this mail I wanted to know what use octavian
>>> > people like me (or not) when need to work with hard processes. 4/4 for
>>> > going in "ssh -X", well, I have to re-think on that!
>>> >
>>> > I said no to "ssh -X" because I'm looking for alternatives on
>>> > connecting with computer/computers that process the data.
>>> >
>>> > - I've never used. (read last line of this mail)
>>> >
>>> > - from ssh -X I read that is innerently insecure, because run a root
>>> > process: the X server, read this (old) review about this [1].
>>> >
>>> > - Also, because is slow (or inefficient), that's because is encrypting
>>> > all the communications to the server.
>>> > But I only want to "pass the code strictly necessary".
>>> > Because if I use the ssh connection as a workstation it's excessive
>>> > for what I want.
>>> > Imagine looking images, sound or video with a ssh connection in a
>>> > "streaming way".
>>> > I could do rsync? ... Yes, but are things that take you off your
>>> > concentration, no? It means, that make you thing on two machines
>>> > constantly.
>>>
>>> I suggest you actually start with ssh and rsync and then solve problems
>>> as they come along.
>>>
>>>
>>> > I would like to imagine "one machine", could be with virtualization?
>>> > cloud computing? "p2p processing" ? Assymetric processing?
>>>
>>> Your problem definition is so broad that even "pocket calculator" seems
>>> to be in the solution set. What do you actually want to do?
>>
>> I wanted to know what are doing other people with the problem of "big
>> calculations", and discuss different methods.
>> And I put that keywords for inspire people to post a solution in that way.
>>
>>> > I don't understand a lot how it works distcc, but see this:
>>> > compilation of kernel distributed: the guy is in his computer, and
>>> > divide tasks to others, after it, they have his output file. [1]
>>> > Compiling means that a personalized task (hardware, processor
>>> > architecture...) but delivered to process by others
>>>
>>> distcc is a specialized tool for a very specific problem. In other
>>> words, it is the complete opposite of your approach.
>>>
>>> > For "ssh -X" users, could they give some tricks to make life easier?
>>> Depends; what problems do you have?
>>>
>>> For the record, I used to have Octave running on a 40 node cluster
>>> several years back. One Octave session was started on each host insided
>>> a screen session with 40 windows. Data was put onto the central master
>>> node,
>>> home directories were NFS-mounted into each node. Numeric results were
>>> saved on each node into the shared home directory and finally images
>>> from this data was generated on the master node. The only data that I
>>> then downloaded from this system was the images. This worked reasonably
>>> well for me. If your problem is different, then your solution will be
>>> different.
>>>
>>> All of this only makes sense if your calculations takes some time. If
>>> you have to wait like 1 minute, then the time to set up your specific
>>> solution will be much larger than the time for the calculations on your
>>> small machine.
>>
>> I appreciate your sharing.
>> And I have a question:
>> How you manage this 40 screens (nodes)? In every screen you put
>> different commands/scripts (by hand), to have the processing
>> distributed?
>> Or what tool / interface you use to do it?
>>
>> Thanks!
>> Pedro
>>
>> [1] 
>> http://askubuntu.com/questions/8653/how-to-keep-processes-running-after-ending-ssh-session
>> [2] 
>> http://serverfault.com/questions/19634/how-to-reconnect-to-a-disconnected-ssh-session
>
> Hey guys, given the comments of running via ssh. Perhaps this is the
> best way to start: install a simply octave on a SSI system. That's why
> I was looking for.

SSI:
https://en.wikipedia.org/wiki/Single_system_image

sorry for double-email.
I think someone will appreciate it.

> Also I add some solutions (Live CD/USB). I found it thanks to the
> keyword beowulf and his lists (take a look to beowulf cluster concept
> [1] !)
>
> easy to go
>     Live CD/USB
>         http://pareto.uab.es/mcreel/PelicanHPC/
>         http://birg1.fbb.utm.my/birghpc/
>         http://birg1.fbb.utm.my/birghpcc/
>             for cuda
>     http://www.kerrighed.org/wiki/index.php/Main_Page
>     OpenSSI
>         http://openssi.org/cgi-bin/view?page=openssi.html
>         https://en.wikipedia.org/wiki/OpenSSI
>
> [1] https://en.wikipedia.org/wiki/Beowulf_cluster