Setting up remote execution¶
visplot and interactive clean can do their work on the data — reading
the measurement set, running tclean, reading and writing images and
masks — in a Jupyter kernel on the computer that holds the data, while the
window stays on your own computer. Only what the window shows crosses the
network.
This chapter covers setting that up with python -m cubevis.setup. Once a
kernel exists, using it is one argument:
from cubevis import visplot, iclean
visplot(vis="sis14_twhya_calibrated_flagged.ms", kernel_name="cvpost106-visplot")
iclean(vis="sis14_twhya_calibrated_flagged.ms", imagename="twhya", ...,
kernel_name="cvpost106-iclean")
Paths (vis, imagename) are the kernel’s: relative paths are found in,
and images are written to, the kernel’s working directory on that
computer.
What you need¶
On your computer:
pip install "cubevis[remote]"(thevisplotandicleanextras include it).A computer that holds the data and that you can reach with
ssh, directly or through a login host.
Nothing needs to be installed on the remote computer beforehand. Setup
finds the Python environments there that have cubevis and makes one when
there is none.
First-time setup¶
Run setup from your own computer, naming the computer the data is on as
you would for ssh:
$ python -m cubevis.setup cvpost106
Without a host name, setup asks for one. It asks before each change, says
what it will change, and tests each step before going on. Pressing Enter
accepts the answer shown in [brackets].
Setup works through these steps:
How to reach the host. The route is taken from your own ssh configuration. A
ProxyJumpthere is used as the login host; otherwise setup tries a direct login and asks whether a login host is needed.Logging in, to the login host and then to the host itself. Where a host asks for a password, setup stores it in a password store on your computer (the macOS Keychain, the Linux desktop keyring,
pass, or a gpg-encrypted file) so that kernels can be started without typing it. Passwords are never written to setup’s own files or logs.Looking around the host, read-only: its name and home directory, its conda tools, and its Python environments, each checked for
cubevis.A cluster? If the host looks like one node of a cluster (
cvpost106suggestscvpost*), setup asks once. A cluster is set up once for all its nodes.The environment the kernel runs in. Setup lists the environments that can be used and, when there is none, offers to make one: a conda environment with
cubevis[visplot,iclean]. Setup never installs into an environment of yours.The kernel: its name, the directory it starts in, and any environment variables. Setup writes the kernel’s launcher on the host and runs it once before recording it.
At the end setup prints how to use the kernel:
kernel cvpost106-iclean (on cvpost106) runs visplot, interactive clean:
from cubevis import iclean
iclean(vis="...", imagename="...", ..., kernel_name="cvpost106-iclean")
Note
Screenshots of a complete setup session will be added here.
Logins that ask for a second factor¶
If a host’s login asks for a code or a push notification, say so:
$ python -m cubevis.setup cvpost106 --2fa
No password is stored for such a host. You log in once in a terminal and the connection is shared by what follows.
What a kernel can run¶
Each application needs its own packages in the kernel’s environment:
visplotneedscubevis[visplot];interactive clean needs
cubevis[iclean]and CASA’s data (below).
Setup checks this when it tests a kernel and records the result
(python -m cubevis.setup status shows it). Each connection checks again:
a kernel whose environment lacks an application’s packages says so when
you start the application, with the pip install that adds them. An
environment that setup made can have interactive clean added to it; a
rerun of python -m cubevis.setup HOST offers that.
CASA’s data¶
Interactive clean needs CASA’s data (the measures). When the kernel’s
environment finds it — the measurespath in your ~/.casa/config.py, a
site configuration, or ~/.casa/data — setup asks whether to use it or
to give the environment a private copy (the casadata package, about
850 MB, never updated), so that cleaning there does not change when your
own data is updated. When none is found, an environment that setup made
gets the private copy; for an environment of yours setup says what to do.
Checking and changing a setup¶
Setup can be run again at any time. Nothing that passes is repeated.
Command |
What it does |
|---|---|
|
Show what is set up: networks, nodes, environments, kernels, and what each kernel can run |
|
Test the logins and kernels and change nothing. |
|
Check the host’s kernels and offer another one |
|
Add kernel |
|
Change the directory a kernel starts in ( |
|
Set a kernel’s environment variables; |
|
Move a kernel to another environment |
|
List the environments setup made on the host and remove the ones you choose. Your own environments are never listed, and one a kernel uses is never removed |
|
Every command and option |
Adding --redo asks again even for steps that pass.
When something does not work¶
Start with check, which tests each login and kernel and reports the
first thing that fails:
$ python -m cubevis.setup check
If the login itself is the problem, diagnose shows how ssh reaches the
host next to what setup has configured. It makes one login attempt and
changes nothing:
$ python -m cubevis.setup diagnose cvpost106
Other situations:
Your ssh configuration changed since setup recorded it.
statussays so, andpython -m cubevis.setup HOST(ornetwork NAME) asks whether to take the change over.A laptop that is sometimes inside the network and sometimes outside. Setup records which address ranges count as inside, and goes through the login host only from outside. To add where the computer is now, after checking that the nodes answer directly from there:
$ python -m cubevis.setup network NAME --inside-here
Python starts slowly on the host (an automounted filesystem):
python -m cubevis.setup HOST --pythonhomemakes its kernels setPYTHONHOME, so that Python skips the search for its installation.diagnose HOST --fulltimes the start.
What setup writes, and where¶
On your computer everything is under ~/.config/cubevis (or
$XDG_CONFIG_HOME/cubevis):
File |
Contents |
|---|---|
|
Networks, clusters, nodes, environments and kernels. This directory may be a home directory shared by several computers |
|
What belongs to one computer: the generated ssh configuration, the ssh wrapper, |
The sockets of shared ssh connections are kept on local disk, in
/tmp/cubevis-<uid>.
On the remote computer setup writes only the kernels’ launchers
(~/.config/cubevis/launch/) and, when it makes one, its own Python
environment, with its build logs in ~/.config/cubevis/make/.
Interactive clean on a remote kernel¶
The parameters are checked on the kernel’s computer, as
tclean’s task checks them: a path that is not there (relative to the kernel’s working directory) is reported with that computer’s name and directory.The images are best written in the kernel’s working directory (a relative
imagename). With current casatasks,tclean’s interactive driver looks for the residual and mask there, and animagenameelsewhere fails at the first cycle.While a cycle runs, the status line shows its progress, read from
tclean’s log on the kernel’s computer: the elapsed time, the minor or major cycle, channels, iterations and the peak residual. The whole text is its tooltip.Stop takes effect after the running cycle;
tcleancannot be interrupted within one.When the window is closed, or Python ends, the remote session is closed: the kernel and its worker stop. The images stay on that computer.