Instructions for running CORHEL-AMCG on AWS

All activities on the AWS GPU instance should be done as the "psi" user. To become "psi", use the command: sudo -u psi -s

The PSI CORHEL environment will automatically be loaded when becoming the "psi" user (through the script /data/psi/load_psi_gpu_env that is sourced in the "psi" user's .bashrc)

In the following instructions, [NEXT] = Click the "Next" button

1. Load interface web site in browser:

   https://corhel-dev.ccmc.mysmce.com/thermo_webapps/thermo_designer

   Enter name, e-mail, and daily session number (use 1)

   [NEXT]

2. On Step Overview page, click "Enter Step 1"

   

3. In this guide, we are assuming there is no pre-existing MHD background run for the event under study, so on the "Map Selection" page, select "Create a new map..."

   

4. Select/enter date and time of event, then select the map source (HMI is default)

   [NEXT]

   

5. Select the event location (active region (AR)) by drawing a box around it

   [NEXT]

   

============= ZERO BETA FLUX ROPES =================================

6. In "Step Overview", now click "Enter Step 2"
7. Click "Active Region 1"
8. After a brief calculation, the polarity inversion lines (PILs) for the AR should be shown
9. Select a point at the desired start location of the PIL, then select a 2nd point at the desired end point. [NEXT]
10. Use the tool tips as a guide to design/construct the RBSL flux rope (This is a long step) [NEXT]
11. Now you are back to the page from step (7) but the "Active Region 1" should be green. [NEXT]
12. The "Zero-Beta Simulation" page will automatically start generating the run tar file for download. Wait for it to complete (the "Download Zero-Beta Archive" button will appear).
    • Click the button to download the run tar ball. You are free to rename the file.
    • Also click the "Download Saveset" button to save the saveset tar ball (you are also free to rename this file).
    • The saveset tar file should be downloadable to the user.
    • The run tar ball should be sent to the GPU AWS instance (either through sending it to the /data drive or placing it in the /share drive which is accessible to the GPU instance.)
    • In this test case, I placed the run tar ball into a folder /data/psi/corhel_amcg_runs/<runtag>/
13. Once the run tar ball is on the AWS GPU instance,
    • Navigate to that directory (as the psi user) and launch the run with the command:
    nohup corunpy archive_file=<ARCHIVE_NAME>_zb.tar nprocs=8 num_procs_node=8 wc_limit=06:00:00 overwrite=1 > <ARCHIVE_NAME>_zb.log &
    • After launching the job, the <ARCHIVE_NAME>_zb.log should look like:

    CORUNPY is launching the run...
    Running CORHEL with command: corhel -cr_ut 2022-01-29T23:32 -res custom -name caplanr_at_predsci.com_20230208_10_zb_multi -model cme -cme_model zb -cme_nprocs 8 -cme_wc_limit 06:00:00 -cme_archive_dir /data/psi/corhel/run/ut202201292332-custom/caplanr_at_predsci.com_20230208_10_zb_multi_launch/archive_zb

    • After a brief setup time, the run should be seen to be running on the GPUs with the "nvidia-smi" command. For example:

    Processes:
    GPU	GI ID	CI ID	PID	Type	Process name	GPU Memory Usage
    0	N/A	N/A	21795	C	.../tools/ps_fortran/bin/mas	910MiB
    1	N/A	N/A	21796	C	.../tools/ps_fortran/bin/mas	910MiB
    2	N/A	N/A	21797	C	.../tools/ps_fortran/bin/mas	908MiB
    3	N/A	N/A	21798	C	.../tools/ps_fortran/bin/mas	908MiB
    4	N/A	N/A	21799	C	.../tools/ps_fortran/bin/mas	908MiB
    5	N/A	N/A	21800	C	.../tools/ps_fortran/bin/mas	908MiB
    6	N/A	N/A	21801	C	.../tools/ps_fortran/bin/mas	904MiB
    7	N/A	N/A	21802	C	.../tools/ps_fortran/bin/mas	904MiB

14. When the run is done, the <ARCHIVE_NAME>_zb.log file should contain something like:

    CORHEL run completed!
    Submission directory:
    /data/psi/corhel/run/ut202201292332-custom/caplanr_at_predsci.com_20230208_10_zb_multi_launch
    
    Simulation run directory:
    /data/psi/corhel/run/ut202201292332-custom/caplanr_at_predsci.com_20230208_10_zb_multi

    One could use the line "CORHEL run completed!" as a way of detecting when the run is completed, or one can check the process "corunpy" is complete.
15. The zero beta run report tar ball is located in the "Simulation run directory" listed in the log file. Copy the report tar ball to a location on the /share drive (possibly renaming it as needed). For example:

    cp /data/psi/corhel/run/ut202201292332-custom/caplanr_at_predsci.com_20230208_10_zb_multi/caplanr_at_predsci.com_20230208_10_zb_multi_zero_beta_report.tar.gz /share/psi/corhel_runs/

    This file needs to be unpacked and the html accessible for the user to view through the CCMC website.
16. Pack up the full simulation run directory into a tar file, and transfer it to S3.
17. Clean up the CORHEL run folder on /data (delete all folders in /data/psi/corhel/run/*)
18. After viewing the report, the user (likely) may want to change the rope and try again.
    • Go to the interface website and start a new daily session number (in this case, 2) but this time, select the saveset tar ball in the "Restore a previously saved session" section.
    • Click through step 1 again, (all previously selected parameters are already set), and modify the AR region if desired.
    • Go to step 2 as above, and make changes to the rope.
19. As before, at the end of step 2, download/store the new run tar ball, and download the new saveset tar ball (which the user should receive).
20. As before, run the new zero-beta simulation on the AWS GPU instance, transfer the report tar file to /share, provide the report to the user, save the simulation run data to S3, and then clean up the CORHEL run folder.
21. Repeat steps (18)-(22) as often as the user desires.

============= THERMODYNAMIC BACKGROUND =================================