Submit Grid Job » History » Version 18
Timo Eronen, 2016-12-08 12:46
1 | 1 | Timo Eronen | h1. Submit Grid Job |
---|---|---|---|
2 | 1 | Timo Eronen | |
3 | 12 | Timo Eronen | *1. Before you can submit Grid jobs you need to prepare your setup.* |
4 | 12 | Timo Eronen | |
5 | 12 | Timo Eronen | Finish first the steps described here: https://p55cc-redmine.utu.fi/projects/user-s-page/wiki/Prepare_For_Grid_Usage |
6 | 12 | Timo Eronen | |
7 | 12 | Timo Eronen | *2. Other preparation* |
8 | 12 | Timo Eronen | |
9 | 12 | Timo Eronen | You use the Grid resources via the ARC (Advanced Resource Connector) middleware, developed by the Nordugrid community. |
10 | 12 | Timo Eronen | |
11 | 12 | Timo Eronen | There are basically two ways to submit Grid job using ARC: |
12 | 12 | Timo Eronen | |
13 | 12 | Timo Eronen | # From ANY other computer but Pleione or Titan Login frontend |
14 | 12 | Timo Eronen | # From Pleione or Titan Login frontend. (i.e. pleione.utu.fi or titan.utu.fi) |
15 | 12 | Timo Eronen | |
16 | 12 | Timo Eronen | h3. 2.1 Using the Grid from other computers |
17 | 12 | Timo Eronen | |
18 | 12 | Timo Eronen | Using the Grid from other computers than Pleione or Titan you need to install ARC client software and configure it. Such tasks |
19 | 13 | Timo Eronen | are documented here: https://research.csc.fi/fgci-arc-middleware#1.3.2 |
20 | 12 | Timo Eronen | |
21 | 12 | Timo Eronen | h3. 2.2 Using the Grid from Pleione or Titan frontend |
22 | 12 | Timo Eronen | |
23 | 14 | Timo Eronen | The rest of this guide is just short version of this guide: https://research.csc.fi/fgci-using-arc-middleware |
24 | 14 | Timo Eronen | |
25 | 1 | Timo Eronen | So you might want to read the comprehensive guide before running the example. |
26 | 15 | Timo Eronen | |
27 | 18 | Timo Eronen | *3. Example of running a simple self made binary as a Grid job from Titan login node* |
28 | 15 | Timo Eronen | |
29 | 17 | Timo Eronen | To run your binary as a Grid job you need the following files: |
30 | 16 | Timo Eronen | |
31 | 16 | Timo Eronen | - compiled program (the binary) |
32 | 16 | Timo Eronen | - job description file |
33 | 16 | Timo Eronen | - batch file |
34 | 16 | Timo Eronen | - possibly input file(s) |
35 | 16 | Timo Eronen | |
36 | 16 | Timo Eronen | |
37 | 16 | Timo Eronen | The c-source of an example program *gtest.c* : |
38 | 16 | Timo Eronen | |
39 | 16 | Timo Eronen | <pre> |
40 | 16 | Timo Eronen | #include <stdio.h> |
41 | 16 | Timo Eronen | |
42 | 16 | Timo Eronen | int main(void) { |
43 | 16 | Timo Eronen | |
44 | 16 | Timo Eronen | char *line = NULL; |
45 | 16 | Timo Eronen | size_t size; |
46 | 16 | Timo Eronen | |
47 | 16 | Timo Eronen | printf("Hello UTU.\n"); |
48 | 16 | Timo Eronen | |
49 | 16 | Timo Eronen | while (getline(&line, &size, stdin) != -1) |
50 | 16 | Timo Eronen | printf("%s", line); |
51 | 16 | Timo Eronen | |
52 | 16 | Timo Eronen | return(0); |
53 | 16 | Timo Eronen | |
54 | 16 | Timo Eronen | } |
55 | 16 | Timo Eronen | </pre> |
56 | 16 | Timo Eronen | |
57 | 16 | Timo Eronen | Compile the source: |
58 | 16 | Timo Eronen | |
59 | 16 | Timo Eronen | <pre> |
60 | 16 | Timo Eronen | gcc -Wall -o gtest gtest.c |
61 | 16 | Timo Eronen | </pre> |
62 | 16 | Timo Eronen | |
63 | 16 | Timo Eronen | |
64 | 16 | Timo Eronen | The job description file *gtest.xrsl* : |
65 | 16 | Timo Eronen | |
66 | 16 | Timo Eronen | <pre> |
67 | 16 | Timo Eronen | &(executable=gtest.sh) |
68 | 16 | Timo Eronen | (jobname=g_test) |
69 | 1 | Timo Eronen | (runtimeenvironment>="ENV/FGCI") |
70 | 16 | Timo Eronen | (join="yes") |
71 | 16 | Timo Eronen | (stdout=std.out) |
72 | 16 | Timo Eronen | (cpuTime="1 hours") |
73 | 17 | Timo Eronen | (count="1") |
74 | 16 | Timo Eronen | (memory="1000") |
75 | 16 | Timo Eronen | (inputfiles= |
76 | 16 | Timo Eronen | ("gtest" "" ) |
77 | 16 | Timo Eronen | ("gtest.txt" "" ) |
78 | 16 | Timo Eronen | ) |
79 | 16 | Timo Eronen | (outputfiles= |
80 | 16 | Timo Eronen | ("gtest.tgz" "" ) |
81 | 16 | Timo Eronen | ) |
82 | 16 | Timo Eronen | </pre> |
83 | 16 | Timo Eronen | |
84 | 16 | Timo Eronen | The batch file *gtest.sh* : |
85 | 16 | Timo Eronen | |
86 | 16 | Timo Eronen | <pre> |
87 | 16 | Timo Eronen | #!/bin/sh |
88 | 16 | Timo Eronen | echo "Running gtest" |
89 | 16 | Timo Eronen | module load OpenMPI |
90 | 16 | Timo Eronen | chmod u+x gtest |
91 | 16 | Timo Eronen | mpirun ./gtest < gtest.txt > gtest.out |
92 | 16 | Timo Eronen | tar czf gtest.tgz gtest.out |
93 | 1 | Timo Eronen | echo "Done" |
94 | 16 | Timo Eronen | exit 0 |
95 | 16 | Timo Eronen | </pre> |
96 | 16 | Timo Eronen | |
97 | 16 | Timo Eronen | |
98 | 17 | Timo Eronen | Input file for this test run *gtest.txt* : |
99 | 16 | Timo Eronen | |
100 | 1 | Timo Eronen | <pre> |
101 | 1 | Timo Eronen | |
102 | 16 | Timo Eronen | Hello FGCI. |
103 | 16 | Timo Eronen | |
104 | 16 | Timo Eronen | </pre> |
105 | 16 | Timo Eronen | |
106 | 17 | Timo Eronen | *Now you are ready to run the job.* |
107 | 17 | Timo Eronen | |
108 | 16 | Timo Eronen | Get proxy to run the job: |
109 | 16 | Timo Eronen | |
110 | 16 | Timo Eronen | <pre> |
111 | 16 | Timo Eronen | |
112 | 16 | Timo Eronen | $ arcproxy |
113 | 16 | Timo Eronen | |
114 | 16 | Timo Eronen | Enter pass phrase for private key: |
115 | 16 | Timo Eronen | |
116 | 16 | Timo Eronen | Your identity: /DC=org/DC=terena/DC=tcs/C=FI/O=Turun yliopisto/CN=Timo Eronen tke@utu.fi |
117 | 16 | Timo Eronen | Proxy generation succeeded |
118 | 16 | Timo Eronen | Your proxy is valid until: 2016-12-08 19:46:02 |
119 | 16 | Timo Eronen | </pre> |
120 | 16 | Timo Eronen | |
121 | 16 | Timo Eronen | Run the job: |
122 | 16 | Timo Eronen | |
123 | 16 | Timo Eronen | <pre> |
124 | 16 | Timo Eronen | $ arcsub gtest.xrsl |
125 | 16 | Timo Eronen | |
126 | 16 | Timo Eronen | Job submitted with jobid: gsiftp://io-grid.fgci.csc.fi:2811/jobs/3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm |
127 | 16 | Timo Eronen | </pre> |
128 | 16 | Timo Eronen | |
129 | 16 | Timo Eronen | Query status of the job: |
130 | 16 | Timo Eronen | |
131 | 16 | Timo Eronen | <pre> |
132 | 16 | Timo Eronen | $ arcstat gsiftp://io-grid.fgci.csc.fi:2811/jobs/3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm |
133 | 16 | Timo Eronen | |
134 | 16 | Timo Eronen | Job: gsiftp://io-grid.fgci.csc.fi:2811/jobs/3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm |
135 | 16 | Timo Eronen | Name: g_test |
136 | 16 | Timo Eronen | State: Finishing |
137 | 16 | Timo Eronen | |
138 | 16 | Timo Eronen | Status of 1 jobs was queried, 1 jobs returned information |
139 | 1 | Timo Eronen | </pre> |
140 | 16 | Timo Eronen | |
141 | 16 | Timo Eronen | Job was not yet finished but after awhile: |
142 | 16 | Timo Eronen | |
143 | 16 | Timo Eronen | <pre> |
144 | 16 | Timo Eronen | $ arcstat gsiftp://io-grid.fgci.csc.fi:2811/jobs/3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm |
145 | 17 | Timo Eronen | |
146 | 16 | Timo Eronen | Job: gsiftp://io-grid.fgci.csc.fi:2811/jobs/3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm |
147 | 16 | Timo Eronen | Name: g_test |
148 | 16 | Timo Eronen | State: Finished |
149 | 16 | Timo Eronen | Exit Code: 0 |
150 | 16 | Timo Eronen | |
151 | 16 | Timo Eronen | Status of 1 jobs was queried, 1 jobs returned information |
152 | 16 | Timo Eronen | </pre> |
153 | 16 | Timo Eronen | |
154 | 16 | Timo Eronen | Get the results: |
155 | 16 | Timo Eronen | |
156 | 1 | Timo Eronen | <pre> |
157 | 16 | Timo Eronen | $ arcget gsiftp://io-grid.fgci.csc.fi:2811/jobs/3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm |
158 | 16 | Timo Eronen | |
159 | 16 | Timo Eronen | Results stored at: 3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm |
160 | 16 | Timo Eronen | Jobs processed: 1, successfully retrieved: 1, successfully cleaned: 1 |
161 | 16 | Timo Eronen | </pre> |
162 | 16 | Timo Eronen | |
163 | 18 | Timo Eronen | And then you are ready to view the results: |
164 | 16 | Timo Eronen | |
165 | 16 | Timo Eronen | <pre> |
166 | 1 | Timo Eronen | $ ls -l 3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm/ |
167 | 1 | Timo Eronen | total 8 |
168 | 16 | Timo Eronen | -rw------- 1 tke admin 148 Dec 8 09:14 gtest.tgz |
169 | 16 | Timo Eronen | -rw------- 1 tke admin 19 Dec 8 09:14 std.out |
170 | 16 | Timo Eronen | |
171 | 17 | Timo Eronen | $ cd 3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm/ |
172 | 16 | Timo Eronen | |
173 | 16 | Timo Eronen | $ cat std.out |
174 | 16 | Timo Eronen | Running gtest |
175 | 16 | Timo Eronen | Done |
176 | 17 | Timo Eronen | |
177 | 17 | Timo Eronen | $ tar xf gtest.tgz |
178 | 16 | Timo Eronen | |
179 | 16 | Timo Eronen | $ cat gtest.out |
180 | 16 | Timo Eronen | |
181 | 16 | Timo Eronen | Hello UTU. |
182 | 16 | Timo Eronen | |
183 | 1 | Timo Eronen | Hello FGCI. |
184 | 1 | Timo Eronen | </pre> |
185 | 18 | Timo Eronen | |
186 | 18 | Timo Eronen | *4. Example of running a Grid job via FGCI runtime environments from Titan login node* |
187 | 18 | Timo Eronen | |
188 | 18 | Timo Eronen | Comprehensive info: |
189 | 18 | Timo Eronen | |
190 | 18 | Timo Eronen | https://research.csc.fi/fgci-using-software-through-runtime-environments |
191 | 18 | Timo Eronen | |
192 | 18 | Timo Eronen | [will be added later] |