Submit Grid Job » History » Version 20
Timo Eronen, 2016-12-08 12:53
1 | 1 | Timo Eronen | h1. Submit Grid Job |
---|---|---|---|
2 | 1 | Timo Eronen | |
3 | 12 | Timo Eronen | *1. Before you can submit Grid jobs you need to prepare your setup.* |
4 | 12 | Timo Eronen | |
5 | 12 | Timo Eronen | Finish first the steps described here: https://p55cc-redmine.utu.fi/projects/user-s-page/wiki/Prepare_For_Grid_Usage |
6 | 12 | Timo Eronen | |
7 | 12 | Timo Eronen | *2. Other preparation* |
8 | 12 | Timo Eronen | |
9 | 12 | Timo Eronen | You use the Grid resources via the ARC (Advanced Resource Connector) middleware, developed by the Nordugrid community. |
10 | 12 | Timo Eronen | |
11 | 12 | Timo Eronen | There are basically two ways to submit Grid job using ARC: |
12 | 12 | Timo Eronen | |
13 | 12 | Timo Eronen | # From ANY other computer but Pleione or Titan Login frontend |
14 | 12 | Timo Eronen | # From Pleione or Titan Login frontend. (i.e. pleione.utu.fi or titan.utu.fi) |
15 | 12 | Timo Eronen | |
16 | 12 | Timo Eronen | h3. 2.1 Using the Grid from other computers |
17 | 12 | Timo Eronen | |
18 | 12 | Timo Eronen | Using the Grid from other computers than Pleione or Titan you need to install ARC client software and configure it. Such tasks |
19 | 13 | Timo Eronen | are documented here: https://research.csc.fi/fgci-arc-middleware#1.3.2 |
20 | 12 | Timo Eronen | |
21 | 12 | Timo Eronen | h3. 2.2 Using the Grid from Pleione or Titan frontend |
22 | 12 | Timo Eronen | |
23 | 14 | Timo Eronen | The rest of this guide is just short version of this guide: https://research.csc.fi/fgci-using-arc-middleware |
24 | 14 | Timo Eronen | |
25 | 1 | Timo Eronen | So you might want to read the comprehensive guide before running the example. |
26 | 15 | Timo Eronen | |
27 | 19 | Timo Eronen | *3. Running self made binaries and FGCI software via runtime environment* |
28 | 1 | Timo Eronen | |
29 | 19 | Timo Eronen | In principle there are two ways to use the Grid resources: |
30 | 19 | Timo Eronen | - running self made binaries |
31 | 20 | Timo Eronen | - running FGCI pre-installed software via runtime environment |
32 | 19 | Timo Eronen | |
33 | 19 | Timo Eronen | *4. Example of running a simple self made binary as a Grid job from Titan login node* |
34 | 19 | Timo Eronen | |
35 | 20 | Timo Eronen | To run your self made binary as a Grid job you need the following files: |
36 | 16 | Timo Eronen | |
37 | 16 | Timo Eronen | - compiled program (the binary) |
38 | 16 | Timo Eronen | - job description file |
39 | 16 | Timo Eronen | - batch file |
40 | 16 | Timo Eronen | - possibly input file(s) |
41 | 16 | Timo Eronen | |
42 | 16 | Timo Eronen | The c-source of an example program *gtest.c* : |
43 | 16 | Timo Eronen | |
44 | 16 | Timo Eronen | <pre> |
45 | 16 | Timo Eronen | #include <stdio.h> |
46 | 16 | Timo Eronen | |
47 | 16 | Timo Eronen | int main(void) { |
48 | 16 | Timo Eronen | |
49 | 16 | Timo Eronen | char *line = NULL; |
50 | 16 | Timo Eronen | size_t size; |
51 | 16 | Timo Eronen | |
52 | 16 | Timo Eronen | printf("Hello UTU.\n"); |
53 | 16 | Timo Eronen | |
54 | 16 | Timo Eronen | while (getline(&line, &size, stdin) != -1) |
55 | 16 | Timo Eronen | printf("%s", line); |
56 | 16 | Timo Eronen | |
57 | 16 | Timo Eronen | return(0); |
58 | 16 | Timo Eronen | |
59 | 16 | Timo Eronen | } |
60 | 16 | Timo Eronen | </pre> |
61 | 16 | Timo Eronen | |
62 | 16 | Timo Eronen | Compile the source: |
63 | 16 | Timo Eronen | |
64 | 16 | Timo Eronen | <pre> |
65 | 16 | Timo Eronen | gcc -Wall -o gtest gtest.c |
66 | 16 | Timo Eronen | </pre> |
67 | 16 | Timo Eronen | |
68 | 16 | Timo Eronen | |
69 | 16 | Timo Eronen | The job description file *gtest.xrsl* : |
70 | 16 | Timo Eronen | |
71 | 16 | Timo Eronen | <pre> |
72 | 16 | Timo Eronen | &(executable=gtest.sh) |
73 | 16 | Timo Eronen | (jobname=g_test) |
74 | 1 | Timo Eronen | (runtimeenvironment>="ENV/FGCI") |
75 | 16 | Timo Eronen | (join="yes") |
76 | 16 | Timo Eronen | (stdout=std.out) |
77 | 16 | Timo Eronen | (cpuTime="1 hours") |
78 | 17 | Timo Eronen | (count="1") |
79 | 16 | Timo Eronen | (memory="1000") |
80 | 16 | Timo Eronen | (inputfiles= |
81 | 16 | Timo Eronen | ("gtest" "" ) |
82 | 16 | Timo Eronen | ("gtest.txt" "" ) |
83 | 16 | Timo Eronen | ) |
84 | 16 | Timo Eronen | (outputfiles= |
85 | 16 | Timo Eronen | ("gtest.tgz" "" ) |
86 | 16 | Timo Eronen | ) |
87 | 16 | Timo Eronen | </pre> |
88 | 16 | Timo Eronen | |
89 | 16 | Timo Eronen | The batch file *gtest.sh* : |
90 | 16 | Timo Eronen | |
91 | 16 | Timo Eronen | <pre> |
92 | 16 | Timo Eronen | #!/bin/sh |
93 | 16 | Timo Eronen | echo "Running gtest" |
94 | 16 | Timo Eronen | module load OpenMPI |
95 | 16 | Timo Eronen | chmod u+x gtest |
96 | 16 | Timo Eronen | mpirun ./gtest < gtest.txt > gtest.out |
97 | 16 | Timo Eronen | tar czf gtest.tgz gtest.out |
98 | 1 | Timo Eronen | echo "Done" |
99 | 16 | Timo Eronen | exit 0 |
100 | 16 | Timo Eronen | </pre> |
101 | 16 | Timo Eronen | |
102 | 16 | Timo Eronen | |
103 | 17 | Timo Eronen | Input file for this test run *gtest.txt* : |
104 | 16 | Timo Eronen | |
105 | 1 | Timo Eronen | <pre> |
106 | 1 | Timo Eronen | |
107 | 16 | Timo Eronen | Hello FGCI. |
108 | 16 | Timo Eronen | |
109 | 16 | Timo Eronen | </pre> |
110 | 16 | Timo Eronen | |
111 | 17 | Timo Eronen | *Now you are ready to run the job.* |
112 | 17 | Timo Eronen | |
113 | 16 | Timo Eronen | Get proxy to run the job: |
114 | 16 | Timo Eronen | |
115 | 16 | Timo Eronen | <pre> |
116 | 16 | Timo Eronen | |
117 | 16 | Timo Eronen | $ arcproxy |
118 | 16 | Timo Eronen | |
119 | 16 | Timo Eronen | Enter pass phrase for private key: |
120 | 16 | Timo Eronen | |
121 | 16 | Timo Eronen | Your identity: /DC=org/DC=terena/DC=tcs/C=FI/O=Turun yliopisto/CN=Timo Eronen tke@utu.fi |
122 | 16 | Timo Eronen | Proxy generation succeeded |
123 | 16 | Timo Eronen | Your proxy is valid until: 2016-12-08 19:46:02 |
124 | 16 | Timo Eronen | </pre> |
125 | 16 | Timo Eronen | |
126 | 16 | Timo Eronen | Run the job: |
127 | 16 | Timo Eronen | |
128 | 16 | Timo Eronen | <pre> |
129 | 16 | Timo Eronen | $ arcsub gtest.xrsl |
130 | 16 | Timo Eronen | |
131 | 16 | Timo Eronen | Job submitted with jobid: gsiftp://io-grid.fgci.csc.fi:2811/jobs/3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm |
132 | 16 | Timo Eronen | </pre> |
133 | 16 | Timo Eronen | |
134 | 16 | Timo Eronen | Query status of the job: |
135 | 16 | Timo Eronen | |
136 | 16 | Timo Eronen | <pre> |
137 | 16 | Timo Eronen | $ arcstat gsiftp://io-grid.fgci.csc.fi:2811/jobs/3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm |
138 | 16 | Timo Eronen | |
139 | 16 | Timo Eronen | Job: gsiftp://io-grid.fgci.csc.fi:2811/jobs/3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm |
140 | 16 | Timo Eronen | Name: g_test |
141 | 16 | Timo Eronen | State: Finishing |
142 | 16 | Timo Eronen | |
143 | 16 | Timo Eronen | Status of 1 jobs was queried, 1 jobs returned information |
144 | 1 | Timo Eronen | </pre> |
145 | 16 | Timo Eronen | |
146 | 16 | Timo Eronen | Job was not yet finished but after awhile: |
147 | 16 | Timo Eronen | |
148 | 16 | Timo Eronen | <pre> |
149 | 16 | Timo Eronen | $ arcstat gsiftp://io-grid.fgci.csc.fi:2811/jobs/3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm |
150 | 17 | Timo Eronen | |
151 | 16 | Timo Eronen | Job: gsiftp://io-grid.fgci.csc.fi:2811/jobs/3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm |
152 | 16 | Timo Eronen | Name: g_test |
153 | 16 | Timo Eronen | State: Finished |
154 | 16 | Timo Eronen | Exit Code: 0 |
155 | 16 | Timo Eronen | |
156 | 16 | Timo Eronen | Status of 1 jobs was queried, 1 jobs returned information |
157 | 16 | Timo Eronen | </pre> |
158 | 16 | Timo Eronen | |
159 | 16 | Timo Eronen | Get the results: |
160 | 16 | Timo Eronen | |
161 | 1 | Timo Eronen | <pre> |
162 | 16 | Timo Eronen | $ arcget gsiftp://io-grid.fgci.csc.fi:2811/jobs/3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm |
163 | 16 | Timo Eronen | |
164 | 16 | Timo Eronen | Results stored at: 3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm |
165 | 16 | Timo Eronen | Jobs processed: 1, successfully retrieved: 1, successfully cleaned: 1 |
166 | 16 | Timo Eronen | </pre> |
167 | 16 | Timo Eronen | |
168 | 18 | Timo Eronen | And then you are ready to view the results: |
169 | 16 | Timo Eronen | |
170 | 16 | Timo Eronen | <pre> |
171 | 1 | Timo Eronen | $ ls -l 3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm/ |
172 | 1 | Timo Eronen | total 8 |
173 | 16 | Timo Eronen | -rw------- 1 tke admin 148 Dec 8 09:14 gtest.tgz |
174 | 16 | Timo Eronen | -rw------- 1 tke admin 19 Dec 8 09:14 std.out |
175 | 16 | Timo Eronen | |
176 | 17 | Timo Eronen | $ cd 3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm/ |
177 | 16 | Timo Eronen | |
178 | 16 | Timo Eronen | $ cat std.out |
179 | 16 | Timo Eronen | Running gtest |
180 | 16 | Timo Eronen | Done |
181 | 17 | Timo Eronen | |
182 | 17 | Timo Eronen | $ tar xf gtest.tgz |
183 | 16 | Timo Eronen | |
184 | 16 | Timo Eronen | $ cat gtest.out |
185 | 16 | Timo Eronen | |
186 | 16 | Timo Eronen | Hello UTU. |
187 | 1 | Timo Eronen | |
188 | 1 | Timo Eronen | Hello FGCI. |
189 | 1 | Timo Eronen | </pre> |
190 | 18 | Timo Eronen | |
191 | 19 | Timo Eronen | *5. Example of running a Grid job via FGCI runtime environments from Titan login node* |
192 | 18 | Timo Eronen | |
193 | 18 | Timo Eronen | Comprehensive info: |
194 | 18 | Timo Eronen | |
195 | 18 | Timo Eronen | https://research.csc.fi/fgci-using-software-through-runtime-environments |
196 | 18 | Timo Eronen | |
197 | 18 | Timo Eronen | [will be added later] |