WEBVTT 1 00:00:16.720 --> 00:00:19.919 Shreyasi Acharya: Your microphone is muted. We cannot hear you. 2 00:00:22.030 --> 00:00:26.750 Deepak Kar: Okay. Yeah. Yeah. Sorry. Okay. So all good. 3 00:00:28.620 --> 00:00:29.370 Shreyasi Acharya: It's perfect. 4 00:00:29.370 --> 00:00:46.049 Deepak Kar: Okay, amazing, amazing. So, thanks for this workshop. I mean, I didn't manage to attend last 2 days. It was very late for us. But but yeah, I learned a lot from the past 2 talks. So that's great. So now we travel from France to South Africa. 5 00:00:46.730 --> 00:01:06.880 Deepak Kar: saving a lot of carbon, I guess. And yeah, I mean South Africa. We are so far away we cannot use trains to go anywhere. We have to fly, unfortunately. But anyway. So yeah, we'll also move from more exciting biodiversity to more mundane computing stuff. 6 00:01:08.003 --> 00:01:09.750 Deepak Kar: Yeah. Now. 7 00:01:10.210 --> 00:01:23.899 Deepak Kar: I'm no expert. I mean, this is, in fact, the 1st time I'm presenting in a meeting like this. So if I use any term not in the way more insiders use adverse apologies now for me, and I feel 8 00:01:24.160 --> 00:01:36.789 Deepak Kar: sustainability means different things to different people. I mean, of course, there are common broad tips, but then more fine grain meaning is probably different in this talk. I'm just going to focus on computing aspects 9 00:01:36.970 --> 00:01:51.989 Deepak Kar: more specifically on our South African colleagues and students. So we made a survey circulated among the community. We got 15 responses, not as many as I wanted. The community is probably twice as large. 10 00:01:52.370 --> 00:01:53.610 Deepak Kar: but also 11 00:01:54.060 --> 00:02:15.210 Deepak Kar: I think it's a fair representation of the community in the sense that we have a lot of master students Phd students. And then so yeah, as you can see on the right, I mean, it's it's kind of we. We got a fair representation from the community. So that's good. Also, I mean, we might think computing does not depend on the country. 12 00:02:15.520 --> 00:02:27.000 Deepak Kar: It's not, I mean, I was a postdoc in Europe for many years, and whether that's the best way of doing things or not, I would just Ssh into Lx plus 13 00:02:27.800 --> 00:02:55.509 Deepak Kar: do. X 11, forwarding. Look at a plot directly, and edit things and look at a plot directly again. When I moved to South Africa. I realized that doesn't work. X 11. Forwarding does not work. The latency is too much. The upload speed is slow. So I had to install root locally. All my students, all my colleagues, I know, do that again. Not a big deal, but just to say, like how your workflow changes, depending on where you are 14 00:02:56.070 --> 00:02:57.140 Deepak Kar: now. 15 00:02:57.800 --> 00:03:04.960 Deepak Kar: We started by asking them what are the biggest problems you face, and I kind of alluded to that. It's essentially 16 00:03:05.270 --> 00:03:13.300 Deepak Kar: low connection, speed and that affects in many ways. If you're running stuff on Lx plus. And 17 00:03:13.820 --> 00:03:18.550 Deepak Kar: the connection gets like, we drop the connection, and then 18 00:03:18.600 --> 00:03:37.869 Deepak Kar: you have to rerun the thing again. Which is not great, as any of us who is working on these things would know. Also, I mean we we don't really have a lot of local clusters and grids. I mean, I know I get. I get a lot of Flack from for saying that. But 19 00:03:37.870 --> 00:03:48.950 Deepak Kar: I mean, the ones we have are incredibly hard to use or get access to and stuff like that. Yeah. So that's probably that. Now 20 00:03:49.010 --> 00:04:03.490 Deepak Kar: I was like, why, that's the case. I mean, Internet is Internet. And I did not know this before I moved to South Africa. So maybe all of you know this, and this is completely redundant. But I found that it is like interesting that our Internet essentially is 21 00:04:03.920 --> 00:04:06.130 Deepak Kar: by undersea cables. 22 00:04:06.320 --> 00:04:18.259 Deepak Kar: quite a few of them, but they are just cables, big bad cables, and whenever a ship or some like ramps into them, and just like breaks a cable, and that apparently happens like 23 00:04:18.399 --> 00:04:20.200 Deepak Kar: more frequently than we know. 24 00:04:20.370 --> 00:04:21.730 Deepak Kar: Our Internet 25 00:04:21.970 --> 00:04:41.940 Deepak Kar: is bad. I mean, there are some times when, like countrywide, Internet is bad for a few days, because one of those cables snapped. I mean, this is this is something we didn't realize before moving to this country and trying to do science. So that's again. Probably everyone knows this. But I don't. Okay. So now I'm going to kind of divide 26 00:04:42.469 --> 00:04:46.069 Deepak Kar: the topic in 3 subtopics in the sense like. 27 00:04:46.300 --> 00:04:49.340 Deepak Kar: what are the good computing practices. 28 00:04:49.610 --> 00:04:53.879 Deepak Kar: So first, st of course, is awareness. Do people know? I mean 29 00:04:54.770 --> 00:05:05.680 Deepak Kar: all all the things we should do? And then are they using that that knowledge so that I'll combine those 2 and then we'll go into. How can we address that? What's the way forward? 30 00:05:05.800 --> 00:05:23.280 Deepak Kar: So I know this is slightly Atlas Lhc specific. But big Panda is essentially our grid, monitoring thing. So we have computing grid, like most of us know, and Atlas has done something very interesting in. I don't say super recent. I think this is a few years old. 31 00:05:23.460 --> 00:05:33.699 Deepak Kar: that when you run a great job it shows your carbon footprint, I mean, of course, as I learned, it's not the exact amount. It's an average amount depending on CPU consumption and whatnot. But 32 00:05:34.200 --> 00:05:38.770 Deepak Kar: it's a number that shows up at that monitoring page, and 33 00:05:39.120 --> 00:05:59.020 Deepak Kar: 73% of people who responded to the survey said they actually aware of it. So that's good. At least they know that they are. There is a carbon footprint in running these great jobs which is relevant for other things, because, if we can run these jobs more optimized way. And then it kind of fits into the fact that. 34 00:05:59.350 --> 00:06:26.779 Deepak Kar: do you know who to ask when you get stuck? Because if you don't, then sometimes you just keep running. The same failed Job again and again, which is not optimal for anything, not for your size, not for your time management, not for carbon footprint. Right? So again a large fraction said, yes, probably the same fraction. So that was good. That made me very happy that whether they reach out to them is, I don't know, but at least they know 35 00:06:28.390 --> 00:06:45.309 Deepak Kar: also Atlas Grid, and I think this is true, for most of the collaborations have a feature where, when you submit a job, a small fraction of jobs run and try to see if the jobs would succeed in terms of memory usage in terms of like finding resources and whatnot. 36 00:06:45.500 --> 00:06:53.439 Deepak Kar: and if those jobs fail, then the rest of the jobs don't run, which is a good thing, which is again, like a good computing practice, and again. 37 00:06:53.920 --> 00:07:04.670 Deepak Kar: a decent fraction like 2 thirds essentially said, they know that that it is a thing. So so again, I think that was definitely a positive. 38 00:07:05.360 --> 00:07:07.369 Deepak Kar: Now we ask them something 39 00:07:08.063 --> 00:07:27.679 Deepak Kar: just to check like what what we do and and I think, as at least as heavy experimentalist. We all have been there. You submit a large number of good jobs before going to bed, and next morning you wake up. Everything is red. Everything is a disaster, and you cry but 40 00:07:27.860 --> 00:07:39.189 Deepak Kar: and then I think the point is to actually try to figure out why they all failed, or you just like. No, it must be something else. And you resubmit everything again. Again. 41 00:07:39.360 --> 00:07:46.500 Deepak Kar: half of more than half essentially said, they actually try to debug rather than blindly submitting. But 42 00:07:47.030 --> 00:07:55.389 Deepak Kar: some some of them also said, They don't. So I think we need to reach out to everyone and see if we can optimize that. That's part of the training. 43 00:07:56.530 --> 00:08:05.280 Deepak Kar: And this is something I learned actually, probably last year that you can visualize contents of a root file. So that means you don't have to download the root, file 44 00:08:05.480 --> 00:08:20.670 Deepak Kar: locally, open it and say it. And then I mean, sometimes you need to do that for your analysis. Sometimes you just want to check quickly if it's empty or a specific branch exists, or or something real quickly, you don't need to do all of that again. 2 3rd said, yes. 45 00:08:20.810 --> 00:08:23.179 Deepak Kar: so all of this is great. 46 00:08:23.310 --> 00:08:29.989 Deepak Kar: Then we ask them, Have you run jobs in Lx plus, which is a login node where you should not be running long jobs 47 00:08:30.130 --> 00:08:42.160 Deepak Kar: 20, like, yeah. And then I think 40% said, Yes, which is a bit concerning, and that ties to the connectivity issue right? Because if you don't do that, it's probably 48 00:08:42.510 --> 00:08:53.840 Deepak Kar: like your. The electron would probably drop unless you're very, very lucky. And then you are trying to rerun everything again. If it's a long job, you're wasting a lot of resources. So that's not great. 49 00:08:55.970 --> 00:09:07.790 Deepak Kar: So what's my overall takeaway from this part. Considering that we have a significant fraction of younger students, I think it's encouraging that they're mostly aware of these issues. 50 00:09:08.350 --> 00:09:35.060 Deepak Kar: I also think because I know the community reasonably well. It's a lot of peer to peer acquired knowledge, which is also great. But the drawback might be that it's somewhat of a closed ecosystem. I mean, if I use a computing practice which is not great, and I'm sure I do, then I pass it on to other people. They use the same thing, and we don't post correct. So that's not great. 51 00:09:35.260 --> 00:09:37.100 Deepak Kar: Which brings us to Prairie. 52 00:09:37.100 --> 00:09:38.340 Greg Hallewell: I know what to do. 53 00:09:39.770 --> 00:09:41.429 Deepak Kar: I'm sorry. Is that a question? 54 00:09:44.870 --> 00:09:47.970 Deepak Kar: Okay, I don't know, anyway. Right. 55 00:09:48.110 --> 00:09:52.599 Deepak Kar: I mean, we can discuss all of these so 56 00:09:53.750 --> 00:10:12.670 Deepak Kar: atlas again, I think it's the same for all the big collaborations we have what we call an induction day, where the new students or collaborators joining are kind of introduced to the, to the collaboration. And there's a computing tutorial which goes on, I believe, for 4 days. So that's super helpful, very exhaustive. 57 00:10:12.800 --> 00:10:18.920 Deepak Kar: And we ask people, especially for people who are native newcomers. Have you done those? 58 00:10:19.040 --> 00:10:21.810 Deepak Kar: And turns out only 60% have done that. 59 00:10:22.060 --> 00:10:45.960 Deepak Kar: So that's again, I mean, that's a good number, but it should ideally be close to 100. If you are joining the collaboration, I think going through these is super useful. So that's one thing which kind of bothered me, and then I asked them, Are you part of an ongoing? Because maybe some some of them are engineers? I I mean, I don't know the identity of the people who replied, so maybe they don't really need to know all of these things, but 60 00:10:46.090 --> 00:10:48.539 Deepak Kar: almost 90%, I believe, said 61 00:10:48.940 --> 00:11:02.449 Deepak Kar: they are involved in a close analysis. So they must do the induction and the tutorial. So that's 1 thing which came out, which is, I think, is a good input for the community. And then I asked them to use grid. 62 00:11:03.030 --> 00:11:20.779 Deepak Kar: I mean, anyone doing analysis should. And they do. So that's good. And okay, this is more of me. Thingy. So when you are doing an analysis in Atlas, or I guess anywhere else, you request a signal sample like a simulator signal sample now 63 00:11:21.530 --> 00:11:38.609 Deepak Kar: and then. Of course, the way is, you make some plots and validate, and essentially to know the signal sample actually does what it should do. So can we get a million of those events? Now, there is software called Rivet. Some of you might be aware of it. It's it's kind of like, 64 00:11:39.160 --> 00:11:57.449 Deepak Kar: like, yeah, there is an Mc. That collaboration which is still there. It's them and Cedar and John Butterworth. So yeah, bunch of people. I work very closely with them, and I think rivet is great. This is likely advertisement for rivet, for analyzing particle level simulation. So if you haven't done it, look it up. 65 00:11:57.490 --> 00:12:15.679 Deepak Kar: and I find it great, but large fraction in the community apparently uses, like the whole Atlas analysis chain, which is fine, which is not incorrect. But I, as I said, I personally feel a bit let down, so I was trying to see what I can do, and sometimes you end up with files in tear. 66 00:12:15.870 --> 00:12:31.819 Deepak Kar: which is like a more static storage. Do you know, I asked them. Do you know what to do it most of them said, no, we use something called R. 2D. 2, very creatively named to retrieve the files and run over it. So again. That's important. 67 00:12:31.950 --> 00:12:38.135 Deepak Kar: So and then we asked at the end, what are the softwares used in your laptop? Of course, root and Paiute 68 00:12:38.730 --> 00:12:43.870 Deepak Kar: are like the dominant ones, but also people use generators like Pythia and Madgrav 69 00:12:43.980 --> 00:12:46.909 Deepak Kar: revert is used by quite a few, not enough. 70 00:12:47.516 --> 00:12:58.669 Deepak Kar: Hopefully, the one uses Gn. 4, and someone wanted to use the Atlas pro like the whole Athena package, which is Atlas software on their Mac, which I don't think is recommended, which is, I don't think is optimal. 71 00:12:58.880 --> 00:13:24.880 Deepak Kar: And then very quickly, am I running out of time? No, I have a couple of minutes. So if you're running machine learning where to run it, most of them, or 1 3rd of them run it on their laptop. And 1 3rd is in like online platforms, which is okay. I think that's an important data point. Then we ask them, do you want to get involved in Atlas computing development, which I think that's useful? Most of them said No. 72 00:13:25.650 --> 00:13:32.260 Deepak Kar: and then but then they use Gpus in. Okay, that's my last slide. So 73 00:13:32.890 --> 00:13:48.639 Deepak Kar: before the pandemic, we actually used to have, like a community workshop where all the Atlas people would gather. No, not just Atlas Atlas, at least we have Atlas and Alice all the Atlas, and at least people would gather students would present would have tutorials and stuff. 74 00:13:48.850 --> 00:14:15.500 Deepak Kar: It stopped during pandemic. Unfortunately, due to many reasons. I'm not going to go into that. We never restarted it, which is a pity. But I think, looking at these things, we we realize that there is a large scope of like training for younger students, because that's our. That's a huge part of our community. So we should organize local training events. Because many of these things can be optimized. And then I just want to end by saying that. 75 00:14:16.120 --> 00:14:38.499 Deepak Kar: as I, as I kind of said at the beginning, I mean local realities play a big role. I mean, we are all sound collaborators. We all have the same resources. But whether people know that resources exist and this is not just distance, I think it's also like the local culture lack of training or lack of it. All of this is important. So I mean, I'm very happy. I went through this this 76 00:14:38.610 --> 00:14:52.739 Deepak Kar: kind of hopefully, we can find some ways in the community to improve the culture and maybe similar exercises in other countries. Other smaller countries, I would say the developing countries would be useful. But that's all I have. Thank you very much. 77 00:14:55.540 --> 00:15:03.820 Shreyasi Acharya: It was really clear, and it was really nice to know the efforts specifically in Atlas Chandra modeling. 78 00:15:04.030 --> 00:15:04.780 Shreyasi Acharya: Please. 79 00:15:05.640 --> 00:15:06.869 Chandramauli: Am I audible? 80 00:15:08.060 --> 00:15:08.770 Shreyasi Acharya: Yes. 81 00:15:08.770 --> 00:15:10.479 Chandramauli: Hi, thank you for the talk. 82 00:15:10.660 --> 00:15:18.570 Chandramauli: So seeing the experiences that you've had would you say that? Is it more or less easier to 83 00:15:18.720 --> 00:15:21.550 Chandramauli: switch to more greener practices for a 84 00:15:22.025 --> 00:15:26.770 Chandramauli: group group that is just starting, such as in South Africa. 85 00:15:27.615 --> 00:15:36.000 Chandramauli: Because I I think that well established group might have be habitual to using more non linear methods. 86 00:15:36.596 --> 00:15:48.779 Chandramauli: And and as you said, that newer groups might have not access to these practices, these more eco-friendly practices. So if you were to say, What do you think? 87 00:15:48.900 --> 00:15:51.110 Chandramauli: Is it more or less easier. 88 00:15:51.830 --> 00:16:02.340 Deepak Kar: Thanks for that question. So I think I'll make. I've made clear 2 points which one of them I should have mentioned that South Africa. Unfortunately, our energy, I mean. 89 00:16:02.490 --> 00:16:04.919 Deepak Kar: is mostly cold 90 00:16:05.367 --> 00:16:22.600 Deepak Kar: which is not great, but, as you can imagine, we have 0 control over it. In fact, people who know South Africa would know that we have. We had like a lot of energy crises like huge power cuts. In last few years. It has become better in last few years. So and also the group, I mean. 91 00:16:22.600 --> 00:16:42.009 Deepak Kar: the community is small, and there is a lot of turnover in terms of students, because we have a lot of math students who just do masters, a lot of students who just do Phds and Masters and Phds and stuff. But the group actually started in 2,008 9 ish, even, maybe earlier. So I mean, we have a long culture. It's just 92 00:16:42.040 --> 00:17:03.159 Deepak Kar: I think it has grown in last 3 years. But yeah, coming back to your question, I think, yeah, because there is a large turnover and a large fraction of younger people, we can actually if we do these trainings properly, I think that would be one of the focus of me and a few of my our colleagues. Then we can. We can actually 93 00:17:03.200 --> 00:17:13.049 Deepak Kar: do some of these things. We can tell younger kids that do this. Not that, at least in our small ways, or we can help. So yeah, thanks. 94 00:17:18.550 --> 00:17:19.890 Shreyasi Acharya: Thank you very much. 95 00:17:21.170 --> 00:17:26.280 Shreyasi Acharya: I see no more questions. And you're also 96 00:17:26.400 --> 00:17:32.020 Shreyasi Acharya: on time. So thank you very much. Deepak, for sharing this talk with us. 97 00:17:32.100 --> 00:18:01.860 Shreyasi Acharya: So now it's time for 15 min break, so we will try to reconvene at in 15 min. So 1010 CST, just bear in mind that we are potentially on time, because there is, one talk which will not happen because the person is not feeling well. So yeah, please take a coffee or take a break, and we reconvene in 15 min.