WEBVTT 1 00:00:00.000 --> 00:00:01.000 Let me see my slides, full screen and repair me well. 2 00:00:01.000 --> 00:00:04.000 Just perfect you can go ahead. 3 00:00:04.000 --> 00:00:18.000 Yes, I'm sold. So first of all, good morning to everyone. My name is Saul Adam and I'm going to talk about how all these people specified on this slide about That's an ability to start this big data process in real time for high energy physics. 4 00:00:18.000 --> 00:00:36.000 But the customer outline of my talks basically includes, first of all, the Hilo project at Valencia I'm going to say what it is, what is the objective and et cetera. And then I will move uh towards the motivation and start issue of the current status of the power consumption status in the park 5 00:00:36.000 --> 00:00:40.000 And then I will try to show the typical partner function. 6 00:00:40.000 --> 00:00:54.000 Which might have on a typical server. And then we'll try to make some studies with different hardware, different utilization level and also some accelerators. 7 00:00:54.000 --> 00:01:02.000 So let's start with the first thing. So this is the disclosure of the project. This is the transpasser project in between Atlas and OLCB. 8 00:01:02.000 --> 00:01:17.000 That we have at Valencia. So we have all the information about that on this slide and the aim of the project is basically the benchmarking of the new hardware architectures and developing fast and high patient algorithms I thought I'd do a consumption. 9 00:01:17.000 --> 00:01:32.000 So some of the activities that are specified on this slide So basically it includes development of different algorithms some faster processing is stuck without a tackle reconstruction and etc. 10 00:01:32.000 --> 00:01:40.000 What is important for this talk is basically the measurement of software and hardware power consumption for high energy basics. 11 00:01:40.000 --> 00:01:55.000 So this is the hardware that we have. Within this project and with which we used to make all our studies so basically this is just a typical server and the potency is that you have to decline GPU cards. 12 00:01:55.000 --> 00:02:09.000 Basically one is 85,000 and another is 86,000 duration So 5,000 is used to also be used for telehe cp in the trigger system. 13 00:02:09.000 --> 00:02:24.000 And then, yep you have some photos of that over there and the last feature all set up is basically the APC metered rock video which literally allows us measuring the overall power consumption of an entire server. 14 00:02:24.000 --> 00:02:32.000 Because it's literally plugged in between the power socket. And the salary itself. 15 00:02:32.000 --> 00:02:40.000 All right so basically if you take a look at some kind of existing studies or some kind of extrapolations for the power consumption. 16 00:02:40.000 --> 00:02:48.000 You will see that quite a drastic increase in the AutoCentral spore consumption is expected. 17 00:02:48.000 --> 00:02:57.000 In your futures. Near future and the significant share of that is basically because of the IT equipments of the service itself. 18 00:02:57.000 --> 00:03:20.000 So they can go throughout all future 60% of the our consumption critical data center according to this And after that we have some grilling systems and lighting etc or the uh second most important is the cooling system certified to potty fibers. 19 00:03:20.000 --> 00:03:31.000 All right, so let's… they find the tools that you can use in order to measure our consumption of the service. So our scientists decided to use the test application, which is ALM. 20 00:03:31.000 --> 00:03:44.000 So this is the sort of we used for the high level trigger one at LHCP. So the reason why you choose this photo is simply because we are the number of sort of collaboration and it's much, much easier path. 21 00:03:44.000 --> 00:04:01.000 Previous the existing software we use basically every day. So disabilities and so basically there are some features of this software first of all it can run on different various architectures It includes both CPU and GPU. 22 00:04:01.000 --> 00:04:11.000 A model of design that allows the execution of sequences of the algorith. So basically the software is split into the set of the algorithm. 23 00:04:11.000 --> 00:04:25.000 Each one is… i mean developed to perform some specific task and then we glue them together to form the sequence So how this flexibility, let's say. 24 00:04:25.000 --> 00:04:38.000 And the total number of the algorithms is over 250k. Solace in the real conditions during the real data taking, we use order of 500 NVIDIA GPU. 25 00:04:38.000 --> 00:04:49.000 I'll save ourselves and video cards. And the last but not least, as in the current status they are done with the LHCP simulation standards. 26 00:04:49.000 --> 00:05:05.000 So what would be the strategy? Order status so basically the power consumption itself it can be studied with different tools in different approaches First of all, we can use some specific dedicated hard one. 27 00:05:05.000 --> 00:05:32.000 So for example it can be the method flower distribution unit that gives us the overall power consumption of the entire cell But it might be interesting to try to look at the individual components. For example, how much power is used only by the GPU or how much power is used only by the cpu so let's talk so in order to answer these questions we might use the device drivers so for example in the nudi case you can use the nvidia adcgm 28 00:05:32.000 --> 00:05:37.000 This link that literally allows us to extract up our consumption consumption only and buy that. 29 00:05:37.000 --> 00:05:53.000 Based on the hardware but the cpu is a bit more trickier about the cpu the modern cpu they have the system which is called the performance counters utilization of that counters we can estimate the approximate power consumption of the CPU. 30 00:05:53.000 --> 00:06:13.000 Course itself so i i believe many of you know that CPU package consists of many many different cores basically can split the power consumption even per course So the goal of the current partnership is basically try to answer this question. Can we reduce the power consumption by optimizing the software? 31 00:06:13.000 --> 00:06:25.000 And or harder. Oops, sorry. So let's… start with the first approach. Let's just try to measure the power consumption It shows the dedicated crowd and you show the dedicated PDU. 32 00:06:25.000 --> 00:06:33.000 And we use sort of a processing. 33 00:06:33.000 --> 00:06:39.000 For this stuff and we process like 300 million events In order to manage the status. 34 00:06:39.000 --> 00:06:44.000 And then if you upload the power cluster star, you will see this kind of the curve. 35 00:06:44.000 --> 00:06:59.000 So you see that we have initialized And then almost instantly so i don't know if you can see my cursor, but basically around 200 seconds you will see another rise in the power consumption. 36 00:06:59.000 --> 00:07:25.000 Which will subsequently transform into the big plateau. So this is how the overall behavior looks like. So we are wondering like what is the causes of this initial rise to the power consumption and then like reduction Because I mean, from the NA point of view, it should be a plateau around something, the system is stable so basically the rest of the curve is more or less expected. 37 00:07:25.000 --> 00:07:34.000 What is the reason of the platinum. So for that, we tried to basically measure development from different components. 38 00:07:34.000 --> 00:07:38.000 So the first system we might use is called a cpr interface. 39 00:07:38.000 --> 00:07:46.000 So this interface a lot of us with the modern motherboards to measure the power consumption using some onboard sensors. 40 00:07:46.000 --> 00:08:00.000 Direct on the motherboard and if you'll try to use this system just basically device driver what is important is that this is the driver doesn't require any additional hardware If you'll try to marshall the partner's option with this system you will see 41 00:08:00.000 --> 00:08:17.000 Quite a good match in between our martial martial with the additional hardware enter this system which basically means that in order to study the total bark consumption or the typical server we don't really necessarily need some additional hardware. 42 00:08:17.000 --> 00:08:22.000 You might just need to write the existing tools, already existing software. 43 00:08:22.000 --> 00:08:50.000 It was in the Uso. Then, we try to look also at the consumption of the CPUs and the memory units So to do that, we use the CPU performance calendar. So basically, this is called appraisal And this system allows us to extract the bulk consumption of each individual component which is specified and a little picture on the left-hand side. 44 00:08:50.000 --> 00:09:19.000 So with these studies, we decided to plot the two cpus Because on that side, I already have physically two CPUs. Two CPU cores, some sort of. And then two different RAM boards um And what we see is that a significant increase in the power consumption was only observed for one of the cpus so basically we were running out of thought we were using only that cpu so we like restricted the execution only to that CPU. 45 00:09:19.000 --> 00:09:23.000 And that explains why the second CPU was idle during the whole process. 46 00:09:23.000 --> 00:09:32.000 And then also… I would say important thing is that we're upset that consumption caused by Dharama. 47 00:09:32.000 --> 00:09:42.000 Is knowledgeable with respect to all the other components. But still, the CPU power consumption behavior is a plateau. 48 00:09:42.000 --> 00:09:48.000 So it doesn't really explain why we have the speaking behavior in the beginning. So we continued our studies. 49 00:09:48.000 --> 00:09:59.000 And we switch to the GPU. And we use this NVIDIA DCVM system consumption of each religion. 50 00:09:59.000 --> 00:10:04.000 And again, you see the plateau. And again, the similar symptoms. 51 00:10:04.000 --> 00:10:21.000 Software execution, we used only one of these views to restrict We basically stick that execution 21 of the gpus And that's why the second one is idle Or still, Kenobi does not explain to us why we have that rice in the beginning. 52 00:10:21.000 --> 00:10:28.000 So we tried to look at the overall system on the spectrum on the bottom of my slide. 53 00:10:28.000 --> 00:10:35.000 So uh this like bright color, I highlighted all the components that we basically measured already. 54 00:10:35.000 --> 00:10:43.000 So the only part which was kind of missing and which could potentially explain the scattered behavior was the fans of the cooling system. 55 00:10:43.000 --> 00:10:59.000 So that's why we decided to basically think how we can measure that. And in the end we ended up with attaching conditional piece of the hardware which is basically a small little device which allows us to measure the wind speed 56 00:10:59.000 --> 00:11:03.000 So now we have the funds in our system. So once the funds are spinning up. 57 00:11:03.000 --> 00:11:22.000 The interest much stronger wind and Then we can measure this event with this device And in this case, we might be able to correlate like the increase in the cooling speed in increase in the cooling power consumption to the rise in the overall power consumption. 58 00:11:22.000 --> 00:11:29.000 Is that exactly what we need? So on the right hand side you can see the dependencies or the temperature or the super temperatures. 59 00:11:29.000 --> 00:11:48.000 Temperature is the time. Cpu and GPU temperature. In blue and orange and in black that is the basically speed merged by that device And you can see that this rise in the power consumption is exactly at the same point of time 60 00:11:48.000 --> 00:12:01.000 As the rise in overall power consumption. So now we are quite confident that this initialize was literally because of the cooling system because of the way how liquid walks on the sacks. 61 00:12:01.000 --> 00:12:11.000 So that's it. So at that point we understood the behavior of our consumption card and then we tried to play around a little bit with that. 62 00:12:11.000 --> 00:12:19.000 So the first thing we tried to do is we tried to use different hardware to run our system and marshall consumption is different hardware. 63 00:12:19.000 --> 00:12:32.000 So we used for that cpu and two gpus and you can see the results on this plot uh so basically Just to summarize these things, we observed that the faster the hardware is. 64 00:12:32.000 --> 00:12:52.000 The smaller overall power consumption will be even though the instantaneous power consumption might be high because it's more power Or still, if you integrate that it will be actually on the lower level. And it's uh can be easily seen for the CPU case so cpu 65 00:12:52.000 --> 00:13:05.000 Has quite a low instantaneous power consumption CPU only execution. What it takes very long time. And that's why in the end, integrated energy consumption is much, much higher than the GPU case. 66 00:13:05.000 --> 00:13:12.000 So apart from that, we also try to measure the different levels of GPU utilization. 67 00:13:12.000 --> 00:13:19.000 So the online software allows us to run the execution with the threads. 68 00:13:19.000 --> 00:13:24.000 And then each CPU thread is basically marked to a CUDA stream to the sum. 69 00:13:24.000 --> 00:13:32.000 Let's say that what would it should be and then if you'll change this number of sets and measure the part consumption push for them. 70 00:13:32.000 --> 00:13:39.000 We might estimate what would be the effect of utilization. So this is exactly what has shown on this slide. 71 00:13:39.000 --> 00:13:47.000 And then for the low utilization for the blue curve You can see that the execution time is much, much larger. 72 00:13:47.000 --> 00:13:51.000 And then acetoshin would coordinate the type of consumption also is higher. 73 00:13:51.000 --> 00:13:56.000 But then if you try to use more threads into our hardware. 74 00:13:56.000 --> 00:14:06.000 Actually supports. This might lead to the fact that we also will be higher in the power consumption. This is related to the fact that we have a complicated system of scheduling. 75 00:14:06.000 --> 00:14:20.000 Because, I mean, all the addition threads that we create, they are not actually executed, but they are stored somewhere in the memory and then the CPU does the thing which is called the context switch when it switches one thread from one thread to another thread. 76 00:14:20.000 --> 00:14:32.000 So it's not really multi-threading, but you are using this overheat overhead For the… or the let's say schedule. 77 00:14:32.000 --> 00:14:40.000 So yeah, so this is for the GPU sync and then the last part of my talk is basically the pg users should have pg for the small power. 78 00:14:40.000 --> 00:14:50.000 Power consumption. So there are ongoing studies. Which I aim to offload some of the question from the GPU side from our SHLT1. 79 00:14:50.000 --> 00:15:06.000 To the dedicated hardware located FPGA chips. So this is basically mostly about the tracking And what we see in that case is that we, I mean, by uploading some tasks from the GPU to the dedicated hardware, we might see that 80 00:15:06.000 --> 00:15:10.000 They could use in the power consumption, in the GB power consumption. 81 00:15:10.000 --> 00:15:27.000 It's just shown on this slide. So basically, we saw the distribution of sawpart increases by 70% and by the same order of magnitude we have that decrease in the power consumption However, the studies with the overall consumption of APGA plus GPU is still ongoing. 82 00:15:27.000 --> 00:15:40.000 All right, to summarize my book. So the first of all is that there was pretty much an important metric for the high energy physics, especially when it comes to the development of the future software. 83 00:15:40.000 --> 00:15:41.000 That sort of optimizations might potentially 84 00:15:41.000 --> 00:15:53.000 Thank you. Up on the 15 minutes actually so it would be great to wrap up Thank you. 85 00:15:53.000 --> 00:15:55.000 Sorry, could you please say again? 86 00:15:55.000 --> 00:16:02.000 Your timer has been up for a while now, so it would be great to wrap up. 87 00:16:02.000 --> 00:16:14.000 Oh, okay. I mean, that's literally the last slide. And I see a little timer in this room which says like minus one minute, which might be kind of correlated to the fact that I'm getting on the last slide but anyways. 88 00:16:14.000 --> 00:16:26.000 So, okay, whatever. So this is the last slide. You can read the text yourself any uptime to pay attention. 89 00:16:26.000 --> 00:16:32.000 Right. So thank you so much for this really interesting talk. 90 00:16:32.000 --> 00:16:41.000 Do we have any questions for Vladimir? 91 00:16:41.000 --> 00:16:44.000 Yes, Zach, please go ahead. 92 00:16:44.000 --> 00:16:50.000 Thanks, Fyra. This is an interesting talk. I wanted to ask you about the FPGA usage. 93 00:16:50.000 --> 00:16:54.000 Because porting a lot of workflows to FPGAs is usually pretty tough. 94 00:16:54.000 --> 00:17:00.000 So you can get an algorithm or two with some work, but you're not going to run the whole reconstruction there. 95 00:17:00.000 --> 00:17:11.000 Something like that. Have you looked into the balance of embodied carbon against operational carbon when it comes to those FPGAs. 96 00:17:11.000 --> 00:17:17.618 To see if you're really doing something good for the environment if you buy it.