Review of last lecture

00:01 uh just uh just to know that the lecture 00:03 is actually recorded 00:05 so you will be visible uh on the 00:07 recorded lecture at the end 00:09 if it doesn't bother you 00:13 just just let you know okay great 00:16 so hello everyone back to our lecture 00:21 last time we had a special lecture a 00:24 guest lecture 00:25 by one of our local data scientists 00:29 and uh he uh which was a fabian host 00:33 and fabian explained us from like a 00:36 hands-on perspective 00:38 because that's his job uh how 00:41 you can detect in 00:45 non-equilibrium systems and the 00:47 non-equilibrium systems that fabric is 00:49 working on are of course 00:50 biological systems and what he 00:54 basically showed is how biology how 00:57 older 00:58 and non-equitable systems manifest 01:00 itself 01:01 in low-dimensional structures in these 01:04 high-dimensional data sets that he 01:05 showed you some 01:07 methods that will also appear on the 01:10 website i didn't get to uploading uh the 01:12 slides in the video yet 01:13 um so so he showed you some methods of 01:17 how to 01:18 reduce dimensionality or how to extract 01:21 hidden dimensions in these 01:24 in the high dimensional data sets and

introduction & slide 1

01:27 so today so i will start by giving you a 01:30 little bit of 01:31 more introduction to data science and 01:35 some of the things that we need for the 01:36 next lecture that's in the first part of 01:38 the lecture 01:40 and in the second part of the lecture i 01:43 will 01:43 give you another hands-on experience for 01:46 a practical 01:47 example of from start to finish of how 01:50 to 01:51 go through such a data science pipeline 01:55 now to start the beginning beginning of 01:58 the lecture 01:59 uh we'll go back to our new york city 02:01 flights 02:02 data set so there's a little gap because 02:05 we had to 02:06 find dates with uh fabian last time so 02:08 there's a little 02:09 gap this lecture is now connected to 02:11 what i told you 02:12 uh two lectures ago and uh 02:16 let me just share the slide i'll tell 02:18 you give you a little a brief 02:20 introduction 02:21 to data visualization 02:24 just a short one because the slides have 02:26 been already 02:28 on the website since two weeks or maybe 02:31 some of you have already looked at them 02:33 let me just share the screen 02:37 there we go 02:40 okay great perfect so so i'll give you a 02:44 quick 02:44 introduction before we go on an enhanced 02:46 on example 02:48 so today we'll have a hands-on data 02:50 science example and next week we'll have 02:52 a hands-on 02:53 field theory combined with data science 02:56 example now that will be 02:57 next week um so today i'll still need to 03:01 introduce you to some methods that we 03:03 will need 03:04 and um if you 03:08 can confirm now that you can see my 03:11 slides 03:12 can somebody confirm is that working 03:15 yes okay perfect yeah so i'll just give 03:18 you a quick introduction uh to 03:20 how to visualize uh data 03:24 and of course you're all in your work 03:25 you're all visualizing data all the time 03:27 but if your data is high dimensional 03:29 very uh uh and uh complex and structure 03:33 uh it actually matters what you use for 03:36 visualization 03:38 yeah and um so 03:41 with this lectures two lectures ago when 03:43 we talked about this new york city 03:45 data set about the flights that were 03:47 departing from the new york city 03:49 airports i showed you all kinds of ways 03:52 of how to 03:53 do very efficient computations 03:56 on these data sets but all these 03:59 competitions like it didn't 04:00 really give us any real insight 04:04 and the reason was that we were dealing 04:05 with plain numbers but we 04:07 never had anything to look at and uh 04:11 today in the first part of this lecture 04:13 i'll quickly 04:15 show you a plotting scheme so to say 04:18 that is very powerful 04:20 in visualizing a general and visualizing 04:23 high dimensional data so

slide 2

04:27 before that just a quick reminder so 04:29 before we do 04:30 anything we always want to make our data 04:33 set uh 04:34 tidy you know so we typically we 04:37 collaborate with experimentalists they 04:39 give us the the 04:40 we obtain the data in a very messy 04:42 format and then the first step is to 04:45 tidy the data and that means that we 04:47 need to bring it in a form where 04:49 every column is an observable or 04:52 variable 04:53 and every row is it is an 04:56 observation or a sample sorry to 04:59 interrupt you did you change your 05:02 first slide yes oh you can't see that 05:05 no we cannot i don't know what this 05:06 always happens now in zoom 05:08 there was a zoom update 05:12 there's something maybe i have to share 05:15 the entire screen 05:17 let's try this um the problem is just 05:21 that i have like 05:21 100 windows on my desktop 05:25 um if somebody has a system that happens 05:30 lately all the time right um that zoom 05:33 is not working properly okay let's share 05:37 the desktop 05:42 okay now i'm not having embarrassing 05:45 stuff on the 05:46 desktop but that's not the case okay so 05:49 you should be seeing my 05:52 uh desktop now 05:58 oh okay okay a lot of 06:01 lot of wind okay okay okay 06:05 now you now you know what i've been 06:07 working on okay 06:08 so 06:12 can you see uh this messy and tidy slide 06:17 and when i change the slides now can you 06:19 see that it's just 06:20 right now it works okay perfect 06:24 so so this is uh so so 06:27 just a just a reminder uh that the first 06:31 step that we always do is to make the 06:33 data tidy now where we if you have the 06:37 data in entire defaults format then we 06:39 can perform 06:40 column wise operations that are in most 06:44 programming language 06:45 highly optimized and very efficient 06:49 to run until to program 06:53 i introduced you to this very simple 06:58 r package that allows you to implement 07:01 all of these operations and data science 07:05 and there are many other packages of 07:06 course and other other statements

slide 3

07:09 and typically such an operation 07:13 consists of three steps so you 07:16 filter the data that is in this data 07:18 table package the first 07:20 parameter then you group 07:24 the columns by some condition 07:27 now that's the third parameter and then 07:29 you perform 07:31 an operation independently on each group 07:33 that is the one in the middle 07:36 so this is a typical step in such a data 07:39 science 07:40 calculation and i showed you how you 07:42 then can combine these steps along 07:44 pipelines 07:46 uh to perform that complex operations

slide 4

07:50 today i want to show you a way of how to 07:53 more intuitively 07:55 interact with the data and the way we 07:57 typically do that you all 07:59 are familiar with uh plotting of course 08:01 and the way we typically do that 08:03 is we have a plot in mind yeah and this 08:06 plot then has a name 08:08 that's a scatter plot a surface plot 08:11 a histogram and then you look for the 08:13 function 08:15 in and you look for the function and 08:17 matlab also that gives you this specific 08:19 kind of plot 08:21 yeah another way to do that and that 08:24 would 08:25 introduced by someone called leland 08:27 wilkins 08:28 in a book is that you don't give the 08:31 plots names 08:33 but you construct you construct these 08:35 plots with a grammar 08:37 that means that you have a set of rules 08:40 that allows you to construct 08:42 step-by-step almost any visualization of 08:45 your data 08:47 and once you have that you don't have to 08:49 remember long names of different kinds 08:51 of plots 08:52 you just add bit by bit like in a 08:54 sentence you add word or 08:56 or there's word by word to make this 08:59 sentence 09:00 more rich in information but the only 09:02 thing that you need to know 09:04 is the grammar itself and this allows 09:06 you to 09:07 create from the simple grammar uh very 09:09 different kinds of visualizations 09:12 now and this idea that you have a 09:14 grammar of graphics 09:16 you know i've got a set of rules that 09:17 allows you to construct 09:19 visualizations uh it is an r 09:22 implemented in the ggplot package 09:25 gtw 2 that's quite famous 09:29 and in python and that's a little bit 09:30 newer i don't know how well it works 09:33 it's called the plot 09:34 9 package now that analyze the same idea 09:39 and in r we just load this using the dg 09:42 plot the library command and then we are 09:45 able to use all of these commands

slide 5

09:47 in this package now so the basic idea is 09:52 that we start with a tidy 09:56 data table or data frame in r 09:59 and we take this and then we construct 10:03 assign different 10:06 visual characteristics of our plot 10:10 two different columns of this data table 10:15 yeah so the first thing we have to do 10:18 is uh the first thing we have to do is 10:21 we have to say 10:22 what to plot a plug the point or a line 10:25 and that's called the geometry 10:28 a point line bar circle whatever and 10:31 that's the geometry 10:33 then we have this mapping that i 10:34 mentioned yeah where we map 10:36 different aesthetic properties of our 10:40 plot 10:41 two different columns in this table here 10:45 now for example we could say that the 10:47 position on the x coordinate 10:50 should be what is in column a 10:53 the y coordinate should be what is in 10:55 column b 10:56 the size of our dots or whatever should 10:59 be whatever is 11:00 reflect whatever is in column d and the 11:03 column 11:04 should miss a c and the column the color 11:07 should be what is in column d 11:12 how these values are then translated 11:16 to specific colors or to specific size 11:19 or so 11:20 is a different question now we have the 11:24 static properties of our plots or our 11:26 dots so where are they located 11:28 how do they look like and then we just 11:30 have to define a coordinate 11:32 system to define where they appear 11:36 on the screen yeah and if you have these 11:39 things together 11:40 we have the simplest version of a plot 11:44 on the right hand side

slide 6

11:48 so the way this works in practice is 11:51 that you 11:52 have these little building blocks 11:55 that you just put together 11:58 line by line now so 12:01 in r this looks like that you first have 12:03 to of course do 12:04 something to create an object and this 12:07 is just this ggplot command 12:10 where you tell the plot what data to use 12:13 as the first argument 12:15 and the second argument is then how to 12:18 map 12:18 different columns of your data 12:21 of your table to different pro visual 12:24 properties 12:25 of your plot and then you add a geometry 12:30 for example point or line and you have 12:32 your first plot already 12:35 you can of course also add more 12:37 properties yeah you can add more 12:39 geometries or you can add more 12:41 uh detailed properties of your plot for 12:44 example 12:45 if you are not happy with cartesian 12:48 coordinates then you can set your own 12:50 coordinate system 12:52 now you can have polar coordinates or 12:53 whatever 12:55 you can have subplots by 12:58 by adding this further rule here it's 13:01 called facets 13:03 you can change how values 13:06 in this data table how they map 13:10 to different properties so which color 13:12 represents 13:13 which value in your table you can change 13:17 themes for example or different uh the 13:20 way 13:20 lines and whatever are plotted and you 13:23 can of course also save your file 13:25 and these different building blocks of 13:27 your plots 13:28 are connected via these plus signs now 13:31 you put that 13:32 just as many as you want of these things 13:34 of these aspects 13:36 after each other and by this you 13:38 construct more and more complex 13:40 plots yeah and for 13:44 everything that you see here below there 13:45 are some sensible defaults for example 13:49 cartesian coordinates that in many cases 13:52 you don't need to touch 13:54 now let's have a little look at our new 13:56 york city data set 13:59 and this new york city data set right 14:02 here we go

slide 7

14:02 now we already discussed that uh 14:06 two weeks ago and in this new york city 14:08 data set we have 14:09 information about flights departing from 14:13 new york city airports for each flight 14:16 we have 14:17 different information we have for 14:19 example the time 14:21 when this flight departed we have the 14:24 origin 14:24 airport we have the number of the plane 14:26 that was used 14:28 the carrier and so on and the delay 14:31 uh for this specific flight and also as 14:35 we already discussed 14:36 you can connect this table that we can 14:39 have 14:40 that you can download from our github uh 14:44 from the from the website um we can 14:47 connect 14:48 these flights then to other sources of 14:50 information for example 14:52 the weather that uh with the weather 14:55 information for a given 14:57 point in time and a given location 15:00 we can also connect that to airport 15:02 information 15:04 uh we can connect that to information 15:06 about the planes 15:07 yeah and we can also get information 15:10 about the airlines if you want 15:12 yeah and that's how we what we do yes we 15:15 reload 15:16 again just like last time where we load 15:19 again 15:20 all of these different files using the 15:23 function f 15:23 read and then we merge them together 15:27 using these merge commands and when we 15:30 merge them together we sometimes have to 15:32 give for example when we merge with the 15:35 planes 15:36 or let me see here for example when we 15:38 move with the airports that we merge 15:41 with the airports we have to say that 15:43 here the airport identifier is in the 15:46 column origin 15:48 and here the airport identifier is in 15:51 the column 15:52 faa now so we motor all of these things 15:56 together and we have a huge data set 15:58 a data table containing all of these 16:01 information line by line and 16:04 in a tiny format we already did that two 16:08 weeks ago 16:10 now let's have a simple look at these 16:13 plots let's just let's say let's have a 16:15 simple plot

slide 8

16:16 and the first thing we can do is uh like 16:19 just like last time we can calculate 16:22 the average delay 16:27 for each month so we group the data by 16:29 month 16:31 and for each month we take the average 16:35 over all departure delays and save that 16:39 average 16:39 in the column mean delay 16:42 now so what we don't get is that for 16:44 each month 16:46 here in the first column we get a mean 16:49 departure delay this is the second 16:51 column 16:52 and below you can see now the simple 16:54 spot you can do you can 16:56 tell this ggplot function to take this 17:00 table 17:01 now a very simple table and map 17:04 the month to the x-axis and the delay to 17:08 the y-axis 17:10 and then you just add a geometry 17:13 which is just a point then you get what 17:16 you see on the right-hand side 17:18 and you see that something is happening 17:20 here in this summer months 17:22 and something is happening over 17:24 christmas apparently 17:26 okay 17:31 so okay there's something something in 17:33 the waiting room right 17:35 okay so something is happening over 17:37 christmas now let's go on

slide 9 & 10: different types

17:40 uh we can of course also add different 17:42 geometries to a plot 17:44 yeah so these different geometries so we 17:46 just use the geometry 17:49 of a line now of a point we can also add 17:52 different geometries 17:54 and for the sake of simplicity what i'm 17:56 doing here i'm using the tools that i 17:58 introduced two weeks ago 18:01 not to do all of these things in one 18:03 line so we take the flights 18:06 data set calculate for each month the 18:08 average delay 18:11 and send everything with this pipe 18:14 operator here 18:16 to the g plot and in the g 18:19 plot we just need to define that this 18:22 aesthetic mapping 18:23 that the x coordinate is the month and 18:25 the y coordinate 18:27 is the delay and we save this everything 18:31 we just save in an object g on the left 18:34 hand side 18:36 and now we can take this g and add 18:38 different things to it 18:39 we can add different geometries on the 18:41 left here top left we have the 18:44 the point as before we can add a 18:47 geometry line 18:48 now then we get a line we can add a bar 18:52 that's called column we can add a bar 18:55 or we can add all of them together to 18:58 the plot 18:59 and then we have all of them together 19:01 this information of what 19:03 what happens with the data that's not 19:06 done in the geometry that's we have done 19:08 once in the beginning and now we can 19:10 just operate on this object and add 19:12 different things and change the part the 19:14 way we like it 19:18 yeah so there are also geometries that 19:21 are 19:21 that involve analysis now for example uh 19:25 if you have a background in biology 19:27 maybe then you know your favorite 19:29 dot box plot on the right hand side that 19:32 summarizes 19:33 different uh properties of the 19:36 statistical distribution 19:38 for example here in the left hand side i 19:41 take the flights 19:42 all this combined information i take it 19:46 use the x the carrier as the x 19:48 coordinate 19:49 and the logarithm of the departure delay 19:52 as the y-coordinate 19:53 and i add this box plot where i have 19:56 automatically 19:57 the median and this is some 20:01 inter-quarter range and then i 20:04 always forget uh forget what this means 20:06 is probably the range of the data 20:08 without 20:08 outliers or so now that's basically in 20:11 some disciplines these box plots are 20:13 used to characterize distributions 20:15 another way to characterize 20:17 distributions are violence 20:19 violent plots that essentially give you 20:22 a plot of the probability distribution 20:24 here 20:25 um just in a vertical manner the thicker 20:29 the violin is 20:30 the more the the higher the probability 20:32 to find a data point 20:34 there yeah and

slide 11 & 12: aesthetic

20:37 uh so of course we can now also play 20:40 with 20:40 how these plots look like so 20:43 for example now i'm and if you look at 20:46 the red 20:47 i'm doing the same operation yeah i'm 20:50 calculating the average departure delay 20:54 for each month and each airport and each 20:57 carrier 20:58 but now i'm just for simplicity i only 21:01 take the big 21:02 three carriers united airlines delta and 21:04 american airlines 21:07 and now i create this plot again now i 21:09 have this aesthetic mapping 21:12 the month should be the x coordinate the 21:14 delay the y coordinate 21:16 and now i have another aesthetic which 21:19 is the color 21:20 i say the color should be the origin 21:23 airport 21:25 and the line type line type should 21:27 correspond to the carrier 21:30 and now i just add the geometry the line 21:32 and i get this plot that you see on the 21:34 bottom here 21:36 you can see that all carriers have a 21:38 problem in the summer month 21:40 and also in the over christmas you know 21:43 except for america so something is going 21:46 on with american 21:47 airlines around march uh no idea what 21:50 this is 21:52 yeah and uh 21:55 no wait that's not american airlines now 21:56 that that's new arc 21:58 newark american airlines has a problem 22:01 in march 22:02 yeah you see it's not the problem the 22:04 plot is not perfect yet 22:06 and uh okay so we can go on now we can 22:09 add 22:10 other aspect we can change other experts 22:13 of the plot 22:14 for example i here say that the film 22:18 should be the airport and then i use a 22:21 box plot 22:22 and then i get an overview over how the 22:25 difference 22:26 airport the airports the different 22:27 airports compared to each other 22:30 for each carrier and what you can see 22:34 is that jfk is doing well for 22:37 some of them but not for all 22:40 you know american airline and for united 22:43 and 22:44 and what is it where is it delta is 22:46 doing well 22:48 um but there is no clear trend here of 22:50 course

slide 13: subplots

22:52 something that's more interesting is if 22:54 you plot these delays 22:57 for the big three carriers yeah 23:00 uh as a as a function of the hour of the 23:04 day 23:06 yeah so here the x-axis the x-coordinate 23:09 is the hour and i turn that into a 23:12 factor 23:13 from numeric to something that is 23:16 discrete just for plotting purposes 23:19 the the fill color is the origin airport 23:24 and i added here a 23:27 subplot now that's called a facet 23:30 by carrier if you remember the two 23:34 lectures ago this is a formula we can 23:36 use a formula in r 23:37 to specify how plots are distributed 23:40 across different subplots 23:42 and you can see that for a delta and 23:44 united airlines 23:46 you can nicely see how these delays 23:49 add up during the day 23:52 yeah and yeah and it looks a little bit 23:54 even that is 23:56 let's have a look at the next slide

slide 14

23:58 maybe uh let's 23:59 have a look at the next slide so if we 24:01 can have also more complicated subplots 24:04 yeah so for example we have we can have 24:06 a grid 24:07 by using a more complicated formula 24:10 where the 24:11 y-axis yeah the y-direction should be 24:13 the origin 24:14 and the x-direction in this grid should 24:17 be the carrier 24:18 yeah and then we got these plots and you 24:21 can see 24:21 how you can actually see how these 24:23 delays 24:25 add up during the day and it seems like 24:28 a little bit yeah it's a speculation 24:30 but because we have a logarithm in the 24:32 y-axis 24:34 and we have linear linear increase 24:37 over time uh in these delays 24:41 uh over over time during the day that 24:44 you have an exponential 24:45 build up of delays now that's quite 24:47 quite interesting

slide 15: plot of 3 variables

24:49 yeah okay so we can do all those kind of 24:53 other fancy thing we can if you have 24:55 more than two variables 24:57 now for example when we calculate the 24:59 average delay 25:02 as a function of the month the hour and 25:05 the origin 25:05 airport yeah then we have more than 25:09 we have then we have then even more 25:11 variables that we want to visualize 25:14 uh we can do that for example with such 25:16 something that's called a heat map 25:19 yeah and this heat map here the 25:24 the the fill and also the color of these 25:27 tiles 25:28 is given by the mean delay 25:31 while the month and the hour are plotted 25:34 on these axes here 25:36 then we add the geometry of the tile to 25:39 get these heat maps 25:41 then you can visualize relationships 25:44 between two variables namely month 25:47 and hour and it seems like this buildup 25:51 of delays is specifically drastic at the 25:54 summer months 25:56 while it's not that evident in other 25:58 months

question on assigning facet

26:02 excuse me i have a question syntax 26:06 in this where you've written facet 26:10 underscore rap the first argument tells 26:13 us 26:13 the x argument and so origin will be 26:16 plotted on the x scale 26:18 yes so so this facet rep 26:21 is just to say okay take 26:25 one column of the data table yeah it is 26:28 already in this case origin 26:31 and group the data according to this 26:33 column your origin 26:35 and then make one plot for each of these 26:39 origin 26:40 airports and put them next to each other 26:44 as many as fit on the screen 26:47 now and if they don't fit on the screen 26:48 go to the next line 26:50 it's basically just this wrap what i'm 26:53 saying here is that you have a 26:54 one-dimensional so to say 26:56 line of plots uh compared to this grip 26:59 that is 27:00 this grid here where was that yeah this 27:03 grid 27:04 is the same thing it's basically the 27:06 same thing 27:07 uh so here we have these two coordinates 27:10 now these two 27:11 directions origin airport on the on the 27:14 y direction 27:17 and carrier on the x direction 27:20 now this is this this formula notation 27:22 and r 27:23 yeah so that's it's a little bit counter 27:25 intuitive but you give it a formula 27:28 uh in order to uh to to tell 27:32 r how all this this package how these 27:34 plots should be distributed 27:36 on your screen or on the on the in the 27:38 pdf file that you export 27:41 yeah and you can if you want you can use 27:46 the reason why the formula i used here 27:48 is that you could do something here 27:51 that you have here carrier plus 27:54 for example um what are we plotting here 28:00 carrier plus uh um 28:04 month or so yeah you can have here 28:08 uh if you do something like this 28:12 you can have more complicated formula 28:15 to say that a combination of carrier and 28:19 month of these two columns 28:21 should be on the x direction here 28:27 and on the y direction you have origin 28:30 now you can 28:30 you can construct more complicated 28:34 grids of plots if you wanted to 28:37 it's very often that's not very useful 28:40 and 28:41 just think about that that this is 28:43 what's on the uh 28:46 that was my that was my question because 28:48 in the next slide the 28:50 original argument the origins have been 28:52 plotted on the x 28:53 scale whereas in this particular case 28:55 the original airport has been plotted 28:56 along the y scale 28:59 so now i exactly so now i'm trying to 29:02 get my mouse cursor back is here 29:06 so okay now i can change the slide 29:08 hopefully 29:10 okay here we go so here i just left away 29:14 the first argument 29:17 and i left the the left one so that was 29:20 originally the y 29:21 direction and now i only have the x 29:24 direction left you know from left to 29:26 right 29:28 and i use here wrap and not grid because 29:30 i want 29:31 this these plots if i have not 29:34 three but like 15 different groups 29:38 i don't want them to be all in the same 29:40 line because i wouldn't be able to see 29:41 them on the screen 29:43 rep means once the screen is full go to 29:46 the next line 29:47 yeah nothing else that's not it's not a 29:50 complicated thing here let's just 29:52 just just make each plot for each origin 29:55 one plug for each origin 29:57 excuse me yes uh on this heat map 30:00 the color bar the minimum value is not 30:03 zero 30:05 does it mean that yes yes 30:08 rights that departed earlier than 30:10 scheduled 30:13 yes there are yes exactly exactly they 30:16 departed earlier oh 30:19 it's not very funny for the passengers 30:22 yes so so but but sometimes that happens 30:25 and it also it's also a question how the 30:27 data is recorded 30:29 um depends on how the data is recorded 30:33 so especially so this is position 30:37 specifically affects the early mornings 30:40 right and the very late times 30:44 now so this negative is negative 30:45 departure delays 30:47 um i mean that sometimes that can happen 30:51 you know so sometimes 30:52 the question is when that's not the time 30:56 when the gates close 30:58 uh it's probably the time when the 31:00 airplane starts or something like this 31:04 yeah and as you as you know if the 31:06 boarding is 31:07 so very often it happens that that once 31:09 boarding is completed 31:11 the the airport they will sometimes the 31:14 airplane leaves a little bit earlier 31:15 than uh than the schedules 31:18 okay thanks but as always there's a good 31:22 thing 31:22 it's always good to to to know how the 31:24 data was actually collected 31:26 now you think that a delay is well 31:28 defined but then you can 31:29 measure this in different ways yeah 31:32 that's always a very important aspect 31:34 aspect and here are also this data set 31:37 they're also 31:37 missing missing numbers a lot of missing 31:41 numbers 31:41 and that's when an airport started 31:44 somewhere but didn't end up on its 31:47 rival location but it's on other 31:49 airports yeah so that's also 31:51 that's also possible

slide 16: error bar

31:54 okay so let's uh go on just uh 31:58 so we can also use uh gg plot you have 32:01 to do statistical computations 32:03 on the fly and this is uh particularly 32:06 useful for computating 32:08 fancy error bars that's that's 32:11 how i use it and so what 32:15 we can do for example here on the top is 32:18 that 32:19 we have the hour on the x-axis the 32:22 departure delay 32:24 on the y-axis and then 32:27 color and uh and fill as the origin 32:30 airport and then for each of these 32:33 combinations because we take on the left 32:35 and raw data 32:36 now we have many different values we 32:38 have many different flights 32:40 and we can then take just a summary 32:43 function 32:44 statistical summary function from the 32:46 gdg plot 32:48 tell this function to calculate the mean 32:53 and use the geometry of a line 32:56 to do the computation for us yeah so for 32:59 so simple things like the mean 33:00 calculation 33:01 that's pretty good but the nice thing is 33:03 that we can also do 33:05 uh summary functions that are more 33:08 complicated yeah for 33:10 example in this here we have a 33:11 bootstrapping 33:13 confidence intervals yeah that's that's 33:16 basically a fancy way of calculating 33:18 confidence intervals 33:20 and we use a geometry of a written 33:23 not to visualize these confidence 33:25 intervals 33:27 and if you deal with confidence 33:28 intervals you know that's quite 33:30 complicated to calculate and then you 33:32 somehow have to bring them into your 33:33 plot 33:34 so here you don't have to worry about 33:36 this you have the most fancy methods 33:37 just 33:38 uh in one line and you get a nice 33:41 visualization of the uncertainty of your 33:43 data 33:45 another thing we can do is also here in 33:47 this upper case our 33:49 x-axis is discrete because it's an hour 33:52 from from 0 to 24 33:55 or 23 or 5 to 23 33:58 but sometimes we have real number values 34:01 yeah and then we need to tell 34:03 these functions which values to put 34:06 together that means we can 34:07 bin the data an example here is the 34:10 temperature 34:11 in fahrenheit on the 34:15 x-axis and we cannot calculate for every 34:19 value of the temperature 34:21 uh we cannot calculate a separate mean 34:23 yeah because temperature is a real value 34:25 it's uh 34:27 a real valued uh quantity 34:30 yeah and we we need to define a bin to 34:33 summarize values of the temperature 34:36 and that we can do automatically also 34:38 with these start summary bin 34:40 functions where we just calculate 34:43 we just tell the function to bin the 34:45 data 34:47 and for each of these bins again 34:48 calculate the mean 34:50 and plot the result with the geometry of 34:53 a nine 34:54 and we can do the same fancy arrow bars 34:57 and confidence interval calculations 34:59 as before and also you know probably 35:03 that 35:03 these kind of processes are quite 35:05 complicated if you have to do them 35:07 yourself 35:10 and here probably the the me the the 35:12 message is that it's 35:14 not good if it's too cold or too hot 35:18 but then there are correlations right so 35:21 when it's hot here in on the right hand 35:25 side 35:25 there's also when the holidays take 35:27 place yeah that's july 35:29 and june yeah in these previous heat 35:32 maps 35:33 where we had these delays yeah so it's 35:35 not 35:36 it's not quite quite clear whether it's 35:38 the temperature that's bad for the for 35:40 the engines or something like this 35:42 or whether it's just the number of 35:44 people that 35:45 go on holiday and block the airport 35:48 and lead to delays there

slide 17: interpolation

35:53 now we can do even more fancy things now 35:55 we can do 35:56 interpolation yeah on the right and the 35:59 left hand side you know i just 36:00 uh had the same plot as before 36:04 we had um yeah we have the month 36:08 wait okay here's a little error the hour 36:11 on the x-axis 36:13 and we do as the hour on the x-axis 36:17 yeah and now we can add like an 36:19 interpolation line 36:20 just in one line with this summary 36:23 function 36:24 and if we wanted to we would be able to 36:26 do like 36:27 non-linear interpolation linear 36:29 interpolation or anything we want just 36:31 with an argument 36:33 and we get our usual nice error bars for 36:36 free 36:38 of course if we can do non-linear fits 36:40 we can also do linear fits so we can fit 36:42 linear models to check for correlations 36:45 and that's what we do on the right hand 36:47 side and on this right hand side 36:50 what is actually quite interesting here 36:53 is that on the x-axis we have the month 36:56 on the y-axis we have the number of 36:59 seats 37:00 in an airplane and the color 37:04 is the origin airport 37:07 now what we find here is that there's a 37:09 perf almost perfect linear relationship 37:12 between the month and the number of 37:14 seats 37:15 yeah there's a perfect linear 37:16 relationship which is positive 37:20 for the for new arc and 37:23 jfk and uh 37:26 negative for lga which is i think 37:29 laguardia 37:30 yeah so no idea where this comes from 37:33 yeah but apparently you are sitting 37:37 in december in a smaller plane 37:40 higher likelihood to sit in a smaller 37:42 plane in december 37:44 if you are departing from laguardia 37:47 while the planes get larger 37:50 throughout the linearly larger 37:52 throughout the year 37:54 for some reason yeah so that's that's 37:56 one of these 37:57 one of these things where you should be 37:58 suspicious and check what is actually 38:01 the underlying 38:02 reasoning for this data yeah so so 38:05 that's that's something i will show you 38:06 also later 38:07 is that it's very important to check 38:10 whether 38:10 when you do statistical computations 38:12 whether they actually make sense or not 38:14 yeah big data gives you every result 38:18 that you want if you just look for it 38:21 now just because you have so many 38:23 dimensions so many samples you can 38:25 find every hypothesis you want in these 38:28 data sets 38:29 if you just keep looking for it

slide 18: scales

38:33 yeah so we can play around with scales 38:36 yeah so that means that we can change 38:38 how our plots look like for example 38:42 uh in this plot here on the top that's 38:44 something we have shown 38:45 seen before that's the box plot we save 38:48 this 38:48 in a variable p now and now you see how 38:51 i this weird arrow assignment operator 38:55 in 38:55 r why why why why the r community 39:00 likes that you can have that you can 39:02 assign in the different direction 39:03 right so i have the plot and i assign 39:06 the results to a variable p 39:09 and that's not something so it's it's an 39:12 asymmetric 39:12 assignment and so now we have our plot 39:16 and we can add different 39:18 color scales and we can add different 39:20 ways of how our data values 39:23 map to visual characteristics of the 39:26 plot 39:28 for example we can add a new scale score 39:31 for some scale color blur then we get 39:33 different 39:34 blue tones we can also add 39:38 like a manual mapping where we say that 39:42 the uwt 39:43 want to have black gray and white as the 39:45 colors for our airports

slide 19: positions

39:48 now so we can add change visual 39:50 characteristics and we can also change 39:52 of course how things are positioned 39:56 relative to each other yeah and i'll go 40:00 quickly over this because that's a 40:01 little bit of a detail 40:03 yeah so we have for example we can 40:05 create this uh plot here on the right 40:07 hand side 40:08 where we for each plot for each month 40:12 origin and carrier carrier calculate the 40:16 average delay 40:17 for the three carriers and then we can 40:20 make a plot 40:23 where we assign the month to the x-axis 40:25 now to the white 40:27 the delay to the y-axis the fill 40:30 color to the origin airport 40:34 and the transparency of this color 40:37 to the carrier yeah and then first 40:41 we can now plot all of this using a bar 40:44 plot 40:45 and if we have a bar plot we can decide 40:47 how to 40:48 put these bars relative to each other 40:51 and i'll just 40:52 give you three examples we can stack 40:55 them on top of each other 40:56 that's on the right hand side 40:59 now we can dodge them 41:02 that means that we put them next to each 41:05 other that's in the middle 41:07 and we can use a fill 41:10 position that means we always fill them 41:13 up to one 41:14 that means we look at the fraction that 41:16 a certain carrier 41:17 and origin airport contribute 41:21 to the total delays and 41:24 uh let me just see if there's any 41:28 yeah you can see here for example so 41:30 that that 41:32 that that here for example a large 41:35 fraction of the delays in march 41:37 comes actually from this newark airport 41:41 uh while other months uh for example in 41:44 the summer month 41:45 the larger fraction of the lace actually 41:47 comes from the other airports 41:49 um jfk and lga 41:53 yeah we can also do something

slide 20: corrdinate system

41:56 we can also change how the data is 42:00 how we can change the coordinate system 42:02 you know so now we always assume that we 42:04 have 42:04 cartesian coordinates but we can of 42:06 course have any other coordinate system 42:09 so here for example we are plotting uh 42:12 as the x-coordinate the wind direction 42:16 as the y coordinate the departure delay 42:20 and then we just calculate the average 42:22 delay 42:23 again using the summary function 42:27 for a certain certain intervals of the 42:30 wind direction 42:33 yeah and we can then plot that for 42:35 example in different ways 42:38 um we can plot that of course in 42:41 cartesian co-coordinates 42:43 something that's more instructive is 42:45 actually when we talk about 42:46 directions is to use polar coordinates 42:50 yeah and you can see that i do that just 42:53 with one adding just one line 42:54 one more rule to the plot 42:58 and now i can add more 43:01 aesthetic mappings for example i can 43:03 separate 43:04 as before these different contributions 43:07 from the wind direction 43:09 by airport and this is what i've done 43:11 here i just added one more aesthetic 43:13 mapping here 43:15 i said that these bars should be next to 43:18 each other and not on top of each other 43:20 or so 43:21 and i have the polar coordinates as 43:23 before 43:24 yeah then you get the plot that you have 43:26 on the right hand side 43:28 and in this plot what you see 43:31 is that there is a relation with the 43:33 wind direction 43:35 of the departure delays specifically 43:38 when the wind 43:39 comes from what is that southwest 43:43 a little southwest west for the two 43:46 airports 43:47 uh lga 43:50 and newark you know whatever is the 43:53 reason for that 43:54 it's actually for all of them yes and 43:56 actually for all of them 43:58 uh it's actually for all of them 44:01 but it's specifically strong for lga and 44:04 new arc 44:05 and if you look at the location of new 44:07 york that's where the sea is 44:09 that's probably also where a lot of the 44:11 wind 44:13 the strong winds come from from this 44:15 direction 44:16 yeah okay yeah this was just playing 44:19 around with the data 44:20 and you get some some insights 44:23 from just looking visualizing the data 44:27 and these insights are of course much 44:29 harder to get if you just look at data 44:31 tables on the console as we did two 44:34 weeks ago 44:35 and what you can also see here if you 44:38 create such plots 44:40 yeah so you can make more and more 44:42 complicated plots 44:44 but the complexity of your code 44:47 never changes yeah it does only 44:50 increases just linearly because you're 44:51 adding just one bit by bit 44:53 to your plot one layer by layer to your 44:56 plot and you can make 44:57 as complicated plots as you want 45:00 from this now without adding 45:04 more and more complexity actually to 45:06 your code or with 45:07 without requiring requiring more and 45:11 more specialized 45:12 functions and that is the advantage of 45:17 having such a 45:19 grammar of graphics now that allows you 45:21 to to have simple rules 45:23 uh that visual rules that allow you to 45:26 add more and more components to a plot 45:29 and then of course we can make these 45:31 plots look

slide 21

45:32 nice yeah we can add things like 45:35 axis labels for all of our columns 45:39 typically you get a data table that is 45:42 that where some experimentalist has used 45:45 their own notation for things 45:47 doesn't make much sense most of the time 45:49 you want to have your own 45:51 um you want to have your own uh 45:54 um names for the for the x's and for the 45:57 colors and for the 45:58 for the legends and uh specifically 46:01 including units if you want to publish 46:03 that and then you can do that easily 46:06 with this labs command and there's also 46:09 a title command if you want 46:11 here you can add a title and 46:14 annotate your plot as much as you want

slide 22: extensions

46:17 and then you can get as complicated as 46:21 you want you can 46:22 download extensions for example some 46:25 nice extensions add 46:26 new geometries and new coordinate 46:29 systems to these plots 46:31 so here these uh plots that are used in 46:33 anatomy 46:34 they add the human body and mouse body 46:37 coordinate systems 46:38 and you can then easily without adding 46:40 having more complexity 46:42 than what i already showed you you can 46:45 have visualize your data 46:49 that you for example imaging data on the 46:52 mouse or 46:53 human body or whatever you want to do 46:55 now that looks like 46:56 a ton of different extensions to this 47:00 okay so this is a very efficient way of 47:02 visualizing data

slide 23: there is also python implementation

47:03 that relies on a grammar or a set of 47:06 rules 47:07 i showed you an r implementation but 47:09 there's also a python implementation 47:12 and the python implementation is rather 47:14 new so it's just 47:16 i don't know what quality this is and 47:20 what we now want to do 47:24 is i want to uh 47:28 i want to show you how to use these 47:31 tools that we 47:33 that we've seen in the last couple of 47:34 lectures in a specific

The following slides are nowhere found in the slides shared on the course website. The lecturer went through the jupyter notebook about a RNA sequencing project.

47:37 data science project and for this we'll 47:40 just 47:41 go through the code of a real data 47:43 science project and this is a project 47:45 that fabian actually did 47:47 while he was in the group and the 47:50 uh starting point of this project 47:54 is a so-called sequencing experiment now 47:57 so i've already showed you this table 47:59 and here on the x-axis yeah 48:02 on the the color the rows in such an 48:05 experiment 48:06 on such that that look that would be so 48:08 to say a matrix that 48:10 experimentalists would send you so here 48:13 every row 48:14 is a different gene and every column is 48:20 a different cell 48:21 now we have now maybe then twenty 48:23 thousand cells thirty seven thousand 48:25 cells 48:26 and for each of these cells we have 48:29 roughly 48:30 ten thousand measurements yeah and 48:33 these measurements correspond to how 48:36 strongly 48:38 uh a certain gene yeah that's on the 48:42 in the row here is expressed in this 48:45 particular cell so these numbers here 48:47 correspond to how many products of these 48:50 genes these experimental techniques 48:53 found in a given cell yeah and the way 48:56 and these genes 48:57 you might hurt they tell us a lot about 49:00 what cells are doing and how they're 49:02 behaving and what kind of cells they are 49:04 so they're very important 49:06 molecular measurements of what's going 49:09 on inside cells now so for example here 49:13 this gene now that has this id here 49:16 that's a cryptic name and row four 49:20 is not expressed it has a little bit 49:23 information in this particular cell but 49:26 not in other cells 49:27 what other genes like this one here have 49:30 very 49:31 high expression values they have very 49:34 high counts of products from these genes 49:38 that were detected in these experiments 49:42 so what i have to tell you is that these 49:44 experiments are extremely messy 49:47 yeah so so especially there are there is 49:49 a step where 49:50 ex the data is exponentially amplified 49:54 and that exponentially amplifies errors 49:57 in these data sets so it's a 50:00 it's a big mess yeah and now we have to 50:02 find some structure 50:05 in these simple in this high dimensional 50:08 data set in these 50:10 genomics experiments and to show you how 50:13 this is working i'll share another file 50:19 um i'll share another screen 50:22 [Music] 50:24 where are we 50:28 here 50:31 here we go now i'll just give you a uh 50:34 like a hands-on look and 50:37 on how this actually works yeah i don't 50:40 tell you too much about the biological 50:43 background in this project because it's 50:45 not yet published 50:47 um so um can you see you can you should 50:50 be able to see 50:51 the browser right 50:54 and you can see that this here is 50:57 actually a combination 51:00 of python yeah the first block 51:03 and r yeah so this here 51:07 here he's loading some r packages 51:10 and here he's loading python packages 51:13 and all of this is a jupyter notebook 51:15 yeah so yeah combining r and python to 51:18 take the best of both worlds 51:23 yeah and then we then there's a lot of 51:25 data loading going on 51:27 yeah so i'll just go through of course 51:29 we don't have to look in detail how the 51:31 data is loaded 51:32 uh some some biological background 51:35 information about what 51:36 different genes are doing and 51:40 so on yeah and now 51:45 we start with the pre-processing of the 51:48 data 51:49 so as total that this data is messy 51:53 and this data has like 51:56 80 percent nonsensical 52:00 the nonsensical information in other 52:03 words this is dominated by 52:05 technical noise our technical noise is 52:08 extremely strong 52:10 and it gives rise to very weird results 52:13 so the first step we always have to do 52:16 and this 52:17 particular example in genomics but also 52:19 in other data sets 52:20 is we have to look at the data and and 52:23 polish it in a way so that we can are 52:27 actually in principle able 52:29 to detect information here 52:34 so for example this plot here on the top 52:38 shows you basically what is the 52:40 percentage 52:42 of all information in a cell 52:46 that goes to certain genes yeah they 52:48 have these weird names they're 52:49 completely 52:50 irrelevant but you see that this gene on 52:53 the top here 52:54 that has an even weirder name 52:57 in some cell comprises eighty percent of 53:00 the 53:02 information yeah and that 53:05 does not make any biological sense but 53:08 because if you have 53:09 thirty thousand genes in a cell it can't 53:11 be that basically the cell 53:13 is packed completely packed with 53:15 products from a single gene you know 53:17 that that's 53:17 cannot happen in real life and that's 53:20 why we see that here there are a lot of 53:22 cells 53:23 yeah so everything where we have more 53:25 than maybe 30 percent here 53:27 where we don't have any reasonable 53:31 information now that means that we need 53:34 to do quality control now we 53:36 need to filter out cells that actually 53:40 have meaningful information and we have 53:42 to 53:42 keep away cells that don't have 53:44 meaningful information 53:46 yeah and what we do is we look at such 53:50 histograms here 53:52 so we took a look at these histograms 53:54 and we calculate probability densities 53:58 over all cells so all columns of this 54:01 matrix 54:02 and on the x-axis here is the 54:06 total amount of information that we have 54:09 for 54:09 for a cell so the total number of 54:11 molecules that we detected 54:13 for a single cell and you can see this 54:15 follows a distribution 54:18 and the first thing that you see there 54:20 are two blocks now some cells are worse 54:22 and some cells are better so there is 54:25 already some variance in the data 54:27 just because the quality of our 54:30 measurement 54:31 is different for two between two groups 54:33 of cells 54:35 but all of these are actually good 54:37 values yeah they're all good 54:39 yeah and uh we just take out some cells 54:42 here that is the 54:43 vertical line that are below one million 54:46 of these counts now these we throw away 54:51 now we can also 54:53 see look at other accounts so um 54:59 so this is for example how many genes 55:01 that we detect 55:02 and here we also cut off 55:06 here we also cut off 55:09 basically cells that are low quality 55:12 in these tails here we just remove them 55:14 from the data set 55:16 because we know that if we kept them in 55:18 the data set in the long term 55:19 we would have problems with 55:21 dimensionality reduction now they would 55:23 these things dominate in the end 55:26 things that are based on machine 55:28 learning clustering 55:29 dimensionality reduction so we remove 55:31 them from the data set 55:34 and now that's sorry excuse me 55:36 replications 55:37 yes so is there any uh is a systematic 55:41 way to set 55:43 the threshold to filter out 55:46 in this case uh it's a matter of 55:49 experience 55:50 yeah there's not a systematic way 55:53 normally 55:53 [Music] 55:55 you would set the threshold in such a 55:57 data set 55:58 you would set the threshold here in the 56:00 middle between the two 56:02 peaks but between but because the two 56:04 peaks are both at reasonable values 56:06 and they're all of the same height yeah 56:09 then 56:10 uh you know i said we would get we would 56:12 lose 50 56:13 of the data and that's a little bit too 56:15 much yeah but they're all reasonable but 56:17 we have to check later 56:19 that if we find two groups of cells in 56:22 the data 56:23 that these two groups are not 56:26 just representing uh these two peaks 56:30 here in the quality of the measurement 56:32 yeah the later we have to we now go go 56:35 on with analysis 56:37 and if there's something is suspicious 56:39 we go back to this stage here 56:42 and we might have to be more rigorous 56:45 with this 56:45 cut off here yeah so in this case in 56:49 this particular case there's no rigorous 56:51 way of doing that 56:52 it's a matter of how much do you expect 56:54 what is the good measurement 56:56 and uh basically now this is this is a 56:59 this is a 57:00 this is a pretty good example in terms 57:03 of these counts here 57:04 now if you if you're working on genomics 57:06 so this is this is actually zebra fish 57:08 so we have less than in other animals 57:11 in total and um but but here we don't 57:15 have basically 57:17 sometimes you have a another peak here 57:20 at very low values 57:22 and that we would then completely cut 57:24 off 57:26 so here the problem more is this one 57:28 here this plot 57:30 we have a lot of cells that have a high 57:33 percentage 57:34 of mitochondrial accounts so that that 57:37 are genes 57:38 or the dna that is in the mitochondria 57:42 and these genes do not produce 57:45 this dna does not produce many gene 57:48 products 57:49 yeah so it suspicious if you have too 57:51 much of that in this 57:53 in these cells and here we take out 57:56 roughly i guess 20 or 30 57:59 20 20 of the cells 58:02 we lose in this step 58:05 yeah and we can also plot both with 58:08 respect to each other 58:09 for example we can here on the x-axis 58:11 have these different values that 58:13 represent the quality of our data 58:15 and then just in half these bars these 58:19 vertical and lines that we had in the 58:20 histograms 58:22 just in this scatter plot here yeah and 58:25 cut 58:25 and see which what are the kinds of 58:27 cells that we use here 58:29 uh visually yeah 58:33 okay now so now now we get rid of the 58:36 bad stuff the other things that are 58:38 totally crazy now the next thing 58:41 we need to do is to make 58:44 cells comparable so cells 58:48 still have different uh cells that 58:51 have different measurement qualities 58:54 they're still 58:55 different based on technical reasons so 58:57 for some cells we have a lot of 58:58 information 59:00 now we have a lot of these counts that 59:02 we detected 59:03 and in other cells we have less but we 59:06 want to make them comparable to each 59:08 other 59:08 and that's why we have to normalize the 59:11 data 59:12 yeah and there are fancy ways in 59:14 genomics there are fancy ways of doing 59:16 this and you can see we're 59:17 doing all of that basically and uh 59:21 so but we have to normalize the data you 59:24 have to make them comparable that's what 59:26 you always have to do 59:27 and what we also do here is because 59:30 these 59:30 these uh data counts you could see 59:34 like in the matrix that i showed you 59:36 there were very large numbers and very 59:38 small numbers 59:40 yeah and what we have so these 59:43 counts they live on an exponential scale 59:46 that's their these distributions are 59:48 very skewed it's a few cells that have a 59:51 huge amount or a few genes 59:52 that have a huge amount of these counts 59:55 of these 59:56 measurements yeah and that's not 59:58 something that works very well with 60:00 dimensionality reduction 60:02 or clustering methods so we take the 60:04 logarithm 60:05 we log transform the data for 60:08 any further processing now that's also 60:11 something that you do if your 60:13 data is too spread the other it comes 60:15 from some 60:16 exponential process also yeah then you 60:19 have the log transformers you want it to 60:21 be something 60:22 like normally distributed something 60:24 symmetrically and 60:25 rather compact as a distribution 60:30 yeah okay let's go on so here we can do 60:33 a variance stabilizing 60:35 uh transformation and we do more stuff 60:37 on the data 60:40 and then we can go and now we can start 60:44 to understand the data the first thing 60:46 we need to do 60:48 is to see does it actually make sense 60:51 what we do 60:52 what we have here are we actually 60:54 looking at biological information and 60:56 real information 60:57 or are we just looking at technical 61:00 aspects 61:01 of the experiments and as a first step 61:05 what we do here is we 61:08 plot a little pca fabian showed that 61:11 last week what a pca is 61:13 and these data on the pca on the 61:15 principal component analysis plots 61:18 they look like this and you can see 61:21 so here i plot for example the total 61:24 amount of these counts in a cell you 61:27 know that's a 61:28 that's a technical that's just measuring 61:31 the quality 61:33 of the measurements and you can see 61:35 there is some variability here so that 61:37 these cells have a little bit more these 61:40 cells are a little bit more 61:41 less and some of this technical 61:43 variability is captured 61:46 by the um by these experiments 61:50 by this principle component analysis so 61:53 here we're fine 61:54 with it it's not extreme we don't have 61:56 disconnected clusters 61:57 so we are fine with this it's already in 62:00 a good shape 62:01 and also we know that in cells can have 62:05 these differences 62:06 in the total number of molecules in the 62:09 cell for biological reasons 62:13 now and then we can look at these plots 62:15 here what is the percentage of variance 62:18 actually explained by certain principal 62:21 components now that's actually the 62:23 y-axis is 62:24 something else but we can order these 62:27 principal components 62:29 based on how much they explain the total 62:31 data 62:33 and what we do and that this is sort of 62:35 saying like a very professional 62:37 way of doing things is we do all other 62:39 calculations not on the real data 62:43 but on the principal components of the 62:45 data yeah that's an 62:46 intermediary step that we do just to get 62:49 cleaner results in the end 62:51 so so what we do here is we take the 62:53 first 62:54 20 25 or so principal components 62:58 that constitute like 99 of the variance 63:01 of the total 63:02 plot and we say the rest is noise that's 63:04 a way of getting rid of the noise 63:06 in the data and now we go on 63:10 yeah and do dimensionality reduction 63:12 further dimensionality reduction 63:16 let me make this larger 63:24 now i hope you can see these plots 63:28 so this is a u map yeah so this is the 63:31 this is a umap 63:32 uh that is a non-linear way of reducing 63:36 the dimensions that 63:37 fabian showed to you last week 63:40 and you can see once we do the 63:42 non-linear dimensionality 63:44 reduction our data looks already much 63:47 more structured so these cells here 63:51 are from actually from the brain these 63:53 are brain cells 63:55 and of course there are different kinds 63:57 of cells in the brain 63:59 and because they're different kinds of 64:01 cells in the brain we also 64:03 expect here a structure 64:06 to pop up in these low-dimensional 64:09 representations 64:11 typically these clusters correspond to 64:13 different kinds of cells 64:16 yeah so we don't know that's just a gray 64:18 bunch of cells here 64:19 of dots and the two dimensional planes 64:21 we don't know what the x's are 64:23 and we don't know what these cells are 64:25 and now we have to dig a little bit 64:26 deeper 64:28 and we do clustering so here is 64:31 clustering 64:32 and that's clustering that's one of 64:35 these community based clustering 64:37 algorithms 64:39 and we did that for several 64:43 resolutions of the clustering yeah so in 64:45 clustering 64:46 in most of the time you have to tell the 64:48 algorithm 64:49 how many clusters you want yeah 64:53 and uh that's that's kind of a 64:54 resolution 64:56 that you give to these algorithms and uh 64:59 by doing this 65:00 you uh and what you see here are 65:03 different clusterings 65:06 with different resolutions now so here 65:09 you say okay 65:10 give me what is that 15 clusters 65:13 then you got this plot on the bottom 65:14 left if you say okay 65:16 give me what is it eight clusters then 65:19 you get this 65:20 these clusters on the top left 65:24 and you can have more clusters if you 65:27 want yeah 65:28 they're different stages but we don't 65:30 know yet 65:31 what makes sense yeah we don't know how 65:33 many real clusters there are in the data 65:37 but we can take all of them and we can 65:39 go one step further 65:41 how do we know that such a cluster is a 65:43 real biological cluster 65:46 we know that if the cells in a cluster 65:50 all share some property that is not 65:53 shared 65:54 by other that is not shown by other 65:57 cells 65:59 now then we know that this cluster is 66:01 something real 66:02 uh that really is going on in the brain 66:06 and the way we do that is 66:09 let's go down then we look at the 66:12 literature 66:13 so now we look at the literature we look 66:15 at papers 66:16 yeah so we say we look at papers and 66:18 then in these papers 66:20 we see okay there are different cells 66:22 and kinds of cells in the brain 66:25 and people have done experiments 66:28 genetic experiments for example where 66:30 they found for example 66:32 that stem cells express a certain gene 66:36 now for example this gluolar gene here 66:38 that's expressed by stem cells 66:40 and now we plot this umap with the color 66:45 representing how much of the products of 66:48 this glue 66:49 gene we found in a certain cell 66:53 and now that makes a little bit sense 66:54 and now here in this corner here on the 66:56 top left 66:58 these are our stem cells 67:01 and then we can see okay so there are 67:03 also neurons in the brain and 67:05 many other things let's what's going on 67:07 yeah so there's another cell type that 67:09 expresses this 67:11 cell yeah that's the more advanced cell 67:13 type 67:14 from stem cells yeah that's all now in 67:17 the next step here let's express 67:18 in these cells here and then you could 67:21 can go down 67:22 and identify all of these clusters 67:27 step by step and identify 67:30 what kind of cells you have in the data 67:34 yeah you can do that with more genes 67:37 even 67:38 so there's a lot of genes like these 67:40 genes here that identify neurons 67:43 and different kinds of neurons so here 67:45 we have a little 67:46 just feature of this plot that is 67:48 identified by this gene 67:50 and if you talk bibologies all of these 67:52 names uh 67:54 are associated with different shapes or 67:56 different functions of cells 67:59 we can also do more fancy stuff and to 68:01 look at different gene scores and add 68:03 groups of genes 68:05 do statistical computations yeah and 68:08 once we've do 68:09 done that we decide okay 68:12 for these set of genes here we have a 68:15 unique 68:16 we fulfill this condition that each of 68:18 these clusters here 68:19 for these clusters we fulfill the 68:21 condition 68:22 that each of these clusters has 68:26 a certain biological function or 68:28 represents a biological function 68:30 because we found a gene in the 68:32 literature 68:33 that corresponds to a certain cell type 68:35 in the body 68:37 that is expressed in one of them but not 68:39 in the others 68:41 yeah so that's uh that's very important 68:45 and then we can give these clusters 68:47 names here for example 68:49 retinol or radioglia cells 68:52 uh uh only oligodendrocyte 68:56 precursor cells and so on and neurons 68:59 and then you find okay so these orange 69:01 ones here are the neurons 69:02 yeah and uh here were the stem cells 69:06 and then you can start thinking okay 69:08 these sensors somehow 69:09 turn here into these neurons and they 69:12 mature 69:13 now they get they get more mature and 69:15 then at the end they turn into these 69:17 neurons here 69:18 and then we have other cell types in the 69:20 brain like microglia 69:23 and so on that we also can find here 69:27 now remember in the u map the 69:31 distances in the u map is the humor 69:35 keeps the global topology intact 69:38 now that means that cells that are close 69:40 in here are 69:42 actually also very similar on uh 69:45 in the in this high dimensional space 69:49 so it's actually tempting that you think 69:50 about these 69:52 paths here as a trajectory that cells 69:55 take while they take from while they go 69:58 from stem cells 70:00 into neurons in the brain 70:04 now so let's go on now there's a lot of 70:06 consistency checks 70:08 yeah so you have to check all kinds of 70:10 genes 70:11 yeah a lot of them and discuss a lot of 70:14 with the people who actually know 70:16 who do nothing else in their lives but 70:17 to look at these cells and the brains 70:19 and they know 70:20 all of these genes and all of the papers 70:22 in these genes 70:23 yeah and i also do more fancy stuff 70:28 yeah and uh then you can do with 70:32 different clustering uh and now we have 70:35 a 70:36 identification and this one i said is 70:38 the one 70:40 that we can live with with these eight 70:42 clusters 70:43 while for these higher clusters now we 70:46 have 70:47 several several clusters representing 70:49 the same cell type 70:51 and we can in principle come back later 70:53 to these higher clusters 70:56 and typically biologists once want you 70:58 to do to get as many clusters 71:02 as you can yeah um 71:05 yeah so now we have these classes and we 71:06 can have some measurements 71:08 of how good these clusters are actually 71:12 and there are specific plots for these 71:15 for this 71:15 for example these are called dot plots 71:18 and 71:19 um what these plots show is on the 71:22 x-axis 71:23 our gene names and on the y-axis 71:27 are the cluster names yeah 71:31 and the color represents how much 71:34 a gene on average is present in the 71:38 cluster 71:40 and the size of these plots tells you 71:44 how what is the fraction of cells that 71:47 have this gene on 71:49 in this cluster now so and if we go now 71:53 here 71:54 what we want to see is that these genes 71:56 that we have here 71:58 are only on in one cluster but not in 72:01 other clusters 72:02 now for example a good thing is this one 72:05 here 72:06 this only go this oc cluster 72:10 now there's only one cluster here 72:13 there's only there are these these genes 72:16 here that we have 72:17 identified for this cluster that is 72:20 called 72:21 marker genes the other computation tools 72:23 to detect them 72:25 they are only you only find that in this 72:28 cluster 72:28 but not in the other cluster same for 72:32 this mg this microglia 72:34 we only find these genes in this single 72:36 this one cluster but not in any other 72:39 cluster 72:40 and while we go for what's a messy one 72:44 opc's 72:45 or we go for this neurons here 72:49 then it's more messy and it's not so 72:53 clearly defined 72:54 so if we go back now for example 72:57 now if we go here if we look at these 72:59 plots then these 73:01 mg's microglia they were very clean 73:04 in this plot that i just showed you yeah 73:07 and that's also represented in this new 73:09 new 73:10 you map they're a different cell type 73:14 that is not produced presumably by the 73:17 same stem 73:18 cells as the neurons 73:21 also these ocs yeah as oligodendrocytes 73:25 i think 73:26 yeah that's separated from the rest so 73:29 there's no overlap here there's a 73:30 distinct cell types 73:33 while for things like that we had an 73:35 overlap between these material neurons 73:38 at the sixth cluster that's called six 73:40 here 73:41 yeah there we found that 73:44 there is an overlap actually in these 73:46 marketers they have the same genes 73:48 that they express and probably that 73:50 means that this cluster six 73:52 is an artifact yeah that we cannot take 73:55 too seriously 73:56 and on the right hand side you can see 73:58 that in the next step now we took these 74:01 i have went a little bit broader and 74:03 took this cluster six 74:04 into the interval neurons here 74:08 now so now of course i showed you 74:10 basically a finalized version 74:12 normally you go back between these steps 74:14 and the earlier steps 74:16 again and again the call it back to the 74:17 quality control 74:19 until you have that do not have any 74:21 trace 74:22 of experimental technical parameters 74:25 in your plots and then also you go back 74:29 between these clusterings your 74:31 experimental friends 74:32 who do the experiments and the 74:35 literature 74:35 until you find something that is really 74:37 corresponding to 74:40 to to what makes biologically sense 74:43 it's not that these techniques give you 74:46 automatically 74:47 something uh that this was like a 74:50 mathematical criterion 74:53 that would tell you uh what makes sense 74:55 and you have to do that always have to 74:56 do that yourself 74:59 that's not that you push a button and 75:00 then everything is 75:02 works automatically and that's why these 75:04 people who do these kind of 75:06 analysis are very much looked looked for 75:09 on the job market 75:12 okay so uh let me just see if there's 75:15 any 75:16 other anything other interesting i think 75:17 there's you can go on and on forever you 75:20 know you can give the experimental lists 75:22 you know so which genes are on and off 75:25 in which cells 75:26 and once the experimentalists have these 75:29 lists they can do more 75:30 experiments uh they can create for 75:33 example 75:34 new animals that lack these genes 75:37 and what i want to show you 75:40 is now something that's on the bottom 75:44 you know so that the analysis is very 75:46 lengthy 75:50 now there are many aspects of course 75:51 that are irrelevant for biology 75:54 um i just want to show you 76:01 a lot of stuff a lot of stuff now that's 76:04 what the biologists are interested in 76:05 right 76:06 so these they hold these jeans um 76:09 all the stuff all the stuff a lot of 76:11 stuff even more stuff 76:20 yeah if more more more more also a lot 76:22 of calculations 76:24 yeah more heat maps to check 76:28 you know which genes are on where and so 76:30 on yeah so that's all 76:32 uh uh consistency checks 76:35 and one thing i want to just to show you 76:38 is uh something that's called let me see 76:40 do we have 76:42 [Music] 76:43 okay is something that's called 76:46 trajectory inference you know that's 76:48 so so so these are cells from a brain 76:52 and what cells do is they start that 76:54 they divide and they produce other cells 76:57 and then they get more specialized over 76:59 time 77:00 so these cells they start as a stem cell 77:03 and then 77:04 they mature until they at some point 77:06 they are in you 77:07 a new one so what we did here is 77:11 uh what you did here and what you do in 77:12 many cases is you take 77:14 okay i have a snapshot data so this 77:17 fish or these fish were killed for the 77:20 measurement 77:21 i don't have a time course or anything 77:23 now i don't uh as a snapshot measurement 77:26 just one measurement 77:28 but i have different cells that are at 77:30 different stages of this dynamic process 77:33 of cell maturation so what then people 77:36 do is uh can we get the temporal 77:39 information back 77:40 yeah and this is the trajectory 77:42 inference of how 77:44 these different clusters and cell types 77:46 relate to each other 77:48 and where you then can calculate rates 77:51 of how 77:51 one cell go one cell type 77:54 the flux that from one cell type leaks 77:57 leads to another cell type 77:59 now and in principle based on this you 78:01 can then come up with stochastic 78:03 models that you can in principle also 78:05 compare them to 78:06 uh to other experiments or theoretical 78:09 work 78:10 now that's i just wanted to show you uh 78:13 at the end 78:13 and let me just go through here if 78:15 there's anything else that's worth 78:17 showing you 78:18 um a lot of stuff now then you can 78:21 compare to humans and other animals 78:24 you can see where there are similarities 78:27 uh 78:27 these fish here that we're looking at 78:29 are very interesting because they can 78:31 regenerate their brain they can build 78:33 new neurons 78:34 that's something we cannot do so it's so 78:36 much that we want 78:38 we want to learn what are the 78:39 similarities and difference why are we 78:41 not 78:41 able to do that as humans yeah and 78:45 um yeah and then uh 78:48 we are already you know at the end of 78:51 this very lengthy analysis 78:53 yeah so this is a typical data science 78:56 project 78:57 from start to end and so you can see 79:00 here a mixture of 79:01 r and python and 79:05 the important thing is that you cannot 79:07 what we showed you 79:09 in the last lectures you cannot just go 79:11 and take the data and 79:12 throw a umap on it or do some machine 79:15 learning on it 79:16 so a large part of the data 79:20 of this pipeline here is to actually 79:24 clean up the data and think about what 79:26 is the part of the data that makes sense 79:29 and what is the part of the data that 79:31 does not make sense 79:33 yeah so for example think about this new 79:34 york city flights 79:36 it doesn't make sense to have a negative 79:37 departure delay and it was a 79:39 it was a very uh that was a very good 79:41 question 79:42 yeah it actually uh makes sense and this 79:44 is so to say a data set 79:46 that is used for for teaching a lot so 79:48 it's already cleaned up a lot but 79:50 typically you expect 79:51 to have a lot of nonsensical 79:54 measurements 79:55 uh in your data yeah or sometimes you 79:57 have a departure delay of 79:59 10 billion years or so that that would 80:02 be the correspondence in a real data 80:04 somebody made a typo somewhere yeah and 80:06 then you have that in your data set and 80:08 you have you have to 80:09 have to filter it out again now this 80:11 happens all the time if you 80:13 are not looking after this yeah so if 80:16 you're not taking care of this 80:18 then all of these nice plots here that i 80:20 showed you 80:21 they won't work now they only work 80:23 because we clean up the data 80:25 we normalize the data we transform the 80:28 data in a statistical way to make it 80:31 uh nicely behaved yeah it's 80:34 no huge outliers or so and then these 80:37 methods 80:38 like uh umap and so on work on the data 80:42 but you always have to do these 80:44 pre-processing steps 80:46 before that to make that work and the 80:47 next step is you you have these fancy 80:49 methods 80:50 but taking a loan they don't make sense 80:53 so you can see here how this order has 80:56 ordered this 80:56 trajectories cells move here along in 80:59 time 81:00 move along this line and turn into 81:02 neurons here 81:04 and of course we have many different 81:05 neurons in the brain 81:07 yeah and but to understand that what's 81:09 happening here this data to make sense 81:11 of this 81:12 now you have to come up with hypothesis 81:15 you have to connect these hypotheses 81:16 with 81:17 what is already out there in the 81:18 literature and then step by step you can 81:21 construct 81:22 an understanding about what are actually 81:24 here the degrees of freedom 81:26 that you see in this data set you know 81:29 and so this is this is an example and 81:32 this 81:32 is an iterative process that you improve 81:36 over time and so this is an example 81:39 that's a purity 81:40 purely data science project or there's 81:43 very little physics in it 81:45 and next time we show you how all of 81:47 these connects to something that we 81:49 actually 81:51 did in the first part of this lecture 81:53 namely field theory 81:54 face transitions criticality and so on 81:58 okay great so i'll stay online and in 82:01 case there are any questions otherwise 82:03 see you next week 82:08 bye 82:21 thank you it was very interesting yeah 82:24 so when are you when are you going

本文收录于以下合集：