这一讲由嘉宾给出，官网没有课件。所以不再做课件帧数的标记了。作为替代，随堂做了笔记，参见笔记部分。
00:07 okay so let's
00:08 uh let's join uh so welcome
00:12 to our uh lecture again now to this uh
00:16 today we have a special guest as we uh
00:19 which is fabian
00:21 ross that you see on the other window
00:23 and as you remember
00:25 uh we before christmas we introduced uh
00:28 theoretical physics concepts that
00:30 explain how order emerges
00:33 in non-equilibrium systems and then last
00:36 week i started
00:37 introducing some basic tools
00:40 from data science namely what are the
00:43 tools that we can use
00:45 to computationally deal with large
00:48 amounts of data
00:50 and today so i need some special help
00:53 from somebody who's a real expert
00:56 which is far beyond you know fabian is a
00:59 trained
01:00 uh physicist and did a phd in
01:02 mathematical biology
01:04 i then did a postdoc in my group at the
01:07 max plain institute in dresden and now
01:10 he's a professional
01:12 uh data scientist and bioinformatician
01:16 at the center for regenerative therapies
01:18 in dresden
01:20 and so he's talking actually about
01:21 something that he's uh
01:23 doing for his professional job which is
01:25 very nice
01:26 and uh over to you fabian thanks a lot
01:29 for
01:30 for sharing your expertise yeah
01:33 thanks thanks for having me here uh for
01:36 giving miss
01:37 uh giving me this opportunity so to give
01:39 a lecture on
01:40 exploring high dimensional data so i
01:42 think uh stefan
01:44 introduced me very well thank you very
01:47 much and
01:48 and also made the point of where we are
01:51 in the course
01:53 so i just wanted to ask you i just would
01:56 uh to say that uh whenever you
01:58 uh wanna ask something i think during
02:00 the course you just like uh unmuted
02:01 yourself and ask the question so that's
02:03 totally fine to me also
02:05 type something in the chat and i try to
02:07 try to answer and then uh
02:09 like if you're not saying anything uh
02:11 please mute yourself to reduce noise
02:14 um
02:18 all right so um okay i'm
02:22 i'm going to start right away
02:25 um so today will be about um exploring
02:29 uh high dimensional data
02:30 right and as many of you
02:34 know with a high dimensional and in high
02:36 dimensions
02:37 strange things can happen so if you for
02:39 instance look at this thing here what
02:41 you see here is a three-dimensional
02:42 projection
02:43 of a four-dimensional object rotating
02:46 and this
02:47 looks very funny right and i mean like
02:49 four dimensions is where we kind of
02:51 still can kind of get a glimpse but but
02:53 what if it gets to
02:54 thousands or millions of dimensions what
02:56 do we do with these kind of data
02:59 um so before i start
03:02 uh let me let me give you a brief
03:04 structure of the course
03:06 today of the lecture um
03:18 might have to click into the window
03:21 no i actually want to share okay
03:25 my ipod screen yes sorry for the delay
03:29 um all right so so i basically start
03:33 by introducing what i mean by high
03:35 dimensional data
03:41 um i will then
03:44 cover two dimension
03:48 dimensionality reduction methods
03:51 [Music]
03:58 and also talk a bit like what's the idea
04:01 of of
04:01 reducing dimensions and third if we
04:04 still have time i'll talk a little bit
04:06 about
04:06 clustering which we kind of can think of
04:08 as a discrete form of dimensionality
04:11 reduction i'd say
04:13 so what do i mean by high dimensional
04:16 data
04:18 well let's let's start with an object
04:21 that i call a data matrix
04:28 um x which is of shape n by m
04:41 and maybe you remember from last week um
04:45 tidy data formats so this um uh
04:49 matrix should be in a tidy format
04:50 already so in the rows we have
04:52 n samples
04:57 and in the columns we have m
04:59 measurements or observables
05:08 um so what do i mean by this so so an
05:11 example could be what you what you
05:12 measure here is for instance you um
05:15 uh take a uh take yourself right and you
05:19 measure for instance the length of your
05:21 thumb and the length of your index
05:22 finger and the length of your middle
05:24 finger and so on you do the same thing
05:26 on there
05:26 with the other hand you measure your
05:28 foot size your body height
05:30 maybe uh the heartbeat rate and so on
05:32 and and uh
05:33 you end up gathering more and more
05:35 observables
05:38 and that's m observables and this gives
05:40 us the dimensionality of our data
05:44 um and now you can do this with yourself
05:47 and you can do this well you can ask
05:49 your friends to do it with them and so
05:50 on and then
05:52 you can maybe do it with all people in
05:55 asia all over the world and this uh
05:57 gives you the number of samples that you
05:58 have right
06:00 um now how do you how do you treat these
06:04 this kind of data right if we go one
06:06 step back
06:07 and uh think about how how can we
06:10 visualize
06:11 and uh stephan uploaded some slides on
06:14 that and and maybe we'll also talk on
06:16 that a little bit but
06:18 what can we what can we actually
06:20 visualize easily
06:22 um
06:28 so if you uh if you would have one
06:31 dimensional data right that's rather
06:33 easy right
06:34 you you can um just uh plot your data
06:38 on a one-dimensional line for instance
06:40 and also
06:41 for two dimensions
06:45 um i think this is pretty
06:47 straightforward right
06:49 um so if this would be the
06:53 entries of our data matrix x1 x2
06:57 um we can just plot them as a scatter
07:00 plot right
07:02 and we could for instance see whether
07:03 they are correlated or not
07:05 but um as soon as we are like
07:08 in in three dimensions or larger like
07:10 three dimensions we can still
07:11 handle we can do a three-dimensional
07:13 plot maybe but this is still
07:15 hard to quantify um this this gets um
07:19 this gets uh non-trivial right and so we
07:22 need techniques to
07:24 to uh think about how to actually look
07:26 at these data and how to analyze these
07:28 data so
07:30 the aim of dimensionality reduction then
07:35 i'd say is to reduce the number of
07:42 observables
07:50 while keeping
07:57 relevant information
08:04 and what this relevant information is
08:06 right this is something uh
08:08 you you have to decide by the problem at
08:11 hand
08:12 um now um
08:15 before i go
08:18 into talking about how you can reduce
08:21 dimensionality
08:22 i wanted to show you some examples
08:26 so i'll share some slides for this
08:33 um and so
08:37 i think as many physicists are in the
08:39 audience one example i wanted to share
08:41 something
08:41 that i think many of you are already
08:44 know
08:45 which is um where we know where many of
08:48 you know that a
08:49 low number of observables actually
08:51 suffice to to describe the system very
08:53 well and that's in the realm of
08:54 statistical physics and you
08:56 saw a lot of uh examples from stefan by
08:58 that
08:59 but even in a rather easy example like
09:01 an and paramagnetic
09:03 iso model right you know that there's uh
09:07 there's many many spins in this system
09:09 right and imagine you could measure all
09:11 these spins
09:12 um you have a very high dimensional
09:15 representation of the state of the
09:17 system then
09:18 but many of these spins will be
09:20 correlated and we know from statistical
09:22 physics that actually you can
09:24 very well describe the microscopic state
09:26 of such a system
09:27 with just the magnetization
09:30 now what will happen if we come from the
09:32 other side from the data science
09:34 perspective and that's a
09:36 paper here an archive by sebastian
09:38 vetzel and what he did was
09:40 um he
09:44 did a simulation of a ferromagnetic
09:46 icing model on a 28 by 28 letters
09:49 at different temperatures and so 28 by
09:52 28 gives
09:53 you 784 spins right and we he recorded
09:57 the state of all these spins and 50 000
09:59 different samples at different
10:00 temperatures
10:02 and now you have a data matrix with 784
10:04 dimensions right and then you can start
10:07 to
10:07 say okay if i just would have this
10:10 matrix and i wouldn't know nothing about
10:11 the system how can i look for order in
10:14 this system and how can i reduce
10:15 dimension and one of these techniques is
10:18 called principal components analysis and
10:20 i'll explain later what this actually is
10:22 but if you do this you get a result like
10:24 this the first principal component
10:27 which is a one-dimensional
10:28 representation of of this
10:31 high-dimensional data matrix perfectly
10:33 scales with the magnetization
10:35 so in this way you would actually be
10:37 able to kind of discover
10:39 an order parameter without knowing
10:42 anything about the system
10:43 by having detailed microscopic data of
10:45 the system
10:47 so this is rather an artificial example
10:50 right because we know very well but how
10:52 how an ising model is behaved but what
10:54 about
10:55 systems where we don't really know how
10:57 they work for instance humans
11:00 um so this is a paper from nature in
11:02 2008
11:04 last authors bustamante and
11:07 what they did was they quantified human
11:09 dna
11:10 so they took 3 000 european humans
11:14 and now you can imagine that the dna is
11:18 a long string of
11:19 characters right and and to a large part
11:22 these this is conserved from one human
11:25 to
11:25 an to another human so the dna will be
11:28 pretty much the same
11:29 but at some positions we have variations
11:31 right the dna from one human to another
11:33 is not exactly the same and at half a
11:35 million positions
11:37 the orthos quantified the dna state
11:40 and so this gives you a a a data set
11:43 with a dimension of half a million
11:45 approximately for 3 000 humans and then
11:48 they did the dimensionality reduction
11:49 and again they did a
11:51 principal component analysis and this is
11:53 a two-dimensional representation of this
11:55 high dimensional data set
11:57 and what i want to ask you now is to
12:00 look at the picture and you don't
12:01 yet have to understand what the
12:03 principal components mean but i i want
12:05 to ask you to type in the ted
12:07 in the chat if you can what you see in
12:10 this picture
12:23 it's a very difficult or very shy
12:25 audience
12:30 what does it look like it look
12:33 if the meat looks like an elephant
12:38 it is a scatter plot yes and uh we see
12:41 some clusters yes
12:43 there's also uh some sensible answers
12:47 yes and it's actually um uh the uh
12:53 now i don't see my mouse anymore i think
12:57 adolfo got it right
12:58 yes i don't forgot it right it's already
13:00 the second answer
13:01 um it resembles a map of europe and
13:09 that's exactly what the title of the
13:11 paper was uh jeans mirrored geography
13:13 within europe right so
13:15 this was uh i think it's quite amazing
13:17 so you just
13:18 record the dna of humans right
13:21 and basically this gives you a map of
13:24 europe
13:25 approximately it kind of makes sense
13:26 right that the genes are
13:28 like um uh like um
13:31 well somewhat remain local
13:36 um so here's another example
13:40 um there's a very i'd say maybe
13:42 unphysical example but i think it
13:44 it works nicely these kind of examples
13:47 using
13:48 images work nicely to explain
13:50 dimensionality reduction techniques
13:52 so this is a fashion data set called
13:55 fashion mnist
13:56 provided by zalando research and
14:01 again you can uh these these images are
14:04 on a 28 by 28 letters
14:06 uh in a gray scale so you can record the
14:10 intensity of each pixel and you end up
14:12 with having 784 pixels on
14:14 of 60 000 pictures and there's different
14:17 kinds of pictures
14:18 like there's uh shirts and tops and
14:21 trousers
14:22 and there's also shoes i think in there
14:24 and then you can do a dimensionality
14:26 reduction and here
14:29 this is an example for a umap a uniform
14:32 manifold approximation i'll explain
14:34 later what this is it's a non-linear
14:36 dimensionality reduction technique
14:38 and again you can see a number of things
14:41 um
14:42 by just taking this as as vectors right
14:45 and
14:45 projecting them in two dimensions you
14:47 see that uh for instance
14:50 um i think this up here which is
14:51 trousers forms one cluster
14:54 which is very different for instance
14:56 from a cluster which consists of
14:59 sandals and sneakers and
15:02 ankle boots so this is the kind of shoes
15:04 cluster
15:05 uh you have bags here which seem to have
15:07 some similarity to
15:09 tops at some point in a certain way
15:13 so um this is a again dimensionality
15:16 reduction right and
15:18 you can see how you can how you can use
15:20 it to group together different kinds of
15:22 things in in this case just pictures uh
15:25 what you also see
15:26 here is that these things cluster so
15:28 there's a discrete kind of order
15:32 and i'll talk about clustering also
15:35 again
15:35 in the second part of the lecture
15:38 um so the now i'll go into
15:42 dimensionality reduction
15:44 and how it works and i'll start with a
15:46 linear dimensionality reduction
15:48 there's a number so so this is just the
15:50 linear transformation of your data
15:52 there's a number of techniques around um
15:55 for instance principle component
15:57 analysis i'll explain this and there's
15:59 other
15:59 uh techniques like uh non-negative
16:02 metrics factorization or independent
16:04 component analysis
16:06 or factor analysis they are they are
16:09 kind of similar in that they are linear
16:12 but i only have time to cover one of
16:14 them which is the easiest one which is
16:15 pca
16:17 um for that i'll again
16:21 share my ipad
16:41 all right what's principal component
16:42 analysis i'll first sketch the idea
16:47 um so for the i for forgetting the idea
16:50 it's sufficient to go into two
16:51 dimensions and i'll
16:53 make a plot a scatter plot of
16:56 two-dimensional data
16:58 where let's assume our data looks a bit
17:02 like this
17:06 all right it's essentially very well
17:08 correlated and
17:09 from looking at this now if you would uh
17:12 give me
17:12 x1 right and i would know x1 i could
17:16 very well predict x2 because i
17:18 i see that these two observables are
17:21 very well correlated and then
17:23 pca is a way to formalize
17:26 this and the first principle component
17:28 is designed in a way that it points in
17:30 the direction
17:32 of highest variability of the data and
17:35 in this example this would be this
17:36 direction
17:38 and this would be principal component
17:40 one and then the next principal
17:41 component
17:42 has to be orthogonal to the uh to pc1
17:46 and again point in the direction of uh
17:48 maximum variability of the data and
17:50 there's only one direction left here
17:52 which is
17:53 uh the orthogonal direction here and
17:54 that's pc2 principle component two
17:58 so then you just change the bases of
18:00 your data
18:03 through the directions of the principal
18:05 components
18:06 and then you end up with your
18:08 representation of your data
18:13 like this
18:19 where now in this directions your data
18:21 would be uncorrelated
18:24 and you would have
18:27 most of the variability in this
18:29 direction right
18:32 so the aim will be
18:38 to find a transformation
18:50 such that the components are
18:54 uncorrelated
19:03 and that and that they are ordered
19:08 by the explained variability
19:28 now how does the math of this look like
19:36 sorry
19:50 mathematically we would start with
19:53 a cr with an eigen decomposition of the
19:56 covariance matrix
19:58 um the
20:01 covariance matrix is an object which we
20:03 get from
20:04 or at least this is proportional to this
20:07 object it's the
20:08 data matrix transformed times the data
20:11 matrix itself
20:13 and you will see that this is um
20:16 a matrix of the shape m by m so it has
20:19 the dimensionality of
20:21 our data and
20:24 if you if you do the computations you
20:27 will see that the
20:28 diagonal elements of these objects are
20:31 proportional to the variability of our
20:33 data in this direction and the
20:35 off-diagonal elements
20:37 are proportional to the covariance of
20:39 components and then we do an item
20:46 decomposition
20:50 and from this we get m eigenvectors
20:53 lambda
20:53 i am eigenvalues lambda and a
20:57 and m i vectors and i group them
20:59 together in a matrix w
21:02 so the columns
21:07 of this matrix are eigenvectors
21:14 of the covariance matrix and importantly
21:18 what we do
21:19 here and we can always do this is we
21:21 sort them by the
21:23 we sort them by the magnitude
21:29 of the eigenvalues lambda i
21:33 and this ensures that the first
21:34 principal component points in the
21:35 direction of highest variability
21:38 now w is just the new uh
21:42 basis for our data and then we can
21:44 transform our data
21:47 so like this the the transformed data
21:51 will look like this we take the data
21:52 matrix
21:53 x times w
21:56 um so in the language of principal
21:58 components this is what's called
22:00 loadings
22:04 or also the rotation
22:09 and this the transformed data is called
22:11 scores
22:20 now i leave it to you as an exercise to
22:22 show that
22:24 the transformed data matrix
22:28 is a dynamic is a diagonal matrix
22:33 which basically means in this transform
22:36 space the components are uncorrelated
22:39 now the data matrix again here is of
22:43 shape
22:43 n by m
22:47 so the dimensionality of this
22:50 is just the same as the dimensionality
22:53 of our original data so how does this
22:56 help in reducing the dimensions
22:58 well there the idea is because the
23:00 eigenvectors are sorted
23:03 by the magnitude of lambda i that we can
23:05 truncate it
23:07 and if we define wr
23:11 which is an n by r matrix as the
23:16 first r columns of w
23:24 then we can get a a projection in lower
23:28 dimensions of our data
23:30 just by using rw
23:34 at wr
23:39 and this will this object will be of
23:40 shape n by r
23:42 and if we for instance select r equals
23:44 two we have a two dimensional
23:46 representation of our data of course
23:47 this
23:48 throws away some information but most of
23:50 the variability of our data is captured
23:59 um yes so so the take home is that that
24:03 is that principle component
24:07 analysis is a linear transformation of
24:09 the data such that the components
24:11 are uncorrelated and ordered by the
24:13 amount of explained variability
24:16 and what i want to do next is i want to
24:20 show you how you can actually do this
24:22 in r which is uh quite easy and you
24:25 don't actually need to know the math for
24:26 this
24:28 um so let me
24:31 um excuse me yes uh could you
24:34 repeat how do we decide uh at what
24:39 at which point to truncate w and what
24:42 exactly do we mean by sorting my
24:43 magnitude of the eigenvalues lambda
24:47 okay um if you
24:54 uh this this is the
24:58 covariance matrix right so essentially
25:00 if i
25:01 if i look at this um i saw you didn't
25:05 do you see my mouse yes yes
25:08 um so lambda 1
25:11 will be proportional to the variability
25:13 in this direction
25:14 okay yes and lambda 2 to the variability
25:18 in this direction
25:21 okay and and so this
25:24 sorting by uh
25:28 by the eigenvalues ensures that the
25:31 first component
25:32 of the first principal component will
25:35 point in this direction of highest
25:36 variability of the data
25:38 right yes
25:42 um so this is what the what the sorting
25:45 means
25:47 uh okay so in this case lambda 1 is
25:51 that the reference that we sort through
25:54 that we get w
25:55 from because lambda 1 is proportional to
25:57 the pc
25:58 one direction um
26:00 [Music]
26:02 lambda lambda 1 is the eigenvalue right
26:05 that corresponds to the
26:06 to the eigenvector which is in this
26:08 direction yes
26:10 yes and it is actually and if you
26:13 if you uh if you if you do kind of
26:17 um
26:17 [Music]
26:20 if you look at this at the transformed
26:22 cobra covariance matrix
26:24 right you will see that in the diagonal
26:26 elements of this you just have the uh
26:29 lambdas right so it's the that's the the
26:32 variable the variance in this direction
26:35 okay
26:36 yes thank you exactly so
26:39 was this all i think i remember
26:42 there was another question uh you also
26:44 asked how to drink it how to decide
26:46 uh uh where we trunk it
26:50 um this is not
26:53 this is up to us um you can kind of you
26:57 you can
26:57 uh what you can what i will show you in
26:59 a moment is that you can decide that you
27:01 wanna
27:02 keep like uh ninety percent of the
27:05 variability of your data and this would
27:06 give you a threshold of where you
27:08 where you stop for instance but
27:11 uh this threshold again is arbitrary so
27:13 you can also just
27:15 i mean one usual decision that you do is
27:17 if you want to visualize your data you
27:18 truncate at two
27:20 because that's easy to visualize um
27:23 you might also take this as an initial
27:25 step
27:26 of dimensionality reduction to maybe
27:29 just like
27:30 going from half a million to 50 or
27:32 something and then do a non-linear
27:33 transformation
27:35 but essentially like in my everyday work
27:37 this is a uh
27:39 it's an arbitrary it's often an
27:40 arbitrary choice and yeah in the end
27:42 have to see whether it works or not
27:44 whether it's sensible
27:45 so that's a tough question and
27:48 there's also some research still on that
27:51 okay um
28:03 all right so i prepared a
28:06 um a
28:10 a document a markdown
28:13 document a notebook in our studio and i
28:15 think stefan briefly introduced our
28:17 studio to do
28:18 to you last week you can also
28:22 get the code on from this url and it's
28:25 also
28:26 by now on the on the course home
28:28 homepage
28:30 um also the code from the last lecture i
28:33 forgot to mention that it's also on
28:34 github
28:37 in the same folder yeah exactly
28:40 and then um well i wanted to show you
28:43 before is that like i i want to just
28:45 show you as an example one of these uh
28:47 famous uh uh data sets you you might
28:51 have heard of the uh it's it's a data
28:52 set where uh
28:53 botanist and statistician fisher
28:56 collected
28:56 uh flowers from a flower called iris
29:01 and there's different species iris
29:04 guinea car aristotoza iris versicolor
29:07 and these have different leaves the
29:10 petal and the sepal
29:12 and he measured the the sample length
29:14 and the sample width and the petal
29:16 length and the petal with
29:17 all these species
29:21 and it's a it's a data set that you can
29:23 easily load in in
29:24 r that's why i used it and you can it's
29:27 four dimensional so here you see the
29:29 data it's uh
29:31 the length width and length and width
29:33 for one species
29:34 and then the other species follow
29:36 somewhere in the data
29:38 and you have this four numbers
29:41 now you can try to visualize this uh in
29:44 a
29:44 for instance pair wise ghetto plots
29:47 plotting the length
29:48 of the sepal against the sample width
29:50 and so on
29:51 and here this is color coded by species
29:54 and you already see some separation
29:56 but it's hard to choose like in this
29:58 four-dimensional plot
30:00 what you actually what to actually look
30:02 at and so you can perform a principal
30:04 component analysis on this
30:06 four-dimensional data set and then r
30:08 this is just a one-liner
30:10 so the principal components analysis is
30:12 done by p or comp
30:14 and here i plug in the four-dimensional
30:17 data set
30:20 and then you get all the things out that
30:23 we just talked about you get this uh
30:25 transformation matrix w called rotation
30:28 here
30:29 um where now you see that the
30:32 the principal the direction of principle
30:34 component one in this four dimensional
30:36 space
30:37 right it's it's a vector um you see
30:40 you can look at the transformed data
30:43 um which is again four dimensional
30:46 at first um here you can
30:50 uh look at the explained uh uh
30:53 variability or explain variance
30:55 um and this is maybe for the question of
30:59 where to
31:00 cut off so that they explained
31:03 variance here you can see the cumulative
31:05 proportion of this
31:07 and with the first two principal
31:09 component you explain
31:10 95 96 of the variability of your data
31:14 and you might say okay
31:16 that's enough with uh and in in pc4
31:19 uh you from pc3 to pc4 you hardly add
31:23 something so the fourth dimension of the
31:24 data set is probably very
31:26 it's very correlated to the first three
31:29 there's not much information left and
31:31 then this is how the data looks now
31:34 in in the direction of principal
31:35 components one and two and what you can
31:37 observe is that
31:39 uh cross principle component one that's
31:41 the main direction which kind of
31:44 um discriminates between the species
31:47 you can also see that the setoza is very
31:50 different from
31:51 versicolor and virginica and here it's
31:54 hard to actually draw a boundary
31:56 but just based on the measurements
32:01 this is what's called a scree plot and
32:03 this is just again a visualization of
32:05 the variance
32:06 explained by each principal component
32:10 now i want to also show you an another
32:13 data set
32:14 where actually principal component
32:16 analysis has a hard job
32:18 to work so this is
32:21 again an image data set where images
32:24 were collected of different objects and
32:25 they are rotated
32:27 right
32:31 and now you can again do a vector
32:34 representation of these images to a
32:35 principal component analysis and this is
32:38 what the
32:38 picture looks like and what the plot
32:41 looks like now and
32:42 and the colors now uh represent
32:45 different objects
32:46 and the dots of the same color are then
32:52 pictures of the objects at different
32:54 angles and you can see
32:56 that if you do this dimensionality
32:58 reduction you can see some structure
33:00 right but it's not really uh doesn't
33:03 really work good in discriminating the
33:05 different
33:06 objects and you would expect that maybe
33:09 you can
33:09 reveal this discrete kind of order
33:11 different kinds of objects
33:15 or that's maybe what you want by doing a
33:18 dimensionality reduction
33:23 here's another example that that i made
33:26 up
33:26 which is uh a dimensionality reduction
33:29 of a swiss roll so what's this with
33:31 actually a swiss roll with a hole so
33:33 what i did here was to say okay i
33:36 um uh randomly place
33:39 uh points on a on a two-dimensional
33:43 surface
33:43 but there's a hole in the middle and
33:45 then i wrap this up
33:47 in 3d like this um
33:50 i transform it like this okay it's
33:53 it's this role and it's it's called it's
33:56 it's a suite actually in in switzerland
33:58 that's why it's called a swiss roll and
34:00 desk you can still see this uh
34:02 this uh hole in the middle now what will
34:04 happen
34:05 if you do a principal components
34:07 analysis of this
34:10 the example is shown here and this is
34:13 the same
34:14 data shown in principle component space
34:21 actually in 3d and so
34:24 what you can see is that
34:28 not much happened right it's pretty much
34:30 the same as before
34:31 just stay the principal component and
34:34 the reason for this is
34:35 that what we do is a linear
34:38 transformation of the data right
34:40 but what i did before was to take two
34:42 dimensional data and
34:43 non-linearly
34:47 projected into 3d well use the
34:50 non-linear function
34:51 and so um projecting this back to 2d
34:54 in a way cannot work with a linear
34:58 dimensionality reduction technique and
34:59 that's why
35:02 non-linear dimensionality reduction
35:04 techniques are also very useful
35:07 so this leads me to the second part of
35:10 dimensionality reduction
35:13 which is about nonlinear dimensionality
35:15 reduction techniques and
35:17 as it's usually the case with non-linear
35:20 things
35:21 there's many of them and then actually
35:23 about one year ago
35:24 i had a look at wikipedia at the
35:28 like a list of some prominent algorithms
35:32 for dimensionality reduction and there
35:34 were 29 i'm sure now
35:36 there's more and this this doesn't claim
35:39 to be complete right so
35:41 um this makes very clear that depending
35:43 on what you want
35:44 there's different things that you can do
35:47 today i will concentrate on uniform
35:49 manifold approximation a rather
35:51 recent one introduced i think in 2018 so
35:54 three years ago
35:58 and it's used a lot in single cell
36:00 sequencing data in single cell genomics
36:03 nowadays
36:05 um so what is this
36:08 technique about so don't be scared
36:11 there's a lot of mathematics on this
36:13 page and you don't have to understand
36:15 all of that right away i won't also i
36:18 also won't have the time to fully
36:20 explain this in detail but i will give
36:22 you an intuition
36:23 of what this does uh
36:26 the basic idea is
36:30 that
36:32 [Music]
36:36 so the basic idea is that we have a
36:39 higher dimensional data set
36:41 but we hypothesize that there's a low
36:43 dimensional manifold so think of
36:45 a for means for instance there could be
36:47 a line in a two-dimensional manifold
36:50 right and then we want to discover this
36:53 dimensional
36:54 manifold so this
36:57 there's a couple of assumptions that are
36:59 made in new map one that the data is
37:01 distributed uniformly on a romanian
37:04 manifold
37:05 that the romanian metric is locally
37:07 constant and that the manifold is
37:09 locally connected
37:11 and what umap then gives you is
37:14 it models this manifold with a fuzzy
37:16 topological structure
37:18 and it finds a low dimensional
37:20 projection of the data
37:22 that has a very similar fuzzy
37:25 topological structure
37:26 to your original data so what does all
37:29 this mean
white board 1
37:30 um let's let's start
37:33 by i will start by by introducing
37:36 briefly very briefly introducing an an
37:40 important idea from topological data
37:42 analysis because
37:43 uh this algorithm is or this idea is
37:47 heavily based on topological data
37:50 analysis and this is simplices
37:53 a zero simplex is a point one simplex is
37:56 a connection of two zeros implices which
37:58 is a line then there's a two simplex
38:00 which is a triangle
38:02 a three simplex which is a tetrahedra
38:05 and you can see that you can easily
38:07 extend this contact
38:09 con concept to higher dimensions so you
38:12 can introduce a four simplex five
38:13 simplex and so on
38:16 and these objects are rather easy
38:19 to construct and the nice thing that you
38:22 can do is you can glue together such
38:24 objects such simplices
38:26 and by gluing them together
38:30 you can basically approximate um
38:33 money folds so um you might all have
38:37 seen already some point where you
38:39 uh triangulate the surface right and
38:41 this is nothing
38:42 then uh approximating this manifold with
38:45 simplices
38:51 now let's look at this example which i
38:54 took from the umap homepage actually
38:56 where you can see points which are
38:59 uniformly sampled actually equidistantly
39:02 sampled i think
39:03 on a not equidistant but uniformly
39:06 sampled on a
39:10 on a line
39:14 in a two-dimensional space right so we
39:16 have a one-dimensional manifold
39:18 in a two-dimensional space and
39:22 uh um so this would be could be our data
39:25 points that were measured
39:26 right and we would be interested in
39:28 finding the topological
39:30 structure of this manifold um so what
39:33 has been done
39:34 here is to draw some spheres around
39:36 these dots
39:38 and the union of all these spheres i
39:40 mean all these spheres are sets
39:42 and the union of all this is what's
39:44 called an open cover
39:45 open cover is just the thing that uh is
39:49 this is a um is a collection of sets
39:52 that span the whole manifold right so
39:54 the line
39:55 that we see here is inside this open
39:57 cover
39:58 and now you can do some heavy
40:00 topological and mathematics
40:02 and you can then prove that if you now
40:05 contract
40:05 uh construct simplices by uh all
40:09 groups of sets that in that that have a
40:12 non empty intersection which in this
40:14 case basically would mean
40:15 you construct a one simplex here line
40:18 here
40:19 and one simplex here and so on you i
40:21 think you get the picture
40:23 then if you do this and this would look
40:25 like
40:26 no um then this
40:29 structure this graph-like structure that
40:32 you get out of
40:33 this captures the topology of the
40:35 manifold
40:37 the information is encoded in there and
40:38 i think in this uh
40:40 two-dimensional example this this is uh
40:44 quite intuitive that the connection of
40:46 all these points
40:51 tells you about uh tells you that this
40:54 is a connected line
40:56 and there's no loops for instance
41:01 um so that's pretty cool and now we
41:04 would like to use this concept to
41:06 uh to learn something about our high
41:09 dimensional data
41:12 is the criterion for the connection
41:14 always like the
41:15 shortest distance between the points
41:18 because i mean here i could
41:20 see it but if i have more complex
41:22 structure
41:23 exactly exactly i'm actually coming to
41:25 this point in a second in this case
41:27 it was a bit like okay let's just guess
41:30 uh the size of the sphere
41:31 make it a good guess such that it works
41:33 right but if you have high dimensional
41:35 data
41:36 what would be the size of this sphere
41:37 that would be super unclear
41:39 so um i'll come back to this question in
41:42 a moment
41:46 it's also connected to a situation with
41:48 like this
41:49 right because um
41:52 our points might not actually be
41:54 uniformly distributed
41:55 okay so um if we would do the same thing
41:59 here
42:00 drawing these circles and
42:04 uh construct the simplices out of that
42:07 we would actually get disconnected parts
42:10 okay
42:11 and this is not what we would
42:13 intuitively like to get right
42:15 because here we
42:19 we saw this line um
42:24 now and this is also this kind of breaks
42:26 the assumption
42:28 of umap which is the uniform
42:30 distribution of uh
42:32 of data points on the money fold so
42:35 how can we circumvent this or how can we
42:37 solve this problem we can
42:38 kind of turn the problem upside down and
42:41 say
42:42 no these data points are uniformly
42:46 distributed we see different distances
42:50 if we apply a constant metric so this
42:52 has to be wrong the
42:53 metric is not constant actually we
42:56 couldn't
42:57 try to construct the metric like this
42:58 and this comes back to your question
43:01 we might say that the distance in this
43:02 case to the fifth
43:04 neighbor should always on average give
43:06 us a distance of five
43:08 right such that we say um distances
43:10 between neighboring data points should
43:12 on average be one
43:15 um and this would mean that that a
43:18 distance of
43:19 five for instance for this uh data point
43:22 would be rather large
43:23 and a distance of 5 for this standard
43:26 point would be rather small
43:28 so we define
43:34 we now use nearest neighbors k nearest
43:37 neighbors
43:38 to define a local metric
43:42 and we then turn this local metric
43:47 again and i'm skipping a lot of details
43:49 which are important here into a
43:53 construction of simplices into a
43:54 construction of graphs where the
43:56 edges now are weighted and the reason
44:00 that they are weighted now is that the
44:02 uh the distances between
44:04 uh points are different and also that uh
44:06 we actually have two measures of
44:09 um of distance one
44:12 using the local metric from here to
44:15 there and one using the local metric
44:16 from here to there because they might
44:18 not be the same
44:22 um but if you properly do the math
44:25 behind all that
44:26 you can show that this construction now
44:29 um captures essential parts of the
44:32 what's now called the fuzzy topology
44:35 of this manifold okay so we have a graph
44:39 representation now
44:41 that captures a topological structure
44:44 of our of the manifold we're interested
44:47 in
44:49 um how can we do that how can we use
44:50 this now to find low dimensional
44:52 representation
44:54 well the idea is that now you start
44:57 um you have this this graph defined and
45:01 now you start
45:02 with a low dimensional representation
45:05 you just place your points randomly
45:07 basically
45:08 and then you construct a similar
45:12 graph representation of your low
45:14 dimensional representation and then you
45:15 shuffle your points around
45:17 until the the topology the graph
45:20 representations kind of match
45:26 so yes you have a graph representation
45:30 of your high dimensional
45:31 data of your low dimensional data you
45:33 interpret the edge weights
45:35 as probabilities and then you minimize
45:38 the cross
45:39 entropy given here where basically
45:42 the important thing is that you try to
45:44 match the
45:46 edge weights in your higher dimensional
45:48 representation with the edge weights and
45:50 the low dimensional representation like
45:52 when this
45:53 ratio gets one the term vanishes and the
45:57 same happens over here
46:01 now how does this work in practice like
46:05 if you want to read more about this
46:06 there's a nice home page
46:11 i just wanted to show an animation here
46:15 which is called understanding umap there
46:18 for instance here you have an example
46:20 in 3d you have um dots
46:23 which run out from a center in a star
46:26 like fashion right and that's a 3d
46:28 represent this is a three-dimensional
46:30 data
46:31 and well actually it's
46:34 10 dimensional data in this case i think
46:37 and this is a three-dimensional
46:38 projection maybe
46:40 and now if you run umap
46:44 these dots are shuffled around randomly
46:47 and you stop
46:48 when you have a topological
46:50 representation which is very similar and
46:51 in this case this is also
46:53 this starship star-shaped pattern but
46:56 now in two dimensions
46:58 and if you want there's many more
47:00 examples here and you can
47:02 play around with this
47:09 um you can of course do the same thing
47:12 in
47:12 r um i i don't know
47:16 like i know this was a lot of
47:18 information and i've
47:19 i didn't go into the details are there
47:21 questions about umap at this point
47:28 otherwise i just show you how to do this
47:31 in r
47:35 it's essentially again a one-liner
47:38 where you have a function umap and what
47:41 i feed in here
47:43 is the swiss roll that i created
47:45 previously
47:47 okay and then um
47:50 i just i i do the um
47:54 dimensionality reduction here with a one
47:56 liner
47:57 then i convert the whole thing in a data
47:59 table object
48:00 to uh to have a nice plotting
48:03 and this is what the result looks like
48:05 now a new map coordinate
48:08 the the swiss role was kind of unrolled
48:11 but it
48:12 heavily lost its shape but what you can
48:15 see is that the topology is preserved
48:17 right
48:18 um so so the hole in the center is still
48:21 there
48:25 um i also want to show you a review map
48:29 representation of the coil 20 data that
48:31 i showed you above so remember this was
48:33 the data set
48:36 where we had pictures which are rotated
48:42 and where the the pca didn't look very
48:46 informative
48:48 now if we do the same thing with a umap
48:50 and we feed in these pictures this is
48:52 what we get
48:54 so most of the images nicely separate
48:57 right different colors mean different
48:59 objects
49:01 they nicely separate and what the umab
49:03 also gives us
49:04 is this kind is is the topology of this
49:07 data in the sense of that we can see
49:09 that this is a rotation
49:10 right so neighboring
49:14 pictures are similar and you
49:18 rotate the object until you come back to
49:20 the other side
49:27 okay i think yes that's it for um
49:30 [Music]
49:31 dimensionality reduction just one
49:35 example
49:36 for linear dimensionality reduction
49:37 which was printable components
49:39 and one for non-linear topology
49:42 preserving dimensionality reduction
49:44 which was umap and i now wanted to talk
49:47 a little bit about clustering
49:50 um so
49:53 you can as you already saw in some of
49:56 the
49:56 reduced dimensionality in in the some of
49:59 the
50:00 representations of the data and reduced
50:01 dimensions uh clusters can appear in
50:04 your data which
50:05 essentially mean you have qualitatively
50:08 different
50:09 objects in your data different different
50:11 samples and
50:15 you sometimes you you
50:18 and and it's good to have a way to group
50:22 different kinds of different classes of
50:24 objects together
50:26 and clustering is nothing
50:29 which is um there is no one definition
50:33 of clustering
50:34 the idea is to find clusters such that
50:37 the data within clusters are similar
50:40 while data from different clusters are
50:42 dissimilar
50:43 but what you define a similarity or
50:46 dissimilarity that's
50:48 basically up to you and so a lot of
50:50 different clustering algorithms exist
50:54 and i just wanted to introduce three of
50:56 them today
50:58 um and you use this a lot in or at least
51:01 i use this a lot in trying to understand
51:03 high dimensional data
51:05 i'm just going to share my
51:10 ipad again
51:27 and i wanted to show you first i want to
51:29 give you three examples and the first
51:30 one would be hierarchical clustering
51:32 which you
51:33 which uh
51:36 we're here i just plotted some
51:40 some data in two-dimensional space to
51:42 illustrate the concept of clustering
51:44 and what you need for hierarchical
51:47 clustering
51:48 is first of all a metric
51:54 which defines distance between your
51:57 objects
51:57 that you have here and this could just
52:00 be for instance the euclidean metric or
52:02 manhattan metric or a correlation
52:04 metric you're basically free to choose
52:06 and then what you do
52:08 is you um you group together
52:12 the two objects which are closest
52:15 together biometrics so with the
52:16 euclidean metric for this example
52:18 this would be those two guys right
52:30 so
52:35 what you then create is a kind of tree
52:37 where where um
52:39 okay we say i said we grow group
52:41 together those two guys so that's the
52:42 first thing to do
52:48 um and now next we need another thing
52:52 which is called the linkage criterion
53:03 um and this tells you how to now treat
53:06 these clusters because
53:08 uh now you need a way to uh
53:11 to give get the distance between the
53:14 clusters
53:21 and this could for instance be the
53:23 minimum distance
53:27 so this would mean if you want to now
53:29 get the
53:30 the distance between uh this cluster
53:33 and uh this object you would
53:36 take the distance between b and d and c
53:39 and d
53:39 and take the minimum distance or you
53:43 could also
53:44 for instance use the centroid there's
53:46 many of the clusters there's many
53:48 choices right
53:51 um now if we kind of go on we maybe take
53:54 the centroid here then the next cluster
53:56 would probably be a grouping of d
54:00 and e
54:06 then i'd say i would add
54:09 those three in one cluster
54:18 then group all these b c d
54:21 e and f until finally we arrive
54:25 at one at an object which
54:29 contains all the elements
54:35 and now if we want to define clusters in
54:38 our data we
54:39 need to define a precision threshold
54:42 along this axis
54:46 and we could for instance put it here
54:53 um and then
55:01 this would cut the tree in this
55:05 in in separate subtrees and we would
55:08 have a
55:08 subtree here that's up three here and
55:11 it's up to here and then this would be
55:12 our three different clusters so we would
55:14 have a cluster one
55:15 cluster two and a cluster three and if
55:18 we uh
55:20 increase the precision like up here then
55:22 we would get more and more clusters
55:28 and as this basically
55:31 just needs some uh
55:34 distance computations um it's it's this
55:37 is rather easily done in in
55:39 uh also in higher dimension i don't know
55:46 um
55:48 next i wanted to show you
55:58 um another important one which is a
56:02 centroid-based clustering called k-means
56:04 clustering and you
56:07 one comes across this quite often as
56:09 well when working with high-dimensional
56:10 data
56:11 so here in this case
56:17 um you pre-specify the number of
56:19 clusters k
56:20 so that's what you have to start with
56:23 how many clusters do you want to get
56:26 you might try out different case in
56:28 practice
56:29 and then you partition your data such
56:31 that the square distance to the cluster
56:34 mean position so that that would be the
56:37 centroid is minimal
56:41 um so how would this look like
56:45 um actually maybe let's look at the
56:49 final result
56:50 first we're kind of uh here you see an
56:53 algorithm converging but what you try to
56:55 do is to
56:55 talk to do a grouping of the data such
56:59 that oh no it doesn't stop there
57:06 sorry so so how do you how do you get
57:08 this partitioning of your data
57:11 one way would be the lloyd algorithm you
57:13 start with random centers that you place
57:15 somewhere so in this case
57:17 this is an example taking from wikipedia
57:20 there's a center here a center there in
57:22 the center there and then
57:23 for each of your data points you assign
57:26 the data point to the cluster
57:28 where the center is nearest right so the
57:30 yellow center is here so everything here
57:32 would be assigned to the yellow
57:34 cluster now once you did this you
57:39 update the centers of your clusters
57:42 now you take the centroid of this yellow
57:45 cluster for instance
57:46 and the yellow center ends up here
57:50 and now you do a new assignment of your
57:54 data points to the nearest center in
57:57 this case all the
57:58 yellow ones up here would be assigned to
58:01 the blue center actually
58:03 so they are assigned to the blue and
58:04 then you iteratively go on and do this
58:07 go on and go and go on until this
58:10 algorithm converges and you end up
58:12 having three clusters defined by this
58:16 criterion of course there's other more
58:17 complicated more sophisticated
58:19 algorithms but this approximates the
58:20 solution
58:27 a third algorithm that i wanted to show
58:29 you and i want to show you this because
58:31 i
58:31 work with it every day comes from social
58:34 systems which is a graph based
58:36 clustering
58:38 when you start with your high
58:39 dimensional data you first need a graph
58:41 representation of your data and we
58:43 already saw this in you
58:44 in the u-map example you could for
58:47 instance
58:48 do a k-nearest neighbor network on your
58:50 data and then
58:51 you partition your data such that the
58:53 modularity
58:54 queue is optimized um
58:59 modularity in this case is defined as
59:02 this object where where e i j are the
59:05 fraction of
59:06 links between cluster i and j
59:09 right and what you try to do then in
59:12 this object is that
59:14 the links between
59:17 cluster i and i which is the internal
59:19 links this
59:20 it's this eii right this is it's the
59:22 links inside the cluster
59:24 you try to maximize this
59:27 and uh you in in this ai term you have
59:30 the
59:30 ij from i to another cluster
59:34 so this is links out out of the cluster
59:36 you minimize this right so you have
59:38 very
59:42 clusters means a cluster in this case
59:45 means that it's
59:46 it's a
59:49 parts of your graph which are densely
59:51 connected inside but only sparsely
59:53 connected to the outside
59:56 there's a particular algorithm which is
59:59 used a lot nowadays which is the longer
60:01 algorithm so how do you solve this
60:03 because this is not
60:05 uh you can't so if this brute force
60:08 um you start with a network like this
60:12 for instance uh this would be your
60:15 nearest neighbor graph that you created
60:17 from your data and then you do copy
60:18 attempts you
60:19 start with node zero so all these now
60:23 you start with all these being separate
60:25 clusters
60:26 and then you copy cluster identity zero
60:29 to neighboring nodes
60:30 and you again compute the modularity and
60:32 if the modularity is increased you
60:34 accept this copy step
60:36 and if it's decreased you uh you reject
60:39 the copy step and you
60:40 do so um until you arrive
60:45 at a steady state like this where copy
60:47 attempts
60:49 don't increase your modularity anymore
60:52 would be in this case something like
60:54 that where you end up with four clusters
60:56 and then you do an aggregation and this
60:59 means
61:00 now that you
61:03 define a new graph representation of
61:05 your data where this
61:07 cluster here the blue cluster is now a
61:09 single node
61:12 it gets four internal connections
61:14 because there's
61:15 one two three four internal connections
61:19 and there is
61:20 four connections to cluster green
61:24 one two three four
61:28 and one connection to the light blue
61:30 cluster
61:31 and so you aggregate this you do the
61:33 copy attempt again
61:35 and so on and so on until you finally
61:38 arrive at a picture where you
61:40 where you um at a state where
61:44 you can't optimize modularity anymore
61:46 and this is um
61:48 how you this is how the algorithm works
61:50 to solve the problem
61:52 of modularity optimization and this
61:54 algorithm
61:56 works very well with many kinds of data
61:58 again for instance it's
62:00 used a lot to identify
62:03 cell types in a single cell genomics
62:12 all right that's it already i realize
62:14 i've been
62:15 quite fast we would have some time but
62:19 i'm i'm sorry
62:22 good timing one hour um
62:27 so just as a summary um the question
62:30 i wanted to talk about is how do you
62:32 explore high dimensional data
62:34 to identify order in this high
62:36 dimensional data set and i showed you
62:39 a couple of dimensionality reduction
62:41 techniques to visualize your
62:42 in-dimensional data
62:44 and that allow you to identify a small
62:46 set of
62:47 observables to describe the data um
62:51 i showed you a couple of approaches for
62:53 clustering to this
62:54 to identify discrete order in high
62:56 dimensional data and
62:58 i think one important point that i want
63:00 to make is the methods to use to work
63:03 with high dimensional data
63:05 depend on the problem at hand so really
63:08 depends on what you're looking for and
63:09 often it's just
63:10 trying out a lot of things to see what
63:12 makes sense
63:14 to understand your data and with this um
63:17 i'm at the end and
63:20 there's some references in the back
63:22 which i will upload uh
63:24 with the slides and with this yes i'm
63:26 i'm happy to take questions
63:29 okay perfect uh thanks a lot fabian yeah
63:32 thanks for taking the time and
63:35 especially since
63:36 uh data people like you are so in such
63:39 high demand these days and very busy so
63:41 it's great that you took the time to
63:43 to show us some of your stuff and
63:46 so uh i think what we usually do is that
63:49 we hang around a little bit
63:51 yeah on zoom and then if you have any
63:53 questions you just stay online
63:56 and ask your questions
63:59 yeah and uh other than uh that
64:02 yeah so then see you all next week and
64:05 there's already a question in the chat
64:13 there were questions in the chat during
64:15 the year during the lecture and i didn't
64:16 see the chat so yes you can't see that
64:18 very well if you're sharing the screen
64:20 yes but i think they were answered
64:22 already by other people
64:24 so so now the latest question is how
64:26 computationally intensive is um
64:29 that's that's a good question and it's
64:31 actually that's another uh
64:33 advantage of umap it's pretty fast
64:38 um it's
64:40 it also it's you don't you don't need a
64:42 lot of memory
64:43 um it scales very well with the
64:46 dimension of your data and also the
64:47 amount of data points so i think
64:49 it's it's um i mean just like
64:56 it's one of the fastest things around at
64:58 the moment um
65:01 i mean if you look at the typical data
65:02 sets uh that that you're looking at
65:05 where you have i mean 15 thousand
65:07 dimensions
65:08 and then maybe a few hundred thousand or
65:11 ten thousands of uh samples yeah you you
65:14 end up
65:15 with a few seconds yes yes on my machine
65:18 like the with the largest data sets i
65:20 have which are
65:21 of the order that you described um it
65:24 takes about
65:25 30 seconds maybe to compute the uh
65:29 compute the u map which is much faster
65:32 for instance there's
65:33 this name i don't know whether you know
65:36 about it but this was used a lot before
65:38 and
65:38 this is much slower usually
65:41 and actually there's um i didn't try
65:44 this out but there's also gpu
65:46 of implementations of umap nowadays and
65:48 this again
65:50 you can easily you can nicely
65:51 parallelize it and so this should be
65:53 much faster again
65:59 you're welcome
66:04 great thanks so are there any other
66:07 questions
66:08 um i have a question it's from
66:12 umap
66:15 so when we uh in all the examples of
66:18 umap we saw approximating a high
66:20 dimensional data
66:21 is there any isn't there any parameter
66:23 in the algorithm that quantifies
66:25 the uh
66:28 the acceptability of the approximation
66:30 whether the approximation is cur
66:32 how how much is it acceptable or not
66:34 because
66:35 or will it approximate any uh high
66:37 dimensional data
66:40 there didn't seem to be any parameter
66:41 that quantified or helped us to
66:44 understand if the result of the umap
66:47 approximation is
66:48 good or not uh obviously concerning the
66:51 problem at hand
66:53 i'm not fully sure whether i completely
66:56 understand the question but here are a
66:57 few thoughts on this um
67:02 so the approximation in umap is that you
67:05 you approximate
67:07 that um
67:10 your your approximation is that the data
67:12 is uniformly distributed
67:14 on the manifold right and the question
67:17 is
67:18 and maybe the question is is this a good
67:19 approximation or is it not a good
67:21 approximation
67:22 the problem is that we that we hardly
67:25 know the real metric
67:27 of the underlying space so
67:30 if you for instance think of these
67:33 examples with
67:34 pictures right what is a distance
67:36 between two pictures
67:37 what is this metric um
67:41 it's it's um i i think this is the you
67:45 you can't actually like
67:46 unless you have a unless you have a good
67:49 microscopic model
67:50 which in this case we don't have it's uh
67:53 not really possible to
67:55 to define a good metric and therefore to
67:58 judge whether whether the
68:00 whether the approximation is a good one
68:02 or not
68:04 um it's not
68:08 um yes i mean the way here that you you
68:12 do it is
68:13 there's no there's no uh right rigorous
68:16 way of doing that and that's why people
68:17 like fabian exist
68:18 who have experience and have consistency
68:21 checks
68:23 that with some experience can check
68:25 whether you map
68:26 makes sense or not yeah or whether it's
68:29 so one of the standard problem is that a
68:31 umap can be dominated by noise by
68:33 technical artifacts for example
68:35 yeah and then you need to have some
68:37 insights into data and some experience
68:40 that tell you whether uh whether uh
68:43 uh what you see actually represents
68:46 something that's real in the data
68:48 or whether that's something just just an
68:50 artifact you know so that's
68:52 uh that's basically one of the the jobs
68:55 that bioinformaticians or data science
68:58 scientists do
68:59 because there's just no rigorous way
69:02 where you just push a button and it
69:03 tells you whether
69:04 whether what you've done is is right or
69:06 not so
69:07 it needs some experience and some some
69:09 insights and understanding
69:11 of the data and the for example the ball
69:14 biology
69:15 uh that is underlying the the data
69:18 so typically with the typical um
69:22 method would be to use multiple
69:24 reduction methods and see which one is
69:26 giving us
69:27 an order which is much more perceptible
69:29 in our in our plots
69:31 you know i guess mean you also have to
69:34 to know the strengths and weaknesses of
69:36 each method
69:38 yeah for example what uh fabian told
69:41 about
69:41 previous uh before t-sne that had a
69:45 tendency
69:45 uh to perform to to give you clusters of
69:49 data points
69:50 yeah it was not very good at
69:52 representing global structures of data
69:54 sets
69:55 yeah and once you know that you can
69:57 interpret
69:58 interpret what you see and then you can
70:00 be careful
70:01 in interpreting uh basically clusters as
70:05 real physical or biological objects
70:09 yeah you maps the you are what you what
70:12 you've seen now
70:12 that overcame this limitation that it's
70:15 basically
70:16 also represented the global structure of
70:18 data quite well
70:20 but i guess also in umab you can have
70:22 artifacts
70:23 especially if your data has noise yeah
70:25 technical noise
70:27 you can have very weird artifacts so you
70:30 need to treat the data
70:32 before you can give it to youmap in in
70:34 typical instances
70:36 yeah so for example log you have to log
70:38 transform data very often
70:40 if your data is says spends huge orders
70:44 of magnitude
70:46 in quant in in in values yeah then then
70:49 sometimes you have the log
70:50 transformations for not the
70:52 the very few very large data points to
70:54 dominate everything
70:57 and so you have to treat the data in a
70:59 way to make it
71:00 say digestible for these methods
71:04 but then maybe i mean
71:07 the the other part of the question was
71:09 in in a way parameters and i didn't talk
71:10 about parameters at all and there are a
71:12 couple of parameters in the algorithm
71:14 um and i i skipped a lot of details but
71:17 the most important parameter
71:19 is the number of neighbors you look at
71:22 in your
71:24 when you construct the graph if you take
71:26 the first neighbor right you you
71:28 kind of um what you do is you
71:32 um estimate your metric very locally
71:36 um so so this is very this is very fine
71:38 but it's also very noisy so you often
71:40 will estimate the wrong metric if you
71:42 if you uh increase the number of
71:45 neighbors in there
71:46 um you will get a more robust estimate
71:50 of the local metric but you will
71:52 miss some fine structure of the data and
71:54 maybe actually there's a there's one
71:55 example that i can show
71:57 uh from here where if you look at
72:01 i don't know whether you can see this
72:02 here this is a
72:04 um maybe you can can you see this or is
72:07 this too small
72:08 i'm sorry you're not sharing the screen
72:12 ah sorry i don't share the screen so you
72:14 can't see anything
72:16 um let me show
72:18 [Music]
72:20 let me show this example um
72:22 [Music]
72:27 kind of if you that's maybe maybe a
72:32 actually a good example for for your
72:33 question where kind of okay here we have
72:35 a circle
72:37 with uh with dots and they on um
72:41 and then like like the distance between
72:43 the dots is not always the same right
72:46 um and so actually if we
72:49 take just this example and we say okay
72:51 here we know the euclidean metric
72:54 uh we can think of how how well does the
72:56 approximation now
72:58 or or how does umap behave if um
73:03 uh in this case right and
73:06 one thing that you can see here is that
73:08 if i increase the number of neighbors
73:13 that this will
73:16 usually lead to a
73:19 actually not
73:25 so you can see that sometimes you manage
73:28 to
73:28 for for some uh there is kind of an
73:31 optimum
73:33 um for looking at the number of
73:35 neighbors which allows you to actually
73:37 capture
73:39 um the the
73:42 circular topology the hole in the middle
73:44 right but
73:46 if you if like if you go too big you'd
73:48 kind of lose
73:49 to start to lose the information about
73:52 the order of these dots
73:54 of these uh measurements right they
73:57 start to get more and more noisy
73:59 if you look at less neighbors
74:03 you get nice ordered lines but you start
74:06 to miss
74:08 things where you have these gaps and you
74:10 start to create clusters
74:11 where there shouldn't be clusters so
74:14 this is the
74:15 most important
74:17 [Music]
74:19 parameter of the of the method
74:22 i'd say there is like if you if you uh
74:27 the umap home page the like from the
74:29 umap algorithm
74:31 they uh feature a nice section of what
74:33 parameters do you have for your map and
74:36 and uh
74:37 what influence do they have but it's
74:40 it is generally relatively robust also
74:44 compared to other methods
74:45 if you go for typical like
74:49 it just but this is just empirical kind
74:51 of if you go with
74:52 about 15 to 30 neighbors
74:56 or something you usually derive good
74:59 representations
75:00 of the topology of your data if you have
75:02 enough data points
75:04 um so it's relatively robust uh to the
75:08 choice of
75:08 of parameters you don't have to play
75:10 around with it too much
75:14 thank you
75:19 okay perfect very nice do we have any
75:21 other questions
75:22 yeah there's a question from kai muller
75:25 from the chat which i will read
75:26 which is uh he asked for high
75:28 dimensional data how does human
75:30 umap determine the manifold dimension
75:36 um
75:38 i don't think that it does
75:41 and
75:47 i don't think that it does and this
75:51 might be related to that it only
75:54 in the u-map algorithm what you do is
75:56 when you when you construct
75:59 this uh the graph representation
76:02 um from the intersecting
76:06 spheres you only look at the
76:10 one simplicity so you only construct a
76:12 graph and you don't look at
76:15 [Music]
76:18 at higher order simplices i'm not sure
76:20 what this could be used to
76:22 to say something about the
76:23 dimensionality
76:26 um of your manifold i'm i'm unsure there
76:31 but you might by default it doesn't tell
76:33 you the dimensionality
76:40 okay um any other question a lot of
76:44 questions today
76:55 okay so if there are no more questions
76:57 you know then uh
76:58 let's end the meeting you know thanks
77:00 all for joining and thanks again for
77:02 fabian thank you well thank you for
77:05 having me
77:06 yeah thank you okay perfect
77:10 yeah see you all next week bye
77:16 you