---
title: "Finetune Stable Diffusion"
video_id: RJbnIndOfVw
url: https://www.youtube.com/watch?v=RJbnIndOfVw
workflow_page: https://workflows-of-ai.com/workflow/finetune-stable-diffusion
language: en
captions: generated
---

0:01 hey there Marcus here um so I've been on
0:04 vacation start a new job and I've been
0:06 sick so it's been a while uh but now I'm
0:10 going to show you
0:11 something uh completely new uh that it
0:16 has some connections to AI interpolator
0:18 but it's it's a New Concept which is
0:20 this part of the hugging face
0:23 module which makes it possible to use
0:27 druple uh not have a GPU at all and to
0:31 be able to fine-tune something and take
0:33 that fine-tune model and use it inside
0:36 Dro B to generate
0:39 stuff uh and this is completely inside
0:42 the GUI no coding needed uh no server
0:45 setup knowledge or anything like that
0:48 needed
0:50 um
0:52 so the reason that I'm it's in the
0:55 hugging face module is basically that
0:58 anything that it generates will be
1:00 hosted at hugging phase so if you if
1:03 you're not aware of these Concepts
1:05 basically hugging phase is uh you can
1:08 think of it as GitHub for AI models so
1:11 it's a repository where you can host uh
1:13 your private AI models uh but it also
1:16 has a lot of public AI models it has uh
1:20 open source AI
1:22 models and what we're going to do is
1:24 we're going to take one of these models
1:27 uh in this case stable diffusion uh any
1:30 of the different versions which is a
1:31 text to image model and teach it A New
1:35 Concept so that's what you can see that
1:37 I have done here so I have fin you on
1:40 images of
1:42 myself I've called myself Marcus
1:44 Johansson JJJ so it's it's it's its own
1:48 token because there is a famous Marcus
1:51 Johansson NHL player so I needed to
1:54 basically have my own token not to
1:56 delute with the knowledge about him and
2:00 then I can write something like that a
2:01 photo on me in front of the touch Mahal
2:03 and it creates this right uh and I can
2:06 use the AI interpolator to also run this
2:09 specific model and do it inside Drupal
2:11 as
2:12 well uh so that's basically what we're
2:15 going to uh what I'm going to Showcase
2:18 today so how this works for you as a
2:22 user is basically you install huging
2:25 face model uh module sorry there is a
2:28 page you go to that
2:30 you describe your New Concept you
2:32 describe which base model you want to
2:34 base it on you upload a lot of
2:36 images uh you crop these to focus on
2:39 whatever it is you're trying to uh
2:41 generate you click on start
2:45 finetuning and you wait some time and
2:49 then it's going to show up on hugging
2:50 face and then you can use AI
2:52 interpolator to generate images with
2:54 this um so what's happening in the
2:58 background then is this that when you
3:01 click generate it's going to start a VM
3:04 in
3:05 runpod and runpod is just a cloud
3:08 provider
3:09 that's specializes in
3:12 AI uh so it specializes in GPU servers
3:15 so servers you most likely don't have
3:18 available uh and that are freaky
3:21 freakishly expensive and that is the
3:24 other reason why I chose runpod because
3:26 runp pod has a credit system which means
3:28 that you have to upload credits um to be
3:32 able to use it
3:34 because even though I'm doing open
3:36 source and open source clearly states
3:38 I'm not responsible for how you use my
3:40 code right I don't want to have this
3:43 scen scenario that someone looks at this
3:47 video starts fine-tuning something there
3:50 is some bug in there that I didn't know
3:52 about uh and it just dies halfway
3:55 through and they don't understand the
3:58 concept of cloud at all so they don't
4:02 understand that oh there was uh the
4:04 garbage handling garbage collection
4:06 didn't work here so there's still a
4:09 server running and these servers as I
4:11 said are frequently expensive and this
4:13 person at the end of the month all of a
4:15 sudden you know get a bill of $2
4:18 $3,000 I don't want that to happen so
4:21 because of the credit system it means
4:22 that you upload $10 for instance first
4:25 case you lose $10 if there's a bug like
4:28 this um
4:30 but the module also has a very clear
4:32 checkbox that you have to check that
4:34 you're aware that this might cause costs
4:36 and that you should also manually go in
4:38 and check so uh the VMS are removed
4:43 automatically otherwise you should
4:45 remove them
4:46 manually but anyway this creates this
4:49 pod takes this Docker image as the image
4:54 um and just so you know I do everything
4:56 open source there is open source uh rep
4:59 for this if you want to see what the
5:01 docker image is doing so it's not you
5:04 know stealing your images or something
5:06 like that um or more importantly it's
5:09 going to have the huging face API token
5:12 on there so I don't steal that of course
5:14 uh so you can see that it's not doing
5:17 that does the fine tuning when it's
5:20 finished with the fine tuning takes this
5:22 mod mod uh model and pushes it to the
5:26 hugging face reple so that's what it's
5:28 doing
5:30 um and the advantage with this of course
5:33 is that you you know you don't need to
5:35 Learn Python a lot of this stuff
5:38 is uh it's not very user friendly
5:42 so a lot of the AI scientists which are
5:45 not usually developers they are very
5:48 very very very very good
5:51 mathematicians
5:53 uh uses python because that's usually a
5:56 language scripting language that you
5:58 learn and
6:00 um
6:02 and because of this they might write a
6:05 script that you know generates this but
6:08 they don't think about usability at all
6:10 so because of this you kind of need to
6:12 know python to do this otherwise there
6:14 are some UI tools for it but most of it
6:17 is anyway based on python at best maybe
6:20 you can get a Docker image which you can
6:22 just start um so in this case you don't
6:25 have to think about it then plus you
6:26 don't have to have the GPU server then
6:31 uh there is one more thing before I'm
6:33 going to show you how to set this up
6:35 then and that there there is this
6:37 affiliate link
6:39 um I'm not in any way trying to get
6:42 twitch on this then I wouldn't be doing
6:44 open source that would be very very
6:46 stupid but when it comes to most of the
6:49 AI uh modules usually I can ask for like
6:53 can I get a free account here just to
6:55 try out some stuff and sometimes they
6:57 give it to me so I don't lose any money
7:00 in this case I lose a lot of money
7:02 testing around and I want to do a lot
7:04 more uh with run pods in terms of one
7:09 fine-tuning so currently it's only doing
7:11 image models I want to be able to do
7:13 large language models which cost a lot
7:16 more money because they require a lot of
7:18 more uh memory better
7:21 Hardware uh and the other thing I have
7:24 done privately or in the company I used
7:27 to work for is working with rmod
7:30 serverless which is a way where you can
7:32 take a model host it but you only pay
7:35 for when you actually use it so it it
7:37 works similar to Amazon Lambda just that
7:39 it has a
7:40 GPU uh so I want to build open source
7:43 stuff with Dro B around this as well it
7:45 would be nice to not lose too much money
7:47 on this so if you're anyway interested
7:50 in using this and you don't mind then
7:53 please use this uh link because from
7:58 everything that you spend I will get 3%
8:00 in credits so it's not money I get I can
8:03 take this out as money but at least I
8:05 don't lose as much money uh when I'm
8:08 doing this open source
8:11 stuff um
8:13 so that's it let's get started then on
8:18 how this
8:21 works so let's start this then uh this
8:24 is a plain droup B standard profile
8:27 admin toolbar install that's all we just
8:30 need the hugging face
8:32 module um I'm running D in this case but
8:37 whatever local environment you have you
8:39 just use composer to get it right uh
8:43 then the hugging face module itself is
8:46 just API module and this comes with sub
8:48 modules that can use it so there's a
8:50 augmentor um plugin for it a module for
8:54 it but what we're going to do in this
8:56 case we're going to use the Run pod uh
8:58 fine-tuning module that is a module
9:01 that's using it and the AI interpolator
9:03 hugging face module that's using it and
9:05 these have dependencies that doesn't get
9:08 loaded automatically so I will just get
9:11 the base AI interpolator in this case
9:14 and also for the uh uh fine-tuning I
9:17 need image widget crop to be able to uh
9:21 specify what part of the image that I
9:23 want to find Jun
9:26 right then I just after getting this I
9:28 click extend uh then there is runp POD
9:32 fine tune I need to install an AI
9:35 interpolator hugging face those two
9:38 modules are what I need one for to fine
9:40 tune and one to
9:41 generate this is going to install a lot
9:43 of dependencies as you see um just
9:47 approve that uh and the next step is we
9:50 have to go to web services hugging face
9:52 config so we have to add our access
9:55 token so we're able to both well save
9:58 the model oh sorry I did I click new
10:02 organization settings is what you want
10:04 but also to upload uh well create the
10:07 finetune but also to read that fine tune
10:09 when we're using the a
10:11 interpolator access
10:13 token uh we create a new fine tune here
10:16 uh fine tune temp I'm going to remove
10:19 this after so you can't copy it and use
10:21 mine uh we're going to show that copy
10:25 paste it here save and we need to do the
10:29 same same them for run pod um it's
10:31 important that you have money in your
10:33 run pod otherwise this will not work so
10:35 you should fill up with like $10 there
10:37 that's under billing I think uh anyway
10:40 I'm going to click settings API key
10:43 create one read and write same thing
10:46 here it's going to be gone by the time
10:48 you see this uh and this needs to be
10:51 able to write of course because it needs
10:53 to start
10:54 servers then it's just under
10:56 configuration content authoring fine
10:58 tune dream booth
11:00 uh this is going to have a fine tune
11:01 models later and dream booth llms and so
11:04 on as a sub uh uh directory but uh yeah
11:08 if you can think of a better place to
11:10 put it please let me know so because I
11:12 have the token it knows that I my name
11:15 space is Ivan boring Swedish comedian by
11:18 the way and then I'm just going to uh
11:21 here create
11:22 airpods uh so what we're going to do
11:24 we're going to take a lot of images of
11:27 uh this is
11:30 this is getting recorded in the middle
11:31 of April so if you see this in the
11:33 future this might be completely off but
11:35 there's rumors of a airpod earphone from
11:39 Apple that's going to have a display on
11:41 the case some people have rendered
11:43 images of this and I want to take those
11:45 and be able to generate my own images of
11:48 the so-called future airpod
11:51 then visibility status I said to private
11:55 uh no I actually will do this public so
11:57 if you ever go there you can actually
11:59 try this afterwards to see how how great
12:01 it works table diffusion 2 we're going
12:03 to base it on right now it's just 1.4
12:06 and two we're going to I'm going to add
12:08 so you can add a lot more text to image
12:10 models that's based on anything that
12:12 dream Booth then can uh find you
12:15 on and this includes custom models as
12:19 well because if I find you in this
12:20 airpods and I realize I also want to
12:23 fine tune on a a a future iPhone that
12:27 might get uh generated but I want those
12:30 two things in the same model so I can
12:31 have the airpods with display and the
12:33 iPhone next to each
12:35 other uh I will be able to F tune that
12:38 concept on top of the model I already
12:41 did right and here we have to write
12:43 something then that connects to it so I
12:46 it's already stored here a phot of the
12:48 in ear earphones so this basically adds
12:51 the taxonomy of what this is right this
12:54 is part of the taxonomy in your
12:57 earphones and I just set that custom
12:59 token to this because airpods already
13:02 exists and I could of course do airpods
13:05 with display but it wouldn't know be so
13:07 certain if it's the airpods with display
13:11 that I'm uploading or if it's the actual
13:13 concept of airpods with display so
13:15 because of this I generate a unique
13:17 token I don't think this uh token or
13:21 this word has ever existed inside the
13:23 training data so that's why
13:27 then um just just going to go here and
13:31 upload a lot of images because I'm using
13:34 Drupal here it doesn't have to be PNG
13:36 512 or 7 768 or whatever is the very
13:41 specific things you need if you run this
13:43 as a script you can just upload any type
13:45 of image then image Styles will take
13:48 care of making this so this will be the
13:50 next step there is advanced settings if
13:52 you're using stable diffusion 1.4 you
13:55 could write uh RTX 309 in this case
13:59 because you don't need such an expensive
14:01 model you don't need so much vram uh and
14:04 you can set something else there uh if
14:07 you're going to use stable diffusion 3
14:10 was just released if this will be able
14:13 to fine tune it probably needs even more
14:15 stuff then there's a lot of uh detailed
14:20 fine-tuning uh parameters you can set
14:22 here you have to read about this if you
14:24 if you're not happy with the result
14:26 start reading about this and you might
14:28 get better result
14:30 basically anyway we will just leave it
14:32 as this is as it is click to The Next
14:35 Step which is basically the cropping
14:38 then so we will make sure we just focus
14:41 on the stuff this is probably not going
14:43 to be the best results because sometimes
14:46 as you can see the earphones are outside
14:48 of the display case sometimes like this
14:51 I have to choose if I want some text in
14:53 there or not uh so if you want a perfect
14:58 mod model you should should um fix the
15:01 images in Photoshop first uh and in this
15:05 case also everyone is guessing different
15:07 you see this is a monochrome display
15:09 this is a color display that's a square
15:11 this is rectangle so on uh
15:15 so uh this will probably not be the best
15:18 result but I will just showcase this as
15:20 a example of what you can
15:23 do uh and when I am finished with those
15:29 there is this check box that I need to
15:31 check also on so this is from my
15:33 perspective what I talked about before
15:35 that you do understand that this cost
15:36 money this starts machines and if
15:39 something fails during this run you have
15:42 to manually go in and stop these
15:43 machines but anyway I will cck uh click
15:47 Start fine tuning now this will take
15:50 forever I'm going to fast forward this
15:52 I'm guessing with a stable diffusion 2
15:54 model takes around 30 minutes it's on
15:57 step two now waiting for the Run pod
15:59 machine to build and start so the funny
16:01 thing is if we go to runp pod and click
16:02 pods now all of a sudden you have a pod
16:06 here so it's actually starting this pod
16:08 now and this is then the place if you
16:10 click here and click stop and then under
16:13 here there's going to be terminate pod
16:14 if you ever need to do that manually
16:17 right uh but anyway this will take 20 30
16:21 minutes now but for you it's going to
16:23 look like it takes 10 seconds
16:47 so we're finished I will put a success
16:50 message here I realized I forgot that
16:53 that's not very good that you don't get
16:55 any feedback that it's done but uh we're
16:58 just going to verify two things so first
17:00 we go to my pods reload and hopefully
17:03 there's no pods left so it has done the
17:06 garbage collection remove the Pod double
17:09 check this anyway and remove it if it's
17:12 still there for whatever reason you see
17:14 it's costed me some money
17:17 here uh then we're going to go to
17:19 hugging face go to my profile and we
17:24 have a new model called Ivan boring
17:27 airpods uh and
17:30 actually uh because we're using stable
17:32 diffusion in this case we can actually
17:34 use the inference API here just to try
17:37 it out so we could try it here a photo
17:40 of the uh in ear
17:44 ear phones uh what did we call it it's
17:48 here airpods with
17:50 display uh on a
17:53 table we just press here this free
17:56 version might or might not be a
17:58 available it needs to start machine
18:01 sometimes uh to be there it worked in
18:04 this case so see we have it here I'm
18:07 just going to show you so we can compare
18:09 so we take stable diffusion to hugging
18:12 face so we're going to compare with the
18:15 exact same
18:18 phrase uh on the base table diffusion
18:22 too let's see if this is started
18:24 otherwise it might take some time um
18:27 because then you can compare and see
18:29 that it's not aware of this so you see
18:31 it it kind of figures out its earphones
18:34 right but it's not the the airpods that
18:38 we generated so it has generated it here
18:41 I mean it's not perfect as I said
18:43 before uh you need to have better
18:46 pictures to create a perfect uh model
18:49 you need more pictures than images than
18:51 10 as well so
18:54 anyway we have now uh fine tuned the
18:58 model so now come comes next step we
19:01 want to start using this mod uh uh model
19:05 as well right so I'm just going to
19:08 create a new content type here I'm going
19:10 to call it
19:13 um airpods image generation right
19:20 uh and uh I'm just going to disable some
19:25 stuff
19:26 here and I'm not going to uh do that
19:30 much chaining if you want to know how to
19:32 chain stuff uh look at my previous
19:35 videos if you like AI stuff please
19:37 subscribe I will release a lot of Drupal
19:40 AI videos in the future hopefully as
19:43 well but um if you want to know how to
19:46 chain so you can have stuff like you
19:48 know asking in natural language for uh
19:52 the the leaked airpods and this becomes
19:56 this uh token that we have airpods with
19:58 display this is something open AI or
20:00 some other LM can fix for you before you
20:03 create the prompt but right now I'm just
20:05 going to create a plain text uh prompt
20:10 take a long
20:11 one uh we call this image
20:17 prompt
20:18 and we just save this oh we should set
20:21 it required maybe so like we know we
20:24 have to fill this out and then I just
20:27 create that image field and that's where
20:28 we want to generate this right so we
20:30 take file upload image and we call this
20:33 the generated image I have it here in my
20:38 history and we're going to use the AI
20:41 interpolator here so in this first case
20:46 I'm going to show you uh how to use it
20:48 for free so we're going to use hugging
20:50 face text to image right uh we use base
20:54 mode and we use the image prompt and we
20:57 just going to cop copy it as it is so we
21:00 just use context we're not going to add
21:01 anything
21:03 else uh we say 100 and we're going to
21:06 save it
21:07 directly and uh we're going to uh use
21:12 the free API inference in this case and
21:16 then we just need to basically search
21:19 for our model here so Ivan boring
21:23 air uh
21:25 pods oh this might have a cache let's
21:28 see if we can find it at least if I save
21:30 here yeah so um this might actually take
21:34 some time before it shows up and now to
21:36 complete because it's cached but anyway
21:38 it's there it's validating that it's
21:41 there uh and then we just need to create
21:45 this we need to add a title here because
21:47 of
21:49 uh we're using a node node is probably
21:52 not what we should uh be using here uh
21:55 and then we're just going to take this
21:58 and on our table do the same thing
22:01 again save so if this gets timeout now
22:06 it's because you're using the free
22:07 version and the free
22:09 version uh you know if they have service
22:13 available they will generate it and that
22:15 happened in this case sometimes they
22:17 don't have servers available and you
22:19 will get a timeout and then you have to
22:20 redo it again so say that you actually
22:24 want this but you want it stable I will
22:26 just show you how this works then so
22:28 what you you can do with your private
22:29 mod model now is you can take deploy and
22:33 uh take a inference end point
22:37 dedicated uh you have to choose a
22:39 machine here and this depends on the
22:41 model so 16 GB is okay I think should be
22:44 okay for stable diffusion inference
22:47 stable infusion 2 we just yeah I just
22:51 this are okay I just make it public
22:55 protected is okay as well because you're
22:56 using the tokens
22:59 um and I'm just going to create this end
23:02 point and you see it cost 50 cents per
23:05 hour so make sure you turn this off
23:08 because here there are no credits this
23:10 will actually just start costing money
23:13 uh right
23:15 away and we just have to wait here and
23:19 this is now going to create that end
23:21 point then and the only thing that's
23:24 different is we go back to the content
23:26 types manage Fields uh
23:29 uh the generated image and under the
23:32 settings instead
23:34 of API inference we take dedicated
23:37 inference and we're just going to have
23:39 to copy this endpoint URL whenever it
23:45 starts let's see this might take some
23:48 time to start
23:57 Maybe e
24:50 and yeah we have a generated image here
24:54 um doesn't do text very well as you can
24:57 see so
24:58 uh that's it basically uh how you can
25:02 fine-tune a model and use that module
25:06 model to generate custom images using
25:09 some completely new uh fine tune token
25:13 and also how you can set up dedicated
25:15 machines that can do it then so once
25:18 again thank you for watching if you like
25:19 it subscribe
25:21 um and see you in next video