Optimizing Your Use Case on Hosted Elasticsearch (Elastic Cloud) and ECE
Articles,  Blog

Optimizing Your Use Case on Hosted Elasticsearch (Elastic Cloud) and ECE


>>SHAY BANON: Elastic Cloud is our SaaS-hosted
Elasticsearch service. You go to Elastic Cloud and we will deploy an Elasticsearch server
for you and manage it for you.  We’ve also introduced Elastic Cloud enterprise, we have
seen quite a lot of organizations that want to go and manage more than one cluster, they
want to be able to manage hundreds of clusters, tens of clusters, all of these aspects of
being able to go and orchestrate an Elasticsearch deployment within your org. Many reasons to
why you’d want to use it.  The challenge that we’ve had, both for our
SaaS-service, and our customers of Elastic Cloud, is that you are slightly limited with
what you can do.  You basically had what we call one slider. This one slide basically
controlled how much memory and how much storage you would want to have for an Elasticsearch
deployment. And as you see here, and what we’ve done, actually a deployment of a cluster
or deployment of our technology is more advanced than that. What about machine learning? What
about different types of data nodes: warm data nodes or hot data nodes when it comes
to logging use cases?  So we wanted you to have more control over the various aspects
of a cluster when you go and deploy it on Elastic Cloud and configure it.
This allows us, for example, to be very aggressive when it comes to logging use cases, and old
data can then have ratios of one gigabyte of memory to 80 or 100 gigabytes of storage,
versus recent data that has ratio of one to 20, or 1 to 24, because your want the results
to come back faster. So we’ve worked over the past year in introducing
these feature and I would like to welcome Andrew on stage to demo it.  Andrew?
(Applause).>>ANDREW MOLDOVAN: OK. Thanks, Shay.  Hello,
everybody. So, many of you will be familiar with this
page already, it is our current Elastic Cloud create cluster page.  And, as Shay was mentioning,
you can create an Elasticsearch cluster on here that has a pre-set, pre-defined ram-to-disk
ratio. And it always goes through the same I/O optimized hardware, and every node has
all the rules.  So, you can’t create dedicated master or dedicated machine learning, for
example.  So, while we do use the best instance types
that AWS and GCP have to offer, a lot of you have asked for more flexibility here.  Maybe
you have a logging use case, maybe you want to use memory or CPU optimized hardware, or
maybe you just want to run machine learning and APM on hardware that fits.  So we have
been listening and we have worked on the newest feature that will allow you to manage each
part of the Elastic Stack individually and to allocate it to specific hardware.
 So, my demo will walk through the logging use case and specifically the hot-warm architecture.
 So, hot-warm is a well-known pattern in Elasticsearch where you have the newer indices
go to more expensive and faster hardware, and you have the older indices that can be
moved to what we call warm nodes, which are cheaper usually, and a bit slower, but they’re
better at storage.  So, I will pick hot-warm and click configure. As you can see, I’m
able to already manage each part of the Stack right here. I have the masters. I’ve got
data hot. I’ve got data warm, machine learning, and Kibana right now. APM and Logstash are
going to be added in the future.  So, I can go ahead and decide that I want
1 gig of RAM for masters is fine, but I’ll put it to three zones, and if you saw that,
the architecture diagram has actually updated. So, I will pick 4 gigabytes of RAM for my
data hot, put that into two zones, and I will do the same for my data warm, just for now,
in two zones.  And as you can also see, each of these instance
configurations is actually going to specific hardware.
So, the master is going to an R4, the data hot is going to an I3, and D2 and M4, and
so different instance types.  This is on AWS, but it certainly could be
on GCP or anything else.  The hardware that we are going to be putting for master, or
data hot, or for the different instances are what we’ve decided are the optimized hardware
for that.  So, machine learning needs a bit more RAM than most, but one zone is fine for
now. Kibana I’ll just leave alone. I’ll go ahead and click create. And so this is
the activity page where you can actually watch this deployment coming online. The deployment
has a number of different steps. I won’t actually wait for everything but I will mention
one part here is that with the hot-warm architecture, you will also be able to manage the indices
in the cloud, so you can tell our index creator, I want to move indices every week from the
hot hardware, to the warm hardware, or whatever you want.
So, I’ve already created a deployment that looks exactly like this one. So I will just
go over to that. It will take a minute and a half, or two minutes to come online.  This
is the deployment overview page, and you can see that we have all of our nice instances
here. So we’ve got — in the different zones that I picked — I have data warm, master,
data hot, master, data warm, machine learning, and so on.
What’s really cool is I can show you that —well, I can also filter, which is kind
of fun —  we can show you the allocator, the host, that is actually where this instance
is living. And I can tell you, or I can show you, that is, what is the instance type of
that.  So we said that the master was supposed to be running on an R4, so I will open up
this host, and we’re going to scroll to the bottom here, and there is the R4 instance
type on that host. And I can do the same for data hot, for example,
so this data hot is supposed to be on some kind of an I3 tagged host, and there is the
I3 tag as well. So this is a very short,very simple demo,
but I think it is extremely powerful example of what we can now do in cloud, you can manage
the pieces and throw them to specific hardware, you have a lot more power in your cloud deployment
than ever before.  All of these changes are coming to Elastic Cloud enterprises and to
Elastic Cloud. To be able to manage all this fun UI, we’ve actually re-vamped the user
console entirely. So it will not look like this, it will actually look like this. It
is both Elastic Cloud enterprise and Elastic Cloud that will use the same code base, and
you will get a lot of other fun usability improvements in there.
I think that’s it.  Thank you very much. (Applause).  
>>SHAY BANON: Thank you very much, Andrew.

Leave a Reply

Your email address will not be published. Required fields are marked *