storage-ha-logs.txt

In this video we will look at some more examples of Openstack multinode architecture. And then about ceph storage and why is a storage needed. Then about log analysis.

Let's start the one from redhatblog url. This is a typical openstack deployment. This is the internet. You have all your services deployed in 3 node and each and every service is isolated from one another. Each an every serivce is deployed in isolate node, I mean a single node. For example, if it horizon, only horizon is deployed in that node. Every green box is one node. We have 3 such node for each service and we use haproxy to load balance the service. Haproxy is a tool, where we can define rules, like if a request is received in a port, redirect the request to any of the three nodes.  When the user try to open the URL , in ideal scenario, the user may enter something like mycloud.example.com. Similary it will the the same URL in openrc file as well. That mycloud.example.com most probably will be a keep alived IP. Keepalived is a small daemon, which will ensure that only one of the 3 haproxy is active. mycloud.examplec.com mostly will be the horizon services VIp. So the user makes an authentication request here, it will strike one of the haproxy which is active and haproxy has all this VIP. It is just a logical entity, it strickes the particular keystone IP. Now haproxy will redirect the request to any one of the 3 backend. Then the keystone server will provide the authentication and authorizon and a catalogue. Even that catalog will contain the same mycloud.example.com url for the endpoints. For example , a endpoint of nova will be mycloud.example.com:8774 . It again goes to haproxy and then haproxy will look for nova backends and send the request to any one of the nova node.  

4:46

Another thing to consider is mysql. We can make mysql high available by deploying a 3 node mysql galera cluster and then create entries for the 3 nodes in haproxy machines. In haproxy we can confirgure the mysql port as mycloud.example.com:3306 and we can put the 3 mysql nodes in the backend. We will use this URL and port name in all the services configuration. For example if cinder want to fetch something from database, it strikes the haproxy, the haproxy will redirect the request to any of the mysql node. Then Rabbitmq, by default it operates as a cluster. YOu can look at the cluster status by using rabbitmqctl command. All the services can connect to any one of the haproxy node to send and receive messages.

We are not going to talk about foreman, it is just an automated deployment tool.

7:03

This is another production level openstack deployment. You can see similar architecture here. The difference here is the here is that in this diagram all the services are deployed in single controller node. But in previous diagram each service is deployed in single node. We have 3 controllers node, and load balanced by haproxy. You also can see that in this environment there are multiple compute nodes , this is how a typical openstack installation will be. There might be 100's and 1000's of nova nodes in typical production environment. Whenever you see more than one service backend, it is powered by haproxy and keepalived. We are not going to cover the additional components here.

09:18

This is an another good guide about the role of haproxy, loadbalancers with respect to openstack high availability. We have haproxy node, controller nodes (more than one controller probalby), compute nodes , with rabbitmq messaging service. We got mariadb for openstack services and mongodb for ceilometer service. Here they are using puppet to deploy it. The other thing we can see is that they are using ceph as a storage backend. 

10:46

There are multiple openstack services which need storage. For example when you boot an instance it uses storage to save virtual machine, block storage service needs storage to create disks, similary swift and glance needs storage. But in production environment, we can't just use local storage and it may get lost if the node is down. Hence we need to use local storage and most Openstack deployments uses Ceph as storage backend. Earlier people used GlusterFs to achieve  it but it is not that effecient. Ceph is designed mostly to help with storage operations similar to openstack, like snapshotting, scalability. 

Here each blue box is a storage node and balck box in monitor node. These monitporing nodes are the one which manage the cliuster. If you try to save 1 gb file, the file is chunked into smaller pieces and spread across the storage nodes and each chunk is replicated thrice by default. For example if you want to save abc, a is stored in one, b in another , c in next and they are all replicated. This cluster is highly scalable, and self healing. IF some node is down, the other two copies of the node will create a third copy. So it is highly reliable. It is bit costly, because if you need to save 1 gb of data, it needs 3 gb space, since it is replicated 3 times. But is is very useful in recovering data in production environment.

14:30

Openstack integrates with this storage using these libraries. Swift uses radosgw to manage storage objects, radosgw in turn uses libradost to access files. Similary cinder uses librbd to create and mange block device, glance and nova also uses librbd. Glance stores the image in ceph, because when you launch an instance, image is copied and data is stored in that. It is fast to make a copy in ceph so instance is launched fastly if glance is also in ceph. Putting this in a single node works , but it will be slow. 

16:37

In some disgrams we saw that the services are deployed over 3 nodes and ther are lot of services. For example , if I want to launch a nova node, First it will come to haproxy, then to keystone in some keystone servver , then again to haproxu and then the request will transfered to one of controller node and then to the compute node. here we will not be very easy to track the entire service/ path logs. For simple errors it is easy, but for complex troubleshooting, it is hard to go through all services and get the logs. 

So in many openstack environment, they use ELK (elasticsearch , logstash and kibana) stack. In all the servers there will be a log forwarded, and the primary purpose of this is to forward the log to logstash server. Here lumberjack is collecting the logs, it is just an engine, which allows us to structure the logs and only send relevant logs to the log server , based on filters. Logstash indexes the logs and it send it to elasticsearch. Elastic search is a search engine for logs. Similar to good it searches the logs and provides us the result in micro seconds, irerespective of how big the log file is. Kibana is just a frontend GUI to display the results. 

logforwarder is a small service , so it is very easy to install it in all the service nodes. And the ELK stack can be deployed in single node. This is ELK GUI, basically you can type some ID here in top corner and in the bottom it will show results from haproxy, compute ,controller and service nodes. You can see all the relavant logs in a single screen.