This post is mainly a recopilation of different sources and what I came up with while creating my first Cassandra Cluster. Hope this helps.
http://www.jonathanhui.com/cassandra, DataStax and the book “Mastering Cassandra” (ISBN : 1782162682)
You must determine or perform the following before starting
1. Choose a name for the cluster.
2. Get the IP address of each node
3. Determine which nodes will be seed nodes. (Cassandra nodes use the seed node list for finding each
other and learning the topology of the ring.)
4. Determine the snitch.
5. If using multiple data centers, determine a naming convention for each data center and rack, for example: DC1, DC2 or 100, 200 and RAC1, RAC2 or R101, R102.
Base Instance Setup
1. Install an Amazon EC2 instance with an Amazon Linux AMI.
2. For the LAB purpose select a Medium instance
Open the following firewall ports for the security group hosting the Cassandra cluster nodes
|8888||OpsCenter website. The opscenterd daemon listens on this port for HTTP requests coming directly from the browser.|
|Cassandra inter-node ports|
|1024 – 65355||JMX reconnection/loopback ports. See description for port 7199.|
|7000||Cassandra inter-node cluster communication.|
|7001||Cassandra SSL inter-node cluster communication.|
|7199||Cassandra JMX monitoring port. After the initial handshake, the JMX protocol requires that the client reconnects on a randomly chosen port (1024+).|
|9160||Cassandra client port (Thrift).|
|Cassandra OpsCenter ports|
|61620||OpsCenter monitoring port. The opscenterd daemon listens on this port for TCP traffic coming from the agent.|
|61621||OpsCenter agent port. The agents listen on this port for SSL traffic initiated by OpsCenter.|
Make sure the Linux instances are created and running, using Putty to open a terminal session
Connecting to Linux/Unix Instances from Windows Using PuTTY: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html
Cluster Installation – Steps
– Update the Amazon Linux
sudo yum update
– Locate the latest stable Cassandra version
– Download Cassandra
NOTE: Identified the latest stable version for the corresponding URL
– Un-tar Cassandra and install Cassandra
tar -xzvf apache-cassandra-2.0.7-bin.tar.gz
sudo mv apache-cassandra-2.0.7 /opt
– Create Cassandra default data directory, cache directory and commit log directory
sudo mkdir -p /var/lib/cassandra/data
sudo mkdir -p /var/lib/cassandra/commitlog
sudo mkdir -p /var/lib/cassandra/saved_caches
sudo chown -R ec2-user.ec2-user /var/lib/cassandra
– Create Cassandra logging directory
sudo mkdir -p /var/log/cassandra
sudo chown -R ec2-user.ec2-user /var/log/cassandra
Edit Cassandra configuration
Change the cluster name
cluster_name: ‘Global Dictionary’
NOTE: if the cluster is started at this point using an incorrect name, refer to the troubleshooting section.
Change the listening address for Cassandra & Thrift
NOTE: listen_address is for communication between nodes
Change the rpc address
NOTE: rpc_address is for client communication
At this point should be possible to install Cassandra locally on each node. Some posts mention to not do this, but worst case scenario is to recreated the data directories later on.
sudo /opt/apache-cassandra-2.0.7/bin/cassandra -f
The listen address defines where the other nodes in the cluster should connect. So in a multi-node cluster it should to changed to it’s identical address of Ethernet interface. The rpc address defines where the node is listening to clients. So it can be same as node IP address or set it to wildcard 0.0.0.0 if we want to listen Thrift clients on all available interfaces. The seeds act as the communication points. When a new node joins the cluster it contact the seeds and get the information about the ring and basics of other nodes. So in multi-node, it needs to be changed to a routable address as above which makes this node a seed. Note: In multi-node cluster, it is better to have multiple seeds. Though it doesn’t mean to have a single point of failure in using one node as a seed, it will make delays in spreading status message around the ring. A list of nodes to be act as seeds can be defined as follows.
Required for production (performance).
l Download jna.jar from
l Add jna.jar $CASSADRA_HOME/lib
l vi /etc/security/limits.conf
$USER soft memlock unlimited
$USER hard memlock unlimited
Repeat the above steps for every node in the ring / cluster
Select a sub-set of ring nodes as seeds. Non-seed nodes contact the seed nodes to join the ring
Define at least one but preferably more for fault tolerance
Seeds are contacted when joining the ring, no other communication with seeds is necessary afterwards
All nodes should have the same seed list
For each nodes, edit cassandra.yaml to add the Cassandra cluster seeds
seeds: “ip1, ip2”
To add a new seed, start the node as a non-seed node with auto_bootstrap to migrate the data first. Then turn auto_bootstrap off and make it to a seed node
Server clock must be synchronized with service like ntp. Otherwise, schema changes may be rejected as out dated
Install System Monitoring Tool
sudo yum -y install sysstat
Change the server timezone
sudo mv localtime localtime.org
sudo ln -sf /usr/share/zoneinfo/US/Pacific localtime
Additional Steps for cluster that spans across networks
Start Cassandra (from seeds to non-seed nodes)
1) Changing the Broadcast addresses to the public IP’s so the nodes can communicate
2) Changing the seed to the public IP address
3) Changing the snitch to EC2MultiRegion Snitch
broadcast_address: (Default: listen_address) If your Cassandra cluster is deployed across multiple Amazon EC2 regions and you use the EC2MultiRegionSnitch, set the broadcast_address to public IP address of the node and the listen_address to the private IP.
listen_address: (Default: localhost) The IP address or hostname that other Cassandra nodes use to connect to this node. If left unset, the hostname must resolve to the IP address of this node using/etc/hostname, /etc/hosts, or DNS. Do not specify 0.0.0.0.
rpc_address: (Default: localhost) The listen address for client connections (Thrift remote procedure calls).
seed_provider: (Default: org.apache.cassandra.locator.SimpleSeedProvider) A list of comma-delimited hosts (IP addresses) to use as contact points when a node joins a cluster. Cassandra also uses this list to learn the topology of the ring. When running multiple nodes, you must change the – seeds list from the default value (127.0.0.1). In multiple data-center clusters, the – seeds list should include at least one node from each data center (replication group)
Use the EC2MultiRegionSnitch for deployments on Amazon EC2 where the cluster spans multiple regions. As with the EC2Snitch, regions are treated as data centers and availability zones are treated as racks within a data center. For example, if a node is in us-east-1a, us-east is the data center name and 1a is the rack location.
You can also specify multiple data centers within an EC2 region using the dc_suffix property in the /etc/dse/cassandra/cassandra-rackdc.properties file. For example, if node1 and node2 are in us-east-1:
This snitch uses public IPs as broadcast_address to allow cross-region connectivity. This means that you must configure each Cassandra node so that the listen_address is set to the private IP address of the node, and the broadcast_address is set to the public IP address of the node. This allows Cassandra nodes in one EC2 region to bind to nodes in another region, thus enabling multiple data center support. (For intra-region traffic, Cassandra switches to the private IP after establishing a connection.)
Additionally, you must set the addresses of the seed nodes in the cassandra.yaml file to that of the public IPs because private IPs are not routable between networks. For example:
seeds: 18.104.22.168, 22.214.171.124
To find the public IP address, run this command from each of the seed nodes in EC2:
When defining your keyspace strategy options, use the EC2 region name, such as “us-east“, as your data center names.
Start Cassandra (from seeds to non-seed nodes)
To verify the status of the ring cluster after all Cassandra servers are started
bin/nodetool -h localhost ring
Address Status State Load Owns Token
10.10.10.1 Up Normal 10.91 KB 70.70% 113716211212737963740265714504910561460
10.10.10.2 Up Normal 6.54 KB 29.30% 163572425264069043502692069140600439631
To monitor the Cassandra log files
tail -f /var/log/cassandra/output.log
tail -f /var/log/cassandra/system.log
Starting up Cassandra
Cassandra Options are configured in
Cassandra environment options are configured in
For production system
l make a copy of cassandra.in.sh as prod.in.sh
l make changes to the copy
l start Cassandra as
To start Cassandra as a non-demon process, use the “-f” option
To kill Cassandra with a script
l Record the process id to a file
cassandra -p /var/run/cass.pid
l Kill the process
kill $(cat /var/run/cass.pid)
n Already used in previous steps. It is important to keep this value updated, so it helps node discovery.
n Define what are the nodes that will serve as seed to help other non seed nodes to discover the topology of the ring/
l Partitioner: The Murmur3Partitioner provides faster hashing and improved performance than the previous default partitioner (RandomPartitioner).
n Murmur3Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
n RandomPartitioner: org.apache.cassandra.dht.RandomPartitioner
n ByteOrderedPartitioner: org.apache.cassandra.dht.ByteOrderedPartitioner
n Ec2Snitch: used for one network, there is also another one that allows cluster running across different networks, it is used to put replicas on very next node of a ring.
n EC2MultiRegionSnitch: Use the EC2MultiRegionSnitch for deployments on Amazon EC2 where the cluster spans multiple regions. As with the EC2Snitch, regions are treated as data centers and availability zones are treated as racks within a data center. For example, if a node is in us-east-1a, us-east is the data center name and 1a is the rack location.
Refer to section Cassandra Cluster Installation. Seed and non seed nodes have the same procedure.
If you just shut the nodes down and rebalance cluster, you risk losing some data, that exist only on removed nodes and hasn’t replicated yet.
Safe cluster shrink can be easily done with nodetool. At first, run:
on the node removed, to stop accepting writes and flush memtables, then:
to move node’s data to other nodes, and then shut the node down, and run on some other node:
to remove the node from the cluster completely. The detailed documentation might be found here:http://wiki.apache.org/cassandra/NodeTool
From my experience, I’d recommend to remove nodes one-by-one, not in batches. It takes more time, but much more safe in case of network outages or hardware failures.
Hard disk capacity
A rough disk space calculation of the user that will be stored in Cassandra involves adding up data stored in three data components on disk: commit logs, SSTable, index file, and bloom filter. When compared to the data that is incoming and the data on disk, you need to take account of the database overheads associated with each data. The data on disk can be about two times as large as raw data. Disk usage can be calculated using the following code:
# Size of one normal column
column_size (in bytes) = column_name_size + column_val_size + 15
# Size of an expiring or counter column
col_size (in bytes) = column_name_size + column_val_size + 23
# Size of a row
row_size (bytes) = size_of_all_columns + row_key_size + 23
# Primary index file size
index_size (bytes) = number_of_rows * (32 + mean_key_size)
# Additional space consumption due to replication
replication_overhead = total_data_size *
(replication_factor – 1)
Apart from this, the disk also faces high read-write during compaction. Compaction is the process that merges SSTables to improve search efficiency. The important thing about compaction is that it may, in the worst case, utilize as much space as occupied by user data. So, it is a good idea to have a large space left.
We’ll discuss this again, but it depends on the choice of compaction_strategy that is applied. For LeveledCompactionStrategy, having 10 percent space left is enough; for SizeTieredCompactionStrategy, it requires 50 percent free disk space in the worst case. Here are some rules of thumb with regard to disk choice and disk operations:
Commit logs and datafiles on separate disks: Commit logs are updated on each write and are read-only for startups, which is rare. A data directory, on the other hand, is used to flush MemTables into SSTables, asynchronously; it is read through and written on during compaction; and most importantly, it might be looked up by a client to satisfy the consistency level. Having the two directories on the same disk may potentially cause a block to the client operation.
RAID 0: Cassandra performs in built replication by means of a replication factor; so, it does not possess any sort of hardware redundancy. If one node dies completely, the data is available on other replica nodes, with no difference between the two. This is the reason that RAID 0 (http:// en.wikipedia.org/wiki/RAID#RAID_0) is the most preferred RAID level. Another reason is improved disk performance and extra space.
Filesystem: If one has choices, XFS (XFS filesystem: http://en.wikipedia. org/wiki/XFS) is the most preferred filesystem for Cassandra deployment. XFS supports 16 TB on a 32-bit architecture, and a whopping 8 EiB (Exabibyte) on 64-bit machines. Due to the storage space limitations, the ext4, ext3, and ext2 filesystems (in that order) can be considered to be used for Cassandra.
SCSI and SSD: With disks, the guideline is faster and better. SCSI is faster than SATA, and SSD is faster than SCSI. Solid State Drives (SSD) are extremely fast as there is no moving part. It is suggested to use rather low-priced consumer SSD for Cassandra, as enterprise-grade SSD has no particular benefit over it.
No EBS on EC2: This is specific to Amazon Web Services (AWS) users. AWS’ Elastic Block Store (EBS: http://aws.amazon.com/ebs/) is strongly discouraged for the purpose of storing Cassandra data—either of data directories or commit log storage. Poor throughput and issues such as getting unusably slow, instead of cleanly dying, is a major roadblock of the network-attached storage.
Instead of using EBS, use ephemeral devices attached to the instance (also known as an instance store). Instance stores are fast and do not suffer any problems like EBS. Instance stores can be configured as RAID 0 to utilize them even further.
Larger memory boosts Cassandra performance from multiple aspects.
More memory can hold larger MemTables, which means that fresh data stays for a longer duration in memory and leads to fewer disk accesses for recent data. This also implies that there will be fewer flushes (less frequent disk IO) of MemTable to SSTable; and the SSTables will be larger and fewer.
This leads to improved read performance as lesser SSTables are needed to scan during a lookup. Larger RAM can accommodate larger row cache,
thus decreasing disk access.
For any sort of production setup, a RAM capacity less than 8 GB is not suggested. Memory above 16 GB is preferred
Cassandra is highly concurrent—compaction, writes, and getting results from multiple SSTables and creation of one single view to clients, and all are CPU intensive. It is suggested to use an 8-core CPU, but anything with a higher core will just be better.
For a cloud-based setup, a couple of things to keep in mind:
A provider that gives a CPU-bursting feature should be preferred. One such provider is Rackspace.
AWS Micro instances should be avoided for any serious work. There are many reasons for this. It comes with EBS storage and no option to use an instance store. But the deal-breaker issue is CPU throttling that makes it useless for Cassandra. If one performs a CPU-intensive task for 10 seconds or so, CPU usage gets restricted on micro instances. However, they may be good (cheap), if one just wants to get started with Cassandra.
Each node in the ring is responsible for a set of row keys. Nodes have a token assigned to them on startup either via bootstrapping during startup or by the configuration file. Each node stores keys from the last node’s token (excluded) to the current node’s token (included). So, the greater the number of nodes, the lesser the number of keys per node; the fewer the number of requests to be served by each node, the better the performance.
In general, a large number of nodes is good for Cassandra. It is a good idea to keep 300 GB to 500 GB disk space per node to start with, and to back calculate the number of nodes you may need for your data. One can always add more nodes and change tokens on each node.[ 112 ]
As with any other distributed system, Cassandra is highly dependent on a network. Although Cassandra is tolerant to network partitioning, a reliable network with less outages are better preferred for the system—less repairs, less inconsistencies.
A congestion-free, high speed (Gigabit or higher), reliable network is pretty important as each read-write, replication, moving/draining node puts heavy load on a network.
Operating system configurations play a significant role in enhancing Cassandra performance. On a dedicated Cassandra server, resources must be tweaked to utilize the full potential of the machine.
Cassandra runs on a JVM, so it can be run on any system that has a JVM. It is recommended to use a Linux variant (CentOS, Ubuntu, Fedora, RHEL, and so on) for Cassandra’s production deployment. There are many reasons for this. Configuring system-level settings are easier. Most of the production servers rely on Linux-like systems for deployment. As of April 2013, 65 percent of servers use it. The best toolings are available on Linux: SSH and pSSH commands such as top, free, df, and ps to measure system performance, and excellent filesystems, for example ext4 and XFS. There are built-in mechanisms to watch the rolling log using tail, and there are excellent editors such as Vim and Emacs. And they’re all free!
l Cassandra: How to fix, ‘Fatal exception during initialization org.apache.cassandra.config. ConfigurationException: Saved cluster name Test Cluster != configured name…’?
l JNA not found. Native methods will be disabled.
l Can’t connect to cassandra – NoHostAvailableException
l Unable to gossip with any seeds
n Make sure the Listen Adress is consistent with Sedds in cassandra.yaml
l If you attempt to change the cluster name in cassandra.yaml after it was started once with a different name it will throw an error.
Saved cluster name [old name] != configured name [new name]
EDIT You can rename the cluster without deleting data by updating it’s name in the system.local table (but you have to do this for each node…)
cqlsh> UPDATE system.local SET cluster_name = ‘test’ where key=’local’;
# flush the sstables to persist the update.
bash $ ./nodetool flush
For the purposes of this LAB I simply removed and recreated the folder
sudo rm -rf cassandra/
and refer to the process to create the folder structure again
l To find out current distro
[ec2-user@ip-172-31-1-109 /]$ cat /etc/*-release
Amazon Linux AMI release 2014.03
Drivers to Connect to Cassandra
Other drivers available: