DEVOPS

 

Kubernetes

 Kubernetes

Kubernetes is a platform to schedule and run containers on cluster of virtual machines. 

It runs on bare metal, VMs,, private datacenter and public cloud.

Daemon Sets -- ensure that all nodes run a copy of a specific pod.

As nodes are added or removed from the cluster, a DaemonSet will add or remove the required pods.

Jobs -- Supervisor process for pods carrying out batch jobs.

Services -- allow communication between one set of deployments with another.

Use a service to get pods in two deployments to talk to each other.

Labels, Selectors, and Namespaces

Selectors - 2 types

Equality-based selectors ( = , !=)

Set-based selectors (IN,  NOT IN, EXISTS)

Namespaces

Working with Kafka

Kafka CLI commands

Create a topic

kafka-topics.sh --zookeeper 127.0.0.1:2181 --topic first_topic --create --partitions 3 --replication-factor 1

List topics

kafka-topics.sh --zookeeper 127.0.0.1:2181 --list

➜  kafka_2.12-2.5.0 kafka-topics.sh --zookeeper 127.0.0.1:2181 --topic first_topic --describe

Topic: first_topic PartitionCount: 3 ReplicationFactor: 1 Configs: 

Topic: first_topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0

Topic: first_topic Partition: 1 Leader: 0 Replicas: 0 Isr: 0

Topic: first_topic Partition: 2 Leader: 0 Replicas: 0 Isr: 0

Delete topic

kafka-topics.sh --zookeeper 127.0.0.1:2181 --topic first_topic --delete

kafka-console-producer

➜  kafka_2.12-2.5.0 kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic first_topic

>hello prashanth

>learning kafkA

>:-)

>^C%    

kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic first_topic --producer-property acks=all

Not recommended

kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic new_topic

Creates the new_topic, with default partitions configured in server.properties

kafka-console-consumer

Reads messages as the producer puts the message

kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic first_topic

Reads the messages from the beginning

kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic first_topic --from-beginning

Kafka consumers in group

kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic first_topic --group my-ap

kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic first_topic --group my-app2 --from-beginning

Will read all the messages from beginning, as it's commits the offset for this group when you run the same command, it doesn't return anything.

kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --list 

 kafka_2.12-2.5.0 kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --describe --group  my-app2

Consumer group 'my-app2' has no active members.

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID

my-app2         first_topic     0          7               7               0               -               -               -

my-app2         first_topic     1          8               8               0               -               -               -

my-app2         first_topic     2          8               8               0               -               -   

Reset the offsets for a consumer group

➜  kafka_2.12-2.5.0 kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group my-app --reset-offsets --to-earliest --execute --topic first_topic

GROUP                          TOPIC                          PARTITION  NEW-OFFSET     

my-app                         first_topic                    0          0              

my-app                         first_topic                    1          0              

my-app                         first_topic                    2          0  

➜  kafka_2.12-2.5.0 kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group my-app --reset-offsets --shift-by -2 --execute --topic first_topic

GROUP                          TOPIC                          PARTITION  NEW-OFFSET     

my-app                         first_topic                    0          5              

my-app                         first_topic                    1          6              

my-app                         first_topic                    2          6     

Kafka Streams

AWS - VPC Security

VPC Security Groups

- assigned at the component level

- Each Network Interface has 5 security groups by default
- 60 inbound or outbound per security group - total of 120 rules [60 inbound, 60 outbound]
- Limited to 2500 security groups per VPC

By default, all inbound traffic on all ports is denied when you create a security group.
You can only configure allow rules within a security group.

Scenario below, Allow 2 EC2 instances communicate with each other using security groups

Network Access Control Lists - NACL

- Apply within the context of a VPC  - it's exists in single VPC, it doesn't span multiple VPCs.
- Apply to one-to-manny subnets - one NACL can apply to many subnets, but within each subnet it should have only one NACL.

-should specify allow or deny traffic

For the above scenario,  if NACL is in place, it allows only 80 and 43 to access MySQL, whereas it will deny for 3306.

The public subnet contains a NAT gateway so the private subnet can get to the internet. 
Now, we configure a NACL that will lock down the private subnet.

 Once implemented, only SSH and ICMP traffic originating from the public subnet will be able to get into the private subnet. 
In addition, all traffic originating from the private subnet will be dropped.

Example Demo:

VPC Flow Logs

To create flow logs,

• Create CloudWatch log group
• Create flow log
• Create IAM policy

fr

AWS - Virtual Private Cloud

Default Virtual Private Cloud

IPv4 CIDR -- Classless Inter-Domain Routing
- when we create an AWS account, a default VPC is created.
- IPv4 CIDR range /16 --- 172.31.0.0/16: 65,536 Private IPs

default Network ACL [Access Control Lists] - allows inbound/outbound traffic

IPv4 Subnet Mask Cheat Sheet

Create Subnet
aws ec2 create-subnet --vpc-id vpc-0d8353c51322e38e8 --cidr-block 192.168.2.0/23 --availability-zone us-east-2a --profile mamidi.dev.admin

aws ec2 create-tags --resources subnet-0bdfe2dbc391a3968 --tags Key=Name,Value=demo-priv-a --profile mamidi.dev.admin

Create Route table

aws ec2 create-route-table --vpc-id vpc-0d8353c51322e38e8 --profile mamidi.dev.admin

Associate RT with Subnet

aws ec2 associate-route-table --route-table-id rtb-0e0c8323e56b5f72a --subnet-id subnet-0bdfe2dbc391a3968 --profile mamidi.dev.admin

Name the RT

aws ec2 create-tags --resources rtb-0e0c8323e56b5f72a --tags Key=Name,Value=demo-priv-rt --profile mamidi.dev.admin

Adding IGW to public subnet

Create IGW

aws ec2 create-internet-gateway --profile mamidi.dev.admin

Add Name to IGW

aws ec2 create-tags --resources igw-0587198c8c30e54a7 --tags Key=Name,Value=demo-igw --profile mamidi.dev.admin

Attach IGW to VPC

aws ec2 attach-internet-gateway --internet-gateway-id igw-0587198c8c30e54a7 --vpc-id vpc-0d8353c51322e38e8 --profile mamidi.dev.admin

Create a new Route for pub route table, so that it points to IGW

aws ec2 create-route --route-table-id rtb-0b66f361a96dacc1c --destination-cidr-block 0.0.0.0/0 --gateway-id igw-0587198c8c30e54a7 --profile mamidi.dev.admin

Configure NAT Gateway Service for private subnets

NAT Options in AWS

Configure VPC Endpoint for S3

PrivateLink - Enables private access to AWS services

VPC Endpoint Benefits
- Private access
- Lower latency
- Simplified network configuration
- Improved security posture
- Available for growing list of services

AWS Network Foundations

AWS

VPC - Virtual Private Cloud

 A VPC is a logically isolated virtual network segment of the AWS Cloud tied to your AWS account. 

Each VPC is contained within a single AWS region. 

When you create a VPC, you specify it's IP address range. 

VPC Components

Subnets - can be private or public, that is, they can contain private or public resources

Route table - can manipulate how traffic flows into and out of subnets 

Internet gateway - to access to the Internet from within a VPC

Egress-only internet gateway  - If you're making use of IPV6 and want to get to the Internet, but want to prohibit inbound connections, you need to use an egress-only gateway

VPC endpoint - If you want to enable private access to other AWS services without traversing the Internet, VPC endpoints are available for a variety of different services. 

Network Address Translation or NAT gateways - NAT as a service
Highly available service which lets resources in a private subnet connect to the Internet

Virtual private gateway - VPG
If you have external resources you wish to connect privately to resources within AWS.

Transit Gateway - 
If you're looking to simplify network management across multiple VPCs and potentially local data centers. 

Peering Connection -  establish connectivity between VPCs

DHCP option sets  - allow you to create your own DHCP options. For instance, if you want to specify your own DNS servers instead of using the AWS provided DNS, you can create an option set and assign it to a VPC. Keep in mind that a VPC can only have one DHCP option set.

Establish private connection

- to connect to an existing infrastructure

External Connection Components

Customer gateway - is a physical networking appliance in an on premises facility, to which all AWS bound network traffic is anchored. 

Virtual private gateway - VPG 
is the virtual counterpart to a customer gateway, it resides inside of AWS, and is the anchor point for all on-premises bound network traffic.

Site-to-site VPN - need a site-to-site VPN, in order for machines in a local data center to communicate with services in AWS. When the VPN connection is established, network traffic flows securely over an encrypted VPN tunnel.

Internet Protocol Security, or Ipsec
VPN tunnel between your existing facilities and your AWS VPC.

Ipsec tunnel

Instead of VPN Tunnel, we can use Direct Connect

Direct Connect - 
- dedicated connectivity to AWS
- Improved network performance
- Reduced bandwidth costs

To avoid SPOF, we can use Two Direct Connect links

Alternatively, If an existing infrastructure is also AWS, then we can use
VPC Peering   - no need of a Gateway or vpn connection

VPC peering connections can span regions.


Transit Gateway

If we need to connect local assets with multiple VPCs. Instead of a VPN connection for each VPC, you can centralize route management using a transit gateway.

Route 53
provide DNS for AWS
- provides name address resolution [ www.google.com --> 192.173.45.35]
- DNS failover - can detect website outage and redirect to different location
- Global traffic management - allows to create traffic policies that optimise user experience

Types of Routing policies:

Failover Routing

Weighted Round-Robin Routing

Latency-Based Routing

Geolocation Routing [image same as above]
routing traffic based on the coordinated ip address to the physical locations

 One way to connect local resources with your AWS account is with an Internet Protocol Security, or Ipsec, VPN tunnel between your existing facilities and your AWS VPC. Let's visualize the components required to make that happen. After creating a VPC, you want to attach it back to an existing data center you operate.