dreadknot/aws

aws's Issues

s3cmd

s3cmd configure

sudo apt-get install s3cmd

s3cmd --configure
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3
Access Key: 
Secret Key: 
Save settings? [y/N] y
Configuration saved to '/home/vagrant/.s3cfg'

Put a file

s3cmd put test s3://bucket
test -> s3://bucket/test  [1 of 1]
 0 of 0     0% in    0s     0.00 B/s  done

List a bucket

vagrant@stormbase:/tmp$ s3cmd ls s3://bucket
2013-12-12 18:27         0   s3://bucket/test

RE:Invent 2014

SEC 315 AWS Config

Knowing something changed

polling describe

inventory of resources

vpc, sg, ec2 -> record change -> config item in state version

history
stream -> sns topic
snapshot point in time
audit at point in time

change impact

What changed
Discovery, record config

Bucket
SNS topic

AWS resource is something you can create, update, delete

Relationships

Config Item:
all attributes at a given time
all in json -> jq that

stream sns topic
dash board
alert when something changes in prod
monitor of activity

Time line view
get history of resource in time range

NOT FREE

APP 306 Cloud Formation

JIRA power Deployment
40,000 change tickets

Long feed back loops
high stress - low trust in deployments
large delta between releases

balance of freedom / responsibility
one product team
one playform team -> who is the platform team?

give access make them responsible

Picture of a pendulum one side has predictability and slow, the other has freedom and fast.

repeatability
flexibility
stack overflow ability
confidence
support
access
patterns
reliable
reproducible

Infrastructure versions makes it accessible on net

service.json-0.1.0
ASG
DNS
SG
Roles
ELB

resources.json-0.1.0
DB
S3
SNS

Primitive PO's

Diff accounts
Teams
Services

VPC in CF

full ssh access

bastion in deticated VPC
peered with VPCs

from laptop to live in 2 mins
github/bbs

BBC has a Project named wormhole

Platform team delivers interface to devs

The opposite of what we have
devs deliver platform

software snapshot
apply config snapshot
tag amis. one ami with software, one with config

tropshere

ansible cloud formation
use ansible to create cloud formation python

bakery

1 snapshot with software bits
1 snapshot with configs

ASG
Update Policy

Test
Live

Indented state in version control
push to S3 for HA
consume with IAM role
customer template

Business driven development

ARC 307 Infrastructure as code

Consistency

SG disaster porn

SG define in group_vars for each stack

DEV | Infra | OPS

Grace Hopper quote, most dangerous phrase is "we've always done it like that"

What infra should be model of AWS resources

Reality reflects model. Audit for diffs apply changes.

BDT 403

10 PB data warehouse in S3

event to s3 15 mins

Log -> suro -> ursula -> s3
Log -> cassandra -> sstables -> aegiela -> s3

Storage seperate from compute

Parquet file format in S3

index data to s3

m1s - netflix uses them all

Presto interactive SQL query from s3 data

techblog.netflix.com

cache work

"needle in hay stacl"

query graph

5% - 10% speed diff S3 vs HDFS

not tried ssds

open source job inspector Inviso

BDT 310

Volume Up
Velocity Up
Variety Up

Real Time Big Data

Data -> ingest -> Store -> Process -> Visualize Answers
Over Time ->

S3 Filesystem + indexer try it out

Cheat Slide. A big list of vendors and open source tools for each component.

Ingest

Store

Process/ Analyze

Visualize

Where is splunk

Types of Data Ingest

Transacrtional

File

Stream

Store

Amazon Kinesis
Apache Kafika

Convert multiple streams into few

Decouple producers / consumers
ordering preserved
replay

What is splunks interface to hot, cold, warm?

How can I inject s3?

abstract buckets to s3

Data characteristics

/hot - ssd, cache, ebs
/warm - s3, file system
/cold - glacier file system

try this too

amazone elastic cache
dynamo DB
hdfs for hot
segment the volumes

Processing
Batch
Stream

Processing tools
Presto
open source
vendors

Visualize
Open source
vendors

Where is splunk?

Write to multiple data stores

Data -> indexer -> DB & S3

get splunk on ingest and store

glacier_cold
s3_warm
HDFS_hot

Ganglia for stacks

MBL 202 Containers

APP 313 Lambda

AWS

• Region
• VPC
• Availability Zone
• Multiple Subnets VLAN Zones
• App
• DB
• LB
• Security Groups
• Allow rules only
• Network Access Control List (NACL)

Use jump hosts to connect.

IAM

IAM

IAM roles for access, boto loads them automatically.

>>> import boto
>>> c = boto.connect_s3()
>>> rs = c.get_all_buckets()
>>> print rs
[<Bucket: cf-templates-9972b0b8kz5j-us-east-1>, <Bucket: elasticbeanstalk-us-east-1-811769099702>]